Evidence or Chance: What the Heck is Hypothesis Testing? (Part 2) 🤔📊

Let's also understand how the 'Bandwagon Effect' impacts voter choices and the difference between KPIs and metrics.

Hello, data enthusiasts and curious minds!

Welcome to the 10th edition of DataPulse Weekly, where we unravel the magic behind data and its impact on our daily lives.

We can’t believe we’ve reached our 10th edition! Our heartfelt gratitude goes out to each one of you for your incredible support. Your encouragement drives us to create even more valuable content. If you haven’t subscribed yet, please do, and share it with friends who would love it. Thank you for being part of our journey!

Our newsletter promises to be a journey through the fascinating intersections of data, narratives, and human experiences. Whether you're an analyst or simply curious about how data shapes our world, you're in the right place.

Before we dive into today’s newsletter, this is part 2 of our series on hypothesis testing, and we covered part 1 in our previous edition due to topic complexity. If you want to follow through with the data case study in this newsletter, we strongly recommend reading the previous edition.

And now, let’s dive straight into today’s Data Menu!

---

Today’s Data Menu

1. Data Case Study: Hypothesis Testing (Part 2)

2. Metric: Z-score

3. Human Bias Focus: Bandwagon Effect

4. Data Nugget: Metrics vs. KPIs

---

Data Case Study: Hypothesis Testing (Part 2)

Before we delve into the second part, here’s a quick recap of what we covered in the last newsletter:

1. Scenario: WebInnovate, a hypothetical company, observed that the average time spent on their website increased from 3 minutes to 3.3 minutes after a key design change.

2. Objective: They want to determine if this design change is truly effective using hypothesis testing.

3. Hypotheses:

   • Null Hypothesis (H0): There is no difference in time spent with the new design.

   • Alternative Hypothesis (H1): The new design has improved the average time spent.

4. Conclusion Possibilities:

   • Reject the Null Hypothesis with statistically significant evidence, supporting the Alternative Hypothesis.

   • Fail to reject the Null Hypothesis due to a lack of statistically significant evidence.

5. Significance Level: Set at 0.05, indicating a 5% risk of concluding that the result is significant when it is actually due to chance.

6. p-Value: The probability of obtaining the observed results (average time spent of 3.3 minutes) if the Null Hypothesis is true.

7. Comparison: We compare the p-value with the significance level to conclude the hypothesis test.

Now, let’s jump straight into calculating the p-value. It involves an interim step: calculating the test statistic.

A test statistic is a standardized value used in hypothesis testing to determine the p-value.

Here are some popular tests to obtain test statistics: z-test, t-test, F-test, and chi-square test. The choice of test depends on the type of test being performed and the characteristics of the data.

We won’t delve into the details of choosing the right test for different data types here, as that could be overwhelming for a single read. We'll cover this in another newsletter. Once we understand one test type, the core concepts can be applied to different scenarios with varying data types.

For now, let’s focus on calculating the test statistic for our specific use case.

In our case, the relevant data includes:

• Original average time spent: 3 minutes

• New average time spent: 3.3 minutes

• New observed sample size: 100

• The standard deviation of sample data: 0.8 minutes

Given this data, the one-sample t-test is the most suitable statistical test. The t-test is used when the sample size is small (< 30) or when the population standard deviation is unknown. Here, since we only have the sample standard deviation, we will use the t-test.

Let’s calculate our test statistic:

This will give us a test statistic of 3.75!

Calculating the P-value:

To calculate the p-value for the t-statistic, we use a t-distribution table or an online calculator.

Consider the t-distribution table as a lookup table for the p-value from the t-statistic. And, apart from the t-statistic, we need one more parameter.

We need to calculate the degrees of freedom - it simply means subtracting 1 from the sample size. Since our sample size is 100, the degrees of freedom (df) is 𝑛−1 = 99.

Using an online p-value calculator, such as socscistatistics, input the t-statistic and degrees of freedom to find the p-value. Our p-value comes about to be 0.00015.

Since the p-value (0.00015) is less than the significance level (0.05), we reject the Null Hypothesis. This provides statistically significant evidence that the design change has indeed improved the average time spent on the website.

And, voila! We are done! Even Bernie Sanders would agree with rejecting the Null Hypothesis 🤓

Key Takeaway:

Hypothesis testing is crucial for assessing changes, like WebInnovate's design update, by comparing new data against population observations. Understanding one method, such as the t-test, helps grasp the steps involved in hypothesis testing.

While sometimes it's evident that changes are effective without formal testing, hypothesis testing becomes essential when stakes are high or sample sizes are small. It provides a reliable tool for making data-driven decisions.

Understanding hypothesis testing allows you to confidently evaluate changes and improve outcomes based on statistical evidence.

Next, we’ll explore the Z-score, which helps measure how far a sample observation lies from its mean value in a normal distribution.

---

Metric of the Week: Z-score

A Z-score measures how many standard deviations a data point is from the mean in a normal distribution. It is calculated as:

where 𝑋 is the data point, 𝜇 is the mean, and σ is the standard deviation.

A Z-score of 0 indicates the data point is exactly at the mean. Positive Z-scores indicate values above the mean, while negative Z-scores indicate values below the mean.

Using the empirical rule (68-95-99.7 rule), about 68% of data in a normal distribution falls within 1 standard deviation (SD) of the mean, 95% within 2 SD, and 99.7% within 3 SD.

Z-scores are vital for standardizing data, comparing different datasets, and identifying outliers.

---

💡 Remember that building a data mindset is effective only when we focus on solving data-related problems. The below question is designed for exactly this kind of practice. We will address this in the last section of this newsletter.

Food for thought:

You are a web analyst at WebInnovate, tasked with increasing user time spent on the website by 30% next quarter. They want to increase their website advertisement revenue by making the user spend more time on the website. Higher Average Time Spent → More Ad Impressions → More Ad Revenue. Numerous parameters are available for monitoring. You need to create a dashboard featuring a single KPI and additional metrics to enhance the team's tracking capabilities. How would you differentiate between a KPI and other metrics?

---

Human Bias Focus: Bandwagon Effect

Did you know there are more than 180 ways your brain can trick you? These tricks, called cognitive biases, can negatively impact the way humans process information, think critically and perceive reality. They can even change how we see the world. In this section, we'll talk about one of these biases and show you how it pops up in everyday life.

The world’s largest election is currently underway in India, where 1 billion eligible voters will decide the country’s fate for the next five years. They will elect 543 members of the Lok Sabha, who will then choose India’s next Prime Minister.

India's vast demographic and geographic diversity means voters can be categorized into three groups:

1. Decided voters: Voters who are certain about whom they will vote for.

2. Non-participants: Voters who have decided not to vote.

3. Undecided voters: Voters who are still considering their voting options.

The third group, the undecided voters, is particularly intriguing. These voters will significantly be influenced by media, expert opinions, and public perception. Many will end up supporting the candidate they perceive as the most popular or the likely winner. A lot of these voters are going to be affected by a human bias known as the Bandwagon Effect.

The term "bandwagon effect" comes from "jump on the bandwagon," which refers to being on the side that is going to win or what everybody is supposedly going to do.

People often choose candidates due to social conformity, fear of missing out, and the desire to back a winner. When they see a candidate gaining momentum, they assume the majority has valid reasons for their support. This herd mentality strengthens the candidate’s momentum, creating a self-fulfilling prophecy.

Recent Examples - Clubhouse, NFTs, and Meme Coins:

The bandwagon effect is evident in recent trends like the popularity of Clubhouse, the surge in NFT investments, and the meme coins phenomenon. These trends show how public opinion can quickly shift based on perceived popularity, driven by hype, social proof, and perceived value.

Remember, recognizing any bias is the first step to overcoming its impact on our decision-making.

This brings us to our last section where we address the question asked earlier.

---

Data Nugget: Metrics vs KPIs

Diving into our previous question: How would you differentiate between a KPI and other metrics?

Let’s first understand the difference and usage of a KPI and Metrics:

• Purpose: Metrics provide operational insights into specific aspects of website performance, while the KPI aligns with the strategic goal of increasing user engagement.

• Scope: Metrics offer detailed data supporting the KPI, which focuses on the broader impact of strategic initiatives.

• Impact: While metrics help monitor and tweak daily operations, the KPI guides long-term strategy and decision-making to boost overall engagement.

In our problem statement, the goal of WebInnovate was to increase the average time spent so that they serve more ads to the users and increase their ad revenue.

This is how the KPI and Metrics for WebInnovate will look like:

KPI (Key Performance Indicator):

• Average Time Spent on Website: This serves as the primary indicator of user engagement, measuring how long visitors stay engaged with the website.

Metrics to Track:

1. Page Views Per Visit: Reveals the depth of user engagement by showing how many pages are viewed per session.

2. Bounce Rate: Measures the percentage of visitors who leave after viewing only one page.

3. Average Time Spent by Device Type: Compare which device types can be leveraged for effective communications

And there can be more!

Improving all the above metrics will boost the main KPI - average time spent and subsequently increasing the ad revenue for WebInnovate

Higher Average Time Spent → More Ad Impressions → More Ad Revenue!

By leveraging both KPIs and metrics, the company can track the right data and make informed decisions to achieve its main goal effectively.

---

That concludes our 10th edition! In our mission to simplify complex data concepts, we took some ambitious goals to tackle one of the toughest topics in our last 2 newsletters. Crafting these explanations in a straightforward manner takes considerable effort. If you found this newsletter beneficial, please consider subscribing and sharing it with just one person who might like it. Your support motivates us to go above and beyond in creating even more valuable content for you.

Recommended Next Read: