Junaid Iqbal | Agentic AI Engineer

Hey everyone! Junaid here. Today, I want to share my recent deep dive into A/B testing, a powerful technique that businesses use all the time to make better decisions. You’ve probably heard about it, especially in the world of online shopping and marketing. It’s how companies figure out if a new button color, a different website layout, or a new ad copy actually makes a difference to their bottom line.

I recently built a project to simulate and analyze an A/B test for an e-commerce platform, and I want to walk you through my process, the challenges I faced, and what I learned. It was a truly hands-on experience, and I think it really helped solidify my understanding.

What Exactly Is A/B Testing?

Imagine you have two versions of something – let’s call them Version A and Version B. Maybe Version A is your current website checkout page, and Version B is a new design you think might encourage more people to complete their purchases.

A/B testing is simply showing Version A to one group of users (the “Control Group”) and Version B to another group (the “Treatment Group”). You then measure how each group performs on a specific metric – in our case, the “conversion rate” (the percentage of users who complete a purchase). The goal is to see if Version B is statistically better than Version A, or if there’s no real difference.

It’s crucial because it takes the guesswork out of design and marketing decisions. Instead of just hoping a new feature works, you prove it with data.

My Project Goal: Simulate & Analyze an E-commerce A/B Test

My main goal for this project was to understand the mechanics of A/B testing from a practical, coding perspective. I wanted to:

1. Simulate A/B test data: Because I didn’t have real-world data readily available, I created my own dataset to mimic what an e-commerce A/B test would look like. This involved generating user IDs, assigning them to control or treatment groups, and then simulating whether they converted or not, based on a presumed underlying conversion rate for each group.
2. Perform statistical analysis: Once I had the data, the next step was to analyze it to determine if any observed differences between the groups were statistically significant, or just random chance.
3. Interpret the results: The numbers are important, but understanding what they mean for a business is critical.

The Tools I Used

I built this project entirely in Python, leveraging some powerful libraries that are staples in data analysis:

• Pandas: For data manipulation and analysis (perfect for handling CSV files and grouping data).
• NumPy: Essential for numerical operations, especially when generating random data.
• Statsmodels: This is where the magic of statistical testing happened, specifically for performing the z-test for proportions.

And of course, VS Code as my coding environment and Git/GitHub for version control and showcasing the project.

My Hands-On Process (The Nitty-Gritty)

Step 1: Simulating the Data (simulate_ab_data.py)

I started by writing a Python script to create my artificial A/B test dataset. Here’s a simplified look at what went into it:

• I decided to have 10,000 users in total, split equally between the Control (A) and Treatment (B) groups.
• I set a “baseline” conversion rate for the Control Group (e.g., 10%) and a slightly higher one for the Treatment Group (e.g., 11.5%) to simulate a positive uplift.
• For each user, I randomly determined if they converted based on their group’s conversion rate.
• Finally, I saved this simulated data into a ab_test_results.csv file.

This part was super helpful because it showed me how raw data might be structured for such an experiment.

Step 2: Analyzing the Results (analyze_ab_test.py)

This was the core of the project. I wrote another Python script to:

1. Load the ab_test_results.csv file using Pandas.
2. Calculate the total number of users and conversions for both the Control and Treatment groups.
3. Determine the observed conversion rate for each group.
4. Perform a proportions z-test using statsmodels. This statistical test helps us figure out if the difference in conversion rates between the two groups is large enough to be considered “real” (statistically significant) or if it could have happened just by chance. I chose an alpha (significance level) of 0.05, meaning I’m looking for a P-value less than 0.05 to consider the results significant.
5. Print out all the key metrics and a clear conclusion.

The Moment of Truth: My A/B Test Results

When I ran my analysis script, here’s what came out:

--- A/B Test Analysis ---
Control Group (A) - Users: 5000, Conversions: 499, Rate: 0.0998
Treatment Group (B) - Users: 5000, Conversions: 572, Rate: 0.1144
Observed Lift (B vs A): 14.63%
Z-statistic: 2.3606
P-value: 0.0091
Significance Level (alpha): 0.05

Conclusion: Reject the Null Hypothesis.
There is a statistically significant difference in conversion rates.
The new checkout page (Version B) performed significantly better.

What does this all mean? My Interpretation:

The Control Group (A) had a conversion rate of about 9.98%, while the Treatment Group (B) had a conversion rate of 11.44%. This means Version B showed an observed lift of 14.63% compared to Version A. That’s a great improvement!

But is this improvement real, or just a fluke? This is where the P-value comes in. My P-value was 0.0091. Since this is much smaller than my chosen significance level of 0.05, I can confidently reject the Null Hypothesis.

In plain English: The Null Hypothesis would be “there is no difference between Version A and Version B.” By rejecting it, I’m saying, “Nope, there is a significant difference.” The new checkout page (Version B) did indeed perform significantly better than the old one.

From a business perspective, this data suggests that rolling out the new checkout page (Version B) would likely lead to more completed purchases, which translates directly to more revenue!

Lessons Learned and Next Steps

This project was incredibly insightful. It taught me:

• The importance of structured data for analysis.
• How to apply statistical tests like the z-test for proportions in a real-world scenario.
• The critical difference between an observed difference and a statistically significant difference.
• How to clearly interpret statistical output for business decision-making.

My next steps with this knowledge might be to explore more complex A/B test scenarios, like tests with multiple variations (A/B/n testing), or dive into Bayesian A/B testing methods.

Wrapping Up

A/B testing isn’t just a buzzword; it’s a fundamental skill for anyone interested in data-driven decision-making, whether in marketing, product development, or general business strategy. Building this project from scratch really helped me grasp its power.

I hope sharing my journey helps you understand A/B testing a bit better too! Feel free to check out the code on my https://github.com/Junaid1991-maker/ab_testing_ecommerce if you’d like to see the full implementation.

Thanks for reading!