A/B Testing for Muzz

Type: Product Experimentation (A/B/C Test)
Tools: Python (Pandas, NumPy, SciPy), Jupyter Notebook, Matplotlib, Seaborn

Overview

This project involved analyzing data from a real-world A/B/C experiment designed to improve Muzz’s phone verification process for new users.
The goal was to evaluate which screen design achieved the highest verification success rate at the lowest average cost per user, while maintaining fairness across user demographics.

The Challenge

Muzz tested three onboarding screens for phone verification:

Each user saw one variant at random after app installation. My task was to interpret the test, evaluate its effectiveness, and recommend which design should be adopted.

Data Used

Three datasets were provided:

1. verification.csv – user ID, test group, selected method, and verification outcome.
2. profiles.csv – user demographics (gender, age, country).
3. costs.csv – per-country unit costs for SMS and WhatsApp verification.

I joined and cleaned these datasets, validated randomization across variants, and engineered new variables such as age group for better insight segmentation.

Approach

1. Exploratory Analysis: Validated data integrity, balanced sample sizes, and checked for bias across gender and age.
2. Feature Engineering: Derived age and age_group from date of birth to understand demographic distribution.
3. Verification Metrics: Measured the verification rate per group and per method (SMS vs WhatsApp).
4. Statistical Testing: Conducted two-proportion z-tests to confirm whether observed differences between groups were statistically significant.
5. Cost Analysis: Combined verification shares with fixed per-country costs to estimate the average cost per verified user.