Sample Ratio Mismatch (SRM) occurs when the actual sample sizes in your A/B test differ significantly from the expected allocation ratio. For instance, if you plan to split traffic evenly between two variants (a 50/50 split) but end up with a 60/40 split, you’ve encountered an SRM. This mismatch signals a problem with the randomisation or tracking in your experiment, which can compromise the validity of your results.
SRM is critical because it indicates that users aren’t being assigned to variations correctly. Causes include technical glitches, biased sampling, or tracking errors. Identifying SRM early ensures you base decisions on accurate data, preserving the integrity of your growth marketing efforts. For more on how statistical methods affect experiment interpretation, see our guide on Frequentist vs Bayesian Statistics for Growth Marketers.
How to Detect Sample Ratio Mismatch
Manually detecting SRM can be time-consuming and error-prone. Fortunately, tools like SRM calculators automate this process. By entering your expected split and actual sample sizes, these calculators quickly determine if there’s a statistically significant mismatch. This allows you to address any issues promptly, ensuring your A/B test results are reliable.
Table of contents
Open Table of contents
Impact of Sample Ratio Mismatch on Your Results
SRM can lead to incorrect conclusions from your A/B tests. If one variant unintentionally receives more traffic, the statistical power and validity of your test are compromised. This can result in false positives or negatives, causing you to implement changes that don’t actually benefit your business.
Common Causes of Sample Ratio Mismatch
Understanding the common causes of SRM can help you prevent it. These causes include:
-
Technical Issues: Bugs in the code that assigns users to variants can cause uneven traffic distribution.
-
User Targeting: Filters and segmentation that exclude certain users might disproportionately affect variants.
-
Cookie Problems: Users who clear cookies or switch devices may be counted multiple times.
-
Bot Traffic: Automated traffic can skew the sample sizes if not properly filtered out.
Best Practices to Avoid SRM
To minimise the risk of SRM:
-
Validate Your Randomisation: Test your assignment logic to ensure users are evenly distributed.
-
Monitor Sample Sizes: Keep an eye on the sample sizes of each variant throughout the test.
-
Use Reliable Tools: Employ trusted A/B testing platforms that handle randomisation effectively.
-
Filter Out Bots: Implement measures to identify and exclude bot traffic from your tests.
How Incrementality Testing Enhances Your Insights
While addressing SRM is essential, incorporating incrementality testing takes your analysis further. Incrementality testing measures the true lift generated by your marketing efforts by comparing performance against a control group that did not receive the intervention. This helps you understand the actual impact of your strategies, distinguishing between causation and mere correlation.
Implementing incrementality testing can:
-
Provide a clearer picture of your marketing ROI.
-
Inform more effective allocation of marketing resources.
-
Identify which strategies genuinely drive growth.
Bringing It All Together
Combining vigilant detection of SRM with incrementality testing ensures your growth marketing initiatives are based on robust data. This approach helps you make informed decisions, optimise your strategies, and ultimately drive genuine value for your business.
How Growth Method Simplifies Your A/B Testing
Navigating the complexities of A/B testing and ensuring data integrity can be challenging. Growth Method is designed to streamline your growth marketing processes, from ideation to experimentation and analytics. Our platform helps you detect issues like Sample Ratio Mismatch early, safeguarding the validity of your tests.
With integrated tools and industry-leading reporting, Growth Method enables you to:
-
Automate the detection of SRM and other anomalies.
-
Implement incrementality testing seamlessly.
-
Focus on execution rather than managing cumbersome workflows.
“We are on-track to deliver a 43% increase in inbound leads this year. There is no doubt the adoption of Growth Method is the primary driver behind these results.” - Laura Perrott, Colt Technology Services
About Growth Method
Growth Method is the only work management platform built for growth marketers. We help companies implement a systematic approach to grow leads and revenue.
To date our customers have recorded over 1000 marketing experiments in Growth Method. Learn more about us on our homepage or book a call with us here. We’re here to help you grow.
“We are on-track to deliver a 43% increase in inbound leads this year. There is no doubt the adoption of Growth Method is the primary driver behind these results.” - Laura Perrott, Colt Technology Services