Swift Heroes 2025

Elevator pitch

In this talk we'll take a critical look at the often overlooked engineering challenges and hidden costs of A/B testing. We'll explore how A/B testing increases code complexity, introduces risks of data corruption, expands the testing matrix, challenges debugging, etc. We'll shed light on the real impacts on developer experience and user satisfaction. I will provide real-life insights that might reshape your perception of A/B testing and convince you that it's a tool that needs to be wielded responsibly and requires a high engineering discipline.

Description

In this talk we'll cover the following topics related to often overlooked costs of A/B testing:
- Code complexity: Introducing variations and experimentation logic into the codebase can increase complexity, making it harder for developers to understand and maintain the code. Removing A/B test code and related infrastructure once a test is completed requires careful cleanup to avoid introducing technical debt or unnecessary complexity into the codebase. Failing to remove redundant code can clutter the codebase and make future development more challenging.
- Debugging Challenges: Debugging issues related to A/B test variations can be more challenging, especially when trying to diagnose discrepancies in behaviour across different test conditions.
- Expanded Testing Matrix: A/B testing expands the testing matrix, requiring QA teams to test multiple variations of features across different devices, platforms, and user segments, which can be time-consuming and resource-intensive.
- Complicates Test Environment Setup: QA teams may need to set up and maintain separate test environments to accurately test A/B test variations, adding overhead in managing and synchronising test environments with production.
- Data Corruption and Loss: Improperly managing the addition or removal of A/B tests could lead to data loss or gaps in the data stream. Adding and removing A/B tests can challenge user data integrity. Experiments can manipulate data that can't be undone once the experiment is removed.
- User Consent and Opt-Out: Ensuring that users are informed about A/B testing activities, their data usage, and providing mechanisms for users to opt-out of participation in tests if desired, to respect user preferences and maintain trust in data integrity practices.

...and more...