Proof of Concept Firebase A/B Testing Implementation

[ajsarkar28]

Objective: Determine the technical feasibility of integrating Firebase A/B Testing with the app using Remote Config. The focus is to understand the setup process, necessary changes to the app’s architecture, and any potential constraints.

AC

• Setup/Implement firebase website and library requirements to trigger an A/B response on a developer screen test and review analytics provided
• If test is implemented in Prod, contact QA to check that Prod is fine

[theodur]

Summary

The mobile app has everything in place for running A/B testing in Firebase. To start a new experiment or see existing experiments, you can visit the A/B Tests tab in the Remote Config section of the Firebase console. Other than adding the appropriate feature toggle to remote config, updating the code to use that feature toggle, and creating the experiment in Firebase, there's nothing else needed in order to run an A/B experiment.

Testing implementation

I created 2 experiments for this spike, 1 for iOS and 1 for Android, since each OS needs its own experiment. Experiments are focused around audiences and variants that are determined by feature toggle values. For the sake of this spike, I used the testFeature feature toggle for the experiments, since it isn't being used. I also used the audience of Staging users so that this would only impact staging, but we can select audiences from this list. We also can create a new audience for whatever experiment we want based on user properties.

I had 2 variants for these experiments, 1 being the baseline (the testFeature feature toggle set to its default value of false) and variant A (testFeature being set to true). When testFeature is true, I displayed a message in the Developer screen. You can add up to 5 variants in an experiment. You can also set the weight distributions of the variants, so if you want a higher percentage of people to see variant A, you can set the percentage higher compared to the baseline. There are also more customizable feature for A/B tests, like setting an activation event, which can be a way of filtering which users are included in the experiment measurements. From Firebase:

An event that activates the experiment for a user. All targeted experiment users will receive an experiment variant treatment from the start of the experiment, but only users who trigger the activation event will be included in experiment measurement.

Potential issues

One issue I ran into when testing was the remote config override being set in the app, which prevents the app from fetching feature toggle values from Firebase, and just uses the local remote config values. This seems to be caused by the waygate logic setting the override value to true when loading the waygate values. This would only impact testing A/B testing in the future, but wouldn't impact rolling out an experiment to prod, so as of now, the app is ready for A/B testing in production if we wanted to. We should still look into this to make sure the override value isn't being unnecessarily set to true to make testing on staging easier.

Metric tracking

Another useful thing about Firebase A/B testing is you can choose the "primary goal" of the experiment, and it can track the results of that. For instance, I chose Retention of 1 day as the goal for testing, but you can choose from a wide list of stuff. You can also set one of the custom analytic events as the primary goal, and it will track metrics around that event. In terms of general metrics, you can see how many users are participating in the experiment and the percentage of users that receive each variant.

Practical use case

A good example of implementing A/B testing would be if you wanted to test a new flow for completing a task, e.g. refilling a prescription, and you couldn't decide which flow would work better for users. You could create an A/B test with two variants, and add a custom analytic event, e.g. vama_refill_time and track how long it takes each flow to complete. Then you'd be able to see the comparison in time between the two variants and determine which flow enables user's to refill prescriptions quicker.

Common questions

Can A/B testing track users from A/B across sessions?

Yes, once a user is assigned to a A/B variant, they will remain in that variant, so it'll be persistent across sessions.

Can you run a single experiment for both Android and iOS

No, unfortunately there's not, so you'll need to duplicate the experiment for both Android and iOS. This is because in Firebase, technically the iOS app and Android app are two different apps, and an experiment can only be applied to one app.

[theodur]

Moved this to code review while it's being reviewed by @ajsarkar28 and @dumathane

[theodur]

After reviewing my finding above, Ameet suggested we run a trial A/A test in production, meaning a test where nothing has changed but confirms that traffic is flowing appropriately. For the A/A test, we decided to use vama_rx_refill_success as the primary goal of the experiment, and vama_rx_request_confirm as the activation event.

I created an experiment for both iOS and Android. After about a week of running the experiment, the iOS experiment has 57,000 total users and the Android experiment had 21,000 users, which accurately correlates to the distribution of iOS/Android user who use the app. Users across each variant were spread out evenly, at 50% receiving the Baseline and 50% receiving variant A. This suggests the experiment distribution and tracking is working.

[theodur]

This work is complete now that we have a clear understanding of how to implement A/B experiments, and what metrics we can get from them