A/B Test your App using Firebase Remote Config

Last week I was in Nairobi and Cape Town giving a talk on Remote Config and Test Lab in Firebase. I had such a great time and thought I should share some of the content I presented online.

What is A/B Testing?

A/B testing is the process of experimentally testing your UI on different audiences in order to determine the best (or most profitable) user experience.  Firebase Remote Config enables us to do A/B testing by allowing us to randomly segment our app audiences easily.

Example A/B Test

In this example, we are going to test out a different version of a checkout button that we have in our app.

For variant A of the experiment, we want to show a pink “Checkout” button. For variant B we want to show a blue “Cart” button. For the rest of our users, we will show the original “Finished” button.

Now for the fun part, setting up the experiment![adwords_square]

Set up Firebase Remote Config for A/B Testing

1. Create a Firebase Project.
2. Head to Remote Config section.
3. Create a parameter – let’s call this parameter experiment_variant. Make the default value return no_experiment.
4. Click “Add value for Condition”. Then click “Define a new Condition”. Give it the name “Variant A”. We will select “User in Random Percentile” as the conditional. We will then choose the users from 0% – 10% that will receive variant A as their experiment. Click “Create Condition“.
5.  Now you will see that conditional appear for the experiment_variant parameter. Assign variant_a to the value that should be returned for that variant.
6. We will repeat the same for variant B, by creating a conditional for the users from 10% – 20%. This means that only 20% of the entire user base will be running the experiment and 80% will see no change in the standard UI. The percentage of the experiment is totally up to you. Then return variant_b for the parameter value:
7. Make sure to update and publish the changes. We have completed the server side setup of our experiment.

Setting up the A/B Test on Android Devices

Once you have configured the server, the hard part is done! The next section will cover how to get setup for Android but the code is really similar for iOS too – so don’t be deterred!

1. In your android app, make sure you have the Firebase dependencies set up. Place the following in your top level build.gradle:

   buildscript {
       // ...
       dependencies {
           // ...
           classpath 'com.google.gms:google-services:3.0.0'
       }
   }

2. Add the following in your app level build.gradle:

   dependencies {
     // ...
     compile 'com.google.firebase:firebase-core:9.6.0'
     compile 'com.google.firebase:firebase-config:9.6.0'
   }
   
   // At the bottom
   apply plugin: 'com.google.gms.google-services'

3. Download google-services.json file from Firebase console and add it to your app/ folder.
4. Create a remote config defaults file in the res/xml folder like the one below. This file will be used if the user has no internet connection when loading up the app for the first time.

   <?xml version="1.0" encoding="utf-8"?>
   <defaultsMap>
       <entry>
           <key>experiment_variant</key>
           <value>no_experiment</value>
       </entry>
   </defaultsMap>

5. Initialise Firebase remote config. This is also where we set the defaults to the file we created in the previous step.

   remoteConfig = FirebaseRemoteConfig.getInstance();
   
   FirebaseRemoteConfigSettings config = new FirebaseRemoteConfigSettings.Builder()
       .setDeveloperModeEnabled(BuildConfig.DEBUG) 
       .build();
   remoteConfig.setConfigSettings(config);
   
   remoteConfig.setDefaults(R.xml.remote_config);

6. Then fetch the remote config values. Firebase will automatically cache the values for the specified time that you provide. By default, it caches for 12 hours.

   remoteConfig.fetch(CACHE_TIME_SECONDS).addOnCompleteListener(new OnCompleteListener<Void>() {
            @Override
               public void onComplete(@NonNull Task<Void> task) {
                   if (task.isSuccessful()) {
                       Log.d(TAG, "Fetch Succeeded");
                       remoteConfig.activateFetched();
                   } else {
                       Log.d(TAG, "Fetch Failed");
                   }
                   runExperiment();
               }
           });

7. In order to run our experiment, we should fetch the experiment_variant parameter from our remote config and use it in our app.

   String experiment = config.getString("experiment_variant");
   
   FirebaseAnalytics.getInstance(this).setUserProperty("Experiment", experiment);
   
   if (experiment.equals("variant_a")) {
       //..
   } else if (experiment.equals("variant_b")) {
       //...
   } else {
       //..
   }

   In this example, we fetch the parameter and set a custom analytics user property with the variant that has been assigned to the current device. This will allow us to properly track how our A/B test is performing.  In the conditional, we are free to make the changes we want to the app. We can set the text to different things and change the colours depending on the variant.

8. Now that we have the experiment running, we need to track when a user clicks on the “Checkout” button in order to determine which button variant a user is more likely to tap on. We then log the event to Firebase analytics which will include the user property that we set previously.

   buttonCheckout.setOnClickListener(new View.OnClickListener() {
       @Override
       public void onClick(final View view) {
           //..
           firebaseAnalytics.logEvent(“CheckoutClicked”, new Bundle());
       }
   });

   It is worth noting that there are a bunch of default events and parameters that you can make use of with Firebase Analytics. [adwords_square]

9. In order to determine the success of your A/B test, you need to analyse your data to decide which UI is the most successful. After logging into the console, we can see there were 24 counts of the “CheckoutClicked” event.After applying the filter for the user property “Experiment = Variant A”, we can see that 19 of the 24 clicks were from variant A of the experiment (granted this was just me clicking around in my sample app – but you get the idea).

For larger data sets where the result of the experiment may not be obvious at first, you can export your events to BigQuery. This will allow you to make more complex deductions from your experiments.

For now at least, we know the clear winner (in this very large test 😬) is Variant A – the pink checkout button wins!

There you have it, an easy way to experiment with your UI using Firebase Remote Config.

Sample code can be found here (It is not very complex as all the hard work is done by the Firebase APIs.) 😊

Happy AB Testing!