A/B Test your App using Firebase Remote Config

Last week I was in Nairobi and Cape Town giving a talk on Remote Config and Test Lab in Firebase. I had such a great time and thought I should share some of the content I presented online.

What is A/B Testing?

A/B testing is the process of experimentally testing your UI on different audiences in order to determine the best (or most profitable) user experience.  Firebase Remote Config enables us to do A/B testing by allowing us to randomly segment our app audiences easily.

Example A/B Test

In this example, we are going to test out a different version of a checkout button that we have in our app.

A/B test we will run using Firebase Remote config.

For variant A of the experiment, we want to show a pink “Checkout” button. For variant B we want to show a blue “Cart” button. For the rest of our users, we will show the original “Finished” button.

Now for the fun part, setting up the experiment![adwords_square]

Set up Firebase Remote Config for A/B Testing

  1. Create a Firebase Project.
  2. Head to Remote Config section.
  3. Create a parameter – let’s call this parameter experiment_variant. Make the default value return no_experiment.Setting up parameter in Firebase Remote Config
  4. Click “Add value for Condition”. Then click “Define a new Condition”. Give it the name “Variant A”. We will select “User in Random Percentile” as the conditional. We will then choose the users from 0% – 10% that will receive variant A as their experiment. Click “Create Condition“.Adding a A/B Testing conditional to use with Firebase Remote Config
  5.  Now you will see that conditional appear for the experiment_variant parameter. Assign variant_a to the value that should be returned for that variant.Conditional in Firebase Remote Config
  6. We will repeat the same for variant B, by creating a conditional for the users from 10% – 20%. This means that only 20% of the entire user base will be running the experiment and 80% will see no change in the standard UI. The percentage of the experiment is totally up to you. Variant B for Firebase Remote ConfigThen return variant_b for the parameter value:All variants in Firebase Remote Config
  7. Make sure to update and publish the changes. We have completed the server side setup of our experiment.

Setting up the A/B Test on Android Devices

Once you have configured the server, the hard part is done! The next section will cover how to get setup for Android but the code is really similar for iOS too – so don’t be deterred!

  1. In your android app, make sure you have the Firebase dependencies set up. Place the following in your top level build.gradle:
    buildscript {
        // ...
        dependencies {
            // ...
            classpath 'com.google.gms:google-services:3.0.0'
        }
    }
  2. Add the following in your app level build.gradle:
    dependencies {
      // ...
      compile 'com.google.firebase:firebase-core:9.6.0'
      compile 'com.google.firebase:firebase-config:9.6.0'
    }
    
    // At the bottom
    apply plugin: 'com.google.gms.google-services'
  3. Download google-services.json file from Firebase console and add it to your app/ folder.
  4. Create a remote config defaults file in the res/xml folder like the one below. This file will be used if the user has no internet connection when loading up the app for the first time.
    <?xml version="1.0" encoding="utf-8"?>
    <defaultsMap>
        <entry>
            <key>experiment_variant</key>
            <value>no_experiment</value>
        </entry>
    </defaultsMap>
  5. Initialise Firebase remote config. This is also where we set the defaults to the file we created in the previous step.
    remoteConfig = FirebaseRemoteConfig.getInstance();
    
    FirebaseRemoteConfigSettings config = new FirebaseRemoteConfigSettings.Builder()
        .setDeveloperModeEnabled(BuildConfig.DEBUG) 
        .build();
    remoteConfig.setConfigSettings(config);
    
    remoteConfig.setDefaults(R.xml.remote_config);
  6. Then fetch the remote config values. Firebase will automatically cache the values for the specified time that you provide. By default, it caches for 12 hours.
    remoteConfig.fetch(CACHE_TIME_SECONDS).addOnCompleteListener(new OnCompleteListener<Void>() {
             @Override
                public void onComplete(@NonNull Task<Void> task) {
                    if (task.isSuccessful()) {
                        Log.d(TAG, "Fetch Succeeded");
                        remoteConfig.activateFetched();
                    } else {
                        Log.d(TAG, "Fetch Failed");
                    }
                    runExperiment();
                }
            });
  7. In order to run our experiment, we should fetch the experiment_variant parameter from our remote config and use it in our app.
    String experiment = config.getString("experiment_variant");
    
    FirebaseAnalytics.getInstance(this).setUserProperty("Experiment", experiment);
    
    if (experiment.equals("variant_a")) {
        //..
    } else if (experiment.equals("variant_b")) {
        //...
    } else {
        //..
    }

    In this example, we fetch the parameter and set a custom analytics user property with the variant that has been assigned to the current device. This will allow us to properly track how our A/B test is performing.  In the conditional, we are free to make the changes we want to the app. We can set the text to different things and change the colours depending on the variant.

  8. Now that we have the experiment running, we need to track when a user clicks on the “Checkout” button in order to determine which button variant a user is more likely to tap on. We then log the event to Firebase analytics which will include the user property that we set previously.
    buttonCheckout.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(final View view) {
            //..
            firebaseAnalytics.logEvent(“CheckoutClicked”, new Bundle());
        }
    });

    It is worth noting that there are a bunch of default events and parameters that you can make use of with Firebase Analytics. [adwords_square]

  9. In order to determine the success of your A/B test, you need to analyse your data to decide which UI is the most successful. After logging into the console, we can see there were 24 counts of the “CheckoutClicked” event.firebase_analytics_ab_testingAfter applying the filter for the user property “Experiment = Variant A”, we can see that 19 of the 24 clicks were from variant A of the experiment (granted this was just me clicking around in my sample app – but you get the idea).

Variant A filters - Remote config

For larger data sets where the result of the experiment may not be obvious at first, you can export your events to BigQuery. This will allow you to make more complex deductions from your experiments.

For now at least, we know the clear winner (in this very large test 😬) is Variant A – the pink checkout button wins!

AB Testing variant A winner

There you have it, an easy way to experiment with your UI using Firebase Remote Config.

Sample code can be found here (It is not very complex as all the hard work is done by the Firebase APIs.) 😊

Happy AB Testing!


Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.


Comments

6 responses to “A/B Test your App using Firebase Remote Config”

  1. sohamnavadiya avatar
    sohamnavadiya

    Thanks for share the tutorial. Very useful.

  2. Angel Romero avatar
    Angel Romero

    Hi Rebecca,

    Thanks for the article.

    Due to the asynchronicity of the value fetching from Remote Config, it could happen that the app is using a local value on start-up until it receives the remote one. Do you have any thoughts about how to handle this wait?

    In your example, if that’s the only screen in the app and we want to make sure that the user is seeing the button in a certain colour, would you create a splash screen to wait to fetch the value? Would you update the UI according to the value of the experiment once received? Any other alternative?

    Thanks,
    Angel

  3. Hi,
    What I would suggest is to fetch the values in the background, and on next launch of the app, use the new values. So that your users aren’t shocked by a sudden change in the UI. Making the user wait for the values to be fetched isn’t such a nice user experience.

  4. Angel Romero avatar
    Angel Romero

    Thanks a lot for your answer. I agree with everything you mention.
    However, depending on the nature of the application (the frequency the customers use it mainly), pulling regularly in the background may not make sense due to the battery and bandwidth consumption (I know it’s minimum, but maybe it doesn’t pay off).
    I think this is a tricky topic and I couldn’t find too much about industry standards or best practices and neither about how other tools handle this.

  5. Eduardo Bonet avatar
    Eduardo Bonet

    Awesome post, but I just have an addendum to people who are designing the experiments: for an experiment to be valid one would need at least 3 more groups, since groups A, B and no_experiment should also have each a control group. If in the case as it is laid out A has more clicks, there’s no statistical guarantee that it is actually the color that induces more clicks, it could be some random hidden variable. If you have two groups that receive the A experiment, and both respond consistently with higher click count, than the probability of the count increase being explained by the color of the button is much higher. I understand the focus of your post is not explaining A/B testing, but how to implement it, which is perfect, since we as Android devs should no be responsible for it, but I’ve seen way too many people, specially stakeholders and “growth hackers”, designing incorrect experiments that might lead to poor decision making.

  6. The purpose of this article was not to explain too deeply the workings of an AB experiment, as this can take on many forms. You could do more in depth research to see what your users prefer (if you have the time and money to do so). But for a simple app where you just aren’t sure which you users would prefer, this would suffice as a good indication rather than just assuming you know full well what your users prefer. Also experiments can be testing more than one thing at a time. This post is indicating how to use Firebase Remote Config to do the experiment and not the technicalities of running a good experiment. You are welcome to write a post describing technicalities and things you should consider for AB testing and link it here to be included.