If your app is doing really well in organic search, you are probably reluctant to mess with what is working.

We totally get it.

In this post, we will show you how you can A/B test app preview videos against your current screenshots, with zero risk. If your new video leads to more downloads in testing, you can upload it.

If not, then you can leave your app as-is and stick with the screenshots. Too good to be true?

Keep reading and we will show you how to do it…

Setup A/B App Preview Video Test

The key to testing a new app promo video is to use an A/B testing service that will allow you to simulate an Apple or Google Play App Store page. This keeps your existing live app profile page the same, while testing new ideas.

…or if you have not launched your app yet, these testing platforms can help you test your app’s assets before you launch.

In this example, we will use TestNest to test a video versus existing screenshots on the App Store. Setting up a new test is really simple.

First, select the store, device and assets that you want to test.

Then choose the app you want to do testing on. In this example, we will use Clash of Clans. TestNest will automatically import of the app’s assets to do your A/B test.

Now you can upload your app preview video to TestNest to test it against your existing screenshots.

From here, TestNest will create pages that look like App Store pages that you can use to test your variations. After you run a test, the platform will give you detailed stats on how people interacted with the pages and how quickly they installed your app.

If you have an app on Google Play, then you can do live testing via Google’s built-in app testing tool. The upside to this is that you are testing with live, organic traffic and you don’t have to sign up for a separate service. The downside is that you can’t run simultaneous experiments and you can only run four types of tests.

Either way, now the question becomes, how do you get people to your pages to conduct a statistically significant test?

I’m glad you asked…

Send Traffic to Your Split Test

Next, you will need to send some traffic to these pages to see which version gets the most downloads. This can be an issue if your existing website doesn’t get much traffic, or you don’t even have a website.

Not to fear, that is where paid ads come in.

In my opinion, the best place to start is with Facebook ads. They can be very affordable and they are hard to beat, when it comes to demographic targeting.

I’m not the only one who feels this way. TestNest recommends starting with Facebook ads and so do many other marketers I know. A great resource for Facebook marketing is Jon Loomer’s site.

How to Know if Your Test is Statistically Significant

The next question that you probably have is: How do I know when I have enough conversions to be sure that my test is statistically significant?

That is not always a straightforward answer, but this post offers some great tips, including the importance of A/A testing. Optimizely also offers a calculator that will give you the sample size that you will need to have to conduct a valid test, based on three simple factors.

Test Multiple Assets

If your app video tested better than your screenshots, then great! You should now add add your video to your app’s live app store page with confidence.

But don’t stop there.

You can split test multiple videos.

Test your app’s description.

Test your app icon.

Remember, the key to App Store Optimization is continuous testing.

Conclusion

So if you are wondering if an app promo video will improve your downloads, but you don’t want to make changes to your live app profile page just yet, then this kind of A/B testing is exactly what you need.

Of course, there is no guarantee that your results will be exactly the same. But since testing tools like TestNest closely mimic app store pages, the results are usually similar.

This is an easy, low risk way to test your ideas…no matter how crazy they are.