Increasing Supported Devices and Bug Fixing on Android – Remote Testing

I just spent the past few days trying to fix support for a large number of Android devices. This is the result (bottom to top):-

ArrowMania2-Device-Test-Report

Looking from the bottom to the top, you can see my initial version worked on 105 devices and the final version 171! Not bad!! Well, as I say, I can’t be 100% sure the app is fixed for ALL those devices until it is tested by players in the real world. I haven’t actually tested the app myself, I actually have access to 5 different Android devices, what I’ve done to test and debug here is to use three online services which are:-

Crashlytics.com – Add their SDK to your app, release a new APK and wait for the crash reports to come in. One more bonus is that this has to be the easiest SDK to install ever.

http://www.apkudo.com – A remote testing service which will test your APK on 219 devices from various generations and Android version. The output is a simple but very useful list of Success or Fail with access to what caused a failure in the form of the exception (including the fingerprint file and line number). The main thing this service is lacking is a copy of the devices screen so you can actually check your app is displaying correctly.

http://app.testobject.com – A similar remote service, this time 30+ real devices to test on but the advantantage being you get a copy of the screen the device was showing when running your app, a very good indication your app was actually running correctly and hadn’t crashed after a few seconds. And even better (though you are limited to 30 test minutes before having to pay) you can try your app on real devices via access to these trough a web portal. This remote control works very well, unlike similar I’ve tried in vein to use like Samsung’s Remote Test Lab.

What Made me Do This?

My initial inspiration came from my Google Analytics data, I’d been looking at the user events for a couple of months in dismay knowing there were a lot of devices showing play time of zero seconds – a sign the app wasn’t running on the device. Along with a steadily dropping rating on the Google Play store page, I know I am getting bad ratings from a lot of unsatisfied customers who had downloaded my app only to find it didn’t run on their device.

What Did I end up Fixing?

The initial crash reports were to do with sound or all things. Concurrent access exceptions on a few devices were showing about 10 DNR (did not run crashes) from 1000 downloads. The function in question was a simple setEffectsVolume. I removed all calls to setEffectsVolume and now rely on the user to use the devices own volume control. A bit of a hack but it stopped the exceptions. One side effect was I now had exceptions, though only 2 per 1000 from the stopEffect function in the sound java! I put a try catch around that, will see if it helps over the next few days.

The next problems came with the Google Play Billing code, almost all in the IabHelper code. I’ve blogged about Google Play Billing before, it truly is the most bugged and crash causing code I have ever used. Anyway, I fixed a number of crashes caused by failed setup, simple tests for null Context variables, a dispose that could happen twice, a try catch around some other parts.

Then came failures of Cocos2d-x to initialise on about 8 devices that were on the apkudo.com website. An edit text wrapper interface just seemed to cause a big crash. As I don’t use the edit text system (on screen input) I found where the wrapper class is created and simply commented it out (see my posts here for more detail).

The result, well I’ve uploaded the app to Google Play, I will see in a few days. I have also reduced my minSdkVersion, before I had this set to 10 as I know 50% of Android 2.2 users would find Arrow Mania 2 didn’t work on their device and I didn’t want bad reviews…. but it seems I am getting bad reviews anyway from 2.3.6+ users for crashes anyway! So what the heck, may as well let the 2.2. users have a p