How to Benchmark Your Android Device

How powerful is your phone? The question would have been nonsensical a decade ago, but as we increasingly depend on smartphones and tablets for some of our everyday computing needs, paying more attention to their performance makes sense. Until recently, choosing a smartphone was mainly a matter of aesthetic taste and mobile OS preference. Today, however, most of the devices available prioritize their display and use similar slate or candy-bar designs. As a result, performance has become a much more prominent differentiator.

Unfortunately, many mobile device owners are falling into the same trap that desktop PC consumers did years ago: They look at a list of specifications, see that a smartphone's processor is clocked higher or has more cores, and assume that the device is faster than one that has what appear to be lower specs. It's as though the MHz myth of the 1990s is back to confuse buyers in a totally new market segment.

The standard SoC (system on a chip) powering one of today's mobile devices, however, is advancing at a rapid pace. A device produced just a few months ago may have the same numbers of processor cores and be clocked at the same frequency as a brand new device, but its underlying architecture may be far less efficient. In addition, the two devices' SoCs may be outfitted with different GPUs, different amounts of memory, and different versions of the mobile OS. Because of these hidden variables, a vendor's list of specs doesn't reveal the whole story. As is true with desktop and laptop PCs, establishing the relative performance of mobile devices requires testing.

Getting Started

Since Android's ecosystem is far more diverse than its mobile OS rivals', we'll focus on testing an Android-based device here. A handful of the tests that we'll be covering here are cross-platform or run within a Web browser, meaning that they'll work on iOS or Windows Phones-based devices--or even on desktops and notebooks--as well.

In evaluating the performance of an Android-based smartphone or tablet (or any other modern smartphone or tablet), it's best to think of the device as a tiny PC. When benchmarking a full-fledged desktop PC, we don't declare that system faster or slower than another after running a single test that stresses a particular component. Instead, we run a host of different tests to gauge how well various individual components work, then we weigh the results, and finally we reach a judgment about overall performance. The same approach is appropriate for Android-based devices.

A number of benchmark tools are available online and in the Google Play market for testing device performance. We'll discuss a handful of free tools here, but many other benchmarking tools are available, too. Some (like Vellamo from Qualcomm) are offered by hardware manufacturers; others are created by students. We recommend using established tools whose makers have demonstrated a capacity and willingness to issue updates; ultimately, however, you should tailor your benchmark suite to your unique usage habits. If Web browsing is your highest priority, use apps that test browser and JavaScript performance. If you spend most of your time on mobile devices playing games, be sure to run graphics/GPU-centric tests.

Benchmarking Procedures

As with PC performance testing, there are some standard procedures to follow when running benchmark tests on Android-based devices. Apps or utilities running in the background on an Android-based device can sap system resources and interfere with the benchmarks, resulting in inconsistent or misleading scores that don't accurately reflect real-world performance. To eliminate this variable, your best course is to shut the background apps down. Many Android-based devices include built-in Task Managers that will turn off unnecessary background apps, thereby freeing up memory and processor resources. Restart your device, wait a few minutes to ensure that everything has loaded completely and then use the task manager to kill unneeded apps and maximize available memory. Wait a few moments longer, and then begin a fresh test.

Processor, Memory, Storage, and Overall Performance Tests

Linpack for Android (available at Google Play store, free)

Linpack for Android measures the CPU's computation speed.

Linpack for Android is a quick and easy-to-run benchmark that measures how fast an Android device can solve a dense N by N system of linear equations. Linpack is purely a CPU benchmark, and it can run in either single-thread mode (to assess the performance of a single CPU core) or multithread mode (to tax all of the available processing cores in a device). To start running the tests, you simply launch the app, choose the single-thread or multithread test, and then click the appropriate button in the main menu.

AnTuTu System Benchmark (available at Google Play store, free)

AnTuTu System Benchmark tests an Android device's CPU, GPU, memory and storage performance.

AnTuTu System Benchmark is an all-in-one benchmark designed to run tests on your device's CPU, GPU, memory, and storage. The CPU benchmark evaluates both integer and floating-point performance. The GPU tests assess 2D and 3D performance. The memory test measures available memory bandwidth and latency. And the storage tests gauge the read and write speeds of a device's flash memory. To run AnTuTu, swipe to the Test tab, select the tests that you'd like to run, and then click the Start button. When the tests are complete, the results will appear on the Scores tab.

CF-Bench (available at Google Play store, free)

CF-Bench tests both native and managed Java code performance.

CF-Bench is another all-in-one benchmark, created by respected Android developer Chainfire (hence the CF). Chainfire originally built the tool for his own use, but a couple of years ago he released it to the public. CF-Bench is a CPU, memory, and storage benchmark tool that can exploit the additional computing resources of multicore SoCs; and it produces consistent, repeatable scores, while testing both native and managed Java code performance. To run CF-Bench, launch the app and click the Full benchmark button. Results will appear at the bottom of the screen, along with a handy comparison to other popular devices.

Next: GPU/graphics, network, and browser performance