Mobile benchmarks ain't nothin' but a number

Okay, everyone, the jig is up: Those benchmark numbers are all a ruse. Mostly. A number of Android manufacturers have been caught cranking up the CPU speeds on their phones and tablets to score higher on processor benchmark tests. Samsung and LG are among the worst offenders, but even HTC has been caught futzing the numbers on its One and One miniphones.

This is exactly why we don’t include benchmarks in our Android device reviews (though we do perform battery run-down tests). Your only determining factor in buying a new device should be how the device performs as you use it, not some arbitrary numbers spit out by a benchmark application. Those numbers won’t always reflect the user experience, and they shouldn’t be the determining factor in whether or not a device is worth buying.

Image: Michael Homnick
Are these phones really as fast as their manufacturers peg them to be?

Take the Samsung Galaxy Note 10.1, for instance. The device scored well in benchmark tests because Samsung had tweaked its processor to make it rev faster for benchmark apps. Once you got your hands on it, though, you had a completely different story. In our own review, we noted that its performance was inconsistent, and it exhibited constant lag as we played games and launched apps.

The Sony Xperia TL was another device that looked great on paper but didn’t deliver on its benchmark results. It was based on the Xperia T handset, which scored well in Geekbench tests and other benchmarking apps. The phone appeared as a promotional product in tandem with the release of the movie Skyfall, but it was hardly a gadget you could refer to as the “Bond phone.” Instead, it was bogged down by a bloated overlay and plagued by constant slowdowns, despite its benchmark numbers. Its design was also awkward and antiquated.

On the other hand, the Moto X didn’t shine as a speed demon in benchmark tests, but as a day-to-day device it’s a relatively snappy phone without any input lag at all. Rather than tweak the processor to yield higher benchmark results, Motorola focused on its always-on, always-listening capabilities—a feature that is just as marketable as processor speed, but more relatable for the common consumer.

Windows Phones could be lumped into this category, too. Most of the latest Nokia Lumia devices are still using comparatively pokey processors, and though they’re usually touted for their photo-taking abilities, they’re all solid phones, and responsive to boot. It’s not the CPU inside that necessarily counts, but how the operating system uses it to serve the user.

At least you can take comfort in the fact that not every company is gaming the system to earn higher benchmark scores. Google’s Nexus and Motorola’s phones play fair, save for the Google Play editions of the HTC One and the Samsung Galaxy S4, and Anandtech points out that it’s mostly the OEMs that are to blame for the inflated numbers.

Benchmarks serve a purpose in testing raw processor performance, but that can be difficult to determine with the variety of hardware configurations among individual devices. The real disappointment here is that the manufacturers have corrupted a system that many have relied on for buying decisions. It makes me wonder whether it’s worth supporting a company that would go to such lengths to make itself look better among its competitors.

The bottom line here is that mobile benchmarks aren't trustworthy, and even if they were, they're a poor indicator of how a device feels to use. It’s the experience that makes a gadget worth your money—not how impressive its benchmark numbers are.

For comprehensive coverage of the Android ecosystem, visit Greenbot.com.

Subscribe to the Smartphone News Newsletter

Comments