MediaTek defends itself after benchmark cheating accusations

After reports emerged suggesting MediaTek has been cheating the benchmarking system, the chipset manufacturer has vehemently defending its position.

It has been alleged in AnandTech that MediaTek has been cheating the mobile enthusiasts with some clever code. In the firmware files, references were found tying benchmark apps to a so-called ‘sports mode’. When triggered (if a benchmark app has been initiated), features on the phone were ramped up to give the impression of better performance.

AnandTech claims the cheating was brought to light thanks to testing two different OPPO Reno 3 devices. The Reno 3 Pro (the European version) beat the Reno 3 (the Chinese version) in the PCMark benchmark utility, despite its Helio P95’s Cortex-A75 CPU cores being two generations older than the Dimensity 1000L’s Cortex-A77 CPU cores. And not only did the Reno 3 Pro has older MediaSet chipsets than the Reno 3 devices, it had half as many.

The difference in the test results were slightly unusual, though when a ‘stealth’ benchmark apps were used, the lower results were confirmed.

Why those in the industry feel it is necessary to cheat benchmarking tests is anybody’s guess. The negatives of being caught far outweigh the gains of impressing a few hyper-geeks, and the cheaters eventually get caught. It is embarrassing and some might ask whether they are a reliable partner. The chipsets in questions have been used in OPPO, Vivo, Xiaomi and Sony devices.

Following the original statement, which you can see at the foot of the article, an expanded blog post was offered to the industry.

“We do find it interesting that AnandTech has called into question the benchmarking optimizations on MediaTek powered devices, when these types of configurations are widely practiced across the industry,” MediaTek said. “If they were to review other devices, they would see, as we have, that our key competitor has chipsets that operate in the exact same way – what AnandTech has deemed cheating on device benchmarking tests.”

Although this is a very reasonable explanation, it is still a bit fishy. It is perfectly understandable for performance to be ramped up for some applications, but the fact the ‘sports mode’ has been linked to the initiation of a benchmarking app as well as other functions (gaming for instance) suggests the aim is to fool the tests. Most reasonable individuals would assume these tests are performed in ‘normal’ mode.

Whether this is an adequate explanation, we’ll let the court of public opinion decide, but it is somewhat of a flimsy excuse.

Original MediaTek statement:

MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets. We work closely with global device makers when it comes to testing and benchmarking devices powered by our chipsets, but ultimately brands have the flexibility to configure their own devices as they see fit. Many companies design devices to run on the highest possible performance levels when benchmarking tests are running in order to show the full capabilities of the chipset. This reveals what the upper end of performance capabilities are on any given chipset.

Of course, in real world scenarios there are a multitude of factors that will determine how chipsets perform. MediaTek’s chipsets are designed to optimize power and performance to provide the best user experience possible while maximizing battery life. If someone is running a compute-intensive program like a demanding game, the chipset will intelligently adapt to computing patterns to deliver sustained performance. This means that a user will see different levels of performance from different apps as the chipset dynamically manages the CPU, GPU and memory resources according to the power and performance that is required for a great user experience. Additionally, some brands have different types of modes turned on in different regions so device performance can vary based on regional market requirements.

We believe that showcasing the full capabilities of a chipset in benchmarking tests is in line with the practices of other companies and gives consumers an accurate picture of device performance.

Huawei caught being naughty with benchmark scores, but it doesn’t seem to care

Four Huawei smartphone models have been delisted from independent benchmark service UL after they were found to be producing artificially high and misleading benchmark scores.

After testing the devices in the UL lab and confirming claims breached rules, the P20, Pro Huawei Nova 3 and Honor Play models have been delisted from UL’s performance rankings. The Huawei P20 has also been removed after AnandTech’s testing and reporting, the outlet which initially discovered Huawei’s suspect behaviour, was taken into account.

“Over the last five years, more than a few smartphone manufacturers have been caught and shamed for trying to manipulate benchmark scores,” the team said in a blog post. “That’s why UL has clear rules for manufacturers that govern how a platform can interact with its benchmarking software.

“After testing the devices in our own lab and confirming that they breach our rules, we have decided to delist the affected models and remove them from our performance rankings. Delisted devices appear unranked, and without scores, at the bottom of our popular list of the best smartphones. 3DMark scores from delisted devices should not be used to compare models.”

Each model was tested with the public version of UL’s benchmarking tool 3DMark, available from Google Play, and a private version of 3DMark that is not available to the public or manufacturers. As you can see from the performance results below, it does appear Huawei has been caught out gamifying the public version of the tool, while the private version unveils the real story.

Huawei Performance UL

When using the public version of the testing tool, there seems to be a hidden ‘performance mode’ that overrides the devices’ usual power profile, cheating the test. Scores using the public version of the tool were up to 47% higher than want UL’s private tool reported. Smartphones are allowed to adjust their performance depending on workloads, but they cannot be taught to maximise their behaviour when specific benchmark apps are running.

Huawei hasn’t necessary created a device which performs better than rivals on the market, it’s just figured out how to trick the tests. UL doesn’t unveil the mechanics of the private version, so Huawei devices would not be able to recognise this version of the app, thus giving a more genuine reading of typical performance.

This is of course not the first time manufacturers have been caught out trying to trick the performance rankings. Back in 2013, Samsung was caught out for the same naughtiness with its S4 model, which kicked off broader scepticism in the industry. Several more investigations were launched and several more manufacturers found out for mischief.

Perhaps the most interesting aspect of this story is Huawei’s response to the original accusation of dodginess. It didn’t offer any denial of wrong-doing, simply said that everyone else was it as well. That’s alright then.

Ultimately you have to wonder why Huawei bothered trying to trick its way up the leadership board. Such rankings are designed to get a very small segment of society fidgety and over-excited, it is a very niche group of nerds. Most ‘normal’ people, Huawei’s target audience as a mainstream brand, probably couldn’t care less. Fancy branding and clever advertising would more likely convince this audience to purchase a Huawei device as opposed to hyper-geeky performance stats.

The gain was seemingly very small, but now Huawei looks like a bit of a d*ck. Not exactly the more risk-sensible decision.