This isn't the first time that phone makers have tried cheating benchmarks to inflate their phones' clock speeds and performance scores. The likes of Samsung, LG, HTC and Sony were caught red-handed doing the same back in 2013 following which benchmark creators implemented new checks and practices not only to catch cheating phone makers, but also to ensure that the "costs of benchmark cheating weren't worth it."
Did Huawei try to pass off a DSLR pic as one of their own?
XDA Developers recently published a post on their site revealing that some phone makers have started cheating benchmark scores yet again. The prime culprits this time are Chinese phone makers OnePlus and Meizu. This was done despite the fact that inflating clock speeds doesn't push up benchmark scores by more than five per cent and the costs of a public backlash are too much to handle once they are caught. According to XDA Developers, this is more like Volkswagen and Fiat Chrysler using software to cheat emission benchmarks only to be caught later and paying tens of billions in fines, aside from facing public humiliation.
Developers were testing performance of Qualcomm's new Snapdragon 821 processor on various devices when they managed to catch OnePlus. They discovered that while the processor boosts CPU clock speeds while opening apps and then resumes idle speeds after opening in devices like Google Pixel XL and Xiaomi Mi Note 2, the processor continued to run high clock speeds in OnePlus 3T even after apps were opened.
OnePlus 3T just got fixed, but Nougat will have to wait
The developers also discovered that while all eight cores in OnePlus 3T's CPU should drop their clock speeds down to 0.31 GHz when the CPU load drops to 0%, they continued to stay above 0.98 GHz during tests. This made them deduce that OnePlus "was entering an alternate CPU scaling mode to pump up their benchmark scores" to target specific applications.
"We brought a OnePlus 3T to Primate Labs’ office in Toronto for some initial analysis. The initial testing included a ROM dump which found that the OnePlus 3T was directly looking for quite a few apps by name. Most notably, the OnePlus 3T was looking for Geekbench, AnTuTu, Androbench, Quadrant, Vellamo, and GFXBench. As by this point we had fairly clear evidence that OnePlus was engaging in benchmark cheating," the developers said.
After OnePlus, ZTE brings in an improved Axon 7 variant
"OnePlus was makings it CPU governor more aggressive, resulting in a practical artificial clock speed floor in Geekbench that wasn’t there in the hidden Geekbench build. It wasn’t based on the CPU workload, but rather on the app’s package name, which the hidden build could fool. While the difference in individual runs was minimal, the thermal throttling relaxations shine in our sustained performance test," they added.
The developers ran similar tests on other devices to see if the issue was more widespread than they imagined. Thankfully, they found that the likes of HTC, Xiaomi, Huawei, Honor, Google and Sony did not indulge in similar malpractices. However, the Meizu Pro 6 Plus which runs Samsung's Exynos 8890 was also caught cheating benchmark scores just like the OnePlus 3T. Developers noticed that the device switched to using its big cores to obtain better performance scores when certain (benchmarking) apps are running.
These are the best performing phones on AnTuTu benchmark
Under normal parlance, the big cores in the device hardly come into play, something which AnandTech noticed which subjecting the phone to benchmark tests. XDA Developers noticed that while Meizu Pro 6 Plus scores around 604 and 2220 on single-core tests, the scores go up to 1473 and 3906 when benchmark apps are opened. This is basically the difference between the phone's on-ground' performance and its performance when benchmark apps are running.
When contacted, OnePlus acknowledged the same and promised not to target specific benchmarking apps which is a good sign for buyers. However, the company said that it will continue to boost CPU clock speeds when targeted gaming apps are open.
‘In order to give users a better user experience in resource intensive apps and games, especially graphically intensive ones, we implemented certain mechanisms in the community and Nougat builds to trigger the processor to run more aggressively. The trigger process for benchmarking apps will not be present in upcoming OxygenOS builds on the OnePlus 3 and OnePlus 3T,’ said OnePlus.
XDA Developers noted that ideally, the OnePlus 3T's CPU should identify situations when it needs to amp up clock speeds but it seems OnePlus will only let higher clock speeds set in when specific apps are open which is not a good sign. Meizu are yet to respond to the allegations as we speak.
Source: XDA Developers