Intel's i9 9900K vs Ryzen 2700X Gaming Benchmarks Are Misleading, Period

A meaty set of competitive benchmarks for Intel's newly announced 9th Gen Core i9-9900K are in, pitting the powerful mainstream gaming CPU against AMD's Ryzen 2700X. There are plenty of them to chew through. Nineteen in fact, comprising many of the most popular PC games out there. Unfortunately, they're all bogus.

Intel's Core i9-9900KIntel

These benchmarks came to light courtesy of PCGamesN which published a selection of results supplied by Intel. Well, sort of. Intel commissioned a third-party -- in this case Principled Technologies -- to conduct exhaustive gaming testing with the Core i9-9900K, Core i7-8700K and a handful of AMD Ryzen processors like the 2700X and Threadripper 2950X.

The headline reads "Intel’s Core i9 9900K is up to 50% faster than AMD’s Ryzen 7 2700X in games." The article is framed mostly as fact.

AMD's Ryzen 2700X currently retails for about $299. Intel's upcoming i9-9900K will launch later this month at $488, or 68% higher than the Ryzen 2700X -- and unlike its AMD competitor, the i9-9900K ships without a cooler.

So when we see a headline like that and consider the price difference, it's fair -- I would even argue mandatory -- to expect that kind of performance advantage. Unfortunately, the comparative testing wasn't conducted fairly. Not remotely.

What They Did Right

Principled Studies was quite verbose about their testing methodology, leaving almost no guesswork to how they conducted their benchmarks. System hardware was nearly identical except for motherboards and CPUs. The latest GeForce graphics drivers was utilized. Each system had its BIOS updated. The median score across three separate runs was used. They used an Nvidia GTX 1080 Ti, limited the resolution to 1080p and relied on medium to high settings (instead of ultra) to expose any CPU bottlenecks.

So far so good. Fair and through. Granted, the only result that backs up this 50% higher performance claim was Ashes of the Singularity, a game I think people benchmark more than they actually play.

What They Did Very Wrong

Reviews of the i9-9900K are still under embargo and we won't see independent testing for more than a week. So that means the only results we have are the ones Intel commissioned. There's no reason to suspect that the FPS numbers we're seeing with the i9-9900K aren't honest, but remember that reviewers have no way of refuting those claims either, as they're bound by an NDA.

But Hardware Unboxed has extensive experience testing the other CPUs in these benchmark tests, and seeing the published scores for both the Ryzen 2700X and i7-8700K set off their, shall we say, B.S. detector.

Steve Walton from Hardware Unboxed got to work replicating a couple of Principled Technologies' test rigs and studied precisely how they conducted their benchmarks. That's when he noticed the fatal flaw.

"They have gone out of their way to handicap Ryzen," says Walton. The problem lies in how the company utilized the system memory in its testing. And it's a serious problem because one crucial step -- activating and configuring an XMP profile -- was conveniently omitted on Ryzen systems.

"[they] left the ridiculously loose default memory timings in place," Walton writes at TechSpot. "These loose timings ensure compatibility so systems will boot up, but after that point you need to enable the memory profile. It’s misleading to conduct benchmarks without executing this crucial step. Still, it would almost be fair if they had done the same for Intel, but they didn’t. For all Intel platforms they first set the memory to XMP and then adjusted the frequency manually, handling Intel a significant performance advantage, particularly for games."

Using his similar test systems and properly configured memory on both platforms, Walton noticed some glaring discrepancies.

In Ashes of the Singularity, his Ryzen 2700X was 18% faster. Even more fascinating is Walton's Assassin's Creed: Origins result. In the published report by Principled Technologies, Intel's i7-8700K was 36% faster than AMD's Ryzen 2700X. Even when Walton overclocked the memory, his result shows the 8700K only being 10% better.

The bottom line is this: always wait for the independent reviews. This was a study clearly commissioned to present Intel's competitive advantage and to make its new product look good. Of course it was. And I believe Intel when it says the i9-9900K will be the fastest mainstreaming gaming CPU on the market. But by how much, exactly, and will it be worth the price?

Thanks to just how transparent Principled Technologies was with their testing methodology, we have the early stages of very concrete proof that the results you may see leading up to the embargo dropping are, to put it plainly, bogus.

Let's block ads! (Why?)