Commissioning Misleading Core i9-9900K Benchmarks

As expected, Intel officially announced their new 9th-gen Core processors this week along with the Z390 chipset. We wrote all about it here with some brief commentary on what to expect considering the embargo for benchmarks and full reviews is not lifted until October 19.

Although we're technically not bound by this as we didn’t sign an NDA to get our sample, for a number of reasons we are going to wait, out of professionalism and respect for other reviewers who will be sinking days upon days of work into their day-one review.

However, when PCGamesN published early Core i9-9900K results today we were a little surprised. The title read "Intel’s Core i9 9900K is up to 50% faster than AMD’s Ryzen 7 2700X in games," sounded bogus to me, but I read on...

"Intel has now officially announced its new Core i9 9900K processor, proclaiming it as the “world’s best gaming processor.” It’s not just some marketing bluster either… well, not entirely. Intel has commissioned Principled Technologies to do a benchmarking sesh on its newest 9th Gen chips, and their competition, across 19 of the most popular PC games."

So Intel can go and publish their own "testing" done suspiciously through a third party ten days before reviews, while reviewers are prohibited from refuting the claims due to the NDA. First bad sign.

Scrolling down PCGamesN says the following when looking over Intel’s commissioned benchmarks...

"But the real point of all this is for Intel to be able to hold out the 9900K as hands down the best gaming processor compared with the AMD competition, and in that it seems to have excelled. On some games, such as Civ 6 and PUBG, the performance delta isn’t necessarily that great, but for the most part you’re looking at between 30 and 50% higher frame rates from the 9900K versus the 2700X."

Right away many of the results looked very suspect to me, having spent countless hours benchmarking both the 2700X and 8700K, I have a good idea of how they compare in a wide range of titles and these results looked very off. Having spotted a few dodgy looking results my next thought was, why is PCGamesN publishing this misleading data and why aren’t they not tearing the paid benchmark report apart? Do they simply not know better?

Over at the Principled Technologies website you can find the full report which states how they tested and the hardware used. Official memory speeds were used which isn’t a particularly big deal, though they have gone out of their way to handicap Ryzen, or at the very least expose its weaknesses.

Ryzen doesn’t perform that well with fully populated memory DIMMs, two modules is optimal. However timings are also important and they used Corsair Vengeance memory without loading the extreme memory profile or XMP setting, instead they just set the memory frequency to 2933 and left the ridiculously loose default memory timings in place. These loose timings ensure compatibility so systems will boot up, but after that point you need to enable the memory profile. It’s misleading to conduct benchmarks without executing this crucial step.

Still, it would almost be fair if they had done the same for Intel, but they didn’t. For all Intel platforms they first set the memory to XMP and then adjusted the frequency manually, handling Intel a significant performance advantage, particularly for games.

The next step in their manipulation of the results was to only test at 1080p with a GTX 1080 Ti using quality presets that were a step or two down from the maximum level. In many cases this simulates the kind of performance we see when testing at 720p using ultra quality presets. Of course, we also test at 1080p and 1440p as well to give readers the full picture.

One of the worst results picked by PCGamesN to show was from Ashes of the Singularity. Let’s ignore the 9900K for now since I can’t show you those results, instead let’s focus on the 8700K and 2700X. Here the 8700K was 29% faster than the 2700X, that’s a much bigger margin than I would expect to find. Principled Technologies are using the built-in benchmark, the CPU focused benchmark with the game running in the DirectX 12 mode with the high quality preset.

So, I installed two Corsair Vengeance Pro DDR4-3200 modules, loaded the XMP profile on both the AMD and Intel platforms and ran the tests using the exact same settings. I also re-ran the tests with the XMP memory timings but at the official memory speeds for each CPU. I’m only going to expose three of the game results, I didn’t feel the need to spend more time on this test after a couple of hours was enough to make a point...

Here are the results for Ashes of the Singularity, let’s talk about them. Compared to the paid results produced by Principled Technologies using the stock memory my 2700X was 18% faster, that’s a shocking result already.

The 8700K, on the other hand, was 4% slower and this meant compared to the 2700X it was 4% faster and up to 9% faster with the higher clocked memory. That however is nothing like the 29% performance advantage Intel enjoyed in the paid testing.

Next I looked at Far Cry 5 and here the change wasn’t quite as extreme for the 2700X, it was just 3% faster in my test and 10% faster with 3200 spec memory. However the 8700K was slower again, 7% slower with the 2666 memory that Principled Technologies used. Even with 3200 memory I couldn’t match their result.

This means while they claim the 8700K is 26% faster in Far Cry 5, in reality it’s more like 14% or 12% with 3200 memory, still a clear win for Intel but not nearly as extreme as the Principled Technologies benchmarks would lead you to believe.

The last set of results that I looked into were for Assassin’s Creed: Origins and again they used the built-in benchmark with the 3rd highest quality preset at 1080p with a GTX 1080 Ti. Here the 8700K was 36% faster according to Principled Technologies when in reality it’s more like 8% with stock memory speeds and 10% with overclocked memory.

So it’s quite obvious that the Principled Technologies results are a load of rubbish and no one should be reproducing them. Of course, the focus here for Intel was to highlight how great the Core i9-9900K is and unfortunately right now I can’t show the real results for that CPU. Needless to say though, it won’t be 50% faster than the 2700X in games.

The 9900K will be faster than the 2700X for gaming, no doubt about it, but it’s also going to cost twice as much once you factor in the motherboard price. On average we found when using tuned memory for both the 2700X and 8700K, the Intel CPU was ~9% faster at 1080p using a GTX 1080 Ti. Realistically, we’re expecting the 9900K to be a few percent faster when compared to the 8700K, at least in games as most titles aren’t coming close to tapping out the 6-core/12-thread processor.

I don’t have too much of an issue with Intel commissioning the report itself, and the Principled Technologies report is very transparent as they clearly state how they tested the games and configured the hardware. The results and testing methods are heavily biased, but they haven’t attempted to hide their dodgy methods. You can dig into the specs and find all the details, it’s still dodgy but it’s a paid report, so it’s somewhat expected.

The misleading benchmarks from Nvidia regarding their GeForce 20 series launch was pretty bad, this though is on another level and I seriously hope we don’t see these results published anywhere else with the intention of promoting the 9900K. Intel is under pressure, we get that, but ultimately this kind of move is making the company a disservice.

We expect more on this subject to pop up in the coming days and it goes without saying, please wait for our independent i9-9900K review and the reviews of other trusted media outlets to arrive late next week. Tomorrow we also expect to follow up with our thoughts on the new unlocked 28-core Xeon workstation CPU and Intel Core X-series processors, see where those stand against Threadripper.

Let's block ads! (Why?)