Ashes of the Singularity: the first DX12 gaming benchmark tested
Up to 75 per cent faster than DX11 - but controversy surrounds the release.
The embargo lifts today on the first DirectX 12 gaming benchmark - a scripted series of sequences taken from Oxide's forthcoming Ashes of the Singularity. Our results paint quite a picture: on the one hand, we see some phenomenal gains on the AMD side, while on the other, Nvidia performance looks somewhat underwhelming. This has provoked a polite exchange of words highlighting a difference in opinion between the GPU vendor and the developer, and Nvidia has even distributed its own 'reviewer's guide' for the benchmark to the press.
Our own testing methodology is fairly straightforward. The key advantages of DX12 we'll see in the first wave of games are primarily in the CPU area - so we wanted to test two equivalent graphics cards from AMD and Nvidia on two different consumer-level CPUs. We went for the Radeon R9 390 and the GTX 970 as our graphics hardware of choice, paired with the Core i7 4790K (up to 4.4GHz with four cores and eight threads) and a somewhat less capable Core i3 4130 - a dual-core, quad-thread CPU running at 3.4GHz. We ran the game on medium settings with no MSAA, selections we made in order to move CPU utilisation to the forefront and to give DX12 an opportunity to shine.
Curiously, the Ashes of the Singularity benchmark also has a DX12 benchmark purely for the CPU - a frame-rate average that completely eliminates the GPU from contention, giving an idea of theoretical top-end performance. On the i7, that's around 78.5fps and on the i3 that drops down to 41.5fps. So in looking at the average scores in the table below, you can see that the i7 is pretty heavily GPU-bound.
One thing that became clear after our testing as we studied video captures is that the AMD card would only run at DX12 with v-sync enabled (all the other benches are run with v-sync off - standard benchmarking practise). So this limits top-end performance on the R9 390 to 60fps, though in practise it rarely reaches this limit. Results will also take a very slight hit as in many places the GPU will be waiting for the next display refresh before it draws the next frame. We've reached out to Oxide about this, but in the meantime, to make a meaningful comparison, we re-benched all of our AMD tests with v-sync enabled to match the presentation we were getting from DX12.
| 1080p/Medium (Low/Avg FPS) | R9 390 DirectX 11 | R9 390 DirectX 12 |
|---|---|---|
| Core i7 4790K (4.4GHz, Four Cores, Eight Threads) | 15.4/28.4 | 32.1/48.8 |
| Core i3 4130 (3.4GHz, Two Cores, Four Threads) | 12.0/24.1 | 29.2/39.9 |
However, even with the v-sync limitation, the results are something of a revelation with the R9 390. Average frame-rates with the Core i3 processor rise from 24fps to 40fps - a 67 per cent boost to performance. On top of that, even the i7 sees a big increase - rising from an average of just 28fps up to 49fps, a 75 per cent uplift. In both cases, lowest recorded frame-rates double - from 12 to 29fps on the i3 and from 15 to 32fps on the i7. But probably the biggest takeaway from the graph is that the i3 running DX12 is significantly outperforming the i7 on DX11 with the same GPU running the same game at the same settings.
It's a CPU-heavy test, but it does demonstrate that DX12 is going to be a big deal - but we do have to bear in mind that DX11 performance from AMD is currently something of an issue - this contributes significantly to the big gulf we're seeing here. To illustrate the extent of this, in the same benchmark, Nvidia's DX11 performance on a GTX 970 is 28 per cent faster on the i3 and 45 per cent faster on the i7. Bearing in mind that the GTX 970 is generally a slightly slower card than the R9 390, that's quite remarkable - and API overhead is the most likely culprit.
So the question is, where does Nvidia stand here in terms of DX12 performance? Well, at the moment, this is an area of controversy. While there are some gains seen on the i3 side of things when paired with a GTX 970, we actually saw a drop in overall performance on the i7 - DX11 actually runs around 14 per cent slower here. Suffice to say, the R9 390 is significantly faster in DX12 (13 per cent on the i7, 4 per cent on the i3) than the Nvidia card in DX12 mode. The green team's in-house testing - contained in its reviewer's guide and run on a six-core Core i7 5930K running at 3.5GHz - also sees a significant drop, though curiously, results vs DX11 improve when clock-speed is dropped down to 2.0GHz. You can look at their numbers here and here.
"The alpha benchmark will run a series of pre-selected scenes and will give you a score at the end to show how well your system ran that series of scenes. The game looks intriguing, but this alpha benchmark is primarily useful to understand how your system runs a series of scenes from the alpha version of Ashes of Singularity," Nvidia said in a statement released to the tech press this weekend.
"We believe there will be better examples of true DirectX 12 performance and we continue to work with Microsoft on their DX12 API, games and benchmarks. The GeForce architecture and drivers for DX12 performance is second to none - when accurate DX12 metrics arrive, the story will be the same as it was for DX11."
For its part, Oxide has published a lengthy blog explaining its position, and dismissing Nvidia's implication that the current build of Ashes of the Singularity is alpha code, likely to receive significant optimisations before release:
"It should not be considered that because the game is not yet publicly out, it's not a legitimate test," Oxide's Dan Baker said. "While there are still optimisations to be had, the Ashes of the Singularity in its pre-beta stage is as or more optimised as most released games. What's the point of optimising code six months after a title is released, after all? Certainly, things will change a bit until release. But PC games with digital updates are always changing, we certainly won't hold back from making big changes post launch if we feel it makes the game better!"
Our take? Well, we've been vocal about our need for AMD to up its game in terms of the efficiency of its driver when evidence emerged of it holding back the performance level of its excellent hardware on less capable processors. If there's one big positive here, it's that with DX12, the firm has jumped back into contention - performance is always king and this is a crucial step forward. With regards Nvidia's puzzling performance, we were hoping that looking at performance in context with our tools would help, but the results we were able to glean defy our best attempts at analysis. Quite frankly, the notion of DX12 running slower than DX11 in some scenarios isn't what we expected to see. Whether it's a game-specific issue, or a driver-related one, you can be sure that Nvidia's engineering team are digging deep into this benchmark now in an effort to figure out what's going on. We'll update with any news.