PDA

View Full Version : GFX 5900 Caught Cheating



thechemgeek
06-02-03, 06:03 AM
saw this here (http://www.megagames.com/news/html/hardware/gfx5900caughtcheating.shtml). interesting...


"We have now established that nVidia's Detonator FX drivers contain certain detection mechanisms that cause an artificially high score when using 3DMark03. We have just published a patch 330 for 3DMark03 that defeats the detection mechanisms in the drivers and provides correct results." this is how Futuremark introduce their latest patch for their popular 3DMark03 software.

Saratoga, Calif.-based Futuremark issued a statement claiming that nVidia tweaked software needed to run its new GeForce FX 5900 chip, in order to distort performance in Futuremark's 3DMark 03 testing application. Futuremark is one of the leaders in software and services for performing PC benchmark tests.

According to Futuremark when the patch was applied a drop of as much as 24.1 per cent was observed in certain nVidia products, while competition products performance-drop stayed within the margin of error of 3 per cent.

An nVidia spokesperson said Since nVidia is not part of the Futuremark beta program (a program which costs of hundreds of thousands of dollars to participate in), we do not get a chance to work with Futuremark on writing the shaders like we would with a real applications developer. We don't know what they did, but it looks like they have intentionally tried to create a scenario that makes our products look bad.

ATI meanwhile are not too keen to capitalize on another, possible, embarassment for their rivals. ATI's Dave Baumann, partly in response to nVidia's statement said: Despite still being a full Futuremark Beta member, ATI did not make it out of the report entirely unscathed either. There is a performance difference of about 8 per cent in Game Test 4, that accounts for about a 2 per cent difference in the final 3DMark03 score, between the new and old versions, indicating that although not visually different something was occurring on this particular test.

In order to add further validity to the importance of benchmarking tests ATI's Christ Evenden stated: The 1.9 per cent performance gain comes from optimization of the two DX9 shaders (water and sky) in Game Test 4 . We render the scene exactly as intended by Futuremark, in full-precision floating point. Our shaders are mathematically and functionally identical to Futuremark's and there are no visual artifacts; we simply shuffle instructions to take advantage of our architecture. These are exactly the sort of optimizations that work in games to improve frame rates without reducing image quality and as such, are a realistic approach to a benchmark intended to measure in-game performance. However, we recognize that these can be used by some people to call into question the legitimacy of benchmark results, and so we are removing them from our driver as soon as is physically possible. We expect them to be gone by the next release of CATALYST.

What does all this mean however? Most knowledgeable gamers do not buy a card based, solely, on the score achieved in a benchmarking test. It is reliability, performance and value for money which usually determine which card is bought. Benchmark tests are mainly utilized in order to claim the performance crown, giving the bearer increased prestige and an advantage created by the publicity. All that cheating on tests will achieve is to render such tests useless and to create consumers who do not trust the manufacturers. Unfortunately, this time, the reputation of the biggest graphics chip manufacturers has been questioned and there is little chance that they will make it out of it without some doubt remaining in the publics mind. Doubt which can only be cleared by the release of solid, reliable and high quality products.

Brent
06-02-03, 09:25 AM
http://www.ice.org/~slothy/oldnews.jpg

ub3r_n00b
06-02-03, 09:40 AM
lol amen brent



-Preet

Cypher
06-02-03, 04:22 PM
When I finally decide to buy my next card I'm gonna trust Brent and the rest of you on that one.

YeOldeStonecat
06-02-03, 08:01 PM
Futuremark changes their mind...not calling it "cheating" anymore.

*************

http://www.anandtech.com/#19622

FutureMark and NVIDIA have released a joint statement on the earlier snafu concerning Nvidia bungling 3DMark03 scores:
Futuremark Statement

For the first time in 6 months, as a result of Futuremark's White Paper on May 23rd, 2003, Futuremark and NVIDIA have had detailed discussions regarding NVIDIA GPUs and Futuremark's 3DMark03 benchmark.

Futuremark now has a deeper understanding of the situation and NVIDIA's optimization strategy. In the light of this, Futuremark now states that NVIDIA's driver design is an application specific optimization and not a cheat.

The world of 3D Graphics has changed dramatically with the latest generation of highly programmable GPUs. Much like the world of CPUs, each GPU has a different architecture and a unique optimal code path. For example, Futuremark's PCMark2002 has different CPU test compilations for AMD's AthlonXP and Intel's Pentium4 CPUs.

3DMark03 is designed as an un-optimized DirectX test and it provides performance comparisons accordingly. It does not contain manufacturer specific optimized code paths. Because all modifications that change the workload in 3DMark03 are forbidden, we were obliged to update the product to eliminate the effect of optimizations identified in different drivers so that 3DMark03 continued to produce comparable results.

However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source. Futuremark will consider whether this approach is needed in its future benchmarks.

NVIDIA Statement

NVIDIA works closely with developers to optimize games for GeForceFX. These optimizations (including shader optimizations) are the result of the co-development process. This is the approach NVIDIA would have preferred also for 3DMark03.

Joint NVIDIA-Futuremark Statement

Both NVIDIA and Futuremark want to define clear rules with the industry about how benchmarks should be developed and how they should be used. We believe that common rules will prevent these types of unfortunate situations moving forward.
*********************

thechemgeek
06-02-03, 09:38 PM
Originally posted by Brent
http://www.ice.org/~slothy/oldnews.jpg Shak'ti'qua

koldchillah
06-02-03, 09:40 PM
Originally posted by Brent
http://www.ice.org/~slothy/oldnews.jpg

:rotfl:

at least I was only a day behind on this one.. :D

you must be getting tired of us slow people by now.. lol

thechemgeek
06-03-03, 01:29 AM
oh yea....well....

http://www.wyattsplace.com/files/oldnews.jpg

A_old
06-03-03, 01:45 AM
Originally posted by thechemgeek
oh yea....well....

http://www.wyattsplace.com/files/oldnews.jpg


VERY funny pic. haha brent is in it.. i want to OPTIMIZE my diablo 2 character hehe