6. What Happens When Benchmarks Contradict?

The DHCAT benchmark represents a serious and sweeping exercise of media capabilities in modern PCs. Even some of the fastest and best media center PCs we've worked with have earned only modest scores on this assessment. That said, Intel informed us that simply switching from Windows XP to Vista on the same hardware will generally increase scores by 15-20% - we plan to put this claim to the test as soon as we get our Vista benchmarking act together, probably some time later this summer. We also can't help but observe that the difference between Nero Media Home and ViiV media servers on the Intel processors creates the kind of margin that, if added to the AMD processors, would maintain the rankings we observed from recent runs for PCMark05 and 3DMark05 on those same systems.

Though we think that DHCAT provides useful measures for media PC capability and performance - especially if you take the time to observe how high-definition media playback looks when anywhere from two to four other tasks are running in the background - we also question the relevance of including media server tasks in the basic line-up of assessments. Certainly, finding a workable media server version proved the biggest challenge in getting DHCAT working. Also, an informal poll of a dozen fellow media PC enthusiasts turned up only three who were actually using a media server on their home networks: two were running MythTV back-ends for multiple front-ends, and only one was running a ViiV media server.

When we questioned Intel about why they decided to make the media server component part of DHCAT, and pointed out the relatively small portion of the user base that actually runs such software, their response was that "a good benchmark must also seek to lead the way into capabilities that will become more commonplace in the future." It's hard to dismiss this notion out of hand, but it's also hard not to notice that the jury is still out as to whether media servers and media extenders will retain their small niche status or be elevated into a common home networking fixture. To our way of thinking, this might or might not happen any time soon, and it seems strange to give something that is still emerging such a central place in what is otherwise a pretty representative, if aggressive, set of common media tasks and activities.

Summary And Conclusions

When it comes to finding a place in our standard collection of benchmarks, we see some value in including DHCAT in the mix, but certainly, only where media PCs are concerned. For such systems, it's interesting to observe our own usage patterns and compare them to the various parallel tasks combined in so many of the DHCAT scenario elements. We do often record one program while viewing another one, which requires a fair amount of disk activity, encoding on at least one stream of data, plus transcoding on at least one stream as well. We seldom, if ever, edit photos or transcode materials on the PC that we use as a digital video recorder, though. Our experience has been that the machines we use to work on are not the same machines we use to view, process or store our media.

In addition to presuming that media servers will become commonplace enough tomorrow to justify their inclusion in benchmarking today, Intel also seeks to assess media PC performance with workloads that exceed those that are typical, based on our own experience and on talking to lots of other media PC aficionados. Maybe we're suffering from a lack of vision, or we haven't yet really caught onto the fullest possible range of uses for media PCs, and maybe Intel has a better idea. If you look closely at the various scenario elements that Intel sets up inside DHCAT, though, many of them go somewhat beyond what we typically ask our media PCs to do, and some of them go further out than that.

In fact, as we take another long, hard look at the mix of tasks that go into the scenarios - where over half of each one's items feature three or more simultaneous tasks underway - it's hard not to come to some interesting conclusions about what kinds of PCs would excel with such workloads. To us, it appears that an ideal DHCAT machine should include the following components:

A quad core processor or a server motherboard with two dual core processors, to provide the best results for handling up to four simultaneous tasks. Copious memory, to provide ample workspace for up to four data- and compute-intensive tasks at once. This would surely help speed streaming video through, and also offer better buffering for encoding, transcoding, compression, and so forth. 4 GB sounds like a good amount, but 8 GB might be even better. A speedy RAID array, to handle the large amounts of data moving into and out of the processor(s). Even a 10,000 RPM Western Digital Raptor might find this challenging. We'd be inclined to try out RAID 1, RAID 5 and RAID 10 to see what works best. A fast, cutting edge GPU with built-in hardware decoders for H.264 and other common video formats should also help quite a bit. We'd vote for nothing less than an 8600 class Nvidia graphics card in such a system, along with whatever complementary/competitive product offering emerges from AMD/ATI. We want to run Windows Vista on our paragon computer, because of the performance advantages for DHCAT that Intel ascribes to that operating system.

If you put all of these things together - and we'd like to, as soon as the opportunity presents - the odds appear good (at least to us) that DHCAT results should improve, possibly even by a substantial amount. But the question remains: how much does such a machine resemble the hundreds of thousands, if not millions, of media PCs already happily chugging away in family and living rooms around the world?

Does this make DHCAT too far out to attend to when benchmarking media PCs? Not really, but it is worth examining DHCAT's detailed results and putting the most weight on those scenario elements that most closely match your typical usage patterns. We think our other benchmarks make a good contrast and complement to DHCAT, and we intend to keep on running those on our media PCs, while we also intend to keep running DHCAT to see how its results compare to our other tests.