3. Battery And Performance Benchmarks

The two main "gotchas" in mobile computing testing are battery life and performance. There are many testing strategies that we might have chosen to assess these, including rolling our own. After looking at the options, we decided to use four existing benchmarking suites - two from the Business Applications Performance Corporation (BAPCo) and two from FutureMark Inc.

BAPCo is a non-profit consortium of many of the major players in computing, including AMD, Intel, HP, Dell, ATI, NVIDIA, Toshiba and Microsoft, that produces benchmarking software. FutureMark is a private corporation that works closely with major manufacturers in its Benchmark Development Program. These include AMD, Intel, Microsoft, NVIDIA, ATI and Dell. BAPCo and FutureMark work cooperatively; each organization is featured on the other's web site.

BAPCo's benchmarking suites are open to examination, and users of the benchmarks are encouraged to send them on to BAPCo for inclusion in a database of benchmarks for different computers. In fact, in order to publish the results of our benchmarking tests, we must report our findings to BAPCo. This not only allows BAPCo to include our tests in its database, but also provides the consortium with information it needs to better design its next round of tests.

We use BAPCo's MobileMark 2005 and SYSmark 2004 SE benchmarking suites. We use MobileMark 2005 to benchmark laptops and notebooks running on batteries, and SYSmark 2004 SE for laptops and notebooks running on external power. Since all notebooks and laptops can run on battery or external power, we run both sets of benchmarks on each model we evaluate.

Like BAPCo, FutureMark is very open; you can find benchmarks for a wide range of computers on the company's web site. We use FutureMark's PCMark05 and 3DMark benchmarking suites. PCMark05 tests the system as a whole, as well as components such as CPU, disks, RAM memory and graphics components. It runs tasks both singly and in combination. PCMark05 is for benchmarking computers designed for home/office use above all else, so it fits nicely into our home and office laptop and notebook evaluation mission. 3DMark05 focuses on how well a computer does gaming and modeling tasks. It is best for laptop and notebook computers with higher end graphics components. Both FutureMark tests are run with the computer on AC power.

A more detailed discussion of our four benchmarks follows.

MobileMark 2005

MobileMark 2005 focuses on the entire laptop or notebook system. It uses popular real world applications to test laptop or notebook battery life and generate a performance rating. Commercial applications are driven by scripts that simulate the actions of a real user, and include pauses to mimic people's work patterns. During any test, the laptop's or notebook's keyboard and mouse must remain untouched or the scripts and the test will terminate, requiring the entire test to be run over again. Additionally, in rare cases and depending on the computer, a particular script may terminate early because focus is lost on a particular window. This also causes a failure and requires a rerun.

MobileMark 2005 includes four benchmarking groups:

Office productivity Research and reading DVD playback Wireless browsing

The office productivity benchmarks use Adobe Photoshop 6.0.1, Macromedia Flash 5.0, McAfee Virus Scan 5.13, WinZip Computing WinZip 8.0 and Microsoft Word, Excel, PowerPoint and Outlook 2002.

In addition to measuring battery life, MobileMark 2005 computes a performance rating for the office productivity benchmark. This rating is based on the average response times required to complete the various operations in the tests, such as the time between clicking the mouse to initiate a recalculation in Excel and the completion of the recalculation. Initiator steps - such as the time it takes to enter text at the keyboard or click the mouse - are not measured, because in the real world they would depend on the person using the computer.

The MobileMark 2005 office productivity rating shows how a laptop or notebook performs while its battery is discharging. A rating of 100 indicates that the computer under test achieved the same response time rating as a calibration system used by the developers of the test. A rating of 200 means the test computer has twice the MobileMark 2005 performance as the calibration computer. (See the MobileMark 2005 white paper referenced below for more on the calibration computer used by BAPCo.)

The research and reading benchmark uses Netscape Communicator 6.01 to access an eBook. This simulates someone doing research and reading the information found. No network connection is used for this benchmark; all browsed data is on the laptop or notebook being tested.

For the DVD benchmark, MobileMark 2005 measures battery life while InterVideo WinDVD 6.0 plays a provided DVD containing commercial video content. The test includes not only video, but sound as well.

The Wireless browsing benchmark is done using a wireless server and the test laptop or notebook. The server is booted off a CD ROM and runs a version of Linux. The server connects to a wireless access point through a standard RJ-45 network cable, while the test computer connects wirelessly to the wireless access point. The version of MIcrosoft's Internet Explorer currently installed on the laptop or notebook is used for wireless browsing benchmarking.

For more information on MobileMark 2005 benchmarks see the BAPCo white paper.

The major function of MobileMark 2005 is to measure battery life in a variety of situations, though it also calculates an office productivity performance rating. All four MobileMark 2005 benchmarks begin only when the laptop's or notebook's battery charge reaches 100% and external power is removed. The tester's only tasks, other than starting MobileMark 2005 and selecting from among the four benchmark groups, are to pull the plug on the laptop or notebook to start testing and plug it back in after the battery has been fully discharged. (Before you apply for a job in our testing labs, you should know that our testers have to be at the ready during the day and sometimes during the night to start a new test or restart an old one that didn't quite make it through all of the scripted scenarios!)

We test a mobile computer with the battery that ships with the specific model we are evaluating. Higher capacity batteries are available as an option for a most computers, and can significantly extend the battery life. Generally, you can expect the battery life of an optional battery to be higher than that of the standard battery. The battery life score of the optional battery should be proportional to that of the standard one in the same ratio as the two batteries' capacity ratings. If a manufacturer provides a higher capacity battery along with the standard one, we run only the MobileMark 2005 Office Productivity benchmark with the larger battery installed.