What Real-World Notebook Tests Do You Want to See?

At LAPTOP, we’ve been talking for a while about how to make our notebook tests more relevant to our readers. For a long time, we’ve been using a standard suite of benchmarks to test laptops. These include PCMark Vantage for general performance, 3DMark for graphics performance, a Wi-Fi transfer rate test from 15/50 feet, and MobileMark 2007 for battery life and additional performance numbers.

We’re not unhappy with our current suite of tests, but we are looking to tweak our procedures and add more real-world testing to the mix. Our reader surveys show that too many users look at benchmark scores, scratch their heads, and say “What does this mean to me?” We’d love your feedback to help us decide which tests to develop and use:

Here are our goals in a nutshell:

• Realism: We know that any test result represents a simulation of user activity, because no two users and no two user sessions are the same. Still, we aim to simulate activities typical users envision themselves performing.
• Scalability: The tests should be usable across a variety of Windows systems and potentially on other platforms. They should work on everything ranging from desktop replacement systems to mini-notebooks. We will need to automate as much as possible so we can conduct the tests both quickly and accurately.
• Objectivity: Whenever possible, tests should produce numbers that are based on performance, not human interpretation.
• Transparency: Any tests we create should be reproducible by our readers, provided they use the same hardware and software. All procedures and scripts we write should be available to the public.

With these goals in mind, we’ve listed a few ideas below that we’re considering and some we’re already implementing. Please post comments with your suggestions and feedback.

The Multitasking Test
This is the test we need your help developing most. We want to develop an objective way to determine how well a system runs common applications simultaneously.

Idea 1: Video Encoding. We could encoding a video while surfing the Web and shuffling MP3s and then use the amount of time the video took to encode as our benchmark.

Idea 2: Virus Scan/Movie Playback. Run a full system virus scan while playing back a DVD and see how long the scan takes. Problem here is that systems come with different amounts of preinstalled software so the antivirus software would have more files to scan on some systems than others, making this test potentially unfair.

We’re looking for ideas in the multitasking department. What are two or more tasks you frequently perform in combination, and which of these could we time accurately over and over again to judge performance?

The Web-Surfing Battery Test
We’ve already developed and started using a Web-surfing test to measure battery life with Wi-Fi on. The Web-surfing test is a simple Windows shell script that cycles through a series of 60 popular Web sites, loading each page and then pausing for 30 seconds to simulate a user reading the page. The browser cache in Firefox is disabled so that the entire page must be downloaded each time. Meanwhile, the script writes to a time-log file every 5 seconds. The test is conducted at the notebook’s default power-saving settings, except that nothing—not the The storage device that holds your OS, programs, and data.
Learn Morehard drive, the screen, or the CPU—is set to go to sleep, and all battery alerts are disabled.

We feel like this test is a step in the right direction when measuring battery life with Wi-Fi turned on, because it offers sustained use of the wireless connection while performing a task, Web surfing, that is arguably the most common mobile task. We are gradually moving toward the Web-surfing test and away from synthetic benchmarks such as MobileMark, not only because we think it offers a good real-world snapshot of battery life, but also because we’ve found that synthetic suites like MobileMark simply will not run on a large percentage of systems we test.

We plan to make the Web-surfing test code available for public download so anyone can reproduce our results. Does the Web-surfing test sound good to you? What might you change?

The DVD Rundown Test
On those notebooks with an optical drive, we also want to do DVD rundown tests to measure battery life during playback. Since the type of movie will impact the endurance, we will likely stick with the same DVD for a period of six to 12 months. Since we’ll also use this film to gauge the quality of the notebook’s display and graphics capabilities we will likely choose a title that has a fair number of fast-paced action sequences, such as 300 or Iron Man (when that comes out).

The Boot Test
Another test we already use but plan to highlight is the boot test. We want to make sure users know exactly how long it will take from the time they hit the power button to the moment they get full control of the desktop. This means that our boot-time number reflects not only the amount of seconds it takes for the OS to load, but also for any preinstalled tray icons to launch.

The Application Launch Time Test
One big frustration we have when using computers is the amount of time it takes to open popular applications. In a recent Eee PC 1000H article, we measured how long five popular apps took to open by pointing a video camera at the screen and then going back and counting the number of frames from the moment the shortcut is first clicked until the application is fully open.

We used Microsoft Word 2007, Adobe Reader (with a large PDF), Premiere, Photoshop, and Firefox 3 as our apps. We could choose others.

Is the application open test interesting to you?

The Heat Test
Because so many users want to put laptops on their laps, we have started measuring ambient heat on some systems by aiming a laser thermometer at the keyboard, touchpad, and underside rear and recording the results. Few publications comment on ambient heat, but it’s an important consideration when shopping for a system (and a bit difficult to test in the store).

Are you interested in finding out how hot the keyboard, touchpad, and underside of a notebook are before you buy it? Should we be measuring the heat of anything else?

The File Transfer Test
To measure the speed of a notebook’s hard drive, we can copy a standard set of files (we’re thinking 5GB of mixed media) to and from the hard drive and measure the time. We would also use this test for solid state drives. Is this something you’d do often in the real world?

The Video Chat Test
We’ve been doing a ton of subjective testing with Skype video calls, but we’d love to find an objective way to measure video-conferencing performance. Any ideas?

The Gaming Test
On systems with Graphics chips are responsible for processing all images sent to your computer?s display.
Learn Morediscrete graphics, we have been measuring the frame rates of F.E.A.R. and Call of Duty 4. And on gaming rigs we typically test more-demanding title like Crysis. Are there games you’d rather see us test with?

Your Feedback Needed
Now that you know what we’re trying to accomplish, please contribute to the conversation by leaving a comment below. We have a lot of ideas we want to implement, but our most important goal is to provide results that are meaningful to our readers.