What Real-World Notebook Tests Do You Want to See?

At LAPTOP, we’ve been talking for a while about how to make our notebook tests more relevant to our readers. For a long time, we’ve been using a standard suite of benchmarks to test laptops. These include PCMark Vantage for general performance, 3DMark for graphics performance, a Wi-Fi transfer rate test from 15/50 feet, and MobileMark 2007 for battery life and additional performance numbers. We’re not unhappy with our current suite of tests, but we are looking to tweak our procedures and add more real-world testing to the mix. Our reader surveys show that too many users look at benchmark scores, scratch their heads, and say “What does this mean to me?” We’d love your feedback to help us decide which tests to develop and use: Here are our goals in a nutshell:

  • Realism: We know that any test result represents a simulation of user activity, because no two users and no two user sessions are the same. Still, we aim to simulate activities typical users envision themselves performing.
  • Scalability: The tests should be usable across a variety of Windows systems and potentially on other platforms. They should work on everything ranging from desktop replacement systems to mini-notebooks. We will need to automate as much as possible so we can conduct the tests both quickly and accurately.
  • Objectivity: Whenever possible, tests should produce numbers that are based on performance, not human interpretation.
  • Transparency: Any tests we create should be reproducible by our readers, provided they use the same hardware and software. All procedures and scripts we write should be available to the public.

With these goals in mind, we’ve listed a few ideas below that we’re considering and some we’re already implementing. Please post comments with your suggestions and feedback. The Multitasking Test This is the test we need your help developing most. We want to develop an objective way to determine how well a system runs common applications simultaneously. Idea 1: Video Encoding. We could encoding a video while surfing the Web and shuffling MP3s and then use the amount of time the video took to encode as our benchmark. Idea 2: Virus Scan/Movie Playback. Run a full system virus scan while playing back a DVD and see how long the scan takes. Problem here is that systems come with different amounts of preinstalled software so the antivirus software would have more files to scan on some systems than others, making this test potentially unfair. We’re looking for ideas in the multitasking department. What are two or more tasks you frequently perform in combination, and which of these could we time accurately over and over again to judge performance? The Web-Surfing Battery Test We’ve already developed and started using a Web-surfing test to measure battery life with Wi-Fi on. The Web-surfing test is a simple Windows shell script that cycles through a series of 60 popular Web sites, loading each page and then pausing for 30 seconds to simulate a user reading the page. The browser cache in Firefox is disabled so that the entire page must be downloaded each time. Meanwhile, the script writes to a time-log file every 5 seconds. The test is conducted at the notebook’s default power-saving settings, except that nothing—not the hard drive, the screen, or the CPU—is set to go to sleep, and all battery alerts are disabled. We feel like this test is a step in the right direction when measuring battery life with Wi-Fi turned on, because it offers sustained use of the wireless connection while performing a task, Web surfing, that is arguably the most common mobile task. We are gradually moving toward the Web-surfing test and away from synthetic benchmarks such as MobileMark, not only because we think it offers a good real-world snapshot of battery life, but also because we’ve found that synthetic suites like MobileMark simply will not run on a large percentage of systems we test. We plan to make the Web-surfing test code available for public download so anyone can reproduce our results. Does the Web-surfing test sound good to you? What might you change? The DVD Rundown Test On those notebooks with an optical drive, we also want to do DVD rundown tests to measure battery life during playback. Since the type of movie will impact the endurance, we will likely stick with the same DVD for a period of six to 12 months. Since we’ll also use this film to gauge the quality of the notebook’s display and graphics capabilities we will likely choose a title that has a fair number of fast-paced action sequences, such as 300 or Iron Man (when that comes out). The Boot Test Another test we already use but plan to highlight is the boot test. We want to make sure users know exactly how long it will take from the time they hit the power button to the moment they get full control of the desktop. This means that our boot-time number reflects not only the amount of seconds it takes for the OS to load, but also for any preinstalled tray icons to launch. The Application Launch Time Test One big frustration we have when using computers is the amount of time it takes to open popular applications. In a recent Eee PC 1000H article, we measured how long five popular apps took to open by pointing a video camera at the screen and then going back and counting the number of frames from the moment the shortcut is first clicked until the application is fully open. We used Microsoft Word 2007, Adobe Reader (with a large PDF), Premiere, Photoshop, and Firefox 3 as our apps. We could choose others. Is the application open test interesting to you? The Heat Test Because so many users want to put laptops on their laps, we have started measuring ambient heat on some systems by aiming a laser thermometer at the keyboard, touchpad, and underside rear and recording the results. Few publications comment on ambient heat, but it’s an important consideration when shopping for a system (and a bit difficult to test in the store). Are you interested in finding out how hot the keyboard, touchpad, and underside of a notebook are before you buy it? Should we be measuring the heat of anything else? The File Transfer Test To measure the speed of a notebook’s hard drive, we can copy a standard set of files (we’re thinking 5GB of mixed media) to and from the hard drive and measure the time. We would also use this test for solid state drives. Is this something you’d do often in the real world? The Video Chat Test We’ve been doing a ton of subjective testing with Skype video calls, but we’d love to find an objective way to measure video-conferencing performance. Any ideas? The Gaming Test On systems with discrete graphics, we have been measuring the frame rates of F.E.A.R. and Call of Duty 4. And on gaming rigs we typically test more-demanding title like Crysis. Are there games you’d rather see us test with? Your Feedback Needed Now that you know what we’re trying to accomplish, please contribute to the conversation by leaving a comment below. We have a lot of ideas we want to implement, but our most important goal is to provide results that are meaningful to our readers.

AUTHOR BIO
Avram Piltch
Avram Piltch
The official Geeks Geek, as his weekly column is titled, Avram Piltch has guided the editorial and production of Laptopmag.com since 2007. With his technical knowledge and passion for testing, Avram programmed several of LAPTOP's real-world benchmarks, including the LAPTOP Battery Test. He holds a master’s degree in English from NYU.
Avram Piltch on
Facebook Twitter Google+
LEAVE A REPLY
Name*
Email* (will not be published)
Website
*Indicates required field
Comments*
Submit Comments

  1. jenny Says:

    Breakability!

    Also, how well the wireless works–I would rather have a laptop that can pick up a super weak signal.

  2. James Kendrick Says:

    I second the call for a WiFi signal test. Maybe to demonstrate how far from the router it still works. This is a big variance among mobile devices.

  3. Bob Says:

    How about how long it takes to launch itunes?

  4. Kung Fu Says:

    - Wi-Fi signal test
    - Heat test
    - Boot up test

    What about a noise test?

  5. tommy Says:

    I would love to see the LCD panel tested for color profiling
    since most laptop screen cant display color properly ( for photo editing )

    Only 2 notebooks, Apple and Sony are the ones that can do this as far as i know

    Also for heat testing, where is the hot spot is important.
    For example , I found most hot spot is at the palm rest area, making it very
    uncomfortable for prolong usage.
    However if the hot spot is at the key pad area, it is atleast tolerable
    since we dont keep our fingers on the same key

  6. admin Says:

    Right now, our plan is to scan three areas of the computer for heat, possibly four.

    1. The touchpad
    2. Between the G and H keys on the keyboard
    3. Bottom rear, nearest the battery
    4. Possibly, bottom front underneat the touchpad.

  7. Derrill Says:

    In the Multi-tasking tests, toss in some Flash testing, watch a Ustream feed or Hulu while trying to browse the web.

  8. Craig Says:

    1) How long does it take to Hibernate, and resume from Hibernate, in Windows XP
    2) How long does it take to launch Word 2007
    3) How long does the battery last using WiFi and refreshing 5 webpages every 30 seconds
    4) How long does the battery last when playing full-screen video streamed over WiFi
    5) How long does it take to transcode a WMV file into Xvid at VGA resolution
    6) How much brightness difference is there when pivoting the screen up and down 5, 10, and 20 degrees?

  9. Curtis Barker Says:

    How does it perform when loaded up with common malware.

  10. Green hornet Says:

    Some sort of index to determine how green a laptop is in addition to battery tests.

    For example, are Sony’s really as environmentally friendly as they claim?

    Does the bamboo Asus really make much of a difference compare to the non-bamboo equivalent?

FIND A REVIEW
Laptops
All Product Types Accessories Cars Digital Camcorders Digital Cameras eReaders GPS Laptops MP3 & Video Players Projectors Smartphones Software Storage Tablets / MIDs VoIP Wi-Fi
All Subcategories
All Subcategories All-Purpose Budget Business Desktop Replacement Gaming Multimedia Netbook Nettop Rugged Student Tablet PCs Ultraportable
Brand
Acer Alienware Apple Archos ASUS Averatec BenQ CTL Corp. Dell Digital Storm eMachines Emtec Everex Fujitsu GammaTech Gateway General Dynamics Getac Gigabyte Hercules HP HTC iBuyPower Intel Lenovo MSI Nokia Nvidia OCZ OLPC OQO Origin Panasonic Sager Samsung Sony Sylvania Systemax TabletKiosk Toshiba Verizon Viewsonic Viliv VooDoo Workhorse PC ZT Systems
Minimum Rating
Any Rating Editor's Choice 4.5 Stars 4.0 Stars 3.5 Stars 3.0 Stars
Screen Size
10 11 12 13 14 15 16 17 18 20 4 5 6 7 8 9
Resolution
1024x576 1024x600 1024x768 1200X800 1280 x 720 1280x1024 1280x768 1280x800 1366x678 1366x768 1440x1050 1440x900 1600x768 1600x900 1680x1050 1680x945 1920x1080 1920x1200 800x400 800x480
Weight Range
10.1 - 12.0 pounds 12.1 - 14.0 pounds 14.1 - 16.0 pounds 2 lbs 2 pounds and under 2+ lbs 2.1 - 4.0 pounds 4.1 - 6.0 pounds 6.1 - 8.0 pounds 8.1 - 10.0 pounds Over 16 pounds Under 2 pounds
more options
SUBSCRIBE