October 1, 2008

Reality Check: Speed Testers - Your Mileage May Vary

By Jason Schnitzer

Just how fast is my broadband connection, really? Is my neighbor getting better performance than I am? Should I switch service providers? It used to be faster; didn't it used to be faster?

In truth, I pay little attention to all of this broadband speed talk. I'll opt for a 5 Mbps connection over some 'Extreme Turbo' tier offering 10 times the data rate. That said, I'm still keen to learn if I can achieve anywhere near my subscribed service speed limit - even if I do prefer to drive in the slow lane.

What I need is an Internet speed tester. Good news: A quick search points me to several sites offering them.

Away I go. Following one of the returned links to broadbandreports.com, I arrive at a popular site where broadband subscribers gather to exchange the latest broadband wag and compare notes on service satisfaction (or otherwise). One source of comparison is a set of speed tester tools offered on the site.

Selecting a speed test is easy, with a visually pleasing menu of possible servers presented. I assume the idea is to select the server closest to me. None found in my neck of the woods, so I go with one in Texas. With a click, the test begins, and shortly after, the results are displayed.

The testing process is quite rewarding; the results are less so. My results are well short of what I understand my subscribed rate to be. Immediately repeating the speed test gets me a different result. This time, my latency measurement has improved by roughly 20 percent, download speed has increased by about 5 percent, and upload speed has gone down slightly. Intrigued by the variation, I repeat the test for a total of 10 samples.

Not a statistically significant sample, but over the 10 speed test measurements taken, downstream speed varied by 14 percent, upstream speed by 3 percent, and latency by just under 100 percent. Interesting.

Trying a second speed test tool, also offered by broadbandreports.com, this time I chose a server in San Francisco. Again, I go for 10 successive speed test runs. With this tool, the results are a bit more consistent, but contrast greatly to those generated by first speed test tool and server in Texas. The average upstream speed was lower by 110 percent, the average downstream speed decreased by 10 percent, and the reported average latency increased by 150 percent.

I didn't change my service provider while I ran these speed tests. So why are these measurements all over the map?

First, these are two different speed test servers measuring over network paths from two different topological locations. In reality, any speed tester server located external to my provider's managed network will produce measurements that include someone else's network. The longer the network distance between the speed test server and my provider's management domain, the less meaningful the result.

Second is the anatomy of the speed test application itself. These are software applications that vary widely in their design and implementation - with the internal details often closed source and unknown. Different protocols, test traffic blends, implementation languages, runtime environments, and host system configurations are just a few of the many things that can determine the application's behavior and influence speed test results.

Finally, there's the subscriber's network configuration. Using a wireless connection to a $40 router in your college dorm room? Chances are you'll see different results than if you plugged directly into your operator's terminating equipment via Ethernet.

I realize that my little speed test experiment isn't hard science. And that's precisely the point. The accuracy, meaning and significance of results produced by these Internet speed testers is open for debate.

Given these factors, should these speed test results really be used as a meter stick to evaluate my service? Perhaps of greater significance, is it appropriate that these speed tests have become an anecdotal metric for evaluating ISP bandwidth management practices as fodder for the great network neutrality debate?

The one consistent result: Mileage may vary.

Jason Schnitzer is the founder and principal of Applied Broadband. Reach him at jason@appliedbroadband.com.