I had a chance this morning to take a quick peak at the latest offering from S.Souders and Blaze.io It’s an attempt to become as they say in their words “a permanent repository for mobile performance data.”
The only problem is that it lacks any meaningful context and does NOT appear to be at all accurate. For example here’s one of the stats that’s posted…
- The New York Times is in the top 5 pages with the most JavaScript with 230 kB on desktop and 326 kB on mobile – 94 kB more JavaScript on mobile than desktop.
Really? How do we know that? Where’s the actual comparison test of the desktop version and the mobile version. To check this out I went to Blaze.io’s site and ran a test using a iOS 4 Here’s the actual report they generated… link
In short, on the iPhone they downloaded 1.4MB of data in 13.82 seconds. If you look at the HAR data (link) it doesn’t even “redirect” to the Mobile web site (something featured as part of the HTTP archive).
So just for grins I ran exactly the same test on my own desktop. I have a 22Mbps Comcast connection. The total download was 1.8MB in 8.83 seconds.
Think about that for a moment. The iPhone is only 5 seconds slower than a 22Mbps connection to a desktop browser. Clearly something is very wrong here.
Next I ran the same test on Firefox with firebug (cache cleared). This time I got 1.1MB in 8 seconds
I’m sorry to say that without a lot more context the HTTP Archive can not be relied on to revel accurate performance data.