Just to be clear – let’s remain “hard on the problem” not on the people.
Mobile performance is a whole new bailiwick. In the coming months and years lots will be written and I’m sure there will be lots of “heated debates”.
Ultimately though it all comes down to your methodology. The title of this post is Blaze.io vs. 5o9’s Browser2. Two companies “two completely different approaches to testing”.
If you want, you can review Blaze’s methodology by going to this link In short they have a collection of Agents (they don’t define what an Agent consists of) that are running on actual devices. They all connect to a central server running webpagetest.org software. All of the Agents connect to the web via “dedicated high speed WI-Fi.
So to recap – There’s an actual device with a high speed Wi-Fi connection running an agent which talks to a central server running http://www.webpagetest.org/ software.
What’s missing?
The explanation of how, or rather whom gets the web page. If you look at webpagetest.org you will see that it’s designed to get web pages. So my “assumption” here is that the Agent is actually something like PCAP (http://pcapperf.appspot.com/) which is actually a packet sniffer that sits on a laptop (or server) and then when the “Agent” makes a request it reads the packets and generates a HAR report.
The key is whose getting the data? As we don’t know what the “Agent” is there’s no way of really telling.
Secondly it’s sitting on a Wi-Fi connection. This is NOT the same as a carrier connection.
Thirdly – we don’t know where the “Agents” are located. Are they all in the same building as the server or are they spread out around the country?
Again the “devil is in the measurement details”.
So now lets take a look at how 5o9’s Browser2 does it. Well first of all we’re a “browser” NOT an Agent. We sit on any Android device and can access the Internet both via Wi-Fi and 3G or 4G carrier networks. When we run a test we access not only device data, but also carrier information and real time geo-location data.
Note – Mobile is “well Mobile”. Testing in a building is NOT mobile.
Our test results – they’re published with all the same detail as Blaze’s except we add even more data. We’ve actually gone inside the browser and captured ALL of the webkit event messages. You can see a sample at this link. Click on the HAR tab at the top and you’ll see more detail than you ever imagined possible about what’s going on inside the Browser.
In short we’ve taken a “show me the data” approach to Mobile Performance.
As the inventors of Mod_Gzip we know a thing or two about Performance. At some point you have to “lay down” your cards and reveal the timing metrics.
Blaze.io does a great job. All they’ve done is taken a different approach to the measurement problem that’s all. Nothing wrong with that. Unless….
Remember “it’s all about the numbers”.
