I'm interested. I built something to track events for analytics long ago ~7 years. Constantly surprised it's stood the test of time. Coincidentally the loading portion is similar to the GA loading portion analyzed in the article.
Lots of questions, I would love to understand how GA are capturing events that are sometimes trapped and not allowed to bubble, etc. Are they capturing events at document.body level, etc? What type of IDs do they use to bring sessions together, do the ID events similarly? Do they queue up events to batch them or do they trickle them in. How do they deliver events when unloading a page, etc.
Google Analytics does not capture any DOM events. Data gets into the "Events" report in the GA interface when a developer explicitly makes a function call like `ga('send', 'event', Category, Action, Label);`. Google Tag Manager can automatically add tracking to several types of DOM interactions. GTM captures events at the document.body, and you would generally need a work-around if something is screwing with event bubbling.
Nowadays, connecting Sessions and Users are handled by storing an ID in a first-party cookie named _ga. There is no such thing as an Event ID in Google Analytics, unless you implement one manually in a Custom Dimension. The Event report has Total Event and Unique Event metrics, and it's up to the user to choose whether they want to pay attention to deduplicated Events or not.
On the web, hits are not batched, they are handled as they come in (except for a throttling limit). The Android and iOS SDK batch hits by default.
On pageload, queued hits can be sent with the "beacon" protocol, which is a HTML standard that exists specifically to solve this exact problem. In browsers that do not use the beacon protocol, the hit is simply dropped.
Any reference to the beacon protocol? I've never seen or heard of this. I've implemented mechanisms to periodically drain queues and clean up onbeforeunload for single page apps. Does GA's event queue span pages?
The "protocol" is just a deferred HTTP POST request that does not block page unload. Please use this as it is widely supported and provides better UX (page is not blocked by sync XHR).
If you could analyze the old ga.js and figure out a way to accurately duplicate the way the old utmz cookie was created, you could probably make some money off your effort. I've yet to see a great solution to this.
How old is "old", and what do you need it for? I've written code to parse the utmz cookie and separate code to recreate the whole organic/referral/utm attribution model in GA using first party log data. The only major drawback AFAIK is that adwords utm parameters are obfuscated and Google doesn't provide a way to resolve glclids to their campaign's utm params.
I've reverse-engineered most of their minified code, and it's a huge pain. Since they use closure compiler, there's a huge object pool that gets referenced for every single thing. It's supposed to make the source a little smaller, but geez.
Unless I'm misremembering, something that I found interesting was that the cookie wasn't randomized, it was a fingerprint based on a large number of params.
Yes, and when I was 5, I took apart my parents rotary telephone and figured out how it worked. I'm sure there was a document somewhere that already explained it... but for me, actually "discovering" it led to a new level of understanding. I'm confused by your use of "Sherlock" as a pejorative. I actually like the author's use of deduction as a learning tool... instead of just vomiting knowledge read elsewhere.
> I'm confused by your use of "Sherlock" as a pejorative.
It's not a pejorative, it's sarcasm. You could replace "Sherlock" with "This genius" in that sentence for essentially same effect.
I'm not really sure why, but for some reason "Sherlock" has become very common to use in a sarcastic context like that. Kinda like how "fat chance" and "slim chance" now mean basically the same thing due to "fat chance" almost always being used sarcastically.
Regardless... even though what the author discovered is common knowledge to those who've done some work with GA tracking (or perhaps simply researched interesting ways to add elements to the DOM). I applaud the curiosity that lead to this blog post.
Nah. The author did the equivalent of saying they would tell us how the iPhone works, but got as far as just opening the box it comes in and powering it on.
Something that has always puzzled me is why only GA follows the saner approach to push all the function calls in an array which the async loaded script can pick up later. Many of the popular analytics solutions (like, Segment, Mixpanel) create a factory function that initialises all API calls with a generic body. It seems rather unnecessary and only adds to the boilerplate. Take a look at the GA's init code and compare it with Segment's [1].
Some libraries (fb, ga) using a single factory method that handles all commands.
Ex: fb('init', {})
The others (segment, heap) use explicit functions for different actions.
Ex: heap.init()
The main benefit is you can detect API violations without loading the async script. If you call an invalid function with the former you have to wait for it to execute at some point in the future.
This allows you to surface errors immediately - breaking current exection - even if it's additional libraries havent loaded.
Having written one of those analytics snippets... Hindsight is 20:20. When I wrote it, having a facade class seemed reasonable. Eventually I realized the GA approach would've been better, but at that point we couldn't change it because we were worried it would freak out our customers.
The reason why the arguments spell isogram is because you can't have two arguments with the same letter. So they chose the most meta isogram, "isogram." Surprised the author missed that and assumed it was just reference to the script itself.
I've noticed several minified JavaScripts that have the variables spell words like the i s o g r a m here. Haven't bookmarked any so I can't find an immediate example. I thought it was just a curiosity, but I think the frequency is increasing. I wonder why. Just for giggles? Or is there any particular reason?
I believe that Google set a trend, 'HotJar' is a newbie on the block. If anyone can find an example that predates 'urchin.js' (or wherever this started) please share!
I don't know where it started for software, but I can give you an example that predates the web.
In the 80s and early 90s my mother was an interior designer at a high end furniture store. There were no prices on the floor models because most things could be customized (fabric, arm shape, wood, etc) before delivery. However there was a base price that the designer could use to (I guess) keep people within their budget.
They used a ten-letter isogram to print the base price covertly on the furniture labels so that associates could figure it out without going to the back office and looking it up in the computer.
The last word I remember them using was Cumberland. C=0, d=9. So if the base price for a dining room table was $15,840 the tag would have urnec written on it somewhere.
that's the kinds of insightful comments I come here for. Side note, I remember in a gift / greeting card shop growing up, everything had long numbers printed on a little sticker on the back, turns out the price was just the last 4 numbers (divided by 1000).
Using 1-character variable names saves a small amount of network bandwidth on a massive scale (for every site that uses Google Analytics). You then might as well make the 1-character variable names spell out a word, and that word has to be an "isogram" in order to provide unique variable names.
They explain what the variables mean in the function header block;
* @param {Window} i The global context object.
* @param {HTMLDocument} s The DOM document object.
* @param {string} o Must be 'script'.
* @param {string} g Protocol relative URL of the analytics.js script.
* @param {string} r Global name of analytics object. Defaults to 'ga'.
* @param {HTMLElement} a Async script tag.
* @param {HTMLElement} m First script tag in document.
It might be easier to read the code if they used different names, but any good refactoring tool will replace them with your own names without any problems.
I have minimised it in my own case to just this (using ga.js instead of analytics.js because this part is shorter—regardless of the fact that ga.js itself is longer—and not worrying about any fanciness that I don’t need):
Piwik is an open source GA alternative [0]. It doesn't share many of the goals of Fair Analytics. I also don't see much actual analytics in Fair Analytics, just data collection.
Agreed, FA looks like raw material compared to piwik. There's a lot of components needed to build a full solution. You need browser capture, data sink, backend aggregation, dashboards, adhoc query capability, etc...
Yes, and it's available on https://sandstorm.io so you don't have to host it yourself either.
I actually much prefer the Piwik interface, I don't need to dig through masses of menu options to find the data I actually care about. Not helping the Google panopticon is a bonus.
I really don't see that being used on anything but hobby sites. Their definition of privacy essentially means it doesn't work.
For me, Google Analytics privacy issue is that it's used everywhere - Google can track users across a fair chunk of the web. Your site running its own private Piwik install? Fill yer boots, not a problem.
For all those here mentioning that he's only "deconstructed" the snippet, and not the actual main linked analytics script, you're the first to observe that he's deconstructed the snippet incorrectly.
Don't get me wrong, it's really great and heartening to see someone new to JS getting back to basics and trying to understand things in proper minute detail (instead of doing another tutorial about a boilerplate React app), but this is hardly HN front-page material? It's a very simple JS snippet.
Anyone not wanting to use the JS might be interested to know that the Google Analytics Measurement Protocol is fully documented, and you can create your own front-end implementation, should you wish:
This is awesome, it includes IP override and what not. I wonder why more people don't invoke this information server side during their own requests instead of loading the third party JS on the frontend.
One thing to be aware of is that latent hits, via the Queue Time "qt" paramter, have a maximum delta of 4 hours:
"Used to collect offline / latent hits. The value represents the time delta (in milliseconds) between when the hit being reported occurred and the time the hit was sent. The value must be greater than or equal to 0. Values greater than four hours may lead to hits not being processed."
Would be nice if that delta time could be greater than 24 hours, eg. a human approves that a low-volume contact form submission isn't spam and that hit is tied to a GA goal.
The reason it is a function that calls itself is to not pollute the global namespace. The rewritten expanded example leaks "gaScript" to the global namespace which is a little rude if you are injecting your script onto a third-party page and using a random or long function name is not as clean a solution since it still leaves dirt and takes more bytes.
I'm not sure if it's actually intentional, but given the command-queue nature of the ga function, i[r].q's resemblance to IRQ could be a cute reference. (Assuming i and r were fixed, p=1/26 for using q... science!)
(Originally noted in some old article that I can't find right now.)
It is pretty obvious that the snippet loads the real script. I expected this article to be about what the actual google analytics tracking script does rather than the tracking script loader.
> Interestingly, the arguments passed to the function spell out i, s, o, g, r, a, m, is a “term for a word or phrase without a repeating letter” (source), which I guess makes sense, given that the script looks like it has the bare minimum of characters possible.
I thought that was a kind of recursive joke (not necessarily a very witty one) : if they want to make the parameters spell out a word, the word has to be an isogram, otherwise you'd define a parameter twice. And since "isogram" is an isogram, maybe the temptation was too great..
85 comments
[ 5.5 ms ] story [ 168 ms ] threadLots of questions, I would love to understand how GA are capturing events that are sometimes trapped and not allowed to bubble, etc. Are they capturing events at document.body level, etc? What type of IDs do they use to bring sessions together, do the ID events similarly? Do they queue up events to batch them or do they trickle them in. How do they deliver events when unloading a page, etc.
Nowadays, connecting Sessions and Users are handled by storing an ID in a first-party cookie named _ga. There is no such thing as an Event ID in Google Analytics, unless you implement one manually in a Custom Dimension. The Event report has Total Event and Unique Event metrics, and it's up to the user to choose whether they want to pay attention to deduplicated Events or not.
On the web, hits are not batched, they are handled as they come in (except for a throttling limit). The Android and iOS SDK batch hits by default.
On pageload, queued hits can be sent with the "beacon" protocol, which is a HTML standard that exists specifically to solve this exact problem. In browsers that do not use the beacon protocol, the hit is simply dropped.
GA does not store any data that persists from page to page other than the client ID cookie.
The "protocol" is just a deferred HTTP POST request that does not block page unload. Please use this as it is widely supported and provides better UX (page is not blocked by sync XHR).
https://deedpolloffice.com/blog/articles/decoding-gclid-para...
Unless I'm misremembering, something that I found interesting was that the cookie wasn't randomized, it was a fingerprint based on a large number of params.
for more on async loading techniques, this is a great blog post.
http://blog.errorception.com/2011/08/reliable-high-performan...
It's not a pejorative, it's sarcasm. You could replace "Sherlock" with "This genius" in that sentence for essentially same effect.
I'm not really sure why, but for some reason "Sherlock" has become very common to use in a sarcastic context like that. Kinda like how "fat chance" and "slim chance" now mean basically the same thing due to "fat chance" almost always being used sarcastically.
that's a nice line and describes some very high karma HN denizens.
[1]: https://prnt.sc/fpj9ec
The others (segment, heap) use explicit functions for different actions. Ex: heap.init()
The main benefit is you can detect API violations without loading the async script. If you call an invalid function with the former you have to wait for it to execute at some point in the future.
This allows you to surface errors immediately - breaking current exection - even if it's additional libraries havent loaded.
I believe that Google set a trend, 'HotJar' is a newbie on the block. If anyone can find an example that predates 'urchin.js' (or wherever this started) please share!
In the 80s and early 90s my mother was an interior designer at a high end furniture store. There were no prices on the floor models because most things could be customized (fabric, arm shape, wood, etc) before delivery. However there was a base price that the designer could use to (I guess) keep people within their budget.
They used a ten-letter isogram to print the base price covertly on the furniture labels so that associates could figure it out without going to the back office and looking it up in the computer.
The last word I remember them using was Cumberland. C=0, d=9. So if the base price for a dining room table was $15,840 the tag would have urnec written on it somewhere.
http://www.builtbylane.com/
Web version: https://isogrammer.com/
Example: (function(i, s, o, g, r, a, m) {...}) -> (function(y, c, o, m, b, i) {...})
Precisely. A trick to reduce the character count, nothing more. Love doing this kind of thing (http://gabrielgambetta.com/tiny_raytracer.html).
> Making a.async truthy ensures
"Truthy"???
Many bugs have been caused by incorrect uses of truthy and falsy.
https://github.com/vesparny/fair-analytics
Hope it comes along; there is surprisingly not much else in the way of Node/JS based open source web stats/analytics solutions.
https://piwik.org
I actually much prefer the Piwik interface, I don't need to dig through masses of menu options to find the data I actually care about. Not helping the Google panopticon is a bonus.
1. Has a large footprint (won't run on tiny instances) 2. Relies on blockable user side JS.
Is there something like that that just analyzes server logs?
But Piwik can use methods other than JavaScript in the browser, including importing server-side logs [1].
[0] https://awstats.sourceforge.io
[1] https://piwik.org/faq/new-to-piwik/#faq_63
For me, Google Analytics privacy issue is that it's used everywhere - Google can track users across a fair chunk of the web. Your site running its own private Piwik install? Fill yer boots, not a problem.
For all those here mentioning that he's only "deconstructed" the snippet, and not the actual main linked analytics script, you're the first to observe that he's deconstructed the snippet incorrectly.
Don't get me wrong, it's really great and heartening to see someone new to JS getting back to basics and trying to understand things in proper minute detail (instead of doing another tutorial about a boilerplate React app), but this is hardly HN front-page material? It's a very simple JS snippet.
Edit: found it. Gets the timestamp[1]
1. https://stackoverflow.com/questions/24182317/multiplication-...
I like mine better :)
https://developers.google.com/analytics/devguides/collection...
"Used to collect offline / latent hits. The value represents the time delta (in milliseconds) between when the hit being reported occurred and the time the hit was sent. The value must be greater than or equal to 0. Values greater than four hours may lead to hits not being processed."
Would be nice if that delta time could be greater than 24 hours, eg. a human approves that a low-volume contact form submission isn't spam and that hit is tied to a GA goal.
(references to undefined variables `a` and `m` in some sort of IIIFE)
https://isogrammer.com/
(Originally noted in some old article that I can't find right now.)
Examples:
· Hackers Make $5M a Day by Faking 300M Video Views | https://news.ycombinator.com/item?id=13219871 (6 months ago)
· Uncovering an advertising fraud scheme | https://news.ycombinator.com/item?id=2333824 (6 years ago)
Discussions:
· Alleged $7.5B fraud in online advertising | https://news.ycombinator.com/item?id=9796102 (2 years ago)
· Inside Google's Secret War Against Ad Fraud | https://news.ycombinator.com/item?id=9628967 (2 years ago)
i opened the article and "oh, really?"
I thought that was a kind of recursive joke (not necessarily a very witty one) : if they want to make the parameters spell out a word, the word has to be an isogram, otherwise you'd define a parameter twice. And since "isogram" is an isogram, maybe the temptation was too great..