HN Explorer

Ask HN: How does email tracking work?

11 points by markwaldron ↗ HN
From my understanding, Sidekick uses a "tracking pixel" that reports back to their servers when loaded, then sends the user a notification. Are there any guides out there on how to program something of this nature? Or are there any open source projects that may provide a more in-depth look into how email tracking works?

Any more information would be greatly appreciated.

16 comments

[ 3.4 ms ] story [ 41.9 ms ] thread
detaro
Basic version is anything that can look at a logfile of a webserver. Each mail gets a specific URL for the image, HTTP request to there turns up in the log.

Or you put some kind of server side app on the server that tracks the request in its own log.

markwaldron
Is the HTTP request for the image or for a blank page? I'm just guessing here, but I guess a new file is created each time an email is created and then a reference to who the email is sent to is stored in a DB somewhere?
detaro
something that isn't an image might show up on the page, even if only 1x1 px big, so it's "cleaner" to send an empty gif or something. Doesn't have to be a new file for everybody though -> one could simply redirect everything to one file, or put the metadata in an URL parameter.
davismwfl
I have designed a few of these over the years and worked on others implementations. The common one I see the most is taking a 1x1 transparent image in the email with query string parameters to provide the specific data. Usually something like a this img src="/domain,com/imagename.png?i=UID. Also sometimes we will use the company logo as the tracking image, simply because it doesn't look like a "tracking pixel" by default.

The other way that I just did was basically the same thing, but the image doesn't really exist, like img-uid.png where uid is the uid I care about for tracking purposes. Then in my webserver setup I pass the details to a server side component to manage the tracking and return a generic 1x1 transparent image. There are lots of ways to do that, but it protects you for those cases where an email client strips query string params on img src's.

Of course if the user doesn't allow the images to load you get no tracking, so that is a limitation but most systems have that limitation. So you should also have link tracking etc.

markwaldron
Awesome and very informative! Thanks for the explanation. I'm starting to get a better grip on this. I found a few posts online explaining how to do this with PHP and GD. In each of those instances, do you create a unique image for each user or each email?
dangrossman
Let's say you assign each subscriber in your database a number. You send a mailing out to the list. In the mail to the first subscriber, you include this tag:

<img src="http://www.example.com/track.php?id=1">

In the mail to the second subscriber, you include this tag:

<img src="http://www.example.com/track.php?id=2">

And so on. There needs only be one file on your server to track opens no matter how many people you mailed: track.php.

That one file reads the "id" parameter from the request to know which subscriber opened the mail, and logs that in your database. It then creates a 1x1 transparent image, in memory, using GD, and outputs that to the browser that loaded the image tag.

The image never existed on disk, and nothing needed to be created in advance of sending a mail.

atomical
GD seems like overkill if the same 1x1 pixel is being generated over and over again.
davismwfl
As a point, at least in my implementation, I actually use a byte stream for the 1x1 that is cached and sent to the browser. We do this so that there is no disk read ever for it. We also don't use php/gd, our code is mostly node.js and a little golang with nginx as our front end.

It may seem a little overkill, but I found doing it this way made a pretty significant impact when you have a reasonably decent volume of tracking going on. A good http server of course would cache the image of course, but we just found this to be easier and more consistent for our implementation.

davismwfl
Your welcome.

dangrossman answered this in detail already and I agree with what he says.

Just as an example, in my system each email that gets sent out has a unique id (in our case a hashed id), we use that as the id value. From there we can get back to campaign, email address, company etc. As for the image itself, it is physically only one image generally (or an image byte stream cached in RAM).

btw -- dangrossman used sequential numbers as the ID only as an example to show you, you wouldn't do that in a production system. Have fun!

siquick
This is also how a lot of affiliate marketing tracking works.
27182818284
Adding to this, non-standard, old tags were used to some success until the major email providers caught on. For example, I was really impressed by someone taking advantage of <bgsound> to do tracking.
iqonik
It is normally an image like others have said, however, Google's caching inside gmail is causing some problems:

https://blog.mailchimp.com/how-gmails-image-caching-affects-...

So I would say it is not 100% anymore, but, no alternative approach is out there either.

tedmiston
FWIW however Intercom tracking email opens seems to circumvent Google's effort. I think it only tracks first open vs. every open though.
Grynn
Google merely prevents you from collecting information about the IP address & device. It still lets you know who opened the email. Since the request comes from Google's servers instead of the client device (or browser) you get whatever HTTP headers google thinks are appropriate (and of course the IP address of a Google server)

AFAIK Google does not pre-load images (for example, even if no one opened the email Google could request the image). This would destroy open tracking since you'd get a lot of false positives.

Since "who" opened is already encoded in the image URL, you still get this info.

Anyhew, there's a lot more people do with these tracking images - for example see KickDynamic.com. You can change the image at time of open (to say reflect the user's location, which you deduce from the IP). This can't be done if it's gmail, since you don't have the client's IP.

It's kind of strange that Apple doesn't block (or anonymise) images though they're so hot about privacy at the moment.

zhte415
Most email clients block these by default, at least in a business setting. My advice is to check your audience first.

Bounce-back report tracking could be useful if you're checking emails are delivered. I've used this when emails that have to be delivered are delivered. This was mainly for compliance reasons - whether they're read or not, or end up in whatever user folder, was of little importance. That these emails are delivered successfully was paramount. The bounce-back reporting showed where customer contact failed and secondary measures were necessary to get back in contact.

Your use-case may be different, but if anyone reading this is in a similar situation to ensure delivery but not readership, implement a simple server-side delivery/non-delivery report.