Hi HN. I run a marketing agency and fell down this rabbit hole after a client's analytics made no sense (50k visitors, 47 sales). I ended up building a simple script to track user behavior and analyzed 200+ small e-commerce sites. The average was 73% bot traffic that standard analytics counts as real.
The bots are getting creepily good at mimicking engagement. I wrote up my findings, including some of the bizarre patterns I saw and the off-the-record conversations I had with ad tech insiders. It seems like a massive, open secret that nobody wants to talk about because the whole system is propped up by it.
I'm curious if other developers, founders, or marketers here have seen similar discrepancies in their own data.
What do you think is the false negative ratio of the remaining 27%? A 0.1% total conversion rate would imply 1/270 visitors buying stuff and I am quite certain I buy stuff in maybe every 10th online store I visit or something.
Does it really matter if it's all fraud? You track 47 sales over some period. What was the ad spend for that period? Combine that with previous data and that should be enough to figure out if it was a successful campaign or not.
When a company puts up a billboard or an ad on the bus, they don't care if the ad is seen by dashcams and dogs. All that matters is impact on the bottom line.
Half my advertising budget is wasted, I just don't know which half...
In many corporate cases vague metrics meets the KPI better than accurate ones
I worked for one of the mag7 doing customer support bot tech. Clients internal metrics around containment consistently showed better results than ours - even though you'd normally expect them to be putting pressure on their vendor. because it was a kpi for their internal team to look good to their bosses
With numbers like, if it scares customers a good strategy is to implement the filtering very gradually over time, over 6 months or a year. The fall off is way less scary, and it can be described as improving bot filtering.
It’s not as honest, but more palatable unfortunately
Too often I have been working on ecommerce gigs where the traffic comes from internal 'SEO' tools. Normally the person in charge is marketing, not technical, so it has always been difficult to get past identifying the problem to fixing it.
Often ecommerce companies are very siloed, so one person in one part of the 'team' is scraping the site with their special tools, only for another person to be doing another scrape with their special tools. You can have the guy doing the newsletter doing his thing, the guy doing organic search doing their thing, the guy doing paid ads doing their thing and someone in sales doing their own thing.
The sad thing is that they are typically just a few SQL joins away from exactly the data they want in a format they can digest. However, due to silo-ing, it can be hard to have that conversation.
On top of that, you do get new bots that need to be dealt with. The Huawei bot will scrape everything yet the store might not be delivering to China. So there are legit bots not doing ad fraud that need to be dealt with.
What I also find interesting is that nobody is interested in the server logs. They come for free, and, although not having CDN cache hits, they still record checkout transactions and any pages that don't have a freshly cached page.
Ad fraud also goes on in companies. I worked for a very successful company once and we only measured sales and what was out of stock. We didn't need open rates for emails or click through rates, our main problem was selling too much, which was a nice problem to have. Note that if you sell too much then you aren't going to get it all out the door in a timely fashion, or you run out of big lorries to put the orders in.
Since then I have not worked on a site that is as successful. Instead we have people getting praise for 'false metrics'. Anything an SEO person creates or a marketeer measures will always have some nonsense aspect to it. Or the accounting is mixed with brick and mortar sales even though free shipping and discounts have been given on each sale, with adwords used to get people through the 'door'.
Since sales manager has to report to someone on the board, if the sales numbers aren't good, the nonsense stats can be used to obfuscate the facts. The board only ever care about profit, so I don't like the way this goes down with false metrics of nonsense.
Another fundamental problem in ecommerce is a lack of basic salesmanship. If you work the shop floor doing specialist sales where you have to listen to the customer's needs, then you gain experience in the art of sales. You haven't got to be good at it, in fact it can be better to know your limitations, for me that is big ticket items where I don't have the product knowledge, however, I could always hand those sales over to a much more capable colleague.
You don't win every sale, but, in retail, you can have some really good streaks where no customer leaves empty handed. Your conversion rate is going to be more like 90% in face to face sales if you have the right product at the right price, with customers that don't buy coming back the following day or week to splash the cash.
In High Street retail there is no way you give customers 15%+ off just for stepping through the door. Yet this is table stakes in ecommerce, particularly for small to medium size shops. This instantly devalues the product.
Often there will be chatbots for whatever reason, and I am sure the likes of Dell can get that right, but your typical small ecommerce site will fluff this up too, so any customer daring to use the chatbot will not get instant help from a sales person.
So what to do?
It depends on your product, however, the goal is to get customers for life, not to churn through them. To achieve this it comes down to product, price, availability, shipping times, customer service and incentives for the customer to advertise for you, with reviews, word of mouth and ...
But what I'm struggling to understand is where is the money coming from to run those bots?
How does the advertising money get to them to make it worth their while to run bots at such a scale?
I mean sure, I suspect that people in the advertisers are doing it, but surely thats massively risky. there must be a grey market for this kind of transaction?
> I had one client spending $12,000 per month on Google Ads
In Google Ads you can just turn off the option to run your ads on non-Google sites; I think it's called their Display Network. Just run your campaign only on Google's search pages.
I'm surprised the article doesn't mention this rather common solution.
I work in e-commerce and completely believe this could be true. The 'Cart Abandonment Bot' is actually something we just had a meeting on today, trying to figure out what's going on here.
With that said, what do you think the error is here? Could your script have had some false positives? Enough to move the dial?
> It seems like a massive, open secret that nobody wants to talk about because the whole system is propped up by it.
I was never really a punk-rebel kid, but a certain part of me rooted in the optimism of the early-internet kinda wants to see ad-models crash and burn.
Even advertising "working normally" always had a psychic odor of exploitation and deceit. Ex: "You absolutely need this product or else your peers will hate you."
I find it really interesting when you describe “good bot” traffic.
> During my investigation, a source from the e-commerce data industry provided a crucial piece of the puzzle. He explained that his former company was responsible for scraping 70 million retailer web pages every single day. This is a legitimate and massive source of automated traffic.
> Why do they do this? For vital business intelligence. Major retailers like Amazon do not always notify vendors when they run out of stock. So, brands pay for data scraping services to monitor their own products. These "good bots" check inventory levels, see who is winning the "buy box," ensure product descriptions are correct, and track search result rankings. They even scrape from different locations and mobile device profiles to analyze what banner ads are being shown to different audiences.
I think a lot of the players involved in this would say those are bad bots. Having your competitors scrape your site for data would probably be something most website owners wouldn’t like, but getting data about THEIR competitors would be something they WOULD like.
All bot traffic is good from SOMEONE’S perspective, otherwise it wouldn’t be happening. Someone had to program the bot, someone has to be running it. They obviously think they are good bots.
The people running AI scraping bots think they are good bots. Many content creators think those are bad bots. Price comparison sites scraping retailers think their bots are good. The sites being scraped often think they are bad bots.
I just don’t think we can clearly separate good and bad bot traffic without specifying whose perspective we are talking about.
We are now at the point where plausibly some bots could be at the behest of an AI agent controlled by a human.
This is not likely to distort the basic numbers in your story, but it makes the premise questionable
If your script can correctly segregate bot traffic from human traffic, and website operators can conclude that bot traffic does not make purchases, then what -- you still don't want to blackhole that traffic right ?
I'm none of those things, 10x or otherwise, and would be ashamed if I were. I do run a personal website, though, and most of the traffic is bots that ignore robots.txt. I wish those developers, founders, and marketers would stop doing that to me.
I do web analytics consulting. One of my first projects at a digital marketing agency in 2021 was investigating weird traffic patterns for a global logistics firm. The findings are summarized in this blog post[1].
Bot traffic has been an issue for years. It has given rise to a new cottage industry of ad fraud detection services, none of which I’ve found particularly valuable. It always comes down to “so what do we do about it?” and no one seems to know how to get bots to stop viewing or clicking on paid media placements. Consumers use Google search and are on FB, IG, TikTok, LinkedIn, etc., and there aren’t really competing ad networks with “fewer bots”. So they keep buying fake traffic knowing that a significant chunk of it is invalid.
I don’t see anything changing unless big tech companies making billions in ad revenue have a large enough incentive to do so. At the moment they have plenty of incentives to keep things as they are.
“Half of the money I spend on advertising is wasted; the trouble is I don’t know which half.”
- John Wanamaker
We've created open-source security analytics [1] that some organizations use for click fraud detection.
The numbers that Google Ads shows as invalid clicks is dramatically different from what our platform detects.
In our cases, most fraudulent clicks come from mobile devices and mobile networks. Also, bots don't load page resources in full, making them easier to detect.
If your organization is struggling with fraud click detection, don't hesitate to reach out by email. We are ready to help.
For any real audit of website traffic (M&A, large advertising deals, etc), you typically don't rely on self reported statistics, but rather 3rd parties (e.g. SimilarWeb). These have actual spywares on top of Google analytics plug-ins to correlate real traffic from noise
I find the article's topic interesting, but the writing style is just... no. It reads like a True Crime transcript or really bad marketing copy. Which makes a certain kind of sense, I guess.
What is really mind blowing is that, if understood correctly, bots would be used to check the availability of a product, that sounds so a "hacky" method, like "seriously people are doing that in 2025".
Why is this in the least surprising? It's just the natural successor to what everyone used to do with the trade magazines thirty years ago. Back then you filled in a profile questionnaire to get a free subscription, so every basement hacker turned into the manager of a 500-person division with control of a $1m capital budget. The magazine didn't want to check because it would damage the demographic numbers that they pitched to advertisers. The advertisers knew that there was some liar's poker being played but everyone just rolled with it.
Interesting. You didn’t give specifics on what anti-bot measures sites implemented, so I’ll add:
Bot prevention measures can be good, but the more hoops you make your users jump though (CAPTCHA etc), the more legitimate users will drop off. Those have significant impacts on conversion rates.
I would think fixing this should involve the analytics and attribution side rather than adding friction to your e commerce flow.
Especially as bot tech continues to get better and more indistinguishable from real traffic.
Would like to see the script. From reading it's impossible to tell if the methodology is sound. Would legitimate users with adblockers or disabling JS get counted is false positives, for example?
That said, 73% doesn't come as a surprise. If anything I expect it to be higher.
I guess this quote sums up the situation
> When I tried to bring this up with a few major ad platforms, the conversation always followed a predictable script. The sales reps were incredibly friendly until I mentioned click fraud or bot traffic. Then, the tone shifted instantly to corporate-speak: "Our AI detection is industry leading" and "We take ad fraud very seriously." It was a polite but firm wall, a clear signal to stop asking questions.
> One rep I had known for years finally admitted the truth off the record. "Dude, we know," he said. "Everyone knows. But if we filtered it all out properly, our revenue would drop 40% overnight, and investors would have a meltdown."
I’m puzzled by this: I thought it was well-understood, at least in the industry, that traffic numbers were at least mostly nonsense, and that ad click metrics especially were suuuper shady, typically more than half fraud; yet OP, in the business of “accurate ad spend analytics”, only just discovered this!?
It just doesn’t ring true. That aspect of the story isn’t novel at all, and someone in that line of work should surely have known all this, right?
Now the section on categorising different bot patterns, that’s more interesting, and I haven’t seen so much said about it.
> scraping 70 million retailer web pages every single day. This is a legitimate and massive source of automated traffic.
Why do they do this? For vital business intelligence. Major retailers like Amazon do not always notify vendors when they run out of stock. So, brands pay for data scraping services to monitor their own products. These "good bots" check inventory levels, see who is winning the "buy box," ensure product descriptions are correct, and track search result rankings. They even scrape from different locations and mobile device profiles to analyze what banner ads are being shown to different audiences.
_---------------_
Guilty as charged. You quickly learn to bypass bot detection measures and create a fully automated system to gather all this information just because amazon doesn't provide it in an accessible manner causing harm to businesses who need this intel and their own internet infra.
It makes the argument of the open internet being unable to function without advertising, quite hard to prop up. Especially when over 70% of traffic if just people gaming the system, to real users detriment.
I don't really believe the main thesis of this article- it reads like much of the fake cliff-hanger pseudo-insight endemic to marketing and business influencers.
Mainly, it avoids the main point- 73% of your traffic is "faked" enough to look real.
Who are the players in that scenario that stand to benefit from your traffic being fake?
You pay for Google (search ads) and Facebook ads but the traffic is faked by them (unlikely)
You pay other publishing networks (maybe adsense?) and the website owners profit from sending fake traffic (maybe true? if the article were really trying to make a case for this, just name them?)
Or, you work inside a company and just want to make your department look good?
I'm not sure I know what the point of this article is besides a click bait title.
Just tell me exactly what the mechanism is for this fake traffic- don't hint at some kind of conspiracy.
I still have to hear a compelling argument about why I should use computers “by hand” and ignore these powerful tools. Price checking, comparison shopping, buy when released for sale… All of these things point me to using bots.
This feels a lot less like “fraud” and a lot more like “the world has moved on”. Maybe it’s time to route traffic that looks like bots to a bot-optimized shopping experience.
After we implemented advanced bot traffic detection and filtering, their reported traffic plummeted by 71%. [...]
But then the sales report came in. Their actual sales went up by 34%.
Their real conversion rate optimization (CRO) efforts had been working all along, but the results were buried under an avalanche of fake clicks. They were not bad at marketing; they were just spending thousands of dollars advertising to robots programmed never to buy anything. Their marketing ROI went from "terrible" to "excellent" overnight.
I don't understand how detecting bot traffic would directly lead to less ad spend.
Can you just tell e.g. Google Ads that you don't want to pay for certain clicks?
Did they modify their targeting to try to avoid bots?
We ran into this problem when running ads for an iOS App only to iOS traffic. Somehow 80% of our iOS only targeted traffic clickthrough was Android...
Went to UGC and never looked back.
Would be great if more data were made available by OP to peer review some of this. That said, making money with failure starts looking like a business model - highly unethical. Why make customers succeed when you loose money doing so.
I'm not sure... but... maybe in this one single instance, I'm rooting for the bots.
I mean, it's burning ad dollars and causing advertisers to rethink their strategy. Who knows, maybe that will eventually lead to the realization that web pages that are 20% content and 80% ads are just luring bots and not customers.
On the other hand, the money being burnt is going to Google, Meta, etc... and helping fund massive surveillance infrastructure. To be honest, I'd prefer it if it all just went to shareholders. Heh, maybe that'll be the sign that we've hit peak surveillance infrastructure: Google and Meta dividend payments go up :-)
But I have trouble sympathizing with someone who writes this:
> Mouse Movements: Did the cursor move in natural, human-like arcs, or did it snap between points?
> Scrolling Patterns: Was the scrolling speed variable, with pauses and upward scrolls, or was it a perfectly smooth, mechanical glide?
> Time Between Interactions: How long did a "user" wait between clicking a link, hovering over an image, or adding an item to the cart?
I read that as: "We're tracking every movement, every hesitation... so that we can feed it to our models and determine how best to keep you addicted".
I knew it was happening, and I know I'm editorializing there... but they are getting closer and closer to just coming out and saying it.
I did work in the ad tech industry for almost 15y and big corp like Google/FB scam their user:
- they don't allow double tracking, so you have to trust their numbers
- if you look at IP from their "clicks", you see often a FB/Google datacenter IP range
- and for most of the traffic they might send you, they did just clever algorithm and heavy profiling to stole your organic traffic. So they get this "amazing" performance by claiming people that would have bought on your site anyway
I have seen and been working in companies trying do to the impact metrics well, but these are outliers
- websites showing ads are annoying their user and get no benefit of it
- stores/brands/people that want to advert pays a bug chunk of money for nothing
- only the middle men are getting benefits
“I spoke to a startup founder who raised $2 million in funding based on "user growth" metrics that he later discovered were 80% bots. He is now trapped, forced to pretend everything is fine because admitting the truth could jeopardize his company and his relationship with his investors.”
If you feel trapped now, just wait until you're in prison!
I remember reading an article years ago that argued with similar examples that the entire website ad market is almost entirely artificial/fraud/bots, but that a lot of jobs, a lot of companies, and basically the whole industry depends on simply pretending this is not true.
A lot of our customers use post purchase surveys and on-site surveys to help with this sort of thing. For example a really common use-case is an attribution survey which appears after a sale is made. The survey will ask something like "how did you hear about us?" which helps determine what actually drove the sale so they can get some clear insights outside of Google and Meta. It's not perfectly reliable but it's an additional data point that helps with the mess out there...
80 comments
[ 4.6 ms ] story [ 55.5 ms ] threadThe bots are getting creepily good at mimicking engagement. I wrote up my findings, including some of the bizarre patterns I saw and the off-the-record conversations I had with ad tech insiders. It seems like a massive, open secret that nobody wants to talk about because the whole system is propped up by it.
I'm curious if other developers, founders, or marketers here have seen similar discrepancies in their own data.
When a company puts up a billboard or an ad on the bus, they don't care if the ad is seen by dashcams and dogs. All that matters is impact on the bottom line.
We at engineering decided to filter out bots. Figures fell dramatically by more than 50%.
In less that a day business mandated us to remove the filter.
Bots are real people after all
I worked for one of the mag7 doing customer support bot tech. Clients internal metrics around containment consistently showed better results than ours - even though you'd normally expect them to be putting pressure on their vendor. because it was a kpi for their internal team to look good to their bosses
It’s not as honest, but more palatable unfortunately
Why not use Microsoft Clarity?
> The bots are getting creepily good at mimicking engagement.
You would think Google Analytics would help a lot with this, but they seem to not care.
Too often I have been working on ecommerce gigs where the traffic comes from internal 'SEO' tools. Normally the person in charge is marketing, not technical, so it has always been difficult to get past identifying the problem to fixing it.
Often ecommerce companies are very siloed, so one person in one part of the 'team' is scraping the site with their special tools, only for another person to be doing another scrape with their special tools. You can have the guy doing the newsletter doing his thing, the guy doing organic search doing their thing, the guy doing paid ads doing their thing and someone in sales doing their own thing.
The sad thing is that they are typically just a few SQL joins away from exactly the data they want in a format they can digest. However, due to silo-ing, it can be hard to have that conversation.
On top of that, you do get new bots that need to be dealt with. The Huawei bot will scrape everything yet the store might not be delivering to China. So there are legit bots not doing ad fraud that need to be dealt with.
What I also find interesting is that nobody is interested in the server logs. They come for free, and, although not having CDN cache hits, they still record checkout transactions and any pages that don't have a freshly cached page.
Ad fraud also goes on in companies. I worked for a very successful company once and we only measured sales and what was out of stock. We didn't need open rates for emails or click through rates, our main problem was selling too much, which was a nice problem to have. Note that if you sell too much then you aren't going to get it all out the door in a timely fashion, or you run out of big lorries to put the orders in.
Since then I have not worked on a site that is as successful. Instead we have people getting praise for 'false metrics'. Anything an SEO person creates or a marketeer measures will always have some nonsense aspect to it. Or the accounting is mixed with brick and mortar sales even though free shipping and discounts have been given on each sale, with adwords used to get people through the 'door'.
Since sales manager has to report to someone on the board, if the sales numbers aren't good, the nonsense stats can be used to obfuscate the facts. The board only ever care about profit, so I don't like the way this goes down with false metrics of nonsense.
Another fundamental problem in ecommerce is a lack of basic salesmanship. If you work the shop floor doing specialist sales where you have to listen to the customer's needs, then you gain experience in the art of sales. You haven't got to be good at it, in fact it can be better to know your limitations, for me that is big ticket items where I don't have the product knowledge, however, I could always hand those sales over to a much more capable colleague.
You don't win every sale, but, in retail, you can have some really good streaks where no customer leaves empty handed. Your conversion rate is going to be more like 90% in face to face sales if you have the right product at the right price, with customers that don't buy coming back the following day or week to splash the cash.
In High Street retail there is no way you give customers 15%+ off just for stepping through the door. Yet this is table stakes in ecommerce, particularly for small to medium size shops. This instantly devalues the product.
Often there will be chatbots for whatever reason, and I am sure the likes of Dell can get that right, but your typical small ecommerce site will fluff this up too, so any customer daring to use the chatbot will not get instant help from a sales person.
So what to do?
It depends on your product, however, the goal is to get customers for life, not to churn through them. To achieve this it comes down to product, price, availability, shipping times, customer service and incentives for the customer to advertise for you, with reviews, word of mouth and ...
The word "revenue" above doesn't sound right. Did you mean "ad impressions", "clicks", "visits", or something else?
If the bot traffic is giving you 40% more revenue, well that's excellent news, right? If it's revenue, then keep doing it.
How does the advertising money get to them to make it worth their while to run bots at such a scale?
I mean sure, I suspect that people in the advertisers are doing it, but surely thats massively risky. there must be a grey market for this kind of transaction?
In Google Ads you can just turn off the option to run your ads on non-Google sites; I think it's called their Display Network. Just run your campaign only on Google's search pages.
I'm surprised the article doesn't mention this rather common solution.
I've nicked that sentence to use as the title above, since it's more neutral and representative of what the article is about.
With that said, what do you think the error is here? Could your script have had some false positives? Enough to move the dial?
I was never really a punk-rebel kid, but a certain part of me rooted in the optimism of the early-internet kinda wants to see ad-models crash and burn.
Even advertising "working normally" always had a psychic odor of exploitation and deceit. Ex: "You absolutely need this product or else your peers will hate you."
> During my investigation, a source from the e-commerce data industry provided a crucial piece of the puzzle. He explained that his former company was responsible for scraping 70 million retailer web pages every single day. This is a legitimate and massive source of automated traffic.
> Why do they do this? For vital business intelligence. Major retailers like Amazon do not always notify vendors when they run out of stock. So, brands pay for data scraping services to monitor their own products. These "good bots" check inventory levels, see who is winning the "buy box," ensure product descriptions are correct, and track search result rankings. They even scrape from different locations and mobile device profiles to analyze what banner ads are being shown to different audiences.
I think a lot of the players involved in this would say those are bad bots. Having your competitors scrape your site for data would probably be something most website owners wouldn’t like, but getting data about THEIR competitors would be something they WOULD like.
All bot traffic is good from SOMEONE’S perspective, otherwise it wouldn’t be happening. Someone had to program the bot, someone has to be running it. They obviously think they are good bots.
The people running AI scraping bots think they are good bots. Many content creators think those are bad bots. Price comparison sites scraping retailers think their bots are good. The sites being scraped often think they are bad bots.
I just don’t think we can clearly separate good and bad bot traffic without specifying whose perspective we are talking about.
This is not likely to distort the basic numbers in your story, but it makes the premise questionable
If your script can correctly segregate bot traffic from human traffic, and website operators can conclude that bot traffic does not make purchases, then what -- you still don't want to blackhole that traffic right ?
I'm none of those things, 10x or otherwise, and would be ashamed if I were. I do run a personal website, though, and most of the traffic is bots that ignore robots.txt. I wish those developers, founders, and marketers would stop doing that to me.
Bot traffic has been an issue for years. It has given rise to a new cottage industry of ad fraud detection services, none of which I’ve found particularly valuable. It always comes down to “so what do we do about it?” and no one seems to know how to get bots to stop viewing or clicking on paid media placements. Consumers use Google search and are on FB, IG, TikTok, LinkedIn, etc., and there aren’t really competing ad networks with “fewer bots”. So they keep buying fake traffic knowing that a significant chunk of it is invalid.
I don’t see anything changing unless big tech companies making billions in ad revenue have a large enough incentive to do so. At the moment they have plenty of incentives to keep things as they are.
“Half of the money I spend on advertising is wasted; the trouble is I don’t know which half.” - John Wanamaker
[1] https://jimalytics.com/industry/youre-buying-bots-an-inconve...
The numbers that Google Ads shows as invalid clicks is dramatically different from what our platform detects.
In our cases, most fraudulent clicks come from mobile devices and mobile networks. Also, bots don't load page resources in full, making them easier to detect.
If your organization is struggling with fraud click detection, don't hesitate to reach out by email. We are ready to help.
1. https://github.com/TirrenoTechnologies/tirreno
I guess there is no incentive in most markets. Facebook, etc make only a token effort to reject non-troublesome bot traffic.
[0] bots and other automated traffic which cannot generate revenue or human ad views
Bot prevention measures can be good, but the more hoops you make your users jump though (CAPTCHA etc), the more legitimate users will drop off. Those have significant impacts on conversion rates.
I would think fixing this should involve the analytics and attribution side rather than adding friction to your e commerce flow.
Especially as bot tech continues to get better and more indistinguishable from real traffic.
That said, 73% doesn't come as a surprise. If anything I expect it to be higher.
I guess this quote sums up the situation
> When I tried to bring this up with a few major ad platforms, the conversation always followed a predictable script. The sales reps were incredibly friendly until I mentioned click fraud or bot traffic. Then, the tone shifted instantly to corporate-speak: "Our AI detection is industry leading" and "We take ad fraud very seriously." It was a polite but firm wall, a clear signal to stop asking questions.
> One rep I had known for years finally admitted the truth off the record. "Dude, we know," he said. "Everyone knows. But if we filtered it all out properly, our revenue would drop 40% overnight, and investors would have a meltdown."
It just doesn’t ring true. That aspect of the story isn’t novel at all, and someone in that line of work should surely have known all this, right?
Now the section on categorising different bot patterns, that’s more interesting, and I haven’t seen so much said about it.
Why do they do this? For vital business intelligence. Major retailers like Amazon do not always notify vendors when they run out of stock. So, brands pay for data scraping services to monitor their own products. These "good bots" check inventory levels, see who is winning the "buy box," ensure product descriptions are correct, and track search result rankings. They even scrape from different locations and mobile device profiles to analyze what banner ads are being shown to different audiences.
_---------------_
Guilty as charged. You quickly learn to bypass bot detection measures and create a fully automated system to gather all this information just because amazon doesn't provide it in an accessible manner causing harm to businesses who need this intel and their own internet infra.
Mainly, it avoids the main point- 73% of your traffic is "faked" enough to look real.
Who are the players in that scenario that stand to benefit from your traffic being fake?
You pay for Google (search ads) and Facebook ads but the traffic is faked by them (unlikely)
You pay other publishing networks (maybe adsense?) and the website owners profit from sending fake traffic (maybe true? if the article were really trying to make a case for this, just name them?)
Or, you work inside a company and just want to make your department look good?
I'm not sure I know what the point of this article is besides a click bait title.
Just tell me exactly what the mechanism is for this fake traffic- don't hint at some kind of conspiracy.
I still have to hear a compelling argument about why I should use computers “by hand” and ignore these powerful tools. Price checking, comparison shopping, buy when released for sale… All of these things point me to using bots.
This feels a lot less like “fraud” and a lot more like “the world has moved on”. Maybe it’s time to route traffic that looks like bots to a bot-optimized shopping experience.
Can you just tell e.g. Google Ads that you don't want to pay for certain clicks?
Did they modify their targeting to try to avoid bots?
If a bot network hits all the conversing events then Google will tailor the traffic to look more like the bot network.
If you filter the bot traffic out then Google can tailor the traffic to look like real converting users instead.
I mean, it's burning ad dollars and causing advertisers to rethink their strategy. Who knows, maybe that will eventually lead to the realization that web pages that are 20% content and 80% ads are just luring bots and not customers.
On the other hand, the money being burnt is going to Google, Meta, etc... and helping fund massive surveillance infrastructure. To be honest, I'd prefer it if it all just went to shareholders. Heh, maybe that'll be the sign that we've hit peak surveillance infrastructure: Google and Meta dividend payments go up :-)
But I have trouble sympathizing with someone who writes this:
> Mouse Movements: Did the cursor move in natural, human-like arcs, or did it snap between points?
> Scrolling Patterns: Was the scrolling speed variable, with pauses and upward scrolls, or was it a perfectly smooth, mechanical glide?
> Time Between Interactions: How long did a "user" wait between clicking a link, hovering over an image, or adding an item to the cart?
I read that as: "We're tracking every movement, every hesitation... so that we can feed it to our models and determine how best to keep you addicted".
I knew it was happening, and I know I'm editorializing there... but they are getting closer and closer to just coming out and saying it.
edit: Added newlines in quoted part.
- they don't allow double tracking, so you have to trust their numbers
- if you look at IP from their "clicks", you see often a FB/Google datacenter IP range
- and for most of the traffic they might send you, they did just clever algorithm and heavy profiling to stole your organic traffic. So they get this "amazing" performance by claiming people that would have bought on your site anyway
I have seen and been working in companies trying do to the impact metrics well, but these are outliers
- websites showing ads are annoying their user and get no benefit of it
- stores/brands/people that want to advert pays a bug chunk of money for nothing - only the middle men are getting benefits
If you feel trapped now, just wait until you're in prison!
A lot of our customers use post purchase surveys and on-site surveys to help with this sort of thing. For example a really common use-case is an attribution survey which appears after a sale is made. The survey will ask something like "how did you hear about us?" which helps determine what actually drove the sale so they can get some clear insights outside of Google and Meta. It's not perfectly reliable but it's an additional data point that helps with the mess out there...
[1] https://www.zigpoll.com/