If any cloudflare staff happen to see this... is Cloudflare Warp coming soon? I was pretty excited to give that a try (and it seemed like something I would be willing to pay for)
"The wait is over. Our product and engineering teams have been working round-the-clock to build a new experience that makes the Internet faster and safer for everyone. If I were to say anything more, it would surely give it away so I’ll leave it at that — you’ll just need to tune in to the blog on Wednesday to find out."
There's a lot more to bots than just scraping... DDoS bots, bots that buy hot sneakers before the public gets a chance and drive up the price, bots that go credential stuffing, bots that play nasty tricks with airline seats ...
Not sure what "legitimate" means here? Legal? Running aggressive web crawlers is in many instances against the rules for consumer cloud servers. For example, AWS requires that you obey robots.txt if you run a crawler there. https://aws.amazon.com/premiumsupport/knowledge-center/repor...
In my experience a lot of bots seem to be running on hacked servers or through hacked/insecure proxies. I'd imagine tracking down the owner or someone upstream of those boxes could be effective in taking them offline.
What does that have to with my point? Bots used to purchase inventory (and that aren't otherwise commiting fraud by using stolen credit cards or something) are not malicious.
Are they following the sneaker website's robots.txt while doing that? If not, they are probably violating the AWS terms regardless of whether you believe that activity is "malicious."
if they're running on AWS, which most crawlers are not
When I've run scraping software in the past I used DigitalOcean, which doesn't contain a requirement to abide by robots.txt. As far as I can tell it's both legal and consistent with their ToS to run a program that makes purchases on a website.
if a single bot controller can buy up an entire stock of limited items legitimately, that is malicious as that company is not longer able to meet the needs of their consumers. That's bad for the company.
If it's profitable for anyone to resell, that implies the company priced below the market price and there would be a shortage without scalpers. So the company is unable to meet the needs of its customers in any event. Scalpers just make it somewhat more efficient.
It is illegal if the website’s TOS for making a purchase prohibits the use of automated software.
It doesn't matter if it's legal, it matters if the website owner doesn't want x doing y on their site. A bot consistently not abiding by owners' intent is inheritly malicious.
>This type of attack hurts multiple targets as well: the ecommerce site has real frustrated users who can’t purchase the in demand item. The real users who are losing out on inventory to an attacker who is just there to skim off the largest profit possible. And the unwitting users who are part of the botnet have their resources, such as their home broadband connection, used without their consent or knowledge.
Have you run this by an economist? It's pretty basic economics that "scalping" increases consumer welfare, despite your cursory claim to the contrary.
If the IPs are part of a botnet, that's one thing. But the biggest residential IP network is luminati, which does have consent for their IPs.
All of the comments there either agree with me or are talking about non-economic factors. E.g. the first one explicitly says it's not about economic harm. The second says they're taking producer surplus, which is correct if you interpret that as potential surplus had the producers priced higher. The third one also explicitly points out that it's not being looked at from an economic perspective.
Then my question to you would be, is it possible for a "legitimate", robots.txt respecting scraper-bot (for a non-profit I'm helping for example) to get caught in CF's detector? If so, is there a way to detect that this is happening and an avenue to get unblocked?
> And the unwitting users who are part of the botnet have their resources, such as their home broadband connection, used without their consent or knowledge.
Perhaps I missed something, but doesn't this mean that a lot of homes and IoT devices that have been compromised will have increased CPU usage as a result of CF implementing the bad bot behaviour envisaged? In other words, the botnet owners won't care in the slightest about your response, but a lot of homes will suddenly get shoddy performance as their router grinds to a halt.
(Yes, of course it'd be best if every home and every IoT device were secure... but that's unrealistically optimistic)
Assuming you're building on the LI<>HiQ case here, the ruling would only be applicable to a subset of cases (public, user-generated content, no authorization...).
Even before the ruling is overturned, we can't say scrapping is legal without applying some qualifying conditions.
I'm not very familiar with the concept of tarpitting. How do they get the bot to run CPU intensive code? By passing in extra Javascript? Can this affect a bot that doesn't run any JS?
Was also curious about this step. I assume they're not going to reveal the nitty gritty details for fear of botters coding around it, but I am curious as to how you can "make them use more CPU" while crawling a website
One way is to use a special very slow TCP handler. Imagine a TCP stack that only lets through one 10 byte packet every 10 seconds. This wastes a connection slot on target machine.
I've been running many non-JS crawlers for the past few years, and there were a few pages that kept pushing the CPU load of my servers to a halt. When I dug into the source code, I saw that the HTML was a convoluted text of tables inside tables inside tables inside more tables, thus making it incredibly time-consuming + CPU-intensive for my DOM parser to parse (I was using Nokogiri, a Ruby gem at the time). Thus Cloudflare could be serving these types of "fake" pages to bad bots.
They could also be doing things like serving fake streaming audio that never ends, or anything that might make it seem like the web page is just a huge page that needs time to load.
Usually via javascript. Many of the credential stuffing and similar bots need to run headless browsers these days to be able to do their job. The folks at Kasada (https://www.kasada.io) have talked over the years at a high level of some of the approaches they've taken, there should be a few conference presentations on YouTube. They don't get into the finer detail though as I assume there's a large amount of secret sauce about what they do too.
I'm not sure what they do for the non-JS use case. They sit in the request path like a CDN though so maybe they just return an error or deliberately slow response times?
This builds up to a bigger question, with anti-competitive implications. What if you want to build a Google competitor? Is Google the only one that's allowed to crawl web pages now, and anyone else is automatically deemed a bad bot? Is CloudFlare the one that decides who can be a search engine, and who can't?
Then, after you can use that corpus and provide useful results and have people coming to your search site, then you can have a discussion with CloudFlare that you will be creating your own crawler.
Not sure how it is now, but when I looked into CommonCrawl, they had a significant delay in storing newer pages. It took at least a month before a web page made it into their index. Which makes creating any type of Google-like search engine very tough.
A big part of the net neutrality fight was "allowing the little guy a chance to play with the big boys".
Part of the service says:- for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline... If the infrastructure provider hosting the bot is part of the Bandwidth Alliance, we’ll share the bot’s IP address so they can shutdown the bot completely. The Bandwidth Alliance allows us to reduce transit costs with partners and, with this launch, also helps us work together with them to make the Internet safer for legitimate users.
My reading of that is if CF decide your IP is bad, they can leverage the providers Bandwidth Alliance status to shutdown the providers customer. What if CF's systems misfire? Will there be a grace period? Will there be an appeals process? Will CF compensate anyone effected and have their hosting withdrawn from a misfire?
No one likes bad bots, but I'm feeling more and more uneasy allowing CF decide which is which.
if this is to be believed [0] less then 50% of google searches result in a click though to the site, so one could argue that because of things like snippets in search results Google don't always drive traffic to your site.
I think that's fair. If the user was able to get what they're looking for without having to click, good for them. I'm betting that Google knows when that happen and uses it as a positive signal towards that site.
I don't have a problem with Cloudflare's reach and power, but would love for them to be transparent as to what they constitute a bad bot. What's the rules? Follow robots.txt, Scrape just X pages a minute, what else?
The problem I see is they won't explicitly say what would trigger being flagged as bad (nor what would be considered allowable) as they would claim (as almost everyone in the anti cheat field does) it would give too much info to the bad guys to avoid detection.
I would love for them to prove me wrong though and be open about such things.
> it would give too much info to the bad guys to avoid detection.
I'm pretty sympathetic to this line of thought, so don't take this as 'you're wrong', but I notice that we would never apply this logic to laws, or environmental standards, or contracts.
You'd never hear someone say, "if we have a clearly defined tax code, that will just make it easier for people to find loopholes."
Where I do hear this argument come up is explicitly in contexts of moderation and abuse policies. And it's something that sounds very reasonable, but it's hard for me to get away from the fact that in most other contexts it sounds problematic to me.
Maybe part of the problem is Cloudflare's scale? Maybe the reason it feels bad to have a police officer pull me over because, "we think you're going too fast" instead of "you went over a posted limit", is because that's a critical infrastructure that I can't avoid.
Cloudflare is a private company, and even if it wasn't, I don't know if it would be big enough presence that I would worry about their policies. But when I hear things like this:
> Our goal is nothing short of making it no longer viable to run a malicious bot on the Internet. And we think, with our scale, we can do exactly that.
That maybe shifts the situation a tiny bit farther away from "moderation policy on a personal blog" towards "policeman pulling me over because I broke a law I didn't know existed."
I dunno. I'm not 100% sure how to feel about it. I do think that Cloudflare should be able to filter traffic however they see fit, but that doesn't mean that every strategy they choose is inherently good, or that it might not be problematic for them to lack transparency about their standards.
Ok my wording was slightly off. What I was trying to say is, just because a crawl doesn't come from a Google IP block doesn't mean its not a legit crawl from someone outside of Google.
The way I take that to mean is: if you're using a Googlebot UA, and your request doesn't come from a Google-owned IP range, you're very unlikely to be a legit crawl from Google, and you'll be detected and dealt with.
Which would be fine imo. But as the person I was replying to says you can already detect a bad bot pretending to be google and deal with them how you wish (my fav is just feeding them random values).
But what are CF (and others like akamai) using to determine you are a bad bots? My objection is more the threat they will use their position to kick bad bots offline.
Now don't get me wrong, I had bad bots as much as the next person. But what about misfires? They will happen. I hope CF have a plan in place to deal with such cases.
Personally, due to family issues I've been working remote-remote (I'm a remote worker anyways, but I've been working remotely away from my normal office). As such I've been using a 4G Mobile broadband connection and the amount of CloudFlare "We are just checking you are not a bot" hoops I've had to jump though recently has opened my eyes on how much of the net is actually behind CF's network because CF still associate "IP as Identity" and my 4G connection is CG-NAT'ed.
I just fear at times we forget that there is the rest of the world out there, not just our little bubble and the choices we make have a huge impact on the usability of the net for others.
EDIT: LOL.. Just checked your profile. The Reg is one of the sites that doesn't like my 4G connection and as such I visit less. What are the chances. (Though you only make me sit there and wait to access the content....)
That's odd, TBQH. Contact webmaster@ with some details about the connection (your IP at the time you're seeing the problem, as well as your UA - all stuff you can get by visiting /cdn-cgi/trace) and I'll see if there's anything we can tweak on our end. No promises tho!
I'm CG-NAT'ed... There is no guarantee from my ISP that 2 requests to the same site in a short period of time will actually come from the same public IP address. But I might next time I have the urge to visit the reg, because honestly waiting for CF's permission to visit your site has reduced my patience to access the content so I go elsewhere and don't even bother clicking on links to the reg any more, So I honestly don't know if I will be arsed to jump though more hoops just to be told "Your CG-NAT'ed... What else do you want than allowing your whole ISP free rein to the site without having to spin CPU cycles just in-case you are a bot." (because what else can you say?)
Sorry if that's not the answer you wanted, but that's how I feel after seeing how the other side of the net is treated. IF the urge takes me, I'll help, but I dunno when that urge will take me though.
<EDIT>Fun Fact: My public IP address appears to have changed in the time it took me to write this comment. Atleast to the reg, How my ISP handles routing is another black box upon itself.</EDIT>
If you crawl sites without a distinctive user-agent; do not obey robots.txt; crawl at an unnecessarily high rate, I _will_ ban you, regardless of your intent.
> Our Trusted Client model that spots whether an HTTP User-Agent is what it says it is.
Well, instead of using curl scripts, crawlers will switch to Selenium and use a real scripted browser
In my dream world, Cloudflare would make this a two way street: you get protection against malicious bots, but you have to allow equitable treatment of all well behaved search engines (i.e. you can use robots.txt to ban lesser search engines, but doing that requires you to ban Google as well).
Disclaimer: I have no current economic interest, but kinda want to run a personal search engine.
Cloudflare's specific methods are described in the article, it's likely some web crypto stuff or an extreme response that takes a long time to parse or handle.
>Another trend we have seen is the increase of the combination of bots with botnets, particularly in the world of inventory hoarding bots. The motivation and willingness to spend for these bot operators is quite high.
>The targets are goods of generally of limited supply and high in demand and in value. Think sneakers, concert tickets, airline seats, and popular short run Broadway musicals. Bot operators who are able to purchase those items at retail can charge massive premiums in aftermarket sales. When the operator identifies a target site, such as an ecommerce retailer, and a specific item, such as a new pair of sneakers going on sale, they can purchase time on the new Residential Proxy as a Service market to gain access to end user machines and (relatively) clean IPs from which to launch their attack.
They then go on to spout some economic nonsense about how such bots are harmful. Actually, resellers make the market more efficient, and cloudflare is doing a disservice by lumping legitimate bots in with malicious ones like their credential stuffing example.
By moving the goods to those willing to pay more for them, which increases consumer surplus, which means the market is more efficient. It's fairly basic economics.
Increasing the price should reduce consumer surplus, not increase it.
If the manufacturer of an item increases its price and thus their profit from it, then in principle the increased profit can drive them to manufacture more of the item, avoiding deadweight loss due to insufficient supply. However, that doesn’t apply if a reseller is the one making the profits! And in any case, some of the concrete examples cited (tickets to concerts and Broadway musicals) have an essentially fixed supply; it’s hard or impossible to simply increase production.
Separately, moving goods toward those willing to pay more for them is to some extent equivalent to moving them toward those who care more about them, which could be said to improve efficiency in terms of how much personal happiness is gained per unit. However, it also moves the goods toward those who are simply wealthier. And there are other possible proxies for personal interest besides price, ones that aren’t as affected by wealth. One of them is a person’s willingness to research in advance when certain tickets will go on sale, and be online at that time to snag them before they sell out – in other words, the system that bots are used to subvert. This system has its own biases (favoring those who are organized, have more flexible schedules, or are simply in a more favorable timezone), but it‘s still considered to be reasonably fair, at least when it’s not being subverted.
Increasing price only reduces consumer surplus when it's the same consumers who buy. This is not the case when there's a shortage of tickets at the original sale price, which is the only scenario where scalping is profitable.
Scalping unambiguously increases Kaldor-Hicks efficiency under a simple model. You can argue that efficiency isn't the best way to be fair, and it's fairer to someone to not have a ticket due to random luck rather than being priced out, perhaps. But scalping increases efficiency, which is all I claimed.
> Increasing price only reduces consumer surplus when it's the same consumers who buy. This is not the case when there's a shortage of tickets at the original sale price, which is the only scenario where scalping is profitable.
I don’t see how that follows. If a consumer can’t buy a ticket from the original seller due to a shortage, there’s no guarantee that the original sale price was at (or even near) the maximum price that that particular consumer was willing to pay. Therefore, that consumer might also be willing to buy from a scalper. It would be in the opposite case, if there wasn’t a shortage, that someone who doesn’t buy could be assumed to be unwilling to pay the price.
In practice, I’d expect a decent fraction of those consumers to be near their maximum price, since if not, we could conclude that the original seller irrationally set their price far too low. (But the original price must have been set somewhat too low if a significant shortage exists.) Still, depending on the markup charged by the scalper, I wouldn’t be surprised if a significant fraction of consumers actually would be willing to pay it.
As a practical example, according to an article from last year about Hamilton tickets [1], among seats originally sold for $69-$179, a few were on sale for >$10,000 each, but the average resale price was $412. $10,000 will obviously price out most consumers, but going from ~$100 to ~$400 might not, considering the extremely high desirability of Hamilton seats.
> Scalping unambiguously increases Kaldor-Hicks efficiency under a simple model. You can argue that efficiency isn't the best way to be fair, and it's fairer to someone to not have a ticket due to random luck rather than being priced out, perhaps. But scalping increases efficiency, which is all I claimed.
If I understand correctly, it increases efficiency if the scalper could have hypothetically purchased their ticket from someone who was deprived the opportunity to buy from the original seller, instead of from the original seller directly, and still made a profit by reselling it.
But that doesn’t seem true. There’s no reason to think that person who “won the lottery” would be willing to sell the ticket for the same price they bought it for. Instead they would likely sell it for a similar price to what the scalper wants to charge. At best the scalper can gain some value on the margin by taking on the risk that the ticket won’t sell, but not much value.
>But that doesn’t seem true. There’s no reason to think that person who “won the lottery” would be willing to sell the ticket for the same price they bought it for. Instead they would likely sell it for a similar price to what the scalper wants to charge. At best the scalper can gain some value on the margin by taking on the risk that the ticket won’t sell, but not much value.
As long as the ticket moves to someone with a higher willingness-to-pay, Kaldor-Hicks efficiency is increased.
Since the ticket holder without scalpers isn't the same as the one with scalpers, they have a lower willingness-to-pay than the price the scalper sells at. Therefore, they'd be willing to sell to the actual ticket holder at that price.
The only difference between this scenario and the one with a scalper is some money that the scalper has instead of the original ticket holder, and transfers don't factor into Kaldor-Hicks so this is irrelevant.
> As long as the ticket moves to someone with a higher willingness-to-pay,
Which, as I said before, is only some of the time, but even under that assumption...
> Since the ticket holder without scalpers isn't the same as the one with scalpers, they have a lower willingness-to-pay than the price the scalper sells at. Therefore, they'd be willing to sell to the actual ticket holder at that price.
Then why don't they? Anyone who buys a ticket intending to go to the play is free to change their mind and resell it. Modern online services make this quite easy. But most people don't resell their tickets.
There are a variety of possible explanations. One is to invoke arguably irrational (but still real) behavior like loss aversion. Another has to do with the inconvenience of rescheduling or the risk of not selling, but that doesn't apply to the Kaldor-Hicks calculation since being deprived of a ticket (as opposed to selling it) doesn't create that inconvenience or risk. But here is one possible situation where even an economically rational actor would prefer to have a ticket than to be awarded a market price:
The value of the ticket to them may be higher than the scalpers' market price. This can be true even if the holder was not originally willing to pay that much, because the holder effectively gained net worth as soon as they "won the lottery" for the ticket, so they have a different amount of spare money to work with. And if this person took the time to buy tickets as soon as they were available, they probably care more about the play than the average person in the secondary market, so they would put a higher value on it relative to their wealth.
This is balanced by the fact that other people in the secondary market will likely be wealthier than them. Indeed, in the case of those rare orders-of-magnitude price hikes, most holders probably would be willing to sell if they could be sure the sale would actually go through. But most scalping margins are much more moderate and may not be enough to get the price over the value to the original holder.
>Which, as I said before, is only some of the time, but even under that assumption...
The only scenario where it's not is if the demand curve is perfectly inelastic. If the price is set at $100, and selling it for $400 instead doesn't raise the average willingness to pay of buyers, then there has to be exactly the same demand at $100 as at $400, which is extremely unlikely.
>Then why don't they? Anyone who buys a ticket intending to go to the play is free to change their mind and resell it. Modern online services make this quite easy. But most people don't resell their tickets.
We're talking about a hypothetical scenario without scalpers. In that scenario, it's difficult to impossible to resell tickets.
>But here is one possible situation where even an economically rational actor would prefer to have a ticket than to be awarded a market price:
Yes, positing an income effect from winning the ticket lottery can produce a model where this result no longer holds. But income effects that strong are insanely rare. To reframe this, you're suggesting if they had found $400 on the street that morning, the rational thing to do given their preferences is to spend $400 on a ticket, but they shouldn't otherwise. You're depicting someone so poor that an additional $400 represents a significant wealth difference, but also where it makes sense to spend $400 on a ticket rather than on other things. In the real world, I think we'd call someone like that irrational.
It might be plausible in other contexts, but it doesn't seem reasonable in the ticket context.
Nice UX, fast, free. Nice domain service if you transfer to them. Fast DNS management. DOS/bot mitigation. Caching. Quick SSL, and affordable upgrade options. 2FA via TOTP.
With all the networking/domain stuff momentum build, it would be nice to be able to spin up servers for apps/db.
Main language at cloudflare is golang? Rust? Any python over there?
Request: Allow changing the super administrator for cloudflare account more easily. At least for early-stage accounts.
I’d still consider the registrar service a “maturing product” at best, similar to how CF was in its early years. Enough time and it’ll be polished to where I can say it’s “production ready”.
Great! Let's add it to the list of obstacles to accessing content for us savages in small, poor, out-of-sight-out-of-mind countries:
- Harrassment with an extra verification step every god damn time you log in into a website you've been a paying customer at for years because some rando risk model thinks you're a bad guy
- Harrassment by reCAPTCHA to respond to US-centric image challenges when it decides it's up for some sadistic fun (Mark all images with a store front or street light? Sure, let me just Google how they are "supposed" to look, because I sure as hell ain't seeing any I'm familiar with here.)
- Blanket IP range bans which serve you a default 503 error page and call it a day, with ripple effects throghout tons of unrelated websites. I always know when my ISP's IP range is added to some new blacklist.
- Harrassment by Cloudflare "verifying my browser", presumably burning my CPU cycles so they can be sure that my browser, which has been hitting their IP ranges for years with not as much as a cookie wipe, has not suddenly turned into a bot
- "Your name is invalid": no, your regex is
- "Our fetishism for credit prevents us from accepting your debit card payment unless you submit a scan of your passport signed in blood"
- (NEW) CloudFlare's anti-bot measures which will surely not misfire because no one thought to test their shiny new model on the traffic patterns of some culture with < 50 million Internet users
Apologize for the off-topic rant, but you wouldn't believe how using the Internet for basic things has gotten difficult in the past few years over here (and from what I hear, in many other "forgotten" geographies). No one cares. Long live colonialism!
I'm sorry you have to put up with all of that, and I don't want to be part of the problem for folks like you. But here's the other side of the coin: I ran a small VPS hosting company for many years, and the "small, poor, out-of-sight-out-of-mind countries" are a huge source of fraud. Someone signing up from an IP in Latvia, for example, was virtually guaranteed to be paying with a stolen card that would later get disputed. Debit card payments can get disputed, too, and disputes can happen as much as 120 days after the initial charge.
For comparison purposes, the dispute rate for US-origin payments (US card, US origin IP, no evidence of proxy use) was under 1%.
If I couldn't do things like "submit a scan of your passport", I would probably just decline any cards from high-risk countries.
Start pushing for more validation from the payment providers so you can be sure that a payment is legit and once you have jumped though the hoops that a payment is legit the bank takes the hit for the fraud.
In the EU one thing we are doing is basically enabling 2FA for card purchases aka "Strong Customer Authentication".
Now I'm not 100% happy with the use of SMS as a 2FA but what will happen is if the bank or site ask for extra verification of a purchase the issuing bank will prompt the user to auth the transaction via another means (Either a SMS Code they enter into the page (think verify by visa but using a one time code that gets delivered to the customer with the amount they are approving), a Rich Communication Services with a Approve / Deny prompt or a push notification to the users banking app on their smart device.
Will it defect card fraud completely? Nope, crooks are smart and will find ways around it like they did with chip and pin. But it will help (as long as we don't keep pushing back the rollout date - https://www.businessinsider.com/uk-delays-strong-customer-au...).
> Start pushing for more validation from the payment providers so you can be sure that a payment is legit and once you have jumped though the hoops that a payment is legit the bank takes the hit for the fraud.
Agreed. There are basically zero digital payment systems in the US that aren't a complete joke, and all of our institutions think the correct answer to that problem is to add more captchas.
Imagine an alternate reality where we discovered that HTTP traffic was trivially monster-in-the-middled, and instead of moving to HTTPS, we all said, "well, we'll just do device fingerprinting to validate the request came from the right person." This is the current digital payment system.
Instead of moving to 2FA for each purchase or overhauling ACH, or changing payments to be a push system instead of a pull system, our solution to credit card fraud is that:
A) payments will take a long time to resolve,
B) customers will be required to be vetted using invasive, privacy-incompatible tools that don't work particularly well, and
C) whenever something goes wrong, the merchant will take the financial hit instead of the payment processor that let the fraud happen in the first place.
So many of our payment/privacy/fraud problems boil down to the fact that we are putting band-aides on a fundamentally broken system.
I don't think you as an individual can do a lot, and I don't blame you as an individual. In your position I would've probably done the same.
The situation with poorer countries is completely clear. As people have less and less honest options of providing a dignified living for themselves and their family, their moral principles give way and they turn to crime. Maslow hierarchy of needs and all that. In my opinion, the blame for a lot of these situations is the aggressive exploitative treatment and bullying of "small guys" by "big guys" on the global scene. But that's a separate political discussion, and a long one at that.
One thing that could help, but I doubt it's under your direct control, is taking advantage of systems such as 3-D Secure, which is SMS-based OTP for card payments. My card has this and I'm always happy to verify my payment this way, but payment processors are free to ignore these capabilities and just charge the card. (Conspiracy theory: this is because it hurts the conversion rates, as it adds another step to the funnel.) I think that a payment verified via 3-D Secure would have a much lower risk associated with it.
But overall, I don't think I can offer any suggestions on how to make the system better. I ranted due to a perceived injustice, but that didn't mean I have ways ready to make the situation better. I think that change must come from the powers that be - and they're unlikely to suddenly start giving a shit, so there goes that.
In any case, I thank you for considering my perspective and offering to help move the status quo. Rest assured, my friend, you are not part of the problem.
> But overall, I don't think I can offer any suggestions on how to make the system better. I ranted due to a perceived injustice, but that didn't mean I have ways ready to make the situation better. I think that change must come from the powers that be - and they're unlikely to suddenly start giving a shit, so there goes that.
> In any case, I thank you for considering my perspective and offering to help move the status quo. Rest assured, my friend, you are not part of the problem.
I disagree with basically all of this. A lot of people here are part of the problem. They are part of "the powers that be". Not that they want to keep anyone down or anything, it's just that they have businesses to run and will do reasonable accommodations but no more.
Since I'm sure everyone here wants the problem to be solved, if it can be done easily enough, there is a good hope of solving it with some smart technologies.
3d secure and looking at cookie reputation are both good ideas. If they are not used as much as you think they should be, we gotta ask why that is? Is it because it's too hard to implement? People haven't heard of it? Can library for doing it easily be made? Can it be pushed on people through add it as default, or including it in code samples?
This is much better alternative. The little bit of CPU time you burn probably won't even step up your fan or impact your battery life noticeably. What it will do is change the economics for bots so they go out of business. If there are less bots, then the other, more annoying countermeasures can be scaled back. Everyone wins, except the bots.
I understand your frustration, but the problem isn't that it's far away geographically, it's that your country is allowing criminals to use your IPs to do bad stuff.
The bit about colonialism was tongue-in-cheek, but it's not a completely nonsensical comment.
My country has such a poor median standard of living that things like computer fraud (or crime, generally) become attractive even to reasonably honest people as they become desperate in finding ways to provide a dignified living for themselves and their families.
Due to largely the same root causes, its government has insufficient infrastructure, bandwidth, and expertise to deal with cases of computer fraud, especially with big tech corporations usually refusing to cooperate/communicate with our officials.
One of the contributors to this situation is a military operation perpetrated by a well-known and powerful transatlantic military alliance as recently as 20 years ago; it resulted in $5 billion of direct economic damage (a big deal when that's your entire GDP) and countless billions of indirect economic loss which rippled throughout the years in the aftermath (not to mention the hundreds of civillian lives lost).
I'm trying not to drag politics and emotions into this too much, and my country surely has many issues unrelated to anyone outside, but I'm expressing my grievances as a common person who, like anyone else, did not choose their birthplace, and does in no way identify with the policies and actions of their government, but suffers greatly in many ways due to them + the constant bullying and pressure by the "colonial forces", which is what I was primarily referring to.
I wasn't implying it's your fault, just saying that it's weird to, say, get mad at Walmart when they put up extra security measures in places where they constantly get robbed.
112 comments
[ 3.1 ms ] story [ 94.9 ms ] thread"The wait is over. Our product and engineering teams have been working round-the-clock to build a new experience that makes the Internet faster and safer for everyone. If I were to say anything more, it would surely give it away so I’ll leave it at that — you’ll just need to tune in to the blog on Wednesday to find out."
What's that mean, 'kicked offline'?
Isn't scraping 'legal'?
Buying products at the price offered is perfectly legitimate, regardless of all the scaremongering to the contrary.
In my experience a lot of bots seem to be running on hacked servers or through hacked/insecure proxies. I'd imagine tracking down the owner or someone upstream of those boxes could be effective in taking them offline.
When I've run scraping software in the past I used DigitalOcean, which doesn't contain a requirement to abide by robots.txt. As far as I can tell it's both legal and consistent with their ToS to run a program that makes purchases on a website.
There's no way you're in this conversation without being aware that scalping is a controversial practice at best.
https://theconversation.com/the-economics-of-ticket-scalping...
https://en.wikipedia.org/wiki/Ticket_scalping
That doesn't make them right.
It doesn't matter if it's legal, it matters if the website owner doesn't want x doing y on their site. A bot consistently not abiding by owners' intent is inheritly malicious.
The LinkedIn case shows that it's not illegal in some cases - would you classify the behavior of the defendant in that case as malicious?
Have you run this by an economist? It's pretty basic economics that "scalping" increases consumer welfare, despite your cursory claim to the contrary.
If the IPs are part of a botnet, that's one thing. But the biggest residential IP network is luminati, which does have consent for their IPs.
Citations needed.
Here's some reading info and links for you to illustrate that you're comments are ignorant of several dimensions of the discussion: https://economics.stackexchange.com/questions/6576/is-scalpi...
> And the unwitting users who are part of the botnet have their resources, such as their home broadband connection, used without their consent or knowledge.
Perhaps I missed something, but doesn't this mean that a lot of homes and IoT devices that have been compromised will have increased CPU usage as a result of CF implementing the bad bot behaviour envisaged? In other words, the botnet owners won't care in the slightest about your response, but a lot of homes will suddenly get shoddy performance as their router grinds to a halt.
(Yes, of course it'd be best if every home and every IoT device were secure... but that's unrealistically optimistic)
Assuming you're building on the LI<>HiQ case here, the ruling would only be applicable to a subset of cases (public, user-generated content, no authorization...).
Even before the ruling is overturned, we can't say scrapping is legal without applying some qualifying conditions.
Steamspy.com seems to trigger one of these basically every time when loaded with a fresh cookie.
https://en.wikipedia.org/wiki/Slowloris_(computer_security)
They could also be doing things like serving fake streaming audio that never ends, or anything that might make it seem like the web page is just a huge page that needs time to load.
I'm not sure what they do for the non-JS use case. They sit in the request path like a CDN though so maybe they just return an error or deliberately slow response times?
> - Identification of well known legitimate bots;
What about non-well-known legitimate bots? If I run my own web crawler, am I at risk of falling into the tarpit (and having my IP address reported)?
Use CommonCrawl https://commoncrawl.org/
Then, after you can use that corpus and provide useful results and have people coming to your search site, then you can have a discussion with CloudFlare that you will be creating your own crawler.
A big part of the net neutrality fight was "allowing the little guy a chance to play with the big boys".
Part of the service says:- for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline... If the infrastructure provider hosting the bot is part of the Bandwidth Alliance, we’ll share the bot’s IP address so they can shutdown the bot completely. The Bandwidth Alliance allows us to reduce transit costs with partners and, with this launch, also helps us work together with them to make the Internet safer for legitimate users.
My reading of that is if CF decide your IP is bad, they can leverage the providers Bandwidth Alliance status to shutdown the providers customer. What if CF's systems misfire? Will there be a grace period? Will there be an appeals process? Will CF compensate anyone effected and have their hosting withdrawn from a misfire?
No one likes bad bots, but I'm feeling more and more uneasy allowing CF decide which is which.
They bring real visitors and customers.
Random bots running on AWS or TOR bring me headaches and bigger bills.
if this is to be believed [0] less then 50% of google searches result in a click though to the site, so one could argue that because of things like snippets in search results Google don't always drive traffic to your site.
[0] https://sparktoro.com/blog/less-than-half-of-google-searches...
Respecting robots.txt, using a well-known ua, coming from publicly declared, google owned IP block.
So if you're a small business that wants to build the next Google, you can't? Because obviously, your User Agent won't be well known when you start.
I would love for them to prove me wrong though and be open about such things.
I'm pretty sympathetic to this line of thought, so don't take this as 'you're wrong', but I notice that we would never apply this logic to laws, or environmental standards, or contracts.
You'd never hear someone say, "if we have a clearly defined tax code, that will just make it easier for people to find loopholes."
Where I do hear this argument come up is explicitly in contexts of moderation and abuse policies. And it's something that sounds very reasonable, but it's hard for me to get away from the fact that in most other contexts it sounds problematic to me.
Maybe part of the problem is Cloudflare's scale? Maybe the reason it feels bad to have a police officer pull me over because, "we think you're going too fast" instead of "you went over a posted limit", is because that's a critical infrastructure that I can't avoid.
Cloudflare is a private company, and even if it wasn't, I don't know if it would be big enough presence that I would worry about their policies. But when I hear things like this:
> Our goal is nothing short of making it no longer viable to run a malicious bot on the Internet. And we think, with our scale, we can do exactly that.
That maybe shifts the situation a tiny bit farther away from "moderation policy on a personal blog" towards "policeman pulling me over because I broke a law I didn't know existed."
I dunno. I'm not 100% sure how to feel about it. I do think that Cloudflare should be able to filter traffic however they see fit, but that doesn't mean that every strategy they choose is inherently good, or that it might not be problematic for them to lack transparency about their standards.
Also for fairness - Does Akamai publicize the internals of BotManager? Imperva gives you much transparency into how DistillNetworks works?
Then fraud detection system cannot work ethically.
Can't become a well-known UA if your shutdown before you can make a name for yourself. And a Google IP block doesn't mean legitimate.
I'm not trying to say that Google is bad, but why can't OP's crawler be legitimate?
It does. 2-way DNS lookup is how you verify legit indexers [0] (as ua itself is meaningless).
[0] https://support.google.com/webmasters/answer/80553
But what are CF (and others like akamai) using to determine you are a bad bots? My objection is more the threat they will use their position to kick bad bots offline.
Now don't get me wrong, I had bad bots as much as the next person. But what about misfires? They will happen. I hope CF have a plan in place to deal with such cases.
Personally, due to family issues I've been working remote-remote (I'm a remote worker anyways, but I've been working remotely away from my normal office). As such I've been using a 4G Mobile broadband connection and the amount of CloudFlare "We are just checking you are not a bot" hoops I've had to jump though recently has opened my eyes on how much of the net is actually behind CF's network because CF still associate "IP as Identity" and my 4G connection is CG-NAT'ed.
I just fear at times we forget that there is the rest of the world out there, not just our little bubble and the choices we make have a huge impact on the usability of the net for others.
EDIT: LOL.. Just checked your profile. The Reg is one of the sites that doesn't like my 4G connection and as such I visit less. What are the chances. (Though you only make me sit there and wait to access the content....)
Sorry if that's not the answer you wanted, but that's how I feel after seeing how the other side of the net is treated. IF the urge takes me, I'll help, but I dunno when that urge will take me though.
<EDIT>Fun Fact: My public IP address appears to have changed in the time it took me to write this comment. Atleast to the reg, How my ISP handles routing is another black box upon itself.</EDIT>
Don't worry, their proprietary AI will figure it all out.
> Identification of well known legitimate bots;
> Hand written rules for simple bots that, however simple, get used day in, day out;
> Our Bot Activity Detector model that spots the behavior of bots based on past traffic and blocks them; and
> Our Trusted Client model that spots whether an HTTP User-Agent is what it says it is.
In my dream world, Cloudflare would make this a two way street: you get protection against malicious bots, but you have to allow equitable treatment of all well behaved search engines (i.e. you can use robots.txt to ban lesser search engines, but doing that requires you to ban Google as well).
Disclaimer: I have no current economic interest, but kinda want to run a personal search engine.
I'm glad to see that if my services are behind cloudflare, I could just turn something on and let _them_ deal with it.
https://en.wikipedia.org/wiki/Tarpit_(networking)
>The targets are goods of generally of limited supply and high in demand and in value. Think sneakers, concert tickets, airline seats, and popular short run Broadway musicals. Bot operators who are able to purchase those items at retail can charge massive premiums in aftermarket sales. When the operator identifies a target site, such as an ecommerce retailer, and a specific item, such as a new pair of sneakers going on sale, they can purchase time on the new Residential Proxy as a Service market to gain access to end user machines and (relatively) clean IPs from which to launch their attack.
They then go on to spout some economic nonsense about how such bots are harmful. Actually, resellers make the market more efficient, and cloudflare is doing a disservice by lumping legitimate bots in with malicious ones like their credential stuffing example.
If the manufacturer of an item increases its price and thus their profit from it, then in principle the increased profit can drive them to manufacture more of the item, avoiding deadweight loss due to insufficient supply. However, that doesn’t apply if a reseller is the one making the profits! And in any case, some of the concrete examples cited (tickets to concerts and Broadway musicals) have an essentially fixed supply; it’s hard or impossible to simply increase production.
Separately, moving goods toward those willing to pay more for them is to some extent equivalent to moving them toward those who care more about them, which could be said to improve efficiency in terms of how much personal happiness is gained per unit. However, it also moves the goods toward those who are simply wealthier. And there are other possible proxies for personal interest besides price, ones that aren’t as affected by wealth. One of them is a person’s willingness to research in advance when certain tickets will go on sale, and be online at that time to snag them before they sell out – in other words, the system that bots are used to subvert. This system has its own biases (favoring those who are organized, have more flexible schedules, or are simply in a more favorable timezone), but it‘s still considered to be reasonably fair, at least when it’s not being subverted.
Scalping unambiguously increases Kaldor-Hicks efficiency under a simple model. You can argue that efficiency isn't the best way to be fair, and it's fairer to someone to not have a ticket due to random luck rather than being priced out, perhaps. But scalping increases efficiency, which is all I claimed.
I don’t see how that follows. If a consumer can’t buy a ticket from the original seller due to a shortage, there’s no guarantee that the original sale price was at (or even near) the maximum price that that particular consumer was willing to pay. Therefore, that consumer might also be willing to buy from a scalper. It would be in the opposite case, if there wasn’t a shortage, that someone who doesn’t buy could be assumed to be unwilling to pay the price.
In practice, I’d expect a decent fraction of those consumers to be near their maximum price, since if not, we could conclude that the original seller irrationally set their price far too low. (But the original price must have been set somewhat too low if a significant shortage exists.) Still, depending on the markup charged by the scalper, I wouldn’t be surprised if a significant fraction of consumers actually would be willing to pay it.
As a practical example, according to an article from last year about Hamilton tickets [1], among seats originally sold for $69-$179, a few were on sale for >$10,000 each, but the average resale price was $412. $10,000 will obviously price out most consumers, but going from ~$100 to ~$400 might not, considering the extremely high desirability of Hamilton seats.
> Scalping unambiguously increases Kaldor-Hicks efficiency under a simple model. You can argue that efficiency isn't the best way to be fair, and it's fairer to someone to not have a ticket due to random luck rather than being priced out, perhaps. But scalping increases efficiency, which is all I claimed.
If I understand correctly, it increases efficiency if the scalper could have hypothetically purchased their ticket from someone who was deprived the opportunity to buy from the original seller, instead of from the original seller directly, and still made a profit by reselling it.
But that doesn’t seem true. There’s no reason to think that person who “won the lottery” would be willing to sell the ticket for the same price they bought it for. Instead they would likely sell it for a similar price to what the scalper wants to charge. At best the scalper can gain some value on the margin by taking on the risk that the ticket won’t sell, but not much value.
[1] https://www.seattletimes.com/entertainment/theater/how-did-h...
As long as the ticket moves to someone with a higher willingness-to-pay, Kaldor-Hicks efficiency is increased.
Since the ticket holder without scalpers isn't the same as the one with scalpers, they have a lower willingness-to-pay than the price the scalper sells at. Therefore, they'd be willing to sell to the actual ticket holder at that price.
The only difference between this scenario and the one with a scalper is some money that the scalper has instead of the original ticket holder, and transfers don't factor into Kaldor-Hicks so this is irrelevant.
Which, as I said before, is only some of the time, but even under that assumption...
> Since the ticket holder without scalpers isn't the same as the one with scalpers, they have a lower willingness-to-pay than the price the scalper sells at. Therefore, they'd be willing to sell to the actual ticket holder at that price.
Then why don't they? Anyone who buys a ticket intending to go to the play is free to change their mind and resell it. Modern online services make this quite easy. But most people don't resell their tickets.
There are a variety of possible explanations. One is to invoke arguably irrational (but still real) behavior like loss aversion. Another has to do with the inconvenience of rescheduling or the risk of not selling, but that doesn't apply to the Kaldor-Hicks calculation since being deprived of a ticket (as opposed to selling it) doesn't create that inconvenience or risk. But here is one possible situation where even an economically rational actor would prefer to have a ticket than to be awarded a market price:
The value of the ticket to them may be higher than the scalpers' market price. This can be true even if the holder was not originally willing to pay that much, because the holder effectively gained net worth as soon as they "won the lottery" for the ticket, so they have a different amount of spare money to work with. And if this person took the time to buy tickets as soon as they were available, they probably care more about the play than the average person in the secondary market, so they would put a higher value on it relative to their wealth.
This is balanced by the fact that other people in the secondary market will likely be wealthier than them. Indeed, in the case of those rare orders-of-magnitude price hikes, most holders probably would be willing to sell if they could be sure the sale would actually go through. But most scalping margins are much more moderate and may not be enough to get the price over the value to the original holder.
The only scenario where it's not is if the demand curve is perfectly inelastic. If the price is set at $100, and selling it for $400 instead doesn't raise the average willingness to pay of buyers, then there has to be exactly the same demand at $100 as at $400, which is extremely unlikely.
>Then why don't they? Anyone who buys a ticket intending to go to the play is free to change their mind and resell it. Modern online services make this quite easy. But most people don't resell their tickets.
We're talking about a hypothetical scenario without scalpers. In that scenario, it's difficult to impossible to resell tickets.
>But here is one possible situation where even an economically rational actor would prefer to have a ticket than to be awarded a market price:
Yes, positing an income effect from winning the ticket lottery can produce a model where this result no longer holds. But income effects that strong are insanely rare. To reframe this, you're suggesting if they had found $400 on the street that morning, the rational thing to do given their preferences is to spend $400 on a ticket, but they shouldn't otherwise. You're depicting someone so poor that an additional $400 represents a significant wealth difference, but also where it makes sense to spend $400 on a ticket rather than on other things. In the real world, I think we'd call someone like that irrational.
It might be plausible in other contexts, but it doesn't seem reasonable in the ticket context.
Nice UX, fast, free. Nice domain service if you transfer to them. Fast DNS management. DOS/bot mitigation. Caching. Quick SSL, and affordable upgrade options. 2FA via TOTP.
With all the networking/domain stuff momentum build, it would be nice to be able to spin up servers for apps/db.
Main language at cloudflare is golang? Rust? Any python over there?
Request: Allow changing the super administrator for cloudflare account more easily. At least for early-stage accounts.
- Harrassment with an extra verification step every god damn time you log in into a website you've been a paying customer at for years because some rando risk model thinks you're a bad guy
- Harrassment by reCAPTCHA to respond to US-centric image challenges when it decides it's up for some sadistic fun (Mark all images with a store front or street light? Sure, let me just Google how they are "supposed" to look, because I sure as hell ain't seeing any I'm familiar with here.)
- Blanket IP range bans which serve you a default 503 error page and call it a day, with ripple effects throghout tons of unrelated websites. I always know when my ISP's IP range is added to some new blacklist.
- Harrassment by Cloudflare "verifying my browser", presumably burning my CPU cycles so they can be sure that my browser, which has been hitting their IP ranges for years with not as much as a cookie wipe, has not suddenly turned into a bot
- "Your name is invalid": no, your regex is
- "Our fetishism for credit prevents us from accepting your debit card payment unless you submit a scan of your passport signed in blood"
- (NEW) CloudFlare's anti-bot measures which will surely not misfire because no one thought to test their shiny new model on the traffic patterns of some culture with < 50 million Internet users
Apologize for the off-topic rant, but you wouldn't believe how using the Internet for basic things has gotten difficult in the past few years over here (and from what I hear, in many other "forgotten" geographies). No one cares. Long live colonialism!
For comparison purposes, the dispute rate for US-origin payments (US card, US origin IP, no evidence of proxy use) was under 1%.
If I couldn't do things like "submit a scan of your passport", I would probably just decline any cards from high-risk countries.
What would you suggest I do instead?
In the EU one thing we are doing is basically enabling 2FA for card purchases aka "Strong Customer Authentication".
Now I'm not 100% happy with the use of SMS as a 2FA but what will happen is if the bank or site ask for extra verification of a purchase the issuing bank will prompt the user to auth the transaction via another means (Either a SMS Code they enter into the page (think verify by visa but using a one time code that gets delivered to the customer with the amount they are approving), a Rich Communication Services with a Approve / Deny prompt or a push notification to the users banking app on their smart device.
Will it defect card fraud completely? Nope, crooks are smart and will find ways around it like they did with chip and pin. But it will help (as long as we don't keep pushing back the rollout date - https://www.businessinsider.com/uk-delays-strong-customer-au...).
Agreed. There are basically zero digital payment systems in the US that aren't a complete joke, and all of our institutions think the correct answer to that problem is to add more captchas.
Imagine an alternate reality where we discovered that HTTP traffic was trivially monster-in-the-middled, and instead of moving to HTTPS, we all said, "well, we'll just do device fingerprinting to validate the request came from the right person." This is the current digital payment system.
Instead of moving to 2FA for each purchase or overhauling ACH, or changing payments to be a push system instead of a pull system, our solution to credit card fraud is that:
A) payments will take a long time to resolve,
B) customers will be required to be vetted using invasive, privacy-incompatible tools that don't work particularly well, and
C) whenever something goes wrong, the merchant will take the financial hit instead of the payment processor that let the fraud happen in the first place.
So many of our payment/privacy/fraud problems boil down to the fact that we are putting band-aides on a fundamentally broken system.
The situation with poorer countries is completely clear. As people have less and less honest options of providing a dignified living for themselves and their family, their moral principles give way and they turn to crime. Maslow hierarchy of needs and all that. In my opinion, the blame for a lot of these situations is the aggressive exploitative treatment and bullying of "small guys" by "big guys" on the global scene. But that's a separate political discussion, and a long one at that.
One thing that could help, but I doubt it's under your direct control, is taking advantage of systems such as 3-D Secure, which is SMS-based OTP for card payments. My card has this and I'm always happy to verify my payment this way, but payment processors are free to ignore these capabilities and just charge the card. (Conspiracy theory: this is because it hurts the conversion rates, as it adds another step to the funnel.) I think that a payment verified via 3-D Secure would have a much lower risk associated with it.
But overall, I don't think I can offer any suggestions on how to make the system better. I ranted due to a perceived injustice, but that didn't mean I have ways ready to make the situation better. I think that change must come from the powers that be - and they're unlikely to suddenly start giving a shit, so there goes that.
In any case, I thank you for considering my perspective and offering to help move the status quo. Rest assured, my friend, you are not part of the problem.
> In any case, I thank you for considering my perspective and offering to help move the status quo. Rest assured, my friend, you are not part of the problem.
I disagree with basically all of this. A lot of people here are part of the problem. They are part of "the powers that be". Not that they want to keep anyone down or anything, it's just that they have businesses to run and will do reasonable accommodations but no more.
Since I'm sure everyone here wants the problem to be solved, if it can be done easily enough, there is a good hope of solving it with some smart technologies.
3d secure and looking at cookie reputation are both good ideas. If they are not used as much as you think they should be, we gotta ask why that is? Is it because it's too hard to implement? People haven't heard of it? Can library for doing it easily be made? Can it be pushed on people through add it as default, or including it in code samples?
I understand your frustration, but the problem isn't that it's far away geographically, it's that your country is allowing criminals to use your IPs to do bad stuff.
My country has such a poor median standard of living that things like computer fraud (or crime, generally) become attractive even to reasonably honest people as they become desperate in finding ways to provide a dignified living for themselves and their families.
Due to largely the same root causes, its government has insufficient infrastructure, bandwidth, and expertise to deal with cases of computer fraud, especially with big tech corporations usually refusing to cooperate/communicate with our officials.
One of the contributors to this situation is a military operation perpetrated by a well-known and powerful transatlantic military alliance as recently as 20 years ago; it resulted in $5 billion of direct economic damage (a big deal when that's your entire GDP) and countless billions of indirect economic loss which rippled throughout the years in the aftermath (not to mention the hundreds of civillian lives lost).
I'm trying not to drag politics and emotions into this too much, and my country surely has many issues unrelated to anyone outside, but I'm expressing my grievances as a common person who, like anyone else, did not choose their birthplace, and does in no way identify with the policies and actions of their government, but suffers greatly in many ways due to them + the constant bullying and pressure by the "colonial forces", which is what I was primarily referring to.
My question is, what if you are starting an alternative search engine or something legit?
Edit: my point is, the rules than make a legit or not bot, crawler, scrapper etc. are not clear at all.