> As an example, here’s a measurement of jitter (variance in round trip time) between two of our data centers, Chicago and Newark, over a transit provider’s network... Average jitter over the pictured 6 hours is 4ms, with average round trip latency of 27ms.
> Here’s the same jitter chart between Chicago and Newark, except this time, transiting the Cloudflare Global Private Backbone... Here we see a jitter measurement of 536μs (microseconds), almost eight times better than the measurement over a transit provider between the same two sites.
This is really impressive. Will be interesting to see what's possible when network roundtrips are 2 orders of magnitude more stable.
Seems more like a sales pitch than anything which conveys real information. And, as they imply (but others posting here dispute), if it's better in all cases, why wouldn't it be on in all cases? Why didn't they mention the downsides to enabling it?
Argo is great. I temporarily switched on on a customer's domain and we saw 50+% improvements on latency!
Question to CF employees, since they usually participate here on HN: Any chance we are ever going to be able to enable Argo on specific domains / paths, similar to Workers?
What would you consider a fair per-GB price for Cloudflare to route data over their own private backbone (which I hastily assume includes their own leased fiber, routers/switch fabric, and other associated fixed and variable costs)?
I gladly pay $0.10/GB for some of my smaller, lower-bandwidth sites.
For these sites, I see a latency improvement of around 25% for around 50% of (smart-routed) traffic.
For a couple of my larger sites (PDF and Image heavy), I am unable to justify the Argo cost. On these sites, over half of my monthly traffic is to bots (Googlebot, Bingbot, Yandex, etc). Traffic costs make up of around half of my hosting bill.
A couple of things are keeping me from transitioning these sites to Argo:
- I'd rather pay to speed up only real-users (not bots)
- I'd rather not pay for traffic not routed over Argo (currently all traffic gets billed under Argo for enabled domains, not just the smart-routed traffic)
I currently pay the following rates from Google Cloud->CloudFlare (depending on the CF datacenter the data gets routed to).
NA: $0.04/GB
EU: $0.05/GB
APAC: $0.06/GB
If I could pay a similar rate for a speedup (for only the Argo-routed traffic), I'd be more likely to sign up for Argo on all of my domains.
We provide a lot of value for free through our free service. Argo is an additional paid service that improves performance for customers that want to pay for that.
Your free service is nice - except that you can't use CF as a registrar for more than one domain without also having to pay for the proxying (even if there's no need). I'd gladly put all my eggs in one basket instead of having them spread out across five different registrars if that were fixed. Though maybe it's by-design?
Not sure I understand "pay for the proxying" - you should still be able to sign up your domains with the "free" plan as long as you use CF as your dns. If you don't want to proxy, your DNS records can be set to grey clouds so it just acts as a DNS service (but yes, you do have to be on Cloudflare's nameservers for now).
At this time we don't have any plans to open up the Registrar to domains not using Cloudflare's DNS. If that changes in the future we'll be sure to announce it.
Is there any benefit to having critical domains registered and DNSed at Cloudflare as opposed to using a separate registar w/ Cloudflare's DNS? The benefit of the latter is that one has optionality of DNS providers, and additionally one can use DNS providers that support zone export for secondary DNS.
There's only one domain allowed for free in the DNS/Proxying plan, there's no way I've seen to add more than one domain without being forced to buy the advanced plan?
That's not true. You have always been able to add as many domains for free to a plan as you want. I think there may be some coded limit in the thousands, but our support team can even waive that if you ask nicely.
Hmm, when Cloudflare registrar first came out I remember not being able to transfer just a domain name to CF without having to upgrade plans (or it was a dark pattern). Maybe I indeed have to recheck.
Hi Avamander - I'm the product manager for Cloudflare's Registrar. There is no limit on the number of domains that can be on the Cloudflare registrar for free accounts. You do however need to add the website first (from your dashboard home page). Once a new site has been added you can transfer the domain into the Registrar. Feel free to reach out to our support team and they can help walk you through it.
But when I have to add a new site I have to pay when I have more than one which basically makes it not-free to have more than one domain on a free account just for the registrar feature?
I see this and am reminded of when you guys raised the price for your pro plan from 5 to 20. How long until you get great adoption, followed by a 4x price increase?
We called that the 20-5-5 deal, and as someone that used Cloudflare and then went on to work there for a number of years, that is the only time I can remember Cf raising prices for something that had been fully released.
From my experience, features are added to the Enterprise offerings, and then eventually brought down to lower plan levels. However, most things are based on usage now, not plan level and I expect that is a better path to monetization for CF compared to simply raising prices.
I noted in my other comment that your product manager for this service does not use cloudflare for his personal domain ( lalkaka.com ) but google domains.
I think this is fair. Correlation between [website] performance and "customer happiness" (convert as you want) is pretty established. Also, I think workers and argo (both starting at ~$5/mo) is the right path of breaking down cloudflare cost from free/pro/business/enterprise to "pay as you go".
Rustam can weigh in here, but I'm not exactly sure what with Argo we'd open source. Argo is a collection of a bunch of different technologies that we use to measure the performance across our transit, peering, and backbone connections. We then send different requests down different paths depending on each customer's particular needs and settings. While I'd imagine we will open source parts of the overall system, there's not really a stand-alone package that we could make available for anyone else to use.
If there's a particular part of the overall Argo that would be helpful for you or others, let us know and we'll certainly consider it. Here's a list of things we've built at Cloudflare and open sourced already: https://cloudflare.github.io/
You should consider to remake those graphs comparing the jitters. The non-backbone graph shows a majority of data points are below 0, which is impossible, so that means there are a small long tail at the high >20ms band, and you intentionally cropped them out. So when you said "oh the backbone graph is much better!". No it is not, it is very confusing actually.
I think that is showing the different from the average, so the variance in the jitter. It's not the actual latency which is not actually revealed with the Backbone in place.
There are similarities between Argo and the Google/Amazon products you mention (as well as Azure's Front Door). All are aiming to solve similar problems inherent to routing on the Internet.
Re: enabling Argo at the network layer, watch this space :) I'd love to hear more about what you'd like to do such a product — email me! rustam@cloudflare.com
Off topic but your personal domain is lalkaka.com (your profile) and it does not have a web page. Now before HN jumps on me for this (I know you just use it for email) there is something about someone who works for cloudflare not only not using cloudflare (dns is google domains not cf) but not having the domain resolve to at least a simple 'hello' on a web page.
Why does any of this matter, even in the slightest, when talking about "Argo and the Cloudflare Global Private Backbone" which is what this post is about.
Not everyone needs to have a web page, nor does an employee of a company have to use their services (especially if they spun up their domain, website, whathaveyou before working for the company) for any sort of personal project.
> Why does any of this matter, even in the slightest, when talking about "Argo and the Cloudflare Global Private Backbone" which is what this post is about.
So I need to explain to someone who doesn't think that it matters that it matters to me?
The fact that a post is about something else has never prevented people who take the time to comment on HN (an unpaid endeavor) from voicing an opinion on something related or ironic in some way.
> nor does an employee of a company have to use their services (especially if they spun up their domain, website, whathaveyou before working for the company) for any sort of personal project.
You are right it is not codified into some law. But as a general rule it makes more practical sense for an employee to use the employers products especially if they are posting a comment on a blog about another service that the employer offers where it's easy to see (such as I did) that they are not using the employers free primarily product that they are known for.
Why should the employer not expect that their employees use their free product (which jgrahamc does use of course for his personal domain JGC.ORG ..).
There is no "general rule"; stop trying to make it seem like there is.
Additionally, many companies learned the hard way that forcing an employee to use your products or services over those they want to use will do nothing but lower morale and cause frustration. Let people use what they want and already are using and they'll be much happier because they don't feel stifled by their employer. That's why most companies have a BYOD policy for things like phones, because nobody wants to switch if they don't have to.
Finally, it's frankly none of your business what products a person, any person chooses to use for their own personal projects. That's their decision to make, not yours, and they don't have to justify it to anyone but themselves.
Meh. I'm not the guy you're replying to, and I think he is introducing a bit of an 'issue' where there really isn't one. However, I think it brings up an interesting topic/observation about Cloudflare.
There certainly is a stigma in many companies that employees should use their employers products. I used to do work at some major beverage manufacturers, and for example, carrying a can of Coke into a Pepsi office will literally have you escorted out of the building. For lunch, Pepsi employees are heavily discouraged from eating at places that serve Dr Pepper or Coke products. Company money absolutely cannot be spent on buying catering/services from caterers that don't have Pepsi, etc. For airlines, I have a friend that works at United and he has said multiple times about how he hopes his colleagues never find out that he flies American sometimes because it's cheaper when flying to his hometown.
More recently and relatable to this forum, there was a post a week or two ago where a GitLab employee got lambasted by the CEO for using Github for his personal projects rather than GL.
If anything, I think this is an interesting observation that Cloudflare might have 'less of a stick up its butt' in terms of policing what its employees do. Is this thread the right place for that discussion? Idk, but it's an interesting discussion nonetheless.
The only reason I can see for wanting an employee to have a site on Cloudflare is so that they get personal experience of using Cloudflare _as a user_. That's very valuable and the primary reason I have jgc.org on Cloudflare: so I get to try every new Cloudflare product.
I actually pay for Cloudflare out of my own pocket like a normal user so I see every wart and new product etc.
You feel that way because it does not move the needle for someone to use your own product on such a small scale. Now let's say that same employee was socializing or coming in contact in some way with a decision maker for a potential large customer (or a current large customer). In that case you wouldn't feel the same way. And if you did feel the same way that wouldn't make much business sense because you know business and making a profit is important to most companies or should be. So sure it's easy to take that attitude because you think it doesn't matter (in this case) or because you aren't considering the very general sense of what it might mean to others.
It's is well known that companies go to great lengths for various reasons to encourage their employees to not only use their own products but to discount those products for an employee. As far as you paying for Cloudflare so you can see the end to end experience that's fine and that makes sense. However Cloudflare should consider reimbursing the nominal amount that you are paying. Further they should consider (Apple and Adobe did this as well as auto manufacturers) giving discounts to 'friends and family' because it's a decent source of feedback on the product. You know what else it does? It encourages friends and family to support the person continuing working for the company (because it's in their best interest because they get a benefit). I know of someone personally whose brother is high up at a well know auto manufacturer and the brother gets a car literally at no cost just to be an ambassador and encourage other sales at that company (and it works I have looked into buying the car as a result as have others).
Also noting that Cloudflare is an internet company for which the competitive landscape and ability to have losses is vastly different than an airline or a beverage manufacturer. There is a reason that old school companies are like that. And to think it's wrong in some way does not consider the situation that they are dealing with trying to keep market share in a business with very thin margins.
No the 'general rule' is that in certain situations you do something not to appear to be disrespectful in some way. So for example if you were to work at Ycombinator the 'general rule' would be to try and encourage others to use yc instead of someone who competes with YC. Ditto if you work for a department store it would be typical to be supportive in some way for your employer (who gives you a job) and not if you can afford to shop at one of their competitors. For some reason this has been lost in the new economy.
Because your use case might be different than what your company targets? I see no problem with that, heck, if the person used cloudflare and only cloudflare, I would probably question if they are making proper desicions of benefits of diffirent products, and if they are drinking company kool-aid.
I would rather read content generated by someone who is able to compare and contrast various technologies in an objective fashion, instead of a yes man.
I’m a customer using Argo, but I have to admit it’s not entirely visible to me how big the benefits are that I’m getting. I’m sure it’s great but.. just how much? Some visualizations would be nice.
Additionally, we have found the Argo tunnel to be relatively unreliable. Once every few months it seems there is an issue with one of the PoPs, and it takes a long time before our tunnels reconnect (several hours even); which imho is fairly inexcusable for something as critical as this. As such we’ve had to migrate away from using the tunnel, leaving me with earlier mentioned questions on the value we’re getting.
Analytics (including some cool visualizations) are available in the Traffic Analytics section of the dashboard. There is always more we can do — drop me a note at rustam@cloudflare.com if there are specific additional things you'd like to see.
Re: tunnel reliability, sorry to hear you've had a poor experience. We've been working hard on hardening things like tunnel reconnection and would love for you to give tunnels another shot.
Looking at how Argo is presented (from the blog post), it seems like you've got a really strong use case for SDN, if you're not already using that already. As far as I can tell, dynamically updating routes across an internal network is one of the best non-internal-to-data-center uses of SDN - can you expand a little more on this, if you do use it, or on why you don't, if I'm completely off base?
The data plane runs on our x86 fleet and its control plane is driven by custom route computation and propagation systems (discussed a little more in the post). We're excited to talk more about the technical underpinnings of Argo in the future.
Ha I was reading the post again thinking about how you all did this, and it boiled down to OpenFlow/OVS/SDN magic, or you REALLY went all in on Cisco offerings.
>We're excited to talk more about the technical underpinnings of Argo in the future.
Looking foward to this update, thanks for the response!
I'm curious if this conflicts with the Argo AI trademark. Does anyone have any insight into how the similarity would be determined? Specifically, I'm thinking that routing and self driving cars are near enough to cause some confusion.
This looks like little more than marketing fluff and with rather confusing messaging at that. For example the post states:
>"One specific example of Argo’s smarts is its ability to distinguish between multiple potential connectivity options as it leaves a given data center. We call this “transit selection”
A "global private backbone" and edge routing have zero to do with each other. They are orthogonal. You can "pref" your transit provider's customer routes from just within POP itself. This has zero to do with whether or not you have a "backbone." The way this works is that you pick your top N destination ASNs and continuously monitor ping time, jitter packet loss etc. Then you prefer the more desirable routes via iBGP. This is not new and not innovative either. Internap has been this 20 for years now. This was the whole "secret sauce" of their PNAP architecture back in the day. In fact they even sold a hardware device to do it called the Internap FCP(flow control platform)it has since been named MIRO [1][2]. Of course many people do exactly this themselves with BIRD/Quagga/Exabgp and some Python. And you certainly don't need "machine learning" to do this.
As a CDN customer I only care about getting the best path from the POP back to the eyeball networks. Why would it matter whether the CDN provider has fiber between two of their POPs? There should be enough transit diversity at each edge location
that fiber between POPs is irrelevant. I should spend as little time on the CDN provider's network as possible i.e "hot potato" routing. This is the whole value proposition of a CDN. Honestly this whole sounds like a cost savings mechanism that they've trying to spin and sell as a product innovation.
Also where is the Cloudflare backbone map? It seems to be conspicuously absent for announcement about a "global private backbone." How many route miles are there? Is it an actual backbone or this just some CWDM gear that only connects POPs in the same city over municipal fiber?
58 comments
[ 3.0 ms ] story [ 128 ms ] thread> Here’s the same jitter chart between Chicago and Newark, except this time, transiting the Cloudflare Global Private Backbone... Here we see a jitter measurement of 536μs (microseconds), almost eight times better than the measurement over a transit provider between the same two sites.
This is really impressive. Will be interesting to see what's possible when network roundtrips are 2 orders of magnitude more stable.
1. We've started using our own backbone of fibre links to carry traffic.
2. The Argo product for route optimization and working around Internet problems.
#2 can take advantage of #1 now that it exists. #2 is a paid product.
Question to CF employees, since they usually participate here on HN: Any chance we are ever going to be able to enable Argo on specific domains / paths, similar to Workers?
For a couple of my larger sites (PDF and Image heavy), I am unable to justify the Argo cost. On these sites, over half of my monthly traffic is to bots (Googlebot, Bingbot, Yandex, etc). Traffic costs make up of around half of my hosting bill. A couple of things are keeping me from transitioning these sites to Argo: - I'd rather pay to speed up only real-users (not bots) - I'd rather not pay for traffic not routed over Argo (currently all traffic gets billed under Argo for enabled domains, not just the smart-routed traffic)
I currently pay the following rates from Google Cloud->CloudFlare (depending on the CF datacenter the data gets routed to). NA: $0.04/GB EU: $0.05/GB APAC: $0.06/GB
If I could pay a similar rate for a speedup (for only the Argo-routed traffic), I'd be more likely to sign up for Argo on all of my domains.
From my experience, features are added to the Enterprise offerings, and then eventually brought down to lower plan levels. However, most things are based on usage now, not plan level and I expect that is a better path to monetization for CF compared to simply raising prices.
If there's a particular part of the overall Argo that would be helpful for you or others, let us know and we'll certainly consider it. Here's a list of things we've built at Cloudflare and open sourced already: https://cloudflare.github.io/
- Tunnel (cloudflared): https://github.com/cloudflare/cloudflared
- Argo as ingress (k8s): https://github.com/cloudflare/cloudflare-ingress-controller/
Can Argo be enabled at network layer, say VPN, and for other protocols say VoIP and not just HTTP/S?
Does Argo take inspiration from published research? If so, which ones?
Thanks.
Re: enabling Argo at the network layer, watch this space :) I'd love to hear more about what you'd like to do such a product — email me! rustam@cloudflare.com
Not everyone needs to have a web page, nor does an employee of a company have to use their services (especially if they spun up their domain, website, whathaveyou before working for the company) for any sort of personal project.
So I need to explain to someone who doesn't think that it matters that it matters to me?
The fact that a post is about something else has never prevented people who take the time to comment on HN (an unpaid endeavor) from voicing an opinion on something related or ironic in some way.
> nor does an employee of a company have to use their services (especially if they spun up their domain, website, whathaveyou before working for the company) for any sort of personal project.
You are right it is not codified into some law. But as a general rule it makes more practical sense for an employee to use the employers products especially if they are posting a comment on a blog about another service that the employer offers where it's easy to see (such as I did) that they are not using the employers free primarily product that they are known for.
Why should the employer not expect that their employees use their free product (which jgrahamc does use of course for his personal domain JGC.ORG ..).
Additionally, many companies learned the hard way that forcing an employee to use your products or services over those they want to use will do nothing but lower morale and cause frustration. Let people use what they want and already are using and they'll be much happier because they don't feel stifled by their employer. That's why most companies have a BYOD policy for things like phones, because nobody wants to switch if they don't have to.
Finally, it's frankly none of your business what products a person, any person chooses to use for their own personal projects. That's their decision to make, not yours, and they don't have to justify it to anyone but themselves.
There certainly is a stigma in many companies that employees should use their employers products. I used to do work at some major beverage manufacturers, and for example, carrying a can of Coke into a Pepsi office will literally have you escorted out of the building. For lunch, Pepsi employees are heavily discouraged from eating at places that serve Dr Pepper or Coke products. Company money absolutely cannot be spent on buying catering/services from caterers that don't have Pepsi, etc. For airlines, I have a friend that works at United and he has said multiple times about how he hopes his colleagues never find out that he flies American sometimes because it's cheaper when flying to his hometown.
More recently and relatable to this forum, there was a post a week or two ago where a GitLab employee got lambasted by the CEO for using Github for his personal projects rather than GL.
If anything, I think this is an interesting observation that Cloudflare might have 'less of a stick up its butt' in terms of policing what its employees do. Is this thread the right place for that discussion? Idk, but it's an interesting discussion nonetheless.
I actually pay for Cloudflare out of my own pocket like a normal user so I see every wart and new product etc.
It's is well known that companies go to great lengths for various reasons to encourage their employees to not only use their own products but to discount those products for an employee. As far as you paying for Cloudflare so you can see the end to end experience that's fine and that makes sense. However Cloudflare should consider reimbursing the nominal amount that you are paying. Further they should consider (Apple and Adobe did this as well as auto manufacturers) giving discounts to 'friends and family' because it's a decent source of feedback on the product. You know what else it does? It encourages friends and family to support the person continuing working for the company (because it's in their best interest because they get a benefit). I know of someone personally whose brother is high up at a well know auto manufacturer and the brother gets a car literally at no cost just to be an ambassador and encourage other sales at that company (and it works I have looked into buying the car as a result as have others).
I would rather read content generated by someone who is able to compare and contrast various technologies in an objective fashion, instead of a yes man.
Additionally, we have found the Argo tunnel to be relatively unreliable. Once every few months it seems there is an issue with one of the PoPs, and it takes a long time before our tunnels reconnect (several hours even); which imho is fairly inexcusable for something as critical as this. As such we’ve had to migrate away from using the tunnel, leaving me with earlier mentioned questions on the value we’re getting.
Re: tunnel reliability, sorry to hear you've had a poor experience. We've been working hard on hardening things like tunnel reconnection and would love for you to give tunnels another shot.
Cheers, argo seems like an interesting project.
The data plane runs on our x86 fleet and its control plane is driven by custom route computation and propagation systems (discussed a little more in the post). We're excited to talk more about the technical underpinnings of Argo in the future.
Ha I was reading the post again thinking about how you all did this, and it boiled down to OpenFlow/OVS/SDN magic, or you REALLY went all in on Cisco offerings.
>We're excited to talk more about the technical underpinnings of Argo in the future.
Looking foward to this update, thanks for the response!
https://developers.cloudflare.com/argo-tunnel/trycloudflare/
>"One specific example of Argo’s smarts is its ability to distinguish between multiple potential connectivity options as it leaves a given data center. We call this “transit selection”
A "global private backbone" and edge routing have zero to do with each other. They are orthogonal. You can "pref" your transit provider's customer routes from just within POP itself. This has zero to do with whether or not you have a "backbone." The way this works is that you pick your top N destination ASNs and continuously monitor ping time, jitter packet loss etc. Then you prefer the more desirable routes via iBGP. This is not new and not innovative either. Internap has been this 20 for years now. This was the whole "secret sauce" of their PNAP architecture back in the day. In fact they even sold a hardware device to do it called the Internap FCP(flow control platform)it has since been named MIRO [1][2]. Of course many people do exactly this themselves with BIRD/Quagga/Exabgp and some Python. And you certainly don't need "machine learning" to do this.
As a CDN customer I only care about getting the best path from the POP back to the eyeball networks. Why would it matter whether the CDN provider has fiber between two of their POPs? There should be enough transit diversity at each edge location that fiber between POPs is irrelevant. I should spend as little time on the CDN provider's network as possible i.e "hot potato" routing. This is the whole value proposition of a CDN. Honestly this whole sounds like a cost savings mechanism that they've trying to spin and sell as a product innovation.
Also where is the Cloudflare backbone map? It seems to be conspicuously absent for announcement about a "global private backbone." How many route miles are there? Is it an actual backbone or this just some CWDM gear that only connects POPs in the same city over municipal fiber?
[1] https://www.inap.com/press-release/internap-rebuilds-patente...
[2] https://www.businesswire.com/news/home/20080206005468/en/Tel...