One interesting implementation detail is that this product is itself implemented as a worker. We're finding that the Workers platform is the easiest way to build first-party Cloudflare features.
This is really cool.
Webapps are basically programs that are executed across the network; having fine-grained control over what happens at each layer (client, client service worker, edge nodes, proxy server, origin server) means more choices and complexity, but also so much more power.
The question I would ask is: once you have the ability to run code and store data on the network itself (which Cloudflare effectively is), why do you need an origin at all?
I could be understanding the crux of your question wrong but I believe the point the article and technology makes over and over is that it offloads nearly all of the stress and communication off of the backend and instead replaces it with a Worker KV which converts into becoming one static call and then everything is handled within the Worker. The applications seemed vast and only limited by creativity, as well.
You work for Cloudflare so that's a fun question but realistically there are still plenty of limitations. The script size, number of routes, KV consistency, write limits, and simple get/put API, etc.
It can work for simple apps but there's a long way to go before any serious enterprise system will be hosted entirely on a FaaS system like Workers.
If you’re having trouble with script size and route limitations, would love to chat. We’re working on providing more flexibility in that realm. rita at cloudflare.
It's moving that shopping cart data from being stored at a single origin, to being stored in the network all around the world. The advantage of that in that use-case is you can render your site just as quickly as if it was a static website, but it can contain the customer's personal shopping cart data.
The shopping cart is still stored somewhere, but when the user has cleared their cookies, etc., what information do you have that will let you it's their shopping cart so you can find it again?
There's a (K,V) pair somewhere, but in order to get V, you need K, which you've lost.
It would work just as most shopping carts do I imagine. You could store a session id in a cookie, and then use the data in KV to map that session id to a user account (if they're logged in) and their cart. So you would have a namespace full of sessions, and a namespace full of carts.
You could also store carts by session id until they're logged in, and then store the carts by account id when they are.
But the person you responded to asked what happens after "I clear my local storage/cookies". So you don't have access to the session id. It was in a cookie that is now gone.
If the user signs in, move their shopping cart data from a key derived from their session ID to a key derived from their account ID. When a user is signed in, data is read from the latter.
The feature aside, this is a really great blog post: learning a bit about the history of the feature, even if only tangentially related (Babbage and the Analytical Engine) in the intro is great.
Last write wins. There are several features for controlling consistency that we have been prototyping. Based on how storage is used we will enable those as the product matures.
So why not strong eventual consistency and CRDTs? As they fit perfectly for such functions running on edge nodes and no silly limitations of 1 write per second necessary.
Indeed. This is just stage 1 of our storage plans. We have some really cool stuff in the works but we wanted to get basic KV functionality out there for people to start using ASAP.
It was pretty trivial. I just stored each page into a value and served them out of there, with some stupid code to infer content-type from file extension. I meant it as a dumb demo because I wasn't feeling creative. I didn't know John would mention it in his blog. :)
Currently this isn't a great way to serve content because the values are limited to 64k, you have to infer content-type (which makes your URLs ugly), etc. So I don't recommend it. We'll come up with something better in the future. :)
So, with a global consistency window of 10s, and 1 write per second, would consistency ever be reached if a key is written to every second? What value would be visible?
One great example of why this would matter to me - could workers ever institute a coordinated rate limiting function using this KV store?
A PN-Counter is a method of coordination. And if the writes were coordinated with the consistency time, that would be sufficient to implement consistent counters (or even a naive counter in a store given atomic operatons).
PN-Counter is coordination-free counter, it can't be used as a method of coordination. But it's ok for global rate limiting, as it doesn't have to guarantee exact rate at any point in time, as long as it eventually averages on specified rate.
One thing to remember is you have at least two other ways of storing values. Each worker has global memory which persists between invocations (local, atomic), and you have access to the Cache API (per-data center, ephemeral).
By combining those with KV to do global coordination on a more course time scale, I think you could.
I had to upvote you just because I plan on getting some mileage out the phrase "YOLO last write wins." It has just the right amount of snark to remind junior developers to not trivialize or discount the value of consistency.
What I mean is: can a process block until a particular key appears, then resume execution after receiving the associated value? That's the main differentiating feature of a tuple space, and it wasn't clear to me whether this system supports that or not (or even has a concept of a process).
No, that's not supported at present, but that's an interesting idea.
Workers don't have the concept of processes. The unit of compute is an individual event -- e.g. handling an individual HTTP request. So we'd have to augment it a bit, but we can probably come up with something useful.
Yes, you'll be able to use etags with if-match for optimistic concurrency. When reading and writing from a Worker, if you do a get() followed by a put() on the same key in the context of the same request, we'll take care of the if-match automatically under the hood, so the put() fails if there was a concurrent write.
(This logic is not quite in production yet but should be within a week or two.)
Are there any plans to provide browser detection in the worker API? I imagine it's possible to implement as a user now, but may be clunky to provide a dataset to work from. This seems like a killer feature to have on edge workers to allow serving optimized builds for based on browser feature support without paying the overhead of client side detection.
It's not finding a parser that I find problematic, it's the dataset needed. My understanding is that the workers are limited to 128MB of ram and that they may be 'restarted' at any time. Datasets for accurate browser detection would be really tight in fitting into that amount of RAM when combined with whatever else you may be doing and you have to deal with load / parse time whenever a worker is restarted. This seems like a common enough need that cloudflare could provide it as an API and their implementation wouldn't have such issues.
Using storage along with worker code to perform the function should make this possible.
Our hope is that anyone could create such an API, not just Cloudflare. Just like we are building storage on top of workers, we hope to eliminate the distinction between what is possible for a Cloudflare employee and what is possible for any of our customers.
> ... understanding is that the workers are limited to 128MB of ram and that they may be 'restarted' at any time. Datasets for accurate browser detection would be really tight in fitting into that amount of RA
I think you might be overestimating that. Bundling the aforementioned https://github.com/faisalman/ua-parser-js wouldn't be significant - maybe hundreds of KB as a script (distinct from memory!) - at most.
The only browser detection possible on the server-side is with the user-agent and other headers, which you already have access to within Workers. You can parse that string reliably for modern browsers, and have a fallback for the rest.
@jgrahamc could you link to the technical post near the top, and maybe copy in that table 'Limits and Pricing' on this post? Because it would be really helpful to have that info in the general post, otherwise it's not clear what the details are (like write rate). Great posts BTW!
well, for one example off the top of my head would be using geohashes for proximity in records... if you could glob every geohash with the same first 6-8 characters, you could do range queries pretty quickly.
I've considered similar for shard keys in terms of proximity related information.
Great job - was trying to something similar by setting up three etcd raft nodes on each continent, but this is like 150+ nodes. Mind sharing of you’re using good clocks, paxos or raft to coordinate?
Would also be great if CF could do websocket / sse fanout at the edges - if I have a couple million websockets connected on ws.cf.com/key1 and I update the value, would be great if the new value could be broadcast. That would help with a lot of media / real-time / sports games, apps and websites.
The simplest example is liveblogging - someone is sitting at the next iPhone launch, and sending out photos and commentary that's being updated as they send on a large number of readers' screens worldwide. The Verge has a good system that they do with polling S3 for new data, but there's really much more effective ways to do it.
The more specific cases include opening ticket sales for movies in India - for some big movies tickets are sold out really quickly, so there'll be hundreds or thousands of people with browsers / phones open on a seat layout and they all need to see which seats are being sold in real-time.
Then there's broadcasting stock market movements and tips to thousands of traders in real-time.
The only existing solutions that seem really scalable are fanout.io - others like pusher.com and pubnub.com have pricing that's not conducive to serving a large number of simultaneous users / broadcasting. Fanout.io gets this right, but would be nice to have cheaper alternatives / make it a commodity.
Given that the CF nodes do support websockets, I assume they terminate and re-create new connections to origins as well - so the nodes are capable of holding a large number of websockets open, right? Can Workers intercept websockets / stream data into them? And given that the KV store is capable of propagating new values to all the nodes, there could be a way expose incoming change notifications on keys, which would handle this use case. Could also implement as continuous addition of new timestamps keys, prefixed with a topic.
Fundamentally what I think we want to do is build the primitives you would need to create your own fanout. That means we need the ability to push events to Worker nodes, and for Workers to be able to terminate WebSockets for you.
If you can accept up to 10s of latency, I agree that KV could solve the 'pushing events' problem for you. (A tuple store is semantically equivalent to message passing, and it will get easier when we support range queries).
Yes, both points make sense. And I assume the 10s latency is also geographically related - i.e. it's worst case 10s for the longest distance on the network topology graph? That should be fine.
Low latency requirements also tend to be geographical, I think - stock markets are mostly specific to a single country, examples like ticketing are usually specific to a single city / edge. For general news being broadcast globally, that kind of latency is fine.
> Fanout.io gets this right, but would be nice to have cheaper alternatives / make it a commodity.
Hi! Fanout founder here. I’m glad you like our way of doing things. If you need better pricing for high volume we are always willing to discuss.
I’m not sure you’d want to maintain a WebSocket inside of a CF worker. At least I can’t see that helping with cost. Execution time limits would get you too. IMO connection management belongs as a separate layer in front of the worker, so workers are only woken when things happen.
And on that note, of course you can use Fanout together with Cloudflare Workers today. At the moment this requires making raw API calls but we plan to update our server libs for CF compatibility soon.
> Hi! Fanout founder here. I’m glad you like our way of doing things. If you need better pricing for high volume we are always willing to discuss.
Hi! Wasn't expecting you to show up like this. Re. pricing for high volume, I'm sure enterprises and publishers could afford it, I'm trying to see if it's possible to offer free or very cheap service to individuals with large audiences. So not so much that I want a high volume discount, I'm trying to figure out a way to make it one or two orders of magnitude cheaper, enough that pricing is no longer relevant. Self hosting Pushpin is an option, but CF might be easier to work with.
> I’m not sure you’d want to maintain a WebSocket inside of a CF worker. At least I can’t see that helping with cost. Execution time limits would get you too.
The CF edges already hold / passthrough websockets - whether the Workers API is conducive to controlling them is the question, but we know it already allows streaming - doesn't seem like big jump from there. And for my purposes one-way streaming is enough, so might not even need to change the API. And based on the current billing model it seems like only CPU time is billed (?) - so if the socket is waiting most of the time it's still cost effective. And connection management itself would still be outside the scope of the worker - I'm saying the worker's event loop should behave exactly the way it would if I using the streaming feature right now to make a 1000 slow network requests from the worker and streamed the responses out as the worker received them. Only the protocol would change.
I actually have an open PR to add WebSocket endpoint support to CF Workers (so a Worker can act as a WebSocket client or server, instead of just a proxy). The main hangup though is that we need to figure out how to charge for long-lived connections. We obviously don't want to apply the normal time limits in this case. Hopefully we'll find time to work through this soon. :)
Nice! My 2 cents would be to charge a cheap rate for idling time and the regular rate for CPU execution. Or maybe use bandwidth as a proxy for CPU usage and an overall connection time charge.
Oh interesting, I didn't realize CF Workers metered by actual CPU usage and not just clock time. That would certainly make a difference in the expected cost.
Even still, handling millions of simultaneous connections, with each connection in its own worker in a multi-tenant execution environment in order to save money sounds crazy. :) The operating cost (and thus the price) of a purpose-built system like ours should be lower than that of a generalized system. I think your easiest path is to just tell us your preferred pricing model.
Depends on how you look at it. The way I see it I would be quite reasonable to start a few droplets / instances in every region Digital Ocean / AWS support and have websockets terminate there. I'd could even assume this is what Fanout/comparable service is currently doing.
From the cloudflare side I'd assume that each edge has 1 > n <= 10 of the CF version of this https://www.top-ix.org/wp-content/uploads/2016/10/Open-Conne... . Imagine CF had 100s to 1000s of these at every edge - they could/would then offer either DO sized droplets or AWS Fargate sized containers at every edge with CF load balancing, and we could be installing Pushpin on every edge. And it wouldn't sound odd. This seems like it's just a logistics problem - it's not inconceivable that 2020 / 2023 birthday week will announce this. (@CF you should totally call each droplet/instance a 'Flare')
In the meantime the Workers seems like a way to offer highly controlled computing on these edges - but other than the tighter control of the executing environment I see no difference from the scenario mentioned above. So if all the load balancing, connection handling and deployment pieces are already in place, I think we can and should try to use it for any possible use case that we would have used regular servers for.
To me there's a big difference between a compute host (VM/containers) and a function host in this context.
Running Pushpin on a bunch of compute instances means being able to operate at maximum performance with minimal impact between tenants. Compared to FaaS, I believe the difference in resource usage should be so significant that any provider of a shared-Pushpin service would be able to win on price.
I'm also still not quite sure how a FaaS-based push system would actually work since there's the problem of getting data around among the workers. You don't want a million workers polling a DB. I suppose you could integrate a messaging/queuing system with your workers, and each worker keeps a long-lived connection with the queuing system. But if you do this, you'll have barely built a push system at all. Instead you'll be dependent upon a millions-connection-capable queuing system, and your workers will simply be doing data conversion between two connections. And of course the queuing system would live outside of the FaaS, provided by someone for a cost.
It's worth realizing the multi-tenant nature of Workers happens within a single process, meaning from a cost perspective it's essentially indistinguishable from running your own hardware. It's actually one _less_ VM than if you were managing your own EC2 instances.
This is very cool, but how do you limit an end user to only being able to read a given set of values. It seems to me that this scheme puts all credentials for access in the browser, and they'll be able to just run roughshod all over your data without constraints in place.
Can you create data spaces/keys for ones that are read-write per-user and others that are read-only for a given user?
You write the Worker code which runs on our infrastructure, not in a browser. You have complete control over who can access which piece of data, as it's your code doing the accessing.
Thanks, I misunderstood that part... very cool indeed. What are the overhead limits for workers?
Aside, it seems like this could be really interesting combined to make an entire API with static delivery from S3 or Azure Blobs, with a backend on a cloud hosted database. With a lot of flexibility in between.
I mean compute/memory for the workers... just a rough comparison of what that might look like. Will probably sign up for the early access and play around with it soon.
With this and the changes to their worker caching API it's now completely possible to write a custom pull origin CDN service that you have complete control over.
I wonder how the Fastly crew feels about all this. Great team over there, and they were originally the super configurable CDN, but it seems that is now completely beaten by Workers.
I'm curious but I haven't seen anything about cache expiration on keys. Eg Expire all keys 'key-1' after 30 minutes or expire at a globally recognized time frame.
Does anyone know if this is possible?
A bit offtopic: could you guys move the "please enable javascript"/captcha page to a separate, but cloudflare owned domain, so at least people who don't want to enable javascript for random websites behind cloudflare could enable it for your domain and pass all your browser checks?
102 comments
[ 3.1 ms ] story [ 143 ms ] threadIf you’re truly replicating to every PoP that’s quite a fan-out and I can see why you’re limited to 1 write per second per key!
You're right for some use cases though.
It can work for simple apps but there's a long way to go before any serious enterprise system will be hosted entirely on a FaaS system like Workers.
At the point that CF replaces the origin, it is the origin.
The confusion in your query is the idea that "origin" is a rigid, fixed concept rather than flexible.
There's a (K,V) pair somewhere, but in order to get V, you need K, which you've lost.
You could also store carts by session id until they're logged in, and then store the carts by account id when they are.
But the person you responded to asked what happens after "I clear my local storage/cookies". So you don't have access to the session id. It was in a cookie that is now gone.
> There's no truth in the rumour that a KV store is a @KentonVarda store. Nor that it maps kentons to vardas.
Currently this isn't a great way to serve content because the values are limited to 64k, you have to infer content-type (which makes your URLs ugly), etc. So I don't recommend it. We'll come up with something better in the future. :)
One great example of why this would matter to me - could workers ever institute a coordinated rate limiting function using this KV store?
By combining those with KV to do global coordination on a more course time scale, I think you could.
Seriously, if you're new to that domain go read:
https://aphyr.com/posts/333-serializability-linearizability-...
https://aphyr.com/posts/313-strong-consistency-models
Workers don't have the concept of processes. The unit of compute is an individual event -- e.g. handling an individual HTTP request. So we'd have to augment it a bit, but we can probably come up with something useful.
https://en.wikipedia.org/wiki/Compare-and-swap
It might be straightforward to do this with an ETag like Firebase:
https://firebase.googleblog.com/2017/07/introducing-conditio...
Not sure what this would require on their backend though.
(This logic is not quite in production yet but should be within a week or two.)
Our hope is that anyone could create such an API, not just Cloudflare. Just like we are building storage on top of workers, we hope to eliminate the distinction between what is possible for a Cloudflare employee and what is possible for any of our customers.
I think you might be overestimating that. Bundling the aforementioned https://github.com/faisalman/ua-parser-js wouldn't be significant - maybe hundreds of KB as a script (distinct from memory!) - at most.
No big dataset needed, just some regex and if/then statements. If you still want a library then you can try https://github.com/faisalman/ua-parser-js
$5/month for
After that it's $0.50/month I think that's well within startup money. I mean, if a company has to give out credits perhaps it's just too expensive :-)Like, "show me all keys that match xyz∗ in this namespace"
I've considered similar for shard keys in terms of proximity related information.
Would also be great if CF could do websocket / sse fanout at the edges - if I have a couple million websockets connected on ws.cf.com/key1 and I update the value, would be great if the new value could be broadcast. That would help with a lot of media / real-time / sports games, apps and websites.
The more specific cases include opening ticket sales for movies in India - for some big movies tickets are sold out really quickly, so there'll be hundreds or thousands of people with browsers / phones open on a seat layout and they all need to see which seats are being sold in real-time.
Then there's broadcasting stock market movements and tips to thousands of traders in real-time.
The only existing solutions that seem really scalable are fanout.io - others like pusher.com and pubnub.com have pricing that's not conducive to serving a large number of simultaneous users / broadcasting. Fanout.io gets this right, but would be nice to have cheaper alternatives / make it a commodity.
Given that the CF nodes do support websockets, I assume they terminate and re-create new connections to origins as well - so the nodes are capable of holding a large number of websockets open, right? Can Workers intercept websockets / stream data into them? And given that the KV store is capable of propagating new values to all the nodes, there could be a way expose incoming change notifications on keys, which would handle this use case. Could also implement as continuous addition of new timestamps keys, prefixed with a topic.
If you can accept up to 10s of latency, I agree that KV could solve the 'pushing events' problem for you. (A tuple store is semantically equivalent to message passing, and it will get easier when we support range queries).
Low latency requirements also tend to be geographical, I think - stock markets are mostly specific to a single country, examples like ticketing are usually specific to a single city / edge. For general news being broadcast globally, that kind of latency is fine.
Hi! Fanout founder here. I’m glad you like our way of doing things. If you need better pricing for high volume we are always willing to discuss.
I’m not sure you’d want to maintain a WebSocket inside of a CF worker. At least I can’t see that helping with cost. Execution time limits would get you too. IMO connection management belongs as a separate layer in front of the worker, so workers are only woken when things happen.
And on that note, of course you can use Fanout together with Cloudflare Workers today. At the moment this requires making raw API calls but we plan to update our server libs for CF compatibility soon.
Hi! Wasn't expecting you to show up like this. Re. pricing for high volume, I'm sure enterprises and publishers could afford it, I'm trying to see if it's possible to offer free or very cheap service to individuals with large audiences. So not so much that I want a high volume discount, I'm trying to figure out a way to make it one or two orders of magnitude cheaper, enough that pricing is no longer relevant. Self hosting Pushpin is an option, but CF might be easier to work with.
> I’m not sure you’d want to maintain a WebSocket inside of a CF worker. At least I can’t see that helping with cost. Execution time limits would get you too.
The CF edges already hold / passthrough websockets - whether the Workers API is conducive to controlling them is the question, but we know it already allows streaming - doesn't seem like big jump from there. And for my purposes one-way streaming is enough, so might not even need to change the API. And based on the current billing model it seems like only CPU time is billed (?) - so if the socket is waiting most of the time it's still cost effective. And connection management itself would still be outside the scope of the worker - I'm saying the worker's event loop should behave exactly the way it would if I using the streaming feature right now to make a 1000 slow network requests from the worker and streamed the responses out as the worker received them. Only the protocol would change.
Even still, handling millions of simultaneous connections, with each connection in its own worker in a multi-tenant execution environment in order to save money sounds crazy. :) The operating cost (and thus the price) of a purpose-built system like ours should be lower than that of a generalized system. I think your easiest path is to just tell us your preferred pricing model.
From the cloudflare side I'd assume that each edge has 1 > n <= 10 of the CF version of this https://www.top-ix.org/wp-content/uploads/2016/10/Open-Conne... . Imagine CF had 100s to 1000s of these at every edge - they could/would then offer either DO sized droplets or AWS Fargate sized containers at every edge with CF load balancing, and we could be installing Pushpin on every edge. And it wouldn't sound odd. This seems like it's just a logistics problem - it's not inconceivable that 2020 / 2023 birthday week will announce this. (@CF you should totally call each droplet/instance a 'Flare')
In the meantime the Workers seems like a way to offer highly controlled computing on these edges - but other than the tighter control of the executing environment I see no difference from the scenario mentioned above. So if all the load balancing, connection handling and deployment pieces are already in place, I think we can and should try to use it for any possible use case that we would have used regular servers for.
Running Pushpin on a bunch of compute instances means being able to operate at maximum performance with minimal impact between tenants. Compared to FaaS, I believe the difference in resource usage should be so significant that any provider of a shared-Pushpin service would be able to win on price.
I'm also still not quite sure how a FaaS-based push system would actually work since there's the problem of getting data around among the workers. You don't want a million workers polling a DB. I suppose you could integrate a messaging/queuing system with your workers, and each worker keeps a long-lived connection with the queuing system. But if you do this, you'll have barely built a push system at all. Instead you'll be dependent upon a millions-connection-capable queuing system, and your workers will simply be doing data conversion between two connections. And of course the queuing system would live outside of the FaaS, provided by someone for a cost.
Can you create data spaces/keys for ones that are read-write per-user and others that are read-only for a given user?
Aside, it seems like this could be really interesting combined to make an entire API with static delivery from S3 or Azure Blobs, with a backend on a cloud hosted database. With a lot of flexibility in between.
When you say overhead limit, are you talking about latency? Our goal is to keep reads on the order of 5ms in the 90th percentile.