Really interesting. I've seen so much disagregated database work, and so so so much of that exposes postgres interfaces. But all the good stuff has been closed source!
I'm very very excited to hear about a team taking this effort to postgres itself, in an open source fashion! From the Architecture[1] section of the README:
> A Neon installation consists of compute nodes and Neon storage engine.
> Compute nodes are stateless PostgreSQL nodes, backed by Neon storage engine.
> *Neon storage engine consists of two major components: A) Pageserver. Scalable storage backend for compute nodes. B) WAL service. The service that receives WAL from compute node and ensures that it is stored durably.
Sounds like a very reasonable disaggregation strategy. Really hope to hear about this wonderful effort for many more years. Ticks the boxes: open-source with a great service offering: nice. Rust: nice.
We are committed to building a durable company and we are well funded. So yes, you will hear from us for years to come as we will be shipping more and more features.
Classic Serverless: The database engine runs within the same process, thread, and address space as the application. There is no message passing or network activity.
Neo-Serverless: The database engine runs in a separate namespace from the application, probably on a separate machine, but the database is provided as a turn-key service by the hosting provider, requires no management or administration by the application owners, and is so easy to use that the developers can think of the database as being serverless even if it really does use a server under the covers.
It sounds like this is neo-serverless but serverless nonetheless.
It’s a tale so old that “server” lost meaning before we could even be in a state of “serverless”ness. First there was a cloud, and then there was no clarity of what conditions a process even runs under. And it wasn’t good, but it became stable. And on the next day, there were databases from on high, as if they’d spontaneously burned a bush.
It has never, ever meant that the software wasn't running on a server. From TFA:
"The phrase 'serverless' doesn’t mean servers are no longer involved. It simply means that developers no longer have to think that much about them. Computing resources get used as services without having to manage around physical capacities or limits."
Marketing doesn't care about your personal preferences :)
And Marketing is King, for bottom-up SaaS businesses (businesses selling to individual developers), which most companies open-sourcing their core projects most often are.
The other key feature of "serverless" is that the store-of-record for a "serverless" system isn't a part of the running system — i.e. the system's state doesn't live on any particular server / cluster that is run as part of the operations of the system.
Instead, a "serverless" system's store-of-record is some external and semantically-abstracted storage system — e.g. "a remote git repository" (GitOps); "an object-storage bucket" (Snowflake); "a document store" (Lambda); "a blockchain"; etc. Where all that's important about this storage system is its API, such that the design is portable to any backing storage service that supports the same API.
This means that, in serverless systems, the mutable state in the cluster is just an ephemeral cache representation of the externally-managed data-at-rest.
Another way to think about this is by thinking of "a server" as a thing with durable state that you have to worry about — e.g. protect from disk corruption, make backups of, etc; vs. something like "an ephemeral immutable-infrastructure container workload" that can die and be recreated with no problems. Serverless systems are systems without any "servers" in this sense — nothing to back up; nothing to disaster-recover; etc. Nobody operating these systems ever needs to think about individual servers. Nobody ever needs to SSH into a server, upgrade a server, restart a server, etc. The operations for such systems can be handled entirely at the "cluster" and "ephemeral workload" levels. Nodes within the cluster that "go bad" can simply be drained and deleted — this may even be automated.
And further, because of this lack of local durable state, there's no need to worry about "allocating" that state, and thereby allocating customers to particular clusters. Serverless compute clusters are usually just one huge cluster (per region), where customers' individual request workloads just get scheduled onto that cluster wherever they'll fit.
Of course, the external store-of-record for a serverless system must have "servers" — the data-at-rest ultimately has to reside durably on some disk somewhere. But 1. they're not your servers, and their ops problems are not your ops problems; and 2. because they're a much lower-level abstraction, they can scaled much more robustly, and so can have far fewer operational problems in the first place; and 3. because they're a much lower-level abstraction, they benefit from economies-of-scale in shared tenancy in ways domain-specific compute/DBMS/etc. clusters usually don't; and so your system can benefit from a storage layer that's hyperscaled + hyper-robust from serving millions of tenants' low-level needs.
Also, simpler answer than I already gave: “elastic” and “scalable” X don’t imply a lack of committed resources, but rather just a control plane that auto-scales your commitment to ensure that you’ve always got more than you need.
E.g. AWS Elastic File Store: an auto-resizing NAS-like abstraction, but at any given point, a committed size where you’re paying for unused space.
Vs. Amazon S3 (which might as well stand for “serverless storage service”) — where you only pay for what you use, with no committed capacity beyond current usage.
Modern marketing makes me mad sometimes. They call something "serverless" while in fact it runs on their servers somewhere in the world, where you have even less control than on your cloud. Though, I believe it was popularized by Amazon, which doesn't care much about being honest in its expansionism.
* we already have compute scale-to-zero (cloudrun, lambda, fly.io).
* Network is default pay for use. Storage (S3) is default pay for use.
* The only piece in the stack that was always-on was the database (only serverless db thus far was firestore, or something like sqlite+litestream)
With something like this we get a solid RDBMS engineered to be scale-to-zero, and with good developer experience.
This opens up a world of try-out mini applications that cost cents to host. serverless db (postgres) + serverless compute (cloud-run) + use as you go storage+network. This is a paradigm-shift stack. Exciting days ahead.
> It automatically starts up, shuts down, and scales capacity up or down based on your application's needs.
The only tradeoff is the additional latency someone will have when connecting to the db after it has shutdown and waiting for it to spin back up and become ready.
> You pay only for the capacity your application consumes.
> Scales down to 0.5
But it actually can't scale down to 0.5 or the DB falls over just existing.. auto scaling won't let you go down that low unless you set 0.5 as the max, which literally makes it not scale up, and it's dead, because the DB can't run with that little CPU.
So it's fair to ask if neon can scale to 0, both in marketing, and in practice.
We do scale compute part down to zero after 5 mins of inactivity now (no active transactions). This 5 mins threshold is a random pick, it could be 1 min or 30 mins later, or even customizable by the end-user. Storage part is heavily multi-tenant, so it's always running and our main objective is to make resource utilization as effective as possible.
It still has a significant latency on the first connection attempt after suspend (1-2 seconds), but we are working on that and it seems to be realistic to put the startup time under 1 sec.
Pricing model is still work-in-progress, so cannot say much about it. Yet, my personal intention is to make it cost-effective for both end-user and us. I'd prefer to don't build a service with claims like 'here is your free-tier serverless Postgres with zero-latency on connect', which actually means that under the hood there is an always-running compute burning the investors money. Hope it's realistic to achieve :)
That's where the separation of storage and compute kicks in, I guess. Startup process of our Postgres instance (compute node) is a bit different from vanilla Postgres. We need to go to the network storage service (pageserver and safekeepers) to get the last known commit LSN, but we don't need to perform any sort of recovery on the compute node side. That way, compute is mostly stateless.
Basically, to start we need to know this LSN and to bootstrap the Postgres processes. This is really that quick. After that compute is ready to accept connections and serve requests, as it's able to get any missing pages from pageserver with GetPage@LSN request.
We do have the whole bunch of problems to solve: queries latency after cold start; startup after the unexpected exit of the heavily loaded Postgres instance could be slower; etc.
Some parts of the PostgreSQL start-up sequence take a long time:
- Initializing shared memory -> We, for now, have only small instances, so that doesn't hit us as hard
- Reading data directories -> We don't have to do that at all
- Replaying WAL from a previous unclean shutdown -> We don't need to do that, PageServer is responsible for that
- When initializing a whole new database: Initializing the data directory -> We have a copy that each instance gets initialized from, which makes the process "copy those ~16MB in the background", which saves us from having to do the costly initialization process.
And there's several more infrastructural optimizations, such as pre-loading the docker images onto the hosts.
That's interesting to hear. That probably works great for my use cases, which is typically wake up to refresh a CDN for guests, but ready to work for a bit if a content creator logs in (e.g. a WordPress instance without comments or non-author logins).
Looking forward to seeing how this works out. I have no issues paying for services, I just hate that the minimum entry level cost is $20... I can't imagine why, at scale, it can't be more affordable for hobby/fun level projects.
Short answer if you don't want to read my post: it constantly uses CPU, it's always on. After creation, waiting 2 days, never logged into it, never ran a script against it, never gave it access to any networks, minimum cost is $43/month because it can't actually scale down to 0.5 units unless you CAP it at 0.5, which makes it unusable, because it consumes all of that capacity just to exist.
It sounds like this Neon offering is exactly what I hoped AWS was offering... Or they are using Language to suggest it and mislead the customer just the same ... If it's the former, if probably sign up and try it out. If it's the latter, I'll probably never touch it for the false hope.
I tried recreating your experiment.. created an aurora Serverless v2 db with 0.5 - 2 ACUs in us-east-1.. since you said it was for Wordpress, I disabled the multi-AZ replication, since AFAIK, WP can't use separate reader/writer connections (mentioning this because you didn't say anything about it in your article)... then I let it sit overnight so it had time to create everything.
It's sitting at 23% of 0.5 ACU.
So its either the replication setting (haven't tested with it on yet), or.. AWS is a shared service... wonder if it's similar to EC2, in that sometimes you get an instance on a machine that's more overloaded and the instance doesn't perform as well.. and you have to destroy it and try again. Might want to try it again.
Edit: I don't think its the replication setting... tried that with a new db and its at 25% on each replica after an hour.
There are plenty of serverless database options already: Firestore, DynamoDB, CosmosDB, FaunaDB, even MongoDB, and there are "newsql" distributed relational systems like CockroachDB and Planetscale with serverless plans.
I've met my share of developers without perspective in time, and hey I was probably one of them for my first 5-6 years as a developer, that I don't think one needs to assume corporate toadying for hyperbolic importance for mankind to be ascribed to a corporate offering.
Can you please stop posting these off-topic comments (which are against the site guidelines, as you'll see if you review https://news.ycombinator.com/newsguidelines.html)? Also, can you please stop posting unsubstantive comments generally? You've been doing that in other threads as well.
I appreciate your concern for the integrity of HN discussions, but the thing to do if you suspect abuse or manipulation is to email hn@ycombinator.com so we can look into it. Posting about it in the threads themselves is explicitly against the rules.
> Are you being serious? That's very hyperbolic if so.
Yes Im serious. This is one of the most foundational & key levels of computing: storing & quering data. Without this, computing isnt good for much.
Getting good at this is a huge task for humanity. Right now that task is almost entirely being fulfilled by far off hyperscalers. Aurora, BigTable, Firebase, DynamoDB, CosmoDB, more special works like DataDog Husky... the world is running off vast super-awesome dataengines. But ones that are not ours, that we cant hack on, that we cant run ourselves. It might as well be the Martian's (little green men's) databases as far as I'm concerned: these are not humanity's heritdge & humanity is cut off from active participation with them.
Right now there are so so so few scalable data systems available for the world, that we have. This seems like a great & novel effort to radically open up the range of human capabilities, in one of the most important sectors of computing: handling data.
We have lots of other cloudware for humankind but data has seemingly been much slower & un-scaled. I agree with this @anilgulecha comment[1]:
There are tons of databases, I just named several and there are hundreds more for every possible niche.
> "But ones that are not ours, that we cant hack on, that we cant run ourselves."
Who is "we"? Is your entire issue that not everything is open-source?
> "these are not humanity's heritdge & humanity is cut off from active participation with them."
This is still incredibly exaggerated. Everything humanity does is humanity's heritage. There are plenty of open-source databases of all types and sizes if you look: https://db-engines.com/en/ranking
Your free to your opinion, but I respectfully do not share your values & sense of collective ownership over remote, far off software I can't see, shape, control or rework.
The most notable feature of Neon, that makes it so exceptionally smart, is that it takes the humanity's existing first best choice default-go-to database, by a country mile, and layers in really smart decoupling of storage to help it scale. Reinventing things from scratch can be a win, but that this is already a well known quantity, much loved & cherished & used by all, hacked by all, grown by all is a planet-sized huge plus mark.
I think you vastly overrate the broad availability of alternatives to this, of potential starting places others might explore. Most of your list is dominated by proprietary whatever. Even skipping that, yes, there is a ton of novelty & promise, the potential for breakthroughs. But this is starting with the best, and pushing it into the cloud-layer, into the troposphere.
> "I think you vastly overrate the broad availability of alternatives to this"
There are dozens of "scalable postgres" alternatives from Redshift to Citus to CockroachDB to Yugabyte. I find it strange that you say its overrating the availability when you're also comparing it to something that isn't even a full product yet.
There is also gigabyte as open source. In our tests (which means nothing in general as it’s specific to our business case), it outperforms cockroachdb.
(Some of these are options I've not looked deeper into - Fauna, Planetscale)
This sentiment is perhaps right, but I was careful about calling out scale-to-zero. We do have options that are zero cost (or pay as you use), but there's a fundamental difference in something that may be zero cost because a cloud provider is using it as a customer-acquisition ploy.
Options like litestream+sqlite+s3, or what Neon seems to be, are verifiable you-pay-for-when-db-is-booted up, else the verifiable cost is storage only.
So the trifecta that will be very productive for masses is 1) database where compute is scale-to-zero, 2)open source or commoditised, and 3) is RDBMS.
What is the effective difference to you? Technically all compute can enter hibernation by dumping RAM (or just using virtualized memory backed only by SSDs).
The effective difference is commoditization of the paradigm - there's confidence in a commodity technology that a proprietary cannot give (standardized use and wide support and community, multiple providers, self-hosting).
Eg: of innovation -> commodity. EC2 was innovative, but is today common-place. S3 was the same. DBs that scale-to-zero have not reached that state yet.
Thanks for the link on cockroachdb - it sounds promising. I wonder what's the minimum self-deployable unit of cockroachdb - will google around a bit.
If you host it yourself, or use their dedicated enterprise clusters then you still have instances which are individual (virtual) servers.
The serverless model has no instances exposed to you and instead is a multitenant architecture that uses a pool of shared compute nodes with a routing layer that intercepts and introspects your connection to load up your specific database context for query processing.
There is a big difference between in architecture between Neon and PlanetScale, CockroachDb, and Yugabyte. Neon is shared storage (storage is distributed but shared) and the others are shared nothing. Shared nothing systems are hard to build with supporting all the features of the base system. E.g. https://vitess.io/docs/13.0/reference/compatibility/mysql-co....
Neon is 100% compatible from Postgres b/c we didn't (or almost didn't) change the Postgres engine.
Hi Mani! For sure there are many serverless options - fewer that separated storage and compute and fewer that are open source end-to-end. Neon is also 100% compatible with Postgres (unlike CockroachDB) because compute is Postgres.
Our intention is to standardize the separation of storage and compute cloud architecture - that's why it's open source under the Apache 2.0 license.
For now we've focused on making the product production-ready on AWS, so that we can get users to try it out. Once the model has been proven we'll likely branch out to other clouds.
However, if you really can't wait to run Neon on Azure, you could contribute the integration yourself: the code is available under Apache 2 at https://github.com/neondatabase/neon/
Haven't done anything with Azure yet. Shouldn't be hard to add though, we don't rely on any special cloud storage features, just simple upload/download of files.
Many teams would prefer a postgresql compatible database with full sql support without compromises, missing features, etc. So, this could break that market open a little. Both AWS and Google are unreasonably expensive for this stuff. Most teams don't need a huge database and would be able to run postgresql on a tiny instance and get away with it. Have 2 of those and failover and backups and it's good enough for a lot of small shops.
Most managed/serverless options begin at hundreds of dollars per month. So, you get lots of companies either just handing over the cash or jumping through hoops to get something more reasonable. The latter is a stupid waste of time if you can afford the former. That's how Google and Amazon make money: they make the expensive option more tempting and the cheap option needlessly hard. They are not interesting in supporting frugal teams. The whole point is squeezing their customers hard.
So, this is potentially very nice if it offers some competition on the cost front. I'd certainly consider using this if it proves reliable. In fact, the whole reason I opted out of a relational database is the above. What I'd need is something that is reasonable in cost relative to the modest data I store and retrieve.
We have a single big bare-metal machine. We run Postgres with a ~1TB DB, moderate load, on a Hetzner AX101 (16C/128GB ram). It has 2* 3.84TB nvme drives (zfs mirror with hourly snapshots) used for postgres storage only, and a seperate pair of mirrored sata drives for system/boot (had to request the extra drive, ask support to change boot option in BIOS, and reinstall OS using rescue system). It's about 100 EUR/mo with unlimited data. We bounce all incoming requests from clients (the machine also runs a node backend) through a digital ocean machine (NGINX proxy), as their peering agreements are better, without this some users in Brazil, Turkey etc. have very slow access. OVH I think would be even better for this use (better? peering and IIRC cheaper data). ZFS snapshots are backed up with sanoid to a machine under my desk with spinning disks. AX101 can be fitted with up six 3.84TB drives, that's almost 12TB of mirrored storage, we should be good for a while. You can (should) use at least lz4 compression on zfs.. can consider zstd-1, bit slower, that could double the effective space. The compression also applies to the in-ram ZFS cache, that can be 100GB+.
We used firestore before.. got a bit tired of some of the limitations (latency, indexing). Cost-wise I don't think it's that different actually, but we aren't using much bandwidth, then self-hosted can be dramatically cheaper. Have to manage some details of course (zfs filesystem parameters, set up backups, config postgres etc.), but I found that stuff quite interesting and it's knowledge that will always be useful.
Yes, ZFS snapshots are atomic, so as long as the database use sane semantics to do disk I/O, you're fine (AFAIK, PostgreSQL and MySQL/InnoDB are ok, MySQL/MyISAM is not though).
A snapshot is effectively identical to a killed database server (due to OOM, or power outage), and database servers should be crash-resistant.
With PostgreSQL, checkpointing the database before taking the snapshot may make the crash recovery quicker when loading the snapshotted database.
I have a similar setup as OP, and I use ZFS snapshots to be able to quickly rebuild the whole machine on another host. Of course, it still requires manual intervention in an outage (it could be automated, theoretically), and I may lose up to 5 minutes of data due to my snapshot schedule.
Ah, nop. If it goes down, we'll figure it out.. 24hrs outage wouldn't be the end of the world. Would have been nice to have a second (read replica) machine in East Asia, to reduce latency for users there, but didn't find a provider like Hetzner.. maybe could take a server in a suitcase and install it in a collocation center there. Bit of a hassle though.
Either you have an existing db running and check the metrics, or you don't have a workload to move and you can just try it out from scratch. You can always migrate away if the billing model doesn't work for you.
Alternatively, it's likely that Planetscale will eventually provide a provisioned billing model, similar to DynamoDB. Then you can choose between scalable usage based pricing or less flexible flat rate pricing. Either way, innovating different billing models for RDBMS is a win.
> This opens up a world of try-out mini applications that cost cents to host
Given how much performance you can squeeze out of a $5/month VPS (I've been spinning them up and indeed down regularly over the last couple of years), is this really a paradigm shift?
> Millions of students and enthusiasts around the world would find that cost sufficiently friction-ful to not try out things
I appreciate there are indeed billions of people for whom $5 is a lot of money, but just how many of them are "students and enthusiasts" itching to get started with Postgres?
I realise that - perhaps particularly here - a $5/month VPS is a deeply unsexy thing. You can, however, achieve (and learn) an awful lot with one.
Nowadays there's a ton of idling compute capacity running client side that could be used to spin up a personal cloud environment with all core services if one has the proper knowledge. For anyone that has a core to spare and a few GB's of memory, which should be easy for most modrn midrange hardware, I would recommend they save that $5 paying for a third party VPS, go deploy an open source hypervisor such as KVM, and run the virtual server on your own hardware.
Free tier instances on various providers provides an option. And if you can't afford $5 a month you really shouldn't be playing with services where there's a risk of huge overages if you face a sudden spike in users.
You cannot sign up on AWS Free Tier without the credit card info and many students end up with a huge bill. This sub reddit.com/r/aws/ has many such posts.
Well, and that is just as much of an issue if you use most serverless offerings. But you're not prepared to carefully manage your use, cloud services should not even be on your radar.
Who absolutely would not want to give a credit card to AWS where a bill is dynamic. If $5/mo is bad they definitely can’t handle the screw up that scales up and runs overnight for $500.
On that $5/month VPS there's management overhead - you need to have at least basic Linux knowledge, and ideally more than that to know not to do stupid things like chmod 777 and database exposed on the public internet. You also need to do your updates, etc.
I'm an (former) SRE and run my own Kubernetes cluster for fun, but still use serverless (containers as a service, static website hosting) depending on the project.
With every cloud service there's a management overhead too. Just different skills you need to learn.
I've done SRE/devops work in various capacities including consulting longer than cloud services have existed, and my experience is that I've consistently earned more from clients who insisted on cloud services because they consistently need more help. Nothing is driving more demand for devops consulting services than cloud providers.
The issue with a full Linux system’s overhead is that if there are any new security vulnerabilities the situation could blow up in your face (e.g. the system is used to send spam, or host malware), so you need to maintain it at least minimally. With a serverless cloud architecture at worst it’ll stop working.
Or you just use Flatcar (a derivative of CoreOS), and don't worry about anything more than rebooting once a new image has been (auto-)installed, and run everything else in app containers where you have to worry about nothing more than what you would in your regular cloud setups.
This is not hard to get right. Yes, you need to learn how to do it, but the amount of money I've made from clients who thought cloud was simple and proceeded to create massive security holes for themselves is fairly substantial. People who think they're reducing their attack surface by using these services need to reevaluate - they're large, complex architectures that very few users understand properly. You need to learn the skills either way.
> On that $5/month VPS there's management overhead - you need to have at least basic Linux knowledge
Don't you need that as well as cloud-specific knowledge if you go serverless?
> and ideally more than that to know not to do stupid things like chmod 777 and database exposed on the public internet.
You still need some arcane knowledge to make sure your serverless doesn't experience cost overruns, right?
IME (your's obviously differs), the amount of cloud-specific + vendor-specific knowledge needed to avoid using a $5/m VM is a lot more in volume and a lot less in stability[1] than learning basic Linux once and using VMs everywhere[2].
[1] How the different cloud providers bill, when they bill, how to control your limits, etc changes much more often than knowing how to keep your server patched. Knowing how to get your serverless DB going on AWS doesn't help when you want to use Azure. And each cloud vendor regularly requires you to update your knowledge. Knowing how to keep a PostgreSQL-on-Linux up-to-date can be learned once and used for years. Even if running a managed DB, you'll still need to gain some of that knowledge anyway.
[2] Once you get to a scale where treating your machines like cattle rather than pets, you'll obviously have the team required to use cloud stuff optimally.
> You still need some arcane knowledge to make sure your serverless doesn't experience cost overruns, right?
I know people who have literally written their own analysis tooling just to figure out what's impacting their AWS spend. It's gotten better, but I could have retired many times over on what I've seen clients overpay to cloud providers because they didn't understand what would drive cost.
> [2] Once you get to a scale where treating your machines like cattle rather than pets, you'll obviously have the team required to use cloud stuff optimally.
The "problem" with the cloud story is that at that point you also have a team that could save you a fortune with a hybrid setup. Cloud providers get their margins off those who don't understand how much they're overpaying or who are too small to a) care or b) have leverage. Those big enough to have leverage who understand either negotiate hefty discounts or build out cheaper setups (basically at the point you're spending 7 figures a year, if you're paying anywhere near list prices for cloud services you're a chump; below that it's hit and miss)
I'm not at all against using cloud services, but I wish more people actually understood their costs and picked based on merits rather than cargo-culting. Some teams benefit greatly from cloud services, but usually if they're not cost sensitive. In my current job we have everything on AWS because we're never going to scale to somewhere where it'll get expensive and it's convenient. We'd save money if I moved it to, say, Hetzner, but the hosting bill is too small to matter. For that use it's fine.
The moment the bill starts to bite people ought to at least price out alternatives, and consider hybrid setups. E.g. I've had setups where even just putting a caching proxy in front of AWS to cache images to cut the egress bill would have paid for a team to keep it running. Their egress cost is still bad, but at the time it was just pure highway robbery.
What's the cold start time for something using sqlite+lightstream on scale-to-zero compute? I think you'd need to pull the db out of storage, so I would be slow to go from 0->1 instance. Anyone know if that's right?
I don't see the benefits over a 5$ VPS.
Even admitting I'll save a few cents over a VPS (which is absolutely not guaranteed), the cost saving is so minimal I won't bother rewriting everything under the serverless paradigm just for it. Of course if your cloud doesn't cost an eye and an arm. I can understand people excited to save on expensive Aws instances but maybe you should just consider dumping Aws.
Scale doesn't matter for mini applications and scaling vertically (=throw money for a bigger server) will work for 99% of the companies.
The 1% who need horizontal scaling will have custom everything regardless and will need to hire experts, not a good niche to release a product.
I do. There are real benefits to using a hosted server for large projects (think hundreds of gigabytes to terabytes of data) because getting sharding, fallbacks, and downtimeless maintenance right is difficult, risky, and expensive.
These are reasons why one might go for a hosted database at AWS/GCE/Azure. There are tons of good servers out there for small to medium projects and I don't think this service is right for those, unless they can make do with the free tier. The real benefit is in the larger cloud application space.
A system that does this type of scaling automatically while also reducing the dependency on a single cloud provider's service can be a gamechanger for some companies with huge database servers that risk getting locked in.
I think using this service for most existing applications will introduce a performance drop, a rise in expenses, and a complicated migration path, but on the other hand I think that developing against this system for new projects that are very likely to grow in scale will end up with some major data control benefits.
The open source nature also allows for competitors in markets like the EU to start serving databases that don't break privacy laws (although most companies don't care until they receive a fine).
I'm not sure how this company will become profitable while giving away its special sauce that can be modified to run at competing companies relatively easily, but that's a whole different story.
I like this perspective a lot & think it's absolutely key here.
We- the world- still pick single-node writer postgres & read replicas when we have to store & query data. There's great Kubernetes postgres operators, but it's still a distinctly pre-cloud pre-scale type of technology, & this decoupling & shared-storage sounds ultra promising, allows independent & radicaly scale up & scale down, sounds principally much more managable.
We already had serverless db for ages and it's called ... Google Sheets. You can even query it with simple SQL-like language.
The problem with most other "serverless" databases is that they don't offer HTTP API to query them from restricted environments like serverless functions.
Just a quick point in defense of SQLite: that last one is almost verbatim possible in SQLite, and it is possible to calculate ordinals, although the syntax is with standard SQL rather than a custom syntax. The SQLite docs mention that they never found a use case for jsonb that ended up being faster or more efficient than json, so they left it out, although they do reserve the BLOB data type for jsonb if such a use case is discovered.
a. PolarDB is based on a similar idea. https://www.cs.utah.edu/~lifeifei/papers/polardbserverless-s.... This paper describes it. The biggest difference that I see glancing through the paper is that we really integrated S3 into the storage. In Neon architecture branches, backups, checkpoint are all the same thing and instant to run. This simplifies a good amount of database management AND deliver on better costs. S3 is cheap.
b. Neon doesn't need a special filesystem. Neon storage is in a way a filesystem, however it doesn't expose filesystem API. It's a key value store - serves 8k pages to Postgres and a consensus - update API to the key value store. Pages are organized in LSM trees and background processes put layers of the LSM trees to S3.
c. The cols start is 2sec right now. There is a dependency on K8S. Bare metal implementation will require new code to orchestrate starts and stops.
The way you describe it, to me, is one of those “this sounds obvious in retrospect”. Sounds completely elegant and “right”. Congratulations on a great idea. I really hope you pull it off!
S3 has its limitations though, like too many small files and the get/delete/list ops get very expensive. There's also an upper-limit on throughput per S3-bucket partition. I guess, sstables that pageserver flushes periodically help work around these issues?
> Neon storage is in a way a filesystem, however it doesn't expose filesystem API.
Genuinely curious: When would anyone consider using filesystems like Amazon FSx for Lustre instead which is backed by S3 anyway over implementing a filesystem-esque abstraction of their own (like neon.tech does, and other solutions like rocketset.com, tiledb.com, xata.io, and quickwit.io do).
> Pages are organized in LSM trees and background processes put layers of the LSM trees to S3.
Curious how merges are handled? Also, are you using RocksDB / some other engine underneath?
> Bare metal implementation will require new code to orchestrate starts and stops.
Speaking of new code... SingleStore started as a very high-throughput OLTP database and eventually evolved to into a HTAP (?) database. Do you see Neon evolving in a similar manner, too?
1. Yes. Our first attempt at storage implementation had a problem with many small file. Then the team rearchitected it around LSM trees and it got a LOT better. Our benchmarks show that we are very close in performance with vanilla Postgres and Aurora. There are some "worst case" scenarios where Neon is worse than vanilla Postgres. Aurora has similar problems too.
2. It's best to custom build a storage system here. External distributed filesystems introduce complexity, cost, and bottlenecks that you don't control.
3. Purpose built. LSM trees also have a temporal dimension - LSN. You can fetch a page by pageId and LSN. This is what allows time machine and branching.
4. I call it convergence when OLTP and OLAP is one system - ultimate dream for a database systems engineer. Since I spent 10 years building it I have both scars and aspirations. I think it will come, but this will take a long time. HTAP is in a way a subset of convergence - most systems will have some HTAP. Neon will have some too, but for now it squarely focused on OLTP and helping developers build apps.
Right now, it should be possible to use anything that is compatible with the S3 API, as our current focus is on getting the product to the market. Once the business model is proven, we'll likely branch out to other clouds, with their storage providers.
If you can't wait that long to run Neon on your own cloud, feel free to contribute an integration to your persistent blob storage: the code is available under APLv2 here: https://github.com/neondatabase/neon/
> c) Heard the cold start is a second (IIRC), how does that value differ if one runs Neon on bare metal instead of k8s?
Yeah, as Nikita mentioned it's 2 seconds now. We did some tests and measurements and on bare metal, it's sub 500 ms usually, so the remaining part is the k8s (+ our own control plane) orchestration overhead. For example, with plain Docker (which we use in CI in addition to k8s) it's around 1 second already.
K8s provides a convenient abstraction layer, though. So I think that we'll continue using it and optimization will come with pods pool / over-provisioning and it'll be realistic to bring the startup time closer to bare-metal.
I'm guessing no, or at most just have set of extensions that you can use (kind of like it's done in AWS RDS or AWS Aurora).
The claim is "serverless" i.e. you don't have access to the server and if you could install any extension you essentially have full access to the server as there's no restriction what you could do in an extension. I don't think that would be allowed.
https://neon.tech/docs/cloud/compatibility/ says "During technical preview Neon has restrictions on user ability to install PostgreSQL extensions. Following PostgreSQL extensions come pre-installed: [..]"
Sure. So far, we just precompiled a few popular extensions, and they are available for installation. Ultimately we want to provide an option to bring your own extensions e.g., by specifying a docker image based on our base image. But that is some work on a security front: with a custom extension, you have access to the corresponding Unix user and can construct malicious WAL, send requests to the control plane, etc.
Thank you. PixelPoint (https://pixelpoint.io/) designed the landing page. Copy stuff may change we want to make the story clearer and highlight more features.
We haven't launched pricing yet. In fact our official launch is in a few weeks. This HN post "leaked". We will launch, allow people to use it for free and introduce pricing later this year.
Seems like this might implement database branching in the way most people would assume: branching both the data and schema? I remember being a bit disappointed to learn that PlanetScale's database "branching" was only for the schema [1], which is still quite useful, but this would be so much cooler!
I couldn't find much info about the replication models available/planned however. I would consider this to be table stakes at this point for a serverless database with the recent trend of pushing compute to the edge. This is much more interesting to me than scaling to 0, which is only really useful during the prototyping phase.
PlanetScale is single primary with eventually consistent read replicas, Fauna has strongly consistent global writes (or regional if you choose, but no option for replication between regions if you do) with a write latency cost, Dynamo/Cosmos are active-active eventual consistently replicated with fast writes globally. All useful in different scenarios, but I'd love to have one DB tech that can operate in all of these modes for different use cases within the same app, using the same programming model to interact with data across the board.
I think the decoupled storage engine here would open up some really interesting strategies around replication. What are the team's plans here?
1. Yes schema and data via "copy on write". This will let you instantly create test environments, backups, and run CI/CD. There is a long video here that shows a prototype with GitLab: https://www.youtube.com/watch?v=JVCN9X-vO1g&t=1s.
2. We don't have this feature at the launch, but Matthias van de Meent
is already working on it. We will publish and RFC and solicit comments from the community.
3. We are working on two: regional read replicas and consistent multi-region writes (together with Dan Abadi who helped design FaunaDB). Former is much, MUCH easier.
4. An obvious one is a time machine - we want to allow you query at LSN (or timestamp). A less obvious one is templates: you can start your project with a pre-populated database. We will allow you to create and publish such "templates". Disclaimer - it might not be called templated when we ship it.
I am trying to understand how it works without digging into the code. It sounds like the disk-backed storage here uses S3 which would introduce some severe latency as well as orders of magnitude more access errors (S3 is not going to be more reliable than EBS, let alone physical disk arrays on a day to day basis). Also how do they mitigate latency from their network to mine? In other words why would I run this over a local install if performance mattered at all to me?
PostgreSQL WAL is sent to 3 'Safekeeper' nodes, which provide temporary persistence of WAL on their local disks. This allows us to provide low commit latencies.
After Safekeepers acknowledge the WAL, a PageServer will receive the WAL from these Safekeepers and transform it into LSM-tree "Layers" - blocks of lookup-optimized changelogs, which (when complete) are sent to S3. At that point, the data is considered fully persisted against most, if not all, outages.
The PageServer (which serves as the long-term data server for the running compute nodes) maintains a local cache of Layers. Still, by design, that is only a cache -- it allows for fast responses but is not strictly necessary for the persistence model.
In our cloud offering logical replication is not yet supported. There's an open epic to support receiving a logical replication stream [0], but we've not yet checked on sending logical or physical replication streams.
Everything you've described sounds unbelievably awesome - this is the only thing I can think of that would make it better. If we can get all of these features (I've been dreaming of branching for years, scale to zero!, unlimited storage - just not having to think ahead about it!) plus easy, even on by default logical replication, you will have found the holy grail of cloud databases imo.
Again I am super impressed with the technology involved but do want to clarify: in order to have D in ACID the update must be sent to S3, right? Is there a mode which makes it so that an INSERT, UODATE, and DELETE do it return until this happens? What kind of latency does that introduce and is that latency affected by throughput at all?
Kind of. S3 is the long-term low-cost durability guarantee, while our Safekeepers (3, each in a different zone) provide a high-cost short-term durability guarantee with their local persistent disks.
Latency from PostgreSQL WAL to S3 depends on WAL throughput and the configured pageserver checkpoint distance (default 256MB, and this config field is not equal to that of PostgreSQL).
When you say short term do you mean for hot data or that the guarantee is short term? As in, once it is written to the Safekeeprs is there any chance that the data will disappear?
We keep it there for a short duration, until the changes are confirmed to also be written to S3.
Writing to 3 instances in 3 availability zones is considered persistent enough while also maintaining a high performance, and even though it does not provide the 11 9s of durability that S3 has, 3 availability zones dropping out with loss of all instance-local storage is considered rare enough that we do not think that it will impact our availability and durability guarantees.
Many distributed systems offer ACID by using distribution + replication for the initial write commit.
It's much faster and cheaper to just have your data on multiple nodes (RAM or local disk) and provides better reliability against crashes. Data can then be compacted and streamed out in an async fashion to more durable storage.
The docs are available on https://neon.tech/docs/cloud/about/ . It contains some information, but we're looking to expand that while working towards a public beta that's arriving soon.
The goal of the "cloud" is to extract maximum $$$ from its users. Once you realise that, it's not hard to see through the marketing propaganda and the rest of the decisions following.
else you incur traffic cost and latency
Thus, they are separated so they can charge you (more) for them separately.
What? This isn't controversial, Snowflake did this years ago and this is the main reason it became a $40B business.
Amazon came out with Redshift, a cloud OLAP database, but it tied compute with data so teams couldn't scale compute and data separately and thus had to pay disproportional costs to their required workloads.
Sure, there's good reasons to keep compute and data together. But there is obviously a massive market for technologies that keep them separate...
Do note that many installations of PostgreSQL already have some form of "separation of storage and compute" through a networked storage solution like NFS, EBS, or other SAN-like systems.
The major part of what Neon does is remove the file system abstraction that is between that Storage and Compute, so that we can better utilize the available resources because we can better select what information is or isn't being lost.
A good example of what removing the file system abstraction enables for us is effectively free PITR, (lagging) replicas, and data branching. This is because PostgreSQL's file-system-based storage engine expects to be the only one working on the data directory, which means that any FS attached to a replica cannot be shared. If you remove that file-system based storage engine and plug in a different storage engine, those expectations are removed too, and after some effort integrating into the smgr-APIs, we're now able to provide a storage layer that only needs to contain one copy of the data for N physical replicas, instead of N copies.
I agree that it does have its limitations, but databases are all about trade-offs.
Talking about the "correctness" of a choice between tradeoffs is weird though. Running your database on hard drives nowadays is not a great choice - yet people still did that because the cost of hard drives was way cheaper than that of memory.
Running your database in a way that doesn't guarantee that a 'committed' response actually retains the changes that it was responding on - yet people still ran their database in such configuration to scrape the barrel on performance.
All Neon does is put up another option: If you don't mind the implications of networked storage, then here's one database system that has zero-cost cloning and does not lose data on single-node failure.
Easy trade-off to get scalability. Writing log off-host is fast with 10G network and nvme on the other side (see Amazon Aurora, Microsoft hyper scale, etc)
This isn't binary as much as it's a continuum of abstractions already. Cloud servers already use virtualized disks where the actual drives can be in an entirely different building from the compute racks. Do you consider that separate?
Separating storage and compute at the database nodes is just one architecture method that allows for simpler and more efficient scaling in modern cloud-based deployments where you have large pools object storage that's fast and close. I think you'd be surprised by what you can achieve with some metadata and caching.
From the teams page, the CEO of Neon is the cofounder of MemSQL/Singlestore which is one of the best database products I've used. Looks like a solid team to get this done. Very similar approach to Yugabyte (real postgres compute layer + custom scale out data layer) and many others in the OLAP space.
Manish Jain of dgraph.io noted that building on top of Postgres or betting on Postgres seems like a necessary condition for database startups to be successful.
Some are commodotizing Postgres' wire format but implementing their own query and storage layers (like CockroachDB / Aurora / AlloyDB), while others are modifying parts of Postgres (like Timescale / EdgeDB / YugaByte), and others still are building atop it (Supabase).
Interesting note but that seems to be recency bias with news than anything concrete. Companies from MongoDB to Snowflake to FaunaDB have been successful. Manish himself is from DGraph which is a brand new graph database with no relational to Postgres.
Yes, you will never run out of storage with Neon. The reason to it that in our tiered storage implementation S3 is the cheapest tier and we will offload data into S3 if it gets too big.
We wrote a custom storage layer for postgres, so in our setup, postgres node (k8s pod actually) doesn't store any data, and it is easy to start/stop/reschedule it. So while we actually are DBaaS, the closest analogy here is Aurora or Alloy, not RDS-like setups.
How fast can a "scaled to zero" database start up? Does Neon use a "uninitialized hot spare" strategy to reduce startup latency like crdb?
How much memory do they expect a typical single postgresql compute instance to take? I saw that Neon is targeting 'thousands' of postgresql processes per server, though with giant multi-TB servers these days that doesn't really narrow it down.
Are the postgresql processes multi-tenant as well, or is multitenancy isolated to the storage layer?
---
Heikki from the Neon team presented a talk about why they chose to develop Neon in Rust and what their experience was in Rust Finland 2022. https://www.youtube.com/watch?v=kAQeout-mh8
Postgres process in single tenant. Right now both provisioning a new Postgres instance (we call it project) and cold start is 2 seconds. We will be improving on that.
Hey Nikita! I was just looking at the docs but I was a bit confused about what the various compute instances were doing. Do they all serve reads and writes? If so, is there data partitioning or does this support distributed transactions?
Various compute instances are different endpoints to separate databases. So for now it's single writer system. You can get a lot of power out of a 128 core compute node. In the future will will also spin up extra compute to scale reads.
In the future after that future we will introduce data partitioning - we have a cool design for it, but one step at a time.
This is really exciting and thank you for making it open source. I am still trying to wrap my head around the Neon, but is there any design document or architecture description? I want to learn more about the Neon storage engine and how it all fits together.
I can imagine a world where it might be practical to have one master db for all of your customers/accounts. But a separate db instance for each customer’s data.
Is that the kind of architecture you think might be workable with your system?
won't the tiny compute units on AWS have relatively slow storage? (no NVMe allowed for them I think) fine for small datasets that fit in ram but benchmarks are needed to show the bigger picture.
Yes, we are discussing either simply using Postgres replication to move data to other regions and use our proxy to rout reads to the datacenter closer to the user (like fly.io). This will have issues with supporting more than ~5 regions.
OR we can separate storage from replication and purpose build a multi-tenant replication service. This will support as many regions as you want (over 200) but it's more work. We will publish an RFC for that.
The company is Neon, Inc., which is registered in the USA. We're a remote company, with a significant portion of the developers being located in Europe.
Privacy policy and related stuff will be ready when we publish the public beta, which we expect to happen soon.
Yep, Delaware corp with top tier US investors. I'm in the Bay Area. Heikki is in Finland. Stas is in Cyprus. Majority of engineering is in Europe, some in the US and Canada.
Having a relational database where you're charged purely for the calls you make is a game-changer.
All of the relational databases I looked at in the past required you to have a gateway node on at all times, which is far too expensive for a simple hobby project.
This is exactly what I was looking for windmill, an OSS tool for multi-step automation from scripts.
By any chance, are documentation to host it on top of nomad be on the Roadmap ? If yes, I would try it immediately to replace the postgres server we currently use.
No documentation of how to self host yet. It's not hard, it requires K8S and S3 compatible object store. We want to harden Neon operationally before documenting and supporting on prem production installs.
Requires k8s as in - uses k8s APIs? Or is supported on k8s?
Apologies haven’t read the docs but wanted to highlight this. A hard requirement on k8s to the exclusion of other schedulers would be a shame and an odd choice
Got it, makes sense. I think we are pursuing similar strategies with OSS/self-hostable products. I would love to get in touch with you about it. Could you drop me an email at ruben@windmill.dev ?
Does this storage layer still require VACUUMing? The Aurora system that AWS has uses a log based storage system, so along with point in time restore there’s no longer a need to go free up old garbage rows. This system says it’s copy on write, wondering if vacuuming is still necessary.
Yes, it still requires VACUUM. I believe Aurora does too.
It would be cool to push down at least parts of the VACUUM down to the storage layer, but it would require more invasive changes Postgres code, which we try to avoid. Maybe in the future. Ideally though, I'd like to improve PostgreSQL itself, to reduce the need for VACUUM in the first place.
What are your thoughts on orioleDB? Which is a slightly modified Postgres that does away with the need for vacuum, amongst other changes. Ostensibly done by a dev who is “solving Postgres’ wicked problems”. Eventually the team behind orioleDB plan to upstream their minor changes to vanilla Postgres such that orioleDB becomes simply just an extension of Postgres itself.
Would you ever consider supporting the orioleDB extension in the future?
If it becomes more widespread and mature yes. We are in contact with Alexander Korotkov - Postgres committer and the auther of OrielDB and compare notes from time to time. Right now it makes a lot more sense to us to build our storage at the page level and have minimal changes in Postgres itself.
The difficulty comes from the fact that VACUUM requires most of the same state that is required when processing a user transaction. For example, VACUUM needs the commit log to check whether a given transaction ID is for a transaction that committed or aborted (VACUUM also truncates the commit log to remove information about older XIDs when it is no longer required).
Aurora Postgres does still have VACUUM, which seems to work in the same way -- which is the same in Neon. AWS have in the past promoted Aurora as having significantly better performance characteristics when VACUUM is run. That may well be true, but the benefits seem to come from not having to generate full page writes in WAL, which are a way of preventing a low-level problem called torn pages. Theoretically you could just turn off full-page writes in standard Postgres if you had hardware that offered atomic page writes, though I don't think that it's a widespread practice.
I spend most of my time working on problems with VACUUM in Postgres itself (I'm one of the committers). An approach to organizing storage within transactional constraints seems necessary to push vacuuming down to storage, and that would more than likely need plenty of work in Postgres itself to be practical -- since it would cross a few layers of abstraction. Currently heapam doesn't specifically try to keep tuples inserted around the same time together, so it's hard to make anything that VACUUM does work implicitly or logically.
Thanks for the explanation, I’m nowhere near your experience level with this, but from what I understand Aurora is an append only log with rolling snapshots (this might be wrong). In an append only log storage system there would be no concept of page writes, right? The VACUUM process might still be necessary to find and mark dead tuples, but this seems very minor compared to filesystem operations to reclaim (sometimes gigabytes) of storage. The reclamation of storage would simply left to the background snapshotting process and would not affect DB performance in any way.
The most important job of VACUUM is usually to make sure that queries remain responsive -- it's roughly comparable to a "full scan" within generational garbage collectors (for managed language runtimes). VACUUM does reclaim space in various ways, but that's often considered to be of secondary importance.
In Postgres, the on-disk representation is virtually the same thing as the in-memory representation used by pages stored in the buffer cache. In a system like Aurora or Neon, the representation of pages in the buffer cache is identical to that of Postgres (or has very minimal divergence to deal with one or two isolated problems). That part doesn't really change, which makes it possible to have a very deep level of compatibility without much effort. So it's radically different in one narrow, scoped way, but otherwise very similar.
While the storage knows how to materialize pages on demand, these are not transactionally consistent pages -- they often need to be interpreted by using metadata about transaction commit status, just like in Postgres.
337 comments
[ 3.0 ms ] story [ 280 ms ] threadI'm very very excited to hear about a team taking this effort to postgres itself, in an open source fashion! From the Architecture[1] section of the README:
> A Neon installation consists of compute nodes and Neon storage engine.
> Compute nodes are stateless PostgreSQL nodes, backed by Neon storage engine.
> *Neon storage engine consists of two major components: A) Pageserver. Scalable storage backend for compute nodes. B) WAL service. The service that receives WAL from compute node and ensures that it is stored durably.
Sounds like a very reasonable disaggregation strategy. Really hope to hear about this wonderful effort for many more years. Ticks the boxes: open-source with a great service offering: nice. Rust: nice.
edit: I found the info here: https://boards.greenhouse.io/neondatabase/jobs/4506003004
Please email beta@neon.tech for the invite code. We will ask for feedback in return.
Here's serverless sqlite: https://www.sqlite.org/serverless.html
Classic Serverless: The database engine runs within the same process, thread, and address space as the application. There is no message passing or network activity.
Neo-Serverless: The database engine runs in a separate namespace from the application, probably on a separate machine, but the database is provided as a turn-key service by the hosting provider, requires no management or administration by the application owners, and is so easy to use that the developers can think of the database as being serverless even if it really does use a server under the covers.
It sounds like this is neo-serverless but serverless nonetheless.
Specifically, the term is 10 years old this October: http://readwrite.com/2012/10/15/why-the-future-of-software-a...
It has never, ever meant that the software wasn't running on a server. From TFA:
"The phrase 'serverless' doesn’t mean servers are no longer involved. It simply means that developers no longer have to think that much about them. Computing resources get used as services without having to manage around physical capacities or limits."
1. scaling does not require thinking about servers; and
2. you don't have to pay for committed capacity measured in servers.
And Marketing is King, for bottom-up SaaS businesses (businesses selling to individual developers), which most companies open-sourcing their core projects most often are.
Instead, a "serverless" system's store-of-record is some external and semantically-abstracted storage system — e.g. "a remote git repository" (GitOps); "an object-storage bucket" (Snowflake); "a document store" (Lambda); "a blockchain"; etc. Where all that's important about this storage system is its API, such that the design is portable to any backing storage service that supports the same API.
This means that, in serverless systems, the mutable state in the cluster is just an ephemeral cache representation of the externally-managed data-at-rest.
Another way to think about this is by thinking of "a server" as a thing with durable state that you have to worry about — e.g. protect from disk corruption, make backups of, etc; vs. something like "an ephemeral immutable-infrastructure container workload" that can die and be recreated with no problems. Serverless systems are systems without any "servers" in this sense — nothing to back up; nothing to disaster-recover; etc. Nobody operating these systems ever needs to think about individual servers. Nobody ever needs to SSH into a server, upgrade a server, restart a server, etc. The operations for such systems can be handled entirely at the "cluster" and "ephemeral workload" levels. Nodes within the cluster that "go bad" can simply be drained and deleted — this may even be automated.
And further, because of this lack of local durable state, there's no need to worry about "allocating" that state, and thereby allocating customers to particular clusters. Serverless compute clusters are usually just one huge cluster (per region), where customers' individual request workloads just get scheduled onto that cluster wherever they'll fit.
Of course, the external store-of-record for a serverless system must have "servers" — the data-at-rest ultimately has to reside durably on some disk somewhere. But 1. they're not your servers, and their ops problems are not your ops problems; and 2. because they're a much lower-level abstraction, they can scaled much more robustly, and so can have far fewer operational problems in the first place; and 3. because they're a much lower-level abstraction, they benefit from economies-of-scale in shared tenancy in ways domain-specific compute/DBMS/etc. clusters usually don't; and so your system can benefit from a storage layer that's hyperscaled + hyper-robust from serving millions of tenants' low-level needs.
E.g. AWS Elastic File Store: an auto-resizing NAS-like abstraction, but at any given point, a committed size where you’re paying for unused space.
Vs. Amazon S3 (which might as well stand for “serverless storage service”) — where you only pay for what you use, with no committed capacity beyond current usage.
Here's Sqlite without a server, running in the client and backed by static file hosting: https://phiresky.github.io/blog/2021/hosting-sqlite-database...
Copy on write, presumably.
This opens up a world of try-out mini applications that cost cents to host. serverless db (postgres) + serverless compute (cloud-run) + use as you go storage+network. This is a paradigm-shift stack. Exciting days ahead.
> It automatically starts up, shuts down, and scales capacity up or down based on your application's needs.
The only tradeoff is the additional latency someone will have when connecting to the db after it has shutdown and waiting for it to spin back up and become ready.
> You pay only for the capacity your application consumes.
> Scales down to 0.5
But it actually can't scale down to 0.5 or the DB falls over just existing.. auto scaling won't let you go down that low unless you set 0.5 as the max, which literally makes it not scale up, and it's dead, because the DB can't run with that little CPU.
So it's fair to ask if neon can scale to 0, both in marketing, and in practice.
It still has a significant latency on the first connection attempt after suspend (1-2 seconds), but we are working on that and it seems to be realistic to put the startup time under 1 sec.
Pricing model is still work-in-progress, so cannot say much about it. Yet, my personal intention is to make it cost-effective for both end-user and us. I'd prefer to don't build a service with claims like 'here is your free-tier serverless Postgres with zero-latency on connect', which actually means that under the hood there is an always-running compute burning the investors money. Hope it's realistic to achieve :)
-- Cloud engineer @ Neon
I tried fast booting of PostgreSQL instances and it always took multiple seconds. So i am really curious!
Basically, to start we need to know this LSN and to bootstrap the Postgres processes. This is really that quick. After that compute is ready to accept connections and serve requests, as it's able to get any missing pages from pageserver with GetPage@LSN request.
We do have the whole bunch of problems to solve: queries latency after cold start; startup after the unexpected exit of the heavily loaded Postgres instance could be slower; etc.
- Initializing shared memory -> We, for now, have only small instances, so that doesn't hit us as hard
- Reading data directories -> We don't have to do that at all
- Replaying WAL from a previous unclean shutdown -> We don't need to do that, PageServer is responsible for that
- When initializing a whole new database: Initializing the data directory -> We have a copy that each instance gets initialized from, which makes the process "copy those ~16MB in the background", which saves us from having to do the costly initialization process.
And there's several more infrastructural optimizations, such as pre-loading the docker images onto the hosts.
Looking forward to seeing how this works out. I have no issues paying for services, I just hate that the minimum entry level cost is $20... I can't imagine why, at scale, it can't be more affordable for hobby/fun level projects.
https://ldoughty.com/2022/05/exploring-aws-aurora-serverless...
Short answer if you don't want to read my post: it constantly uses CPU, it's always on. After creation, waiting 2 days, never logged into it, never ran a script against it, never gave it access to any networks, minimum cost is $43/month because it can't actually scale down to 0.5 units unless you CAP it at 0.5, which makes it unusable, because it consumes all of that capacity just to exist.
It sounds like this Neon offering is exactly what I hoped AWS was offering... Or they are using Language to suggest it and mislead the customer just the same ... If it's the former, if probably sign up and try it out. If it's the latter, I'll probably never touch it for the false hope.
Edit: lots of typos from phone keyboard
It's sitting at 23% of 0.5 ACU.
So its either the replication setting (haven't tested with it on yet), or.. AWS is a shared service... wonder if it's similar to EC2, in that sometimes you get an instance on a machine that's more overloaded and the instance doesn't perform as well.. and you have to destroy it and try again. Might want to try it again.
Edit: I don't think its the replication setting... tried that with a new db and its at 25% on each replica after an hour.
All serverless, scale-to-zero or pay for demand...
Neon seems like a vast vast improvement & great & desperately needed potential leap for mankind.
> "great & desperately needed potential leap for mankind"
Are you being serious? That's very hyperbolic if so.
Thank you for the comment rektide. We of course would love to see constructive comments and criticism.
- Neon CEO
I appreciate your concern for the integrity of HN discussions, but the thing to do if you suspect abuse or manipulation is to email hn@ycombinator.com so we can look into it. Posting about it in the threads themselves is explicitly against the rules.
but please take a look at the flagging attack on all of my comments outside this thread. the timing is too peculiar here
somebody really didnt like what i wrote above ;)
Yes Im serious. This is one of the most foundational & key levels of computing: storing & quering data. Without this, computing isnt good for much.
Getting good at this is a huge task for humanity. Right now that task is almost entirely being fulfilled by far off hyperscalers. Aurora, BigTable, Firebase, DynamoDB, CosmoDB, more special works like DataDog Husky... the world is running off vast super-awesome dataengines. But ones that are not ours, that we cant hack on, that we cant run ourselves. It might as well be the Martian's (little green men's) databases as far as I'm concerned: these are not humanity's heritdge & humanity is cut off from active participation with them.
Right now there are so so so few scalable data systems available for the world, that we have. This seems like a great & novel effort to radically open up the range of human capabilities, in one of the most important sectors of computing: handling data.
We have lots of other cloudware for humankind but data has seemingly been much slower & un-scaled. I agree with this @anilgulecha comment[1]:
> This is the missing piece on cloud for masses
[1] https://news.ycombinator.com/item?id=31537313
> "But ones that are not ours, that we cant hack on, that we cant run ourselves."
Who is "we"? Is your entire issue that not everything is open-source?
> "these are not humanity's heritdge & humanity is cut off from active participation with them."
This is still incredibly exaggerated. Everything humanity does is humanity's heritage. There are plenty of open-source databases of all types and sizes if you look: https://db-engines.com/en/ranking
The most notable feature of Neon, that makes it so exceptionally smart, is that it takes the humanity's existing first best choice default-go-to database, by a country mile, and layers in really smart decoupling of storage to help it scale. Reinventing things from scratch can be a win, but that this is already a well known quantity, much loved & cherished & used by all, hacked by all, grown by all is a planet-sized huge plus mark.
I think you vastly overrate the broad availability of alternatives to this, of potential starting places others might explore. Most of your list is dominated by proprietary whatever. Even skipping that, yes, there is a ton of novelty & promise, the potential for breakthroughs. But this is starting with the best, and pushing it into the cloud-layer, into the troposphere.
There are dozens of "scalable postgres" alternatives from Redshift to Citus to CockroachDB to Yugabyte. I find it strange that you say its overrating the availability when you're also comparing it to something that isn't even a full product yet.
Cockroach DB license - https://github.com/cockroachdb/cockroach/blob/2c4e2c6/LICENS...
Mongo license - https://github.com/mongodb/mongo/blob/39e4b70/LICENSE-Commun...
IIRC, backup was enterprise but is now part of the core. However, restore is still enterprise.
This sentiment is perhaps right, but I was careful about calling out scale-to-zero. We do have options that are zero cost (or pay as you use), but there's a fundamental difference in something that may be zero cost because a cloud provider is using it as a customer-acquisition ploy.
Options like litestream+sqlite+s3, or what Neon seems to be, are verifiable you-pay-for-when-db-is-booted up, else the verifiable cost is storage only.
So the trifecta that will be very productive for masses is 1) database where compute is scale-to-zero, 2)open source or commoditised, and 3) is RDBMS.
CockroachDB already does true scale-to-zero if that's your requirement: https://www.cockroachlabs.com/blog/how-we-built-cockroachdb-...
Eg: of innovation -> commodity. EC2 was innovative, but is today common-place. S3 was the same. DBs that scale-to-zero have not reached that state yet.
Thanks for the link on cockroachdb - it sounds promising. I wonder what's the minimum self-deployable unit of cockroachdb - will google around a bit.
If you host it yourself, or use their dedicated enterprise clusters then you still have instances which are individual (virtual) servers.
The serverless model has no instances exposed to you and instead is a multitenant architecture that uses a pool of shared compute nodes with a routing layer that intercepts and introspects your connection to load up your specific database context for query processing.
Neon is 100% compatible from Postgres b/c we didn't (or almost didn't) change the Postgres engine.
Our intention is to standardize the separation of storage and compute cloud architecture - that's why it's open source under the Apache 2.0 license.
I noticed you mention Azure BS in your RFCs as a potential backend. Have you done much work towards that yet?
However, if you really can't wait to run Neon on Azure, you could contribute the integration yourself: the code is available under Apache 2 at https://github.com/neondatabase/neon/
Most managed/serverless options begin at hundreds of dollars per month. So, you get lots of companies either just handing over the cash or jumping through hoops to get something more reasonable. The latter is a stupid waste of time if you can afford the former. That's how Google and Amazon make money: they make the expensive option more tempting and the cheap option needlessly hard. They are not interesting in supporting frugal teams. The whole point is squeezing their customers hard.
So, this is potentially very nice if it offers some competition on the cost front. I'd certainly consider using this if it proves reliable. In fact, the whole reason I opted out of a relational database is the above. What I'd need is something that is reasonable in cost relative to the modest data I store and retrieve.
We used firestore before.. got a bit tired of some of the limitations (latency, indexing). Cost-wise I don't think it's that different actually, but we aren't using much bandwidth, then self-hosted can be dramatically cheaper. Have to manage some details of course (zfs filesystem parameters, set up backups, config postgres etc.), but I found that stuff quite interesting and it's knowledge that will always be useful.
A snapshot is effectively identical to a killed database server (due to OOM, or power outage), and database servers should be crash-resistant.
With PostgreSQL, checkpointing the database before taking the snapshot may make the crash recovery quicker when loading the snapshotted database.
I've always struggled estimating my DB size, curious what you mean by your estimate.
1 - Is not an actual relational DB
2- Doesn't really scale to zero
Planetscale does scale to zero, but has a ridiculous billing model.
Alternatively, it's likely that Planetscale will eventually provide a provisioned billing model, similar to DynamoDB. Then you can choose between scalable usage based pricing or less flexible flat rate pricing. Either way, innovating different billing models for RDBMS is a win.
Given how much performance you can squeeze out of a $5/month VPS (I've been spinning them up and indeed down regularly over the last couple of years), is this really a paradigm shift?
I appreciate there are indeed billions of people for whom $5 is a lot of money, but just how many of them are "students and enthusiasts" itching to get started with Postgres?
I realise that - perhaps particularly here - a $5/month VPS is a deeply unsexy thing. You can, however, achieve (and learn) an awful lot with one.
Some links and discussions:
- https://cloudirregular.substack.com/p/please-fix-the-aws-fre...
- https://twitter.com/alexwlchan/status/1399095011178958851
- https://news.ycombinator.com/item?id=27044371
I'm an (former) SRE and run my own Kubernetes cluster for fun, but still use serverless (containers as a service, static website hosting) depending on the project.
I've done SRE/devops work in various capacities including consulting longer than cloud services have existed, and my experience is that I've consistently earned more from clients who insisted on cloud services because they consistently need more help. Nothing is driving more demand for devops consulting services than cloud providers.
This is not hard to get right. Yes, you need to learn how to do it, but the amount of money I've made from clients who thought cloud was simple and proceeded to create massive security holes for themselves is fairly substantial. People who think they're reducing their attack surface by using these services need to reevaluate - they're large, complex architectures that very few users understand properly. You need to learn the skills either way.
Don't you need that as well as cloud-specific knowledge if you go serverless?
> and ideally more than that to know not to do stupid things like chmod 777 and database exposed on the public internet.
You still need some arcane knowledge to make sure your serverless doesn't experience cost overruns, right?
IME (your's obviously differs), the amount of cloud-specific + vendor-specific knowledge needed to avoid using a $5/m VM is a lot more in volume and a lot less in stability[1] than learning basic Linux once and using VMs everywhere[2].
[1] How the different cloud providers bill, when they bill, how to control your limits, etc changes much more often than knowing how to keep your server patched. Knowing how to get your serverless DB going on AWS doesn't help when you want to use Azure. And each cloud vendor regularly requires you to update your knowledge. Knowing how to keep a PostgreSQL-on-Linux up-to-date can be learned once and used for years. Even if running a managed DB, you'll still need to gain some of that knowledge anyway.
[2] Once you get to a scale where treating your machines like cattle rather than pets, you'll obviously have the team required to use cloud stuff optimally.
I know people who have literally written their own analysis tooling just to figure out what's impacting their AWS spend. It's gotten better, but I could have retired many times over on what I've seen clients overpay to cloud providers because they didn't understand what would drive cost.
> [2] Once you get to a scale where treating your machines like cattle rather than pets, you'll obviously have the team required to use cloud stuff optimally.
The "problem" with the cloud story is that at that point you also have a team that could save you a fortune with a hybrid setup. Cloud providers get their margins off those who don't understand how much they're overpaying or who are too small to a) care or b) have leverage. Those big enough to have leverage who understand either negotiate hefty discounts or build out cheaper setups (basically at the point you're spending 7 figures a year, if you're paying anywhere near list prices for cloud services you're a chump; below that it's hit and miss)
I'm not at all against using cloud services, but I wish more people actually understood their costs and picked based on merits rather than cargo-culting. Some teams benefit greatly from cloud services, but usually if they're not cost sensitive. In my current job we have everything on AWS because we're never going to scale to somewhere where it'll get expensive and it's convenient. We'd save money if I moved it to, say, Hetzner, but the hosting bill is too small to matter. For that use it's fine.
The moment the bill starts to bite people ought to at least price out alternatives, and consider hybrid setups. E.g. I've had setups where even just putting a caching proxy in front of AWS to cache images to cut the egress bill would have paid for a team to keep it running. Their egress cost is still bad, but at the time it was just pure highway robbery.
Is there any cold start delay for neon?
Lambda has many limitations.
In particular, for some reason AWS is allergic to providing a container deployment service that actually scales to zero.
Isn't this what Fargate is?
Scale doesn't matter for mini applications and scaling vertically (=throw money for a bigger server) will work for 99% of the companies. The 1% who need horizontal scaling will have custom everything regardless and will need to hire experts, not a good niche to release a product.
These are reasons why one might go for a hosted database at AWS/GCE/Azure. There are tons of good servers out there for small to medium projects and I don't think this service is right for those, unless they can make do with the free tier. The real benefit is in the larger cloud application space.
A system that does this type of scaling automatically while also reducing the dependency on a single cloud provider's service can be a gamechanger for some companies with huge database servers that risk getting locked in.
I think using this service for most existing applications will introduce a performance drop, a rise in expenses, and a complicated migration path, but on the other hand I think that developing against this system for new projects that are very likely to grow in scale will end up with some major data control benefits.
The open source nature also allows for competitors in markets like the EU to start serving databases that don't break privacy laws (although most companies don't care until they receive a fine).
I'm not sure how this company will become profitable while giving away its special sauce that can be modified to run at competing companies relatively easily, but that's a whole different story.
I like this perspective a lot & think it's absolutely key here.
We- the world- still pick single-node writer postgres & read replicas when we have to store & query data. There's great Kubernetes postgres operators, but it's still a distinctly pre-cloud pre-scale type of technology, & this decoupling & shared-storage sounds ultra promising, allows independent & radicaly scale up & scale down, sounds principally much more managable.
Or for try-out apps, as you mention, you could just run Postgres next to your app in the same container.
This might be possible with fly.io, or will soon, I think.
I’m not sure how comfortable I am using a custom flavor of Postgres (even if it’s just the storage layer).
The problem with most other "serverless" databases is that they don't offer HTTP API to query them from restricted environments like serverless functions.
What a time to be a developer.
I did email Heikki the following questions, in case if someone from Neon is around here.
a) How does Neon compare to polardb https://github.com/ApsaraDB/PolarDB-for-PostgreSQL.
b) The readme mentions a component "Repository - Neon storage implementation". Does it use any special FileSystem? Any links to read more about it?
c) Heard the cold start is a second (IIRC), how does that value differ if one runs Neon on bare metal instead of k8s?
a. PolarDB is based on a similar idea. https://www.cs.utah.edu/~lifeifei/papers/polardbserverless-s.... This paper describes it. The biggest difference that I see glancing through the paper is that we really integrated S3 into the storage. In Neon architecture branches, backups, checkpoint are all the same thing and instant to run. This simplifies a good amount of database management AND deliver on better costs. S3 is cheap.
b. Neon doesn't need a special filesystem. Neon storage is in a way a filesystem, however it doesn't expose filesystem API. It's a key value store - serves 8k pages to Postgres and a consensus - update API to the key value store. Pages are organized in LSM trees and background processes put layers of the LSM trees to S3.
c. The cols start is 2sec right now. There is a dependency on K8S. Bare metal implementation will require new code to orchestrate starts and stops.
S3 has its limitations though, like too many small files and the get/delete/list ops get very expensive. There's also an upper-limit on throughput per S3-bucket partition. I guess, sstables that pageserver flushes periodically help work around these issues?
> Neon storage is in a way a filesystem, however it doesn't expose filesystem API.
Genuinely curious: When would anyone consider using filesystems like Amazon FSx for Lustre instead which is backed by S3 anyway over implementing a filesystem-esque abstraction of their own (like neon.tech does, and other solutions like rocketset.com, tiledb.com, xata.io, and quickwit.io do).
> Pages are organized in LSM trees and background processes put layers of the LSM trees to S3.
Curious how merges are handled? Also, are you using RocksDB / some other engine underneath?
> Bare metal implementation will require new code to orchestrate starts and stops.
Speaking of new code... SingleStore started as a very high-throughput OLTP database and eventually evolved to into a HTAP (?) database. Do you see Neon evolving in a similar manner, too?
Thanks!
2. It's best to custom build a storage system here. External distributed filesystems introduce complexity, cost, and bottlenecks that you don't control.
3. Purpose built. LSM trees also have a temporal dimension - LSN. You can fetch a page by pageId and LSN. This is what allows time machine and branching.
4. I call it convergence when OLTP and OLAP is one system - ultimate dream for a database systems engineer. Since I spent 10 years building it I have both scars and aspirations. I think it will come, but this will take a long time. HTAP is in a way a subset of convergence - most systems will have some HTAP. Neon will have some too, but for now it squarely focused on OLTP and helping developers build apps.
Will it be possible to use something else in place of S3? I'm thinking on-premise or what some would call a private cloud.
If you can't wait that long to run Neon on your own cloud, feel free to contribute an integration to your persistent blob storage: the code is available under APLv2 here: https://github.com/neondatabase/neon/
will page size be tunable on neon cloud for larger datasets?
I can't find it on Youtube, do you have the link?
edit: I found the link, seems it is not on the Youtube yet: https://www.pgcon.org/events/pgcon_2022/schedule/session/236...
Yeah, as Nikita mentioned it's 2 seconds now. We did some tests and measurements and on bare metal, it's sub 500 ms usually, so the remaining part is the k8s (+ our own control plane) orchestration overhead. For example, with plain Docker (which we use in CI in addition to k8s) it's around 1 second already.
K8s provides a convenient abstraction layer, though. So I think that we'll continue using it and optimization will come with pods pool / over-provisioning and it'll be realistic to bring the startup time closer to bare-metal.
-- Cloud engineer @ Neon
The claim is "serverless" i.e. you don't have access to the server and if you could install any extension you essentially have full access to the server as there's no restriction what you could do in an extension. I don't think that would be allowed.
Disclaimer: Neon co-founder
very well executed! congrats!
I couldn't find much info about the replication models available/planned however. I would consider this to be table stakes at this point for a serverless database with the recent trend of pushing compute to the edge. This is much more interesting to me than scaling to 0, which is only really useful during the prototyping phase.
PlanetScale is single primary with eventually consistent read replicas, Fauna has strongly consistent global writes (or regional if you choose, but no option for replication between regions if you do) with a write latency cost, Dynamo/Cosmos are active-active eventual consistently replicated with fast writes globally. All useful in different scenarios, but I'd love to have one DB tech that can operate in all of these modes for different use cases within the same app, using the same programming model to interact with data across the board.
I think the decoupled storage engine here would open up some really interesting strategies around replication. What are the team's plans here?
[1] https://docs.planetscale.com/concepts/branching
1. Yes schema and data via "copy on write". This will let you instantly create test environments, backups, and run CI/CD. There is a long video here that shows a prototype with GitLab: https://www.youtube.com/watch?v=JVCN9X-vO1g&t=1s.
2. We don't have this feature at the launch, but Matthias van de Meent is already working on it. We will publish and RFC and solicit comments from the community.
3. We are working on two: regional read replicas and consistent multi-region writes (together with Dan Abadi who helped design FaunaDB). Former is much, MUCH easier.
4. An obvious one is a time machine - we want to allow you query at LSN (or timestamp). A less obvious one is templates: you can start your project with a pre-populated database. We will allow you to create and publish such "templates". Disclaimer - it might not be called templated when we ship it.
https://www.postgresql.org/docs/current/datatype-pg-lsn.html
What version of Postgres are they targeting? And do they have a strategy for keeping up to date with new versions?
We're looking into supporting PG15 when that comes out too, but I wouldn't hold my breath on that as we still have a lot to do.
PostgreSQL WAL is sent to 3 'Safekeeper' nodes, which provide temporary persistence of WAL on their local disks. This allows us to provide low commit latencies.
After Safekeepers acknowledge the WAL, a PageServer will receive the WAL from these Safekeepers and transform it into LSM-tree "Layers" - blocks of lookup-optimized changelogs, which (when complete) are sent to S3. At that point, the data is considered fully persisted against most, if not all, outages.
The PageServer (which serves as the long-term data server for the running compute nodes) maintains a local cache of Layers. Still, by design, that is only a cache -- it allows for fast responses but is not strictly necessary for the persistence model.
[0] https://github.com/neondatabase/neon/issues/1632
Latency from PostgreSQL WAL to S3 depends on WAL throughput and the configured pageserver checkpoint distance (default 256MB, and this config field is not equal to that of PostgreSQL).
Writing to 3 instances in 3 availability zones is considered persistent enough while also maintaining a high performance, and even though it does not provide the 11 9s of durability that S3 has, 3 availability zones dropping out with loss of all instance-local storage is considered rare enough that we do not think that it will impact our availability and durability guarantees.
It's much faster and cheaper to just have your data on multiple nodes (RAM or local disk) and provides better reliability against crashes. Data can then be compacted and streamed out in an async fashion to more durable storage.
No. just no.
Compute where the data lives, else you incur traffic cost and latency.
A lesson learned in life, known from the dawn of time..
else you incur traffic cost and latency
Thus, they are separated so they can charge you (more) for them separately.
Amazon came out with Redshift, a cloud OLAP database, but it tied compute with data so teams couldn't scale compute and data separately and thus had to pay disproportional costs to their required workloads.
Sure, there's good reasons to keep compute and data together. But there is obviously a massive market for technologies that keep them separate...
Apples and oranges.
> "no. Just no."
Conveys the feeling that there is no scenario where doing this is feasibly and you yourself just said it's acceptable for BI but not AI training.
So the "no. Just no." actually means, "my use case does not allow for this, so I believe no one else should use this" which is a fallacy on its own.
In conclusion, this is fairly usable, but like everything else, it's not for all use cases.
The major part of what Neon does is remove the file system abstraction that is between that Storage and Compute, so that we can better utilize the available resources because we can better select what information is or isn't being lost.
A good example of what removing the file system abstraction enables for us is effectively free PITR, (lagging) replicas, and data branching. This is because PostgreSQL's file-system-based storage engine expects to be the only one working on the data directory, which means that any FS attached to a replica cannot be shared. If you remove that file-system based storage engine and plug in a different storage engine, those expectations are removed too, and after some effort integrating into the smgr-APIs, we're now able to provide a storage layer that only needs to contain one copy of the data for N physical replicas, instead of N copies.
The latency of NFS & EBS or EFS is actually the reason many businesses *do not* use them for their databases.
I've seen deployments that had to go bare-metal because the tiny latency of EBS caused their compute times to rise exponentialy (doing AI training).
Talking about the "correctness" of a choice between tradeoffs is weird though. Running your database on hard drives nowadays is not a great choice - yet people still did that because the cost of hard drives was way cheaper than that of memory.
Running your database in a way that doesn't guarantee that a 'committed' response actually retains the changes that it was responding on - yet people still ran their database in such configuration to scrape the barrel on performance.
All Neon does is put up another option: If you don't mind the implications of networked storage, then here's one database system that has zero-cost cloning and does not lose data on single-node failure.
Separating storage and compute at the database nodes is just one architecture method that allows for simpler and more efficient scaling in modern cloud-based deployments where you have large pools object storage that's fast and close. I think you'd be surprised by what you can achieve with some metadata and caching.
Some are commodotizing Postgres' wire format but implementing their own query and storage layers (like CockroachDB / Aurora / AlloyDB), while others are modifying parts of Postgres (like Timescale / EdgeDB / YugaByte), and others still are building atop it (Supabase).
https://twitter.com/manishrjain/status/1496174276474732544
https://fly.io/blog/globally-distributed-postgres/
Or in their case they might refer to horizontal scalability of their storage layer which is independent from computing.
A bit more details are on https://github.com/neondatabase/neon
How much memory do they expect a typical single postgresql compute instance to take? I saw that Neon is targeting 'thousands' of postgresql processes per server, though with giant multi-TB servers these days that doesn't really narrow it down.
Are the postgresql processes multi-tenant as well, or is multitenancy isolated to the storage layer?
---
Heikki from the Neon team presented a talk about why they chose to develop Neon in Rust and what their experience was in Rust Finland 2022. https://www.youtube.com/watch?v=kAQeout-mh8
Nobody knew Rust, so they started out by hiring someone who did. Good move.
Business idea: consultancy that hires out competent Rust devs to new projects.
We have been hard at work and looking to open the service to the public soon.
In the future after that future we will introduce data partitioning - we have a cool design for it, but one step at a time.
Super interested in this space since we're always looking for ways to evolve our pg!
Also, how do I get an invite code to try?
edit: found this to get started - https://neon.tech/docs/storage-engine/architecture-overview/
I can imagine a world where it might be practical to have one master db for all of your customers/accounts. But a separate db instance for each customer’s data.
Is that the kind of architecture you think might be workable with your system?
We are already working with customers that do that. This is for sure a great use case for Neon.
OR we can separate storage from replication and purpose build a multi-tenant replication service. This will support as many regions as you want (over 200) but it's more work. We will publish an RFC for that.
I'm trying to find out if you're a company and where you are located. Is there no legal entity behind this? Do you have a privacy policy?
Privacy policy and related stuff will be ready when we publish the public beta, which we expect to happen soon.
Postgres is a global phenomenon.
All of the relational databases I looked at in the past required you to have a gateway node on at all times, which is far too expensive for a simple hobby project.
Apologies haven’t read the docs but wanted to highlight this. A hard requirement on k8s to the exclusion of other schedulers would be a shame and an odd choice
That does not mean that you cannot run Neon outside k8s, but we are not actively maintaining nor supporting other hosting options.
It would be cool to push down at least parts of the VACUUM down to the storage layer, but it would require more invasive changes Postgres code, which we try to avoid. Maybe in the future. Ideally though, I'd like to improve PostgreSQL itself, to reduce the need for VACUUM in the first place.
Disclaimer: Neon co-founder
Would you ever consider supporting the orioleDB extension in the future?
Aurora Postgres does still have VACUUM, which seems to work in the same way -- which is the same in Neon. AWS have in the past promoted Aurora as having significantly better performance characteristics when VACUUM is run. That may well be true, but the benefits seem to come from not having to generate full page writes in WAL, which are a way of preventing a low-level problem called torn pages. Theoretically you could just turn off full-page writes in standard Postgres if you had hardware that offered atomic page writes, though I don't think that it's a widespread practice.
I spend most of my time working on problems with VACUUM in Postgres itself (I'm one of the committers). An approach to organizing storage within transactional constraints seems necessary to push vacuuming down to storage, and that would more than likely need plenty of work in Postgres itself to be practical -- since it would cross a few layers of abstraction. Currently heapam doesn't specifically try to keep tuples inserted around the same time together, so it's hard to make anything that VACUUM does work implicitly or logically.
Disclaimer: I work for Neon
In Postgres, the on-disk representation is virtually the same thing as the in-memory representation used by pages stored in the buffer cache. In a system like Aurora or Neon, the representation of pages in the buffer cache is identical to that of Postgres (or has very minimal divergence to deal with one or two isolated problems). That part doesn't really change, which makes it possible to have a very deep level of compatibility without much effort. So it's radically different in one narrow, scoped way, but otherwise very similar.
While the storage knows how to materialize pages on demand, these are not transactionally consistent pages -- they often need to be interpreted by using metadata about transaction commit status, just like in Postgres.