When I started working on Baserow (this seems similar based on the roadmap), a couple of years ago, I thought it would be a big challenge to quickly render a million rows in the browser. Introducing a system that fetches a page of rows based on the scroll offset, and with a small debounce did the trick. We only had a couple of field types, and it was all incredibly fast
The thing that make performance complicated for a no-code database is when you have 30 interconnected tables, some tables with 200 fields, containing many formulas or other computed fields like lookups or rollups. Updating a single cell, can result in thousands of other rows that must be updated across different tables. If there are 30 users making constant changes, locking PostgreSQL rows under the hood while the formulas are recalculated, and then a couple of n8n workflows making a many API requests to those tables, that's when things get interesting. Especially in combination with features like webhooks, real-time updates, 100+ filters, grouping, 26 field types, date dependencies, aggregations, importing/exporting whole databases.
When implementing a new feature, I've heard users say that's not complicated because it's just adding a checkbox. Making to run it at scale and keeping things performant is what's making it complicated.
Terrible from the front-end side of implementation:
- performs worse than your average arbitrary-amount-of-rows-that-won't-fit-on-the-screen library (it should perform the same no matter if its 1k, 1m or 1mm rows)
- is seemingly buggy
- is pointless on its own, because THIS demo is a client-side demo, and no one loads that much data on the client-side.
Revisit this when this demo is performant AND data is loaded from the backend.
Ignoring that, every front-end JS developer should explore these kinds of libs and also try to implement them themselves, because they're basically front-end 101.
Possible bug report: on Safari mobile, when I grab the scroll bar and move it down a bit, the website reloads; if I do it again, I get an error message (“a problem repeatedly occurred…”).
One of the rare times HN can come together and agree on something. Which is: that this is a loose weekend project thrown together quickly that doesnt solve any problem in particular
You basically reinvented something Elixir Streams already nails out of the box.
Streams in Elixir are lazy, chunked, and backpressure-friendly, so you can process any size dataset without loading it all into memory...whether it's a million or a trillion rows. The trick is you never try to render them all in the browser (that's where virtualization comes in).
So yeah...neat work, but battle-tested versions of this have been around for a while.
28 comments
[ 4.2 ms ] story [ 47.5 ms ] threadCoincidentally I worked on a large table renderer too this weekend: https://github.com/markwylde/react-massive-table
I noticed you didn't quite get to a million rows. For me, it cut off at 671088.
Same thing happened when I built my one.
I came across the same thing. In the end I just manual made the rows appear at their absolute position. Seemed to work well.
The thing that make performance complicated for a no-code database is when you have 30 interconnected tables, some tables with 200 fields, containing many formulas or other computed fields like lookups or rollups. Updating a single cell, can result in thousands of other rows that must be updated across different tables. If there are 30 users making constant changes, locking PostgreSQL rows under the hood while the formulas are recalculated, and then a couple of n8n workflows making a many API requests to those tables, that's when things get interesting. Especially in combination with features like webhooks, real-time updates, 100+ filters, grouping, 26 field types, date dependencies, aggregations, importing/exporting whole databases.
When implementing a new feature, I've heard users say that's not complicated because it's just adding a checkbox. Making to run it at scale and keeping things performant is what's making it complicated.
To whom it may concern: I scrolled a bit with the scrollbar on iOS and the page immediately crashed.
It seems this is just a minimum implementation of a ‘virtual list’.
Terrible from the front-end side of implementation: - performs worse than your average arbitrary-amount-of-rows-that-won't-fit-on-the-screen library (it should perform the same no matter if its 1k, 1m or 1mm rows) - is seemingly buggy - is pointless on its own, because THIS demo is a client-side demo, and no one loads that much data on the client-side.
Revisit this when this demo is performant AND data is loaded from the backend.
Ignoring that, every front-end JS developer should explore these kinds of libs and also try to implement them themselves, because they're basically front-end 101.
The column text length too should be trimmed to a uniform max-length except when clicked on. You could make it pup out on the page with CSS.
A better color scheme too won't hurt.
Had the same idea when I saw https://github.com/rowyio/rowy.
Stumbled on an idea while reading a HN entry a few days back and now I will merge them into a niche product idea.
└── Dey well
Streams in Elixir are lazy, chunked, and backpressure-friendly, so you can process any size dataset without loading it all into memory...whether it's a million or a trillion rows. The trick is you never try to render them all in the browser (that's where virtualization comes in).
So yeah...neat work, but battle-tested versions of this have been around for a while.