We encountered this problem when converting audio only LLM applications to visual + audio.
The visuals would increase latency by a lot since they need to be parsed completely before displaying, whereas you can just play audio token by token and wait for the LLM to generate the next one while audio is playing.
I've always liked the idea of putting latency requirements in to API specifications. Maybe that could help delimit what is and is not automatically inlined as the author proposes.
It's not about being picky. It's about communicating needs, and setting boundaries that are designed to satisfy those needs without overwhelming anybody's system to the point of saturation and degraded performance.
Right, but what if some of that SLA information actually directed the code itself.
In the context of this blog post, what if the SLA was <100ms for an initial response, with some mandatory fields, but then any additional information which happens to be loaded within that 100ms automatically is included. With anything outside the 100ms is automatically sent in a followup message?
Came up at work a few weeks ago when a co-worker used "ndjson" which I'd never heard of before, but I knew "jsonl" which he'd never heard of before: As far as I could tell with some searching, they are basically the same thing and have two different names because they came from two different places. "ndjson" was a full-on spec, while "jsonl" was more informal - kind of like an enterprise vs open source, that converged on the same idea.
From wikipedia, "ndjson" used to include single-line comments with "//" and needed custom parsers for it, but the spec no longer includes it. So now they are the same.
ndjson has an actual spec (however bitrotted), everything else in that space makes rookie mistakes like not specifying that a newline is a required message terminator -- consider receiving "13\n42", is that truncated or not?
None of the above is actually good enough to build on, so a thousand little slightly-different ad hoc protocols bloom. For example, is empty line a keepalive or an error? (This might be perfectly fine. They're trivial to program, not like you need a library.)
Very cool point, and it applies to any tree data in general.
I like to represent tree data with parent, type, and data vectors along with a string table, so everything else is just small integers.
Sending the string table and type info as upfront headers, we can follow with a stream of parent and data vector chunks, batched N nodes at a time. Tye depth- or breadth-first streaming becomes a choice of ordering on the vectors.
I'm gonna have to play around with this! Might be a general way to get snappier load time UX on network bound applications.
You can even alternate between sending table and node chunks! This will effectively allow you to reveal the tree in any order including revealing children before parents as well as representing arbitrary graphs! Could lead to some interesting applications.
If you send the tree in preorder traversal order with known depth, you can send the tree without node ids or parent ids! You can just send the level for each node and recover the tree structure with a stack.
Well the whole point is to use a breadth first order here. I don't think there's a depth vector analogue for breadth first traversals. Is there?
But, indeed, depth vectors are nice and compact. I find them harder to work with most of the time, though, especially since insertions and deletions become O(n), compared to parent vector O(1).
That said, I do often normalize my parent vectors into dfpo order at API boundaries, since a well-defined order makes certain operations, like finding leaf siblings, much nicer.
I’m not familiar with depth vectors, but wouldn’t the breadth first traversal analogue of each entry specifying its depth (in a depth first format) be each entry specifying the number of immediate children it has?
Yeah, it has its limits for sure. I like it for the streaming aspect.
I think you can still have the functionality described in the article: you would send “hole” markers tagged with their level. Then, you could make additional requests when you encounter these markers during the recovery phase, possibly with buffering of holes. It becomes a sort of hybrid DFS/BFS approach where you send as much tree structure at a time as you want.
I'm being a bit coy about it but the article aims to describe key ideas in the RSC wire protocol, which is an implementation detail of React and isn't actually beholden to JSON itself. JSON is just a nice starting point to motivate it. However, I think reusing JSON for object notation kind of makes sense (and allows native JSON.parse calls for large objects).
I feel like in an ideal world, this would start in the DB: your query referencing objects and in what order to return them (so not just a bunch of wide rows, nor multiple separate queries) and as the data arrives, the back end could then pass it on to the client.
Many db has done that, 4d.com is one that comes to mind. It is kinda like socket.io + PostgreSQL + node/ruby/php (middleware layer) all in one. In db there is also concept of cursor, etc.
Seems like it is never about merit of technological design. As some CS professor put it, tech is more about fashion than tech now days. IMHO that is true, and often also comes down to the technological context surrounding the industry at the time, and now days if the code is open sourced/FOSS.
99.9999%* of apps don't need anything nearly as 'fancy' as this, if resolving breadth-first is critical they can just make multiple calls (which can have very little overhead depending on how you do it).
* I made it up - and by extension, the status quo is 'correct'.
Having progressive or partial reads would dramatically speed up applications, especially as we move into an era of WASM on the frontend.
A proper binary encoded format like protobuf with support for partial reads and well defined streaming behavior for sub message payloads would be incredible.
It puts more work on the engineer, but the improvement to UX could be massive.
Sure, if you’re the 0.00001% that need that. It’s going to be over engineering for most cases. There are so many simpler and easier to support things that can be done before trying this sort of thing.
Following the example, why is all the data in one giant request? Is the DB query efficient? Is the DB sized correctly? How about some caching? All boring, but if rather support and train someone on boring stuff.
>We technically didn't need more than 640K either.
That old chestnut again - this was true for MSDOS PCs in 1981 when the quote was said. It was still true 10 years later for whatever version of DOS was current then. People keep bringing it up as though Bill Gates said 'no one will need > 640k for all time to come'.
To be clear, I wouldn't suggest someone to implement this manually in their app. I'm just describing at the high level how the RSC wire protocol works, but narratively I wrapped it in a "from the first principles" invention because it's more fun to read. I don't necessarily try to sell you on using RSC either but I think it's handy to understand how some tools are designed, and sometimes people take ideas from different tools and remix them.
I'm already thinking of whether there's any ideas here I might take for CSTML -- designed as a streaming format for arbitrary data but particularly for parse trees
I get that. Originally my comment was a response to another but I decided to delete and repost it at the top level — however I failed to realize that not having that context makes the tone rather snarky and/or dismissive of the article as a whole, which I didn't intend.
There's nothing wrong with "accidentally-overengineering" in the sense of having off-the-shelf options that are actually nice.
There is something wrong with adding a "fancy" feature to an off-the-shelf option, if said "fancy" feature is realistically "a complicated engineering question, for which we can offer a leaky abstraction that will ultimately trip up anybody who doesn't have the actual mechanics in mind when using it".
> There's nothing wrong with "accidentally-overengineering" in the sense of having off-the-shelf options that are actually nice.
Your comment focuses on desired outcomes (i.e., "nice" things), but fails to acknowledge the reality of tradeoffs. Over engineering a solution always creates problems. Systems become harder to reason with, harder to maintain, harder to troubleshoot. For example, in JSON arrays are ordered lists. If you onboard an overengineered tool that arbitrarily reorders elements in a JSON array, things can break in non-trivial ways. And they often do.
I think the issue with the example json is that it's sent in OOP+ORM style (ie nested objects), whereas you could just send it as rows of objects, something like this;
{
header: "Welcome to my blog",
post_content: "This is my article",
post_comments: [21,29,88], # the numbers are the comment ID's
footer: "Hope you like it",
comments: {21: "first", 29: "second", 88: "third" }
}
But then you may as well just go with protobufs or something, so your endpoints and stuff are all typed and defined, something like this;
And with this style, you don't necessarily need to embed the comments in 1 call like this, and you could cleanly do it in 2 like parent-comment suggests (1 to get page+post, second to get comments), which might be aided with `int32 post_comment_count = 4;` instead (so you can pre-render n blocks).
I think the point is that GraphQL solves the problem, a client only actually needing a subset of the data, by allowing the client to request only those fields.
So if not graphql, then what's the latest "in-favor" thinking to solve the problem of underfetching and overfetching? Especially in an environment with multiple kinds of frontends?
Everywhere I worked with GraphQL it was always a pain for the backend team to keep the graphql server updated and also a pain to use in the frontend, simple REST apis or JSON-RPC are much better.
> Why did you not have these pains with other tech?
You don't need a new layer between database -> backend -> frontend
So GraphQL became a backend-for-frontend layer that needs maintaince. The team knowing GraphQL or not is not what causes this, but definitely makes it worse as they use the tool, which is quite complex, wrongly.
NextJS is the "evolution" (or more like growth? as a tumor) of this backend-for-frontend approach, glued with the mentioned RSC in this post. Great recipe to fight accidental complexity all days and nights.
So because they were not generating resolvers from the schema, a maintenance burden was created. You CAN create resolvers manually but that’s really not the best idea when automated approaches exist. That’s poor engineering, not bad technology.
GraphQL federation is where the thing really shines.
From what I recall, GraphQL has a feature that's similar (@defer) but I'm not familiar enough to compare them. RSC was definitely inspired by GraphQL among other things.
The GraphQL payload has redundant copies of the objects within. Relay has an experimental (dead) exploration of normalising data on the server[1] in order to avoid that transport cost.
For @defer, the transport is a stream[2] where subsequent payloads are tracked ("pending") and as those payloads arrive they are "patched" into the existing data at the appropriate path.
Both of these are effectively implementations of this Progressive JSON concept.
I can picture a future where both the JSX and object payloads are delivered progressively over the wire in the same stream, and hydrate React and Relay stores alike. That could potentially simplify and eliminate the need for Relay to synchronise it's own updates with React. It would also potentially be a suitable compile target for decoupling, React taking ownership of a normalised reactive object store (with state garbage collection which is a hard problem for external stores to solve) with external tools like Relay and TanStack Query providing the type-generation & fetching implementation details.
Using `new Promise()` in the client-side store would also mean you could potentially shrink the component API to this. `useFragment(fragment IssueAssignee on Issue { assignee { name }}, issue)` could instead be `const assignee = use(issue.assignee)` and would suspend (or not) appropriately.
In the recent Web 2.0 2.0 submission <https://news.ycombinator.com/item?id=44073785> there was some HATEOAS poo-poo-ing, but maybe this is the delivery mechanism which makes that concept easier to swallow, 'cause JSON
Json is just a packing format that does have that limitation. If you control the source and the destination, could you possibly use a format that supports streaming better like Protobuf?
I’ve written a lot of APIs. I generally start with CSV, convert that to XML, then convert that to JSON.
CSV is extremely limited, and there’s a lot of stuff that can only be expressed in XML or JSON, but starting with CSV usually enforces a “stream-friendly” structure.
I had invented a variant of DER called DSER (Distinguished Streaming Encoding Rules), which is not compatible with DER (nor with BER) but is intended for when streaming is needed.
The type and value are encoded the same as DER, but the length is different:
- If it is constructed, the length is omitted, and a single byte with value 0x00 terminates the construction.
- If it is primitive, the value is split into segments of lengths not exceeding 255, and each segment is preceded by a single byte 1 to 255 indicating the length of that segment (in bytes); it is then terminated by a single byte with value 0x00. When it is in canonical form, the length of segments other than the last segment must be 255.
Protobuf seems to not do this unless you use the deprecated "Groups" feature, and this is only as an alternative of submessages, not for strings. In my opinion, Protobuf also seems to have many other limits and other problems, that DER (and DSER) seems to do better anyways.
Nothing, really, but I don’t have the bandwidth to write JSAX. I wonder why it hasn’t already been done by someone more qualified than I am. I suspect that I’d find out, if I started doing it.
You can do that, in a specialized manner, with PHP, and Streaming JSON Parser[0]. I use that, in one of my server projects[1]. It claims to be JSON SAX, but I haven’t really done an objective comparison, and it specializes for file types. It works for my purposes.
I'll try to explain why this is a solution looking for a problem.
Yes, breadth-first is always an option, but JSON is a heterogenous structured data source, so assuming that breadth-first will help the app start rendering faster is often a poor assumption. The app will need a subset of the JSON, but it's not simply the depth-first or breadth-first first chunk of the data set.
So for this reason what we do is include URLs in JSON or other API continuation identifiers, to let the caller choose where in the data tree/graph they want to dig in further, and then the "progressiveness" comes from simply spreading your fetch operation over multiple requests.
Also often times JSON is deserialized to objects so depth-frst or breadth-first doesn't matter, as the object needs to be "whole" before you can use it. Hence again: multiple requests, smaller objects.
In general when you fetch JSON from a server, you don't want it to be so big that you need to EVEN CONSIDER progressive loading. HTML needs progressive loading because a web page can be, historically especially, rather monolithic and large.
But that's because a page is (...was) static. Thus you load it as a big lump and you can even cache it as such, and reuse it. It can't intelligently adapt to the user and their needs. But JSON, and by extension the JavaScript loading it, can adapt. So use THAT, and do not over-fetch data. Read only what you need. Also, JSON is often not cacheable as the data source state is always in flux. One more reason not to load a whole lot in big lumps.
Now, I have a similar encoding with references, which results in a breadth-first encoding. Almost by accident. I do it for another reason and that is structural sharing, as my data is shaped like a DAG not like a tree, so I need references to encode that.
But even though I have breadth-first encoding, I never needed to progressively decode the DAG as this problem should be solved in the API layer, where you can request exactly what you need (or close to it) when you need it.
>The app will need a subset of the JSON, but it's not simply the depth-first or breadth-first first chunk of the data set.
Right. Closer to the end of the article I slightly pivot to talk about RSC. In RSC, the data is the UI, so the outermost data literally corresponds to the outermost UI. That's what makes it work.
It's encoded like progressive JSON but conceptually it's more like HTML. Except you can also have your own "tags" on the client that can receive object attributes.
It's a fait accompli; they've already implemented all this. They're not really looking for feedback at this point, more like describing how and why it works.
why send the footer above the comments? Maybe its not a footer then but a sidebar? Should be treated as a sidebar then? Besides this could all kinda be solved by still using plain streaming json and sending .comments last?
Part of the point I'm making is that an out-of-order format is more efficient because we can send stuff as it's ready (so footer can go as soon as it's ready). It'll still "slot in" the right place in the UI. What this lets us do, compared to traditional top-down streaming, is to progressively reveal inner parts of the UI as more stuff loads.
Seems like some people here are taking this post literally, as in the author (Dan Abramov) is proposing a format called Progressive JSON — it is not.
This is more of a post on explaining the idea of React Server Components where they represent component trees as javascript objects, and then stream them on the wire with a format similar to the blog post (with similar features, though AFAIK it’s bundler/framework specific).
This allows React to have holes (that represent loading states) on the tree to display fallback states on first load, and then only display the loaded component tree afterwards when the server actually can provide the data (which means you can display the fallback spinner and the skeleton much faster, with more fine grained loading).
(This comment is probably wrong in various ways if you get pedantic, but I think I got the main idea right.)
Yup! To be fair, I also don't mind if people take the described ideas and do something else with them. I wanted to describe RSC's take on data serialization without it seeming too React-specific because the ideas are actually more general. I'd love if more ideas I saw in RSC made it to other technologies.
do you think a new data serialization format built around easier generation/parseability and that also happened to be streamable because its line based like jsonld could be useful for some?
I don’t know! I think it depends on whether you’re running into any of these problems and have levers to fix them. RSC was specifically designed for that so I was trying to explain its design choices. If you’re building a serializer then I think it’s worth thinking about the format’s characteristics.
I've used React in the past to build some applications and components. Not familiar with RSC.
What immediately comes to mind is using a uniform recursive tree instead, where each node has the same fields. In a funny way that would mimic the DOM if you squint. Each node would encode it's type, id, name, value, parent_id and order for example. The engine in front can now generically put stuff into the right place.
I don't know whether that is feasible here. Just a thought. I've used similar structures in data driven react (and other) applications.
It's also efficient to encode in memory, because you can put this into a flat, compact array. And it fits nicely into SQL dbs as well.
Awesome, thanks! I do keep running on the issues, but the levers as you say make it harder to implement.
As of right now, I could only replace the JSON tool calling on LLM's on something I fully control like vLLM, and the big labs probably are happy to over-charge a 20-30% tokens for each tool call, so they wouldn't really be interested on replacing json any time soon)
also it feels like battling against a giant which is already an standard, maybe there's a place for it on really specialized workflows where those savings make the difference (not only money, but you also gain a 20-30% extra token window, if you don't waste it on quotes and braces and what not
One way is to eagerly call JSON.parse as fragments are coming in. If you also split on json semantic boundaries like quotes/closing braces/closing brackets, you can detect valid objects and start processing them while the stream continues.
Can you offer some detail into why you find this approach useful?
From an outsider's perspective, if you're sending around JSON documents so big that it takes so long to parse them to the point reordering the content has any measurable impact on performance, this sounds an awful lot like you are batching too much data when you should be progressively fetching child resources in separate requests, or even implementing some sort of pagination.
> You don’t actually want the page to jump arbitrarily as the data streams in. For example, maybe you never want to show the page without the post’s content. This is why React doesn’t display “holes” for pending Promises. Instead, it displays the closest declarative loading state, indicated by <Suspense>.
> In the above example, there are no <Suspense> boundaries in the tree. This means that, although React will receive the data as a stream, it will not actually display a “jumping” page to the user. It will wait for the entire page to be ready.
However, you can opt into a progressively revealed loading state by wrapping a part of the UI tree into <Suspense>. This doesn’t change how the data is sent (it’s still as “streaming” as possible), but it changes when React reveals it to the user.
[…]
> In other words, the stages in which the UI gets revealed are decoupled from how the data arrives. The data is streamed as it becomes available, but we only want to reveal things to the user according to intentionally designed loading states.
Smalltalk UIs used to work with only one CPU thread. Any action from the user would freeze the whole UI while it was working, but the positive aspect of that is that it was very predictable and bug free. That's helpful since Smalltalk is OOP.
Since React is functional programming it works well with parallelization so there is room for experiments.
> Especially if it involves content jumping around.
I remember this from the beginning of Android, you'll search for something and click on it and the time it takes you to click the list of results changed and you clicked on something else. Happens with adds on some websites, maybe intentionally?
> And the most annoying antipattern is showing empty state UI during loading phase.
Some low quality software even show "There are no results for your search" when the search didn't even start or complete.
Ember did something like this but it made writing Ajax endpoints a giant pain in the ass.
It’s been so long since I used ember that I’ve forgotten the terms, but essentially the rearranged the tree structure so that some of the children were at the end of the file. I believe it was meant to handle DAGs more efficiently but I may have hallucinated that recollection.
But if you’re using a SAX style streaming parser you can start making progress on painting and perhaps follow-up questions while the initial data is still loading.
Of course in a single threaded VM, you can snatch Defeat from the jaws of Victory if you bollocks up the order of operations through direct mistakes or code evolution over time.
I've never really thought about how all the common ways we serialise trees in text (JSON, s-expressions, even things like tables of content, etc), serialise them depth-first.
I suppose it's because doing it breadth-first means you need to come up with a way to reference items that will arrive many lines later, whereas you don't have that need with depth-first serialisation.
Progressive JPEG make sense, because it's a media file and by nature is large. Text/HTML on the other hand, not so much. Seems like a self-inflicted solution where JS bundles are giant and now we're creating more complexity by streaming it.
Things can be slow not because they're large but because they take latency to produce or to receive. The latency can be on the server side (some things genuinely take long to query, and might be not possible or easy to cache). Some latency may just be due to the user having poor network conditions. In both cases, there's benefits to progressively revealing content as it becomes available (with intentional loading stages) instead of always waiting for the entire thing.
Agree with everything you're saying here, but to be fair I think the analogy with Progressive JPEG doesn't sit quite right with your concept. What you're describing sounds more like "semantic-aware streaming" - it's as if a Progressive JPEG would be semantically aware of its blob and load any objects that are in focus first before going after data for things that are out of focus.
I think that's a very contemporary problem and worth pursuing, but I also somehow won't see that happening in real-time (with the priority to reduce latency) without necessary metadata.
It’s not an exact analogy but streaming outside-in (with gradually more and more concrete visual loading states) rather than top-down feels similar to a progressive image to me.
It's data (JPEG/JSON) VS software (HTML/CSS/JS)... you can choose to look at HTML/CSS/JS as just some chunks of data, or you can look at it as a serialized program that wants to be executed with optimal performance. Your blog post makes it seem like your focus is on the latter (and it's just quite typical for react applications to fetch their content dynamically via JSON), and that's where your analogy to the progressive mode of JPEGs falls a bit flat and "streaming outside-in" doesn't seem like all you want.
Progressively loaded JPEGs just apply some type of "selective refinement" to chunks of data, and for Progressive selective refinement to work it's necessary to "specify the location and size of the region of one or more components prior to the scan"[0][1]. If you don't know what size to allocate, then it's quite difficult(?) to optimize the execution. This doesn't seem like the kind of discussion you'd like to have.
Performance aware web developers are working with semantic awareness of their content in order to make tweaks to the sites loading time. YouTube might prefer videos (or ads) to be loaded before any comments, news sites might prioritize text over any other media, and a good dashboard might prioritize data visualizations before header and sidebar etc.
The position of the nodes in any structured tree tells you very little about the preferred loading priority, wouldn't you agree?
I used this analogy more from the user's perspective (as a user, a gradually sharpening image feels similar to a website with glimmers gradually getting replaced by revealing content). I don't actually know how JPEG is served under the hood (and the spec is too dense for me) so maybe if you explain the point a bit closer I'll be able to follow. I do believe you that the analogy doesn't go all the way.
RSC streams outside-in because that's the general shape of the UI — yes, you might want to prioritize the video, but you have to display the shell around that video first. So "outside-in" is just that common sense — the shell goes first. Other than that, the server will prioritize whatever's ready to be written to the stream — if we're not blocked on IO, we're writing.
The client does some selective prioritization on its own as it receives stuff (e.g. as it loads JS, it will prioritize hydrating the part of the page that you're trying to interact with).
This appears conceptually similar to something like line-delimited JSON with JSON Patch[1].
Personally I prefer that sort of approach - parsing a line of JSON at a time and incrementally updating state feels easier to reason and work with (at least in my mind)
The article doesn't advocate sending it progressively to make it smaller on the wire.
The motivating example is one where some of the data (e.g. posts) is available before the rest of the data in the response (e.g. comments). Rather than:
- Sending a request for posts, then a request for comments, resulting in multiple round trips (a.k.a. a "waterfall"), or,
- Sending a request for posts and comments, but having to wait until the commends have loaded to get the posts,
...you can instead get posts and comments available as soon as they're ready, by progressively loading information.
The message, though, is that this is something a full-stack web framework should handle for you, hence the revelation at the end of the article about it being a lesson in the motivation behind React's Server Components.
What is innovative trying to build a framework that does it for you.
Progressive loading is easy, but figuring out which items to progressively load and in which order without asking the developer/user to do much extra config is hard.
> Progressive loading is easy, but figuring out which items to progressively load and in which order without asking the developer/user to do much extra config is hard.
Do developers even control the order in which stuff is loaded? Tha depends on factors beyond a developer's control, such as the user's network speed, the origin server's response speed, which resources are already cached, how much data each request fetches for user A or user B, etc.
Right, which is why I describe a framework that does it for you (RSC) at the end of the article. The article itself is meant as an explanation of how RSC works under the hood.
> Now reuse the same connection to request the nested data, which can all have more nested links in them, and so on.
This still involves multiple round-trips though. The approach laid out in the article lets you request exactly the data you need up-front and the server streams it in as it becomes available, e.g. cached data first, then data from the DB, then data from other services, etc.
When you have an HTTP/2 connection already open a 'round-trip' is not really a gigantic concern performance-wise. And it gives the client application complete control and ver what nested parts it wants to get and in what order. Remember that the article said it's up to the server what order to stream the parts? That might not necessarily be a good idea on the client side though. It would probably be better for the client to decide what it wants and when. Eg, it can request the header and footer, then swap in a skeleton facade in the main content area, then load the body and swap it in when loaded.
Round trips for parallel requests work fine over HTTP/2. (As long as there aren't vast numbers of tiny requests, for example every cell in a spreadsheet).
However, sequentially-dependent requests are about as slow with HTTP/2 as HTTP/1.1. For example, if your client side, after loading the page, requests data to fill a form component, and then that data indicates a map location, so your client side requests a map image with pins, and then the pin data has a link to site-of-interest bubble content, and you will be automatically expanding the nearest one, so your client side requests requests the bubble content, and the bubble data has a link to an image, so the client requests the image...
Then over HTTP/2 you can either have 1 x round trip time (server knows the request hierarchy all the way up to the page it sends with SSR) or 5 x round trip time (client side only).
When round trip times are on the order of 1 second or more (as they often are for me on mobile), >1s versus >5s is a very noticable difference in user experience.
With lower latency links of 100ms per RTT, the UX difference between 100ms and 500ms is not a problem but it does feel different. If you're on <10ms RTT, then 5 sequential round trips are hardly noticable, thought it depends more on client-side processing time affecting back-to-back delays.
Assuming a stable connection, there is no meaningful performance difference between a request/response round-trip from the client to the server, and a response streamed from the server to the client, amortized over time.
But it was too graphql-coupled and didn't really take off, even for my own projects.
But it might be worth revisiting this kind of protocol again someday, it can tag locations within a JSON response and send updates to specific fields (streaming changes).
243 comments
[ 3.2 ms ] story [ 259 ms ] threadIt's not about being picky. It's about communicating needs, and setting boundaries that are designed to satisfy those needs without overwhelming anybody's system to the point of saturation and degraded performance.
In the context of this blog post, what if the SLA was <100ms for an initial response, with some mandatory fields, but then any additional information which happens to be loaded within that 100ms automatically is included. With anything outside the 100ms is automatically sent in a followup message?
From wikipedia, "ndjson" used to include single-line comments with "//" and needed custom parsers for it, but the spec no longer includes it. So now they are the same.
https://github.com/ndjson/ndjson.github.io/issues/1#issuecom...
None of the above is actually good enough to build on, so a thousand little slightly-different ad hoc protocols bloom. For example, is empty line a keepalive or an error? (This might be perfectly fine. They're trivial to program, not like you need a library.)
I like to represent tree data with parent, type, and data vectors along with a string table, so everything else is just small integers.
Sending the string table and type info as upfront headers, we can follow with a stream of parent and data vector chunks, batched N nodes at a time. Tye depth- or breadth-first streaming becomes a choice of ordering on the vectors.
I'm gonna have to play around with this! Might be a general way to get snappier load time UX on network bound applications.
But, indeed, depth vectors are nice and compact. I find them harder to work with most of the time, though, especially since insertions and deletions become O(n), compared to parent vector O(1).
That said, I do often normalize my parent vectors into dfpo order at API boundaries, since a well-defined order makes certain operations, like finding leaf siblings, much nicer.
I think you can still have the functionality described in the article: you would send “hole” markers tagged with their level. Then, you could make additional requests when you encounter these markers during the recovery phase, possibly with buffering of holes. It becomes a sort of hybrid DFS/BFS approach where you send as much tree structure at a time as you want.
Better to rethink it from scratch instead of trying to put a square peg in a round hog.
Seems like it is never about merit of technological design. As some CS professor put it, tech is more about fashion than tech now days. IMHO that is true, and often also comes down to the technological context surrounding the industry at the time, and now days if the code is open sourced/FOSS.
* I made it up - and by extension, the status quo is 'correct'.
Having progressive or partial reads would dramatically speed up applications, especially as we move into an era of WASM on the frontend.
A proper binary encoded format like protobuf with support for partial reads and well defined streaming behavior for sub message payloads would be incredible.
It puts more work on the engineer, but the improvement to UX could be massive.
Following the example, why is all the data in one giant request? Is the DB query efficient? Is the DB sized correctly? How about some caching? All boring, but if rather support and train someone on boring stuff.
That old chestnut again - this was true for MSDOS PCs in 1981 when the quote was said. It was still true 10 years later for whatever version of DOS was current then. People keep bringing it up as though Bill Gates said 'no one will need > 640k for all time to come'.
There is something wrong with adding a "fancy" feature to an off-the-shelf option, if said "fancy" feature is realistically "a complicated engineering question, for which we can offer a leaky abstraction that will ultimately trip up anybody who doesn't have the actual mechanics in mind when using it".
Your comment focuses on desired outcomes (i.e., "nice" things), but fails to acknowledge the reality of tradeoffs. Over engineering a solution always creates problems. Systems become harder to reason with, harder to maintain, harder to troubleshoot. For example, in JSON arrays are ordered lists. If you onboard an overengineered tool that arbitrarily reorders elements in a JSON array, things can break in non-trivial ways. And they often do.
I think the issue with the example json is that it's sent in OOP+ORM style (ie nested objects), whereas you could just send it as rows of objects, something like this;
But then you may as well just go with protobufs or something, so your endpoints and stuff are all typed and defined, something like this; And with this style, you don't necessarily need to embed the comments in 1 call like this, and you could cleanly do it in 2 like parent-comment suggests (1 to get page+post, second to get comments), which might be aided with `int32 post_comment_count = 4;` instead (so you can pre-render n blocks).GraphQL could use Progressive JSON to serialize subscriptions.
You don't need a new layer between database -> backend -> frontend
So GraphQL became a backend-for-frontend layer that needs maintaince. The team knowing GraphQL or not is not what causes this, but definitely makes it worse as they use the tool, which is quite complex, wrongly.
NextJS is the "evolution" (or more like growth? as a tumor) of this backend-for-frontend approach, glued with the mentioned RSC in this post. Great recipe to fight accidental complexity all days and nights.
GraphQL federation is where the thing really shines.
For @defer, the transport is a stream[2] where subsequent payloads are tracked ("pending") and as those payloads arrive they are "patched" into the existing data at the appropriate path.
Both of these are effectively implementations of this Progressive JSON concept.
I can picture a future where both the JSX and object payloads are delivered progressively over the wire in the same stream, and hydrate React and Relay stores alike. That could potentially simplify and eliminate the need for Relay to synchronise it's own updates with React. It would also potentially be a suitable compile target for decoupling, React taking ownership of a normalised reactive object store (with state garbage collection which is a hard problem for external stores to solve) with external tools like Relay and TanStack Query providing the type-generation & fetching implementation details.
Using `new Promise()` in the client-side store would also mean you could potentially shrink the component API to this. `useFragment(fragment IssueAssignee on Issue { assignee { name }}, issue)` could instead be `const assignee = use(issue.assignee)` and would suspend (or not) appropriately.
[1]: https://github.com/facebook/relay/blob/main/packages/relay-r...
[2]: https://github.com/graphql/graphql-wg/blob/main/rfcs/DeferSt...
0: https://jsonapi.org/format/#document-links
I got really, really sick of XML, but one thing that XML parsers have always been good at, is realtime decoding of XML streams.
It is infuriating, waiting for a big-ass JSON file to completely download, before proceeding.
Also JSON parsers can be memory hogs (but not all of them).
I’ve written a lot of APIs. I generally start with CSV, convert that to XML, then convert that to JSON.
CSV is extremely limited, and there’s a lot of stuff that can only be expressed in XML or JSON, but starting with CSV usually enforces a “stream-friendly” structure.
* I do acknowledge you qualified the question with "better".
The type and value are encoded the same as DER, but the length is different:
- If it is constructed, the length is omitted, and a single byte with value 0x00 terminates the construction.
- If it is primitive, the value is split into segments of lengths not exceeding 255, and each segment is preceded by a single byte 1 to 255 indicating the length of that segment (in bytes); it is then terminated by a single byte with value 0x00. When it is in canonical form, the length of segments other than the last segment must be 255.
Protobuf seems to not do this unless you use the deprecated "Groups" feature, and this is only as an alternative of submessages, not for strings. In my opinion, Protobuf also seems to have many other limits and other problems, that DER (and DSER) seems to do better anyways.
[ ["aaa", "bbb"], { "name", "foo" } ]
You can do that, in a specialized manner, with PHP, and Streaming JSON Parser[0]. I use that, in one of my server projects[1]. It claims to be JSON SAX, but I haven’t really done an objective comparison, and it specializes for file types. It works for my purposes.
[0] https://github.com/salsify/jsonstreamingparser
[1] https://github.com/LittleGreenViper/LGV_TZ_Lookup/blob/main/...
In response to "Json is just a packing format that does have that [streaming] limitation".
?page=3&size=100
Yes, breadth-first is always an option, but JSON is a heterogenous structured data source, so assuming that breadth-first will help the app start rendering faster is often a poor assumption. The app will need a subset of the JSON, but it's not simply the depth-first or breadth-first first chunk of the data set.
So for this reason what we do is include URLs in JSON or other API continuation identifiers, to let the caller choose where in the data tree/graph they want to dig in further, and then the "progressiveness" comes from simply spreading your fetch operation over multiple requests.
Also often times JSON is deserialized to objects so depth-frst or breadth-first doesn't matter, as the object needs to be "whole" before you can use it. Hence again: multiple requests, smaller objects.
In general when you fetch JSON from a server, you don't want it to be so big that you need to EVEN CONSIDER progressive loading. HTML needs progressive loading because a web page can be, historically especially, rather monolithic and large.
But that's because a page is (...was) static. Thus you load it as a big lump and you can even cache it as such, and reuse it. It can't intelligently adapt to the user and their needs. But JSON, and by extension the JavaScript loading it, can adapt. So use THAT, and do not over-fetch data. Read only what you need. Also, JSON is often not cacheable as the data source state is always in flux. One more reason not to load a whole lot in big lumps.
Now, I have a similar encoding with references, which results in a breadth-first encoding. Almost by accident. I do it for another reason and that is structural sharing, as my data is shaped like a DAG not like a tree, so I need references to encode that.
But even though I have breadth-first encoding, I never needed to progressively decode the DAG as this problem should be solved in the API layer, where you can request exactly what you need (or close to it) when you need it.
Right. Closer to the end of the article I slightly pivot to talk about RSC. In RSC, the data is the UI, so the outermost data literally corresponds to the outermost UI. That's what makes it work.
It's encoded like progressive JSON but conceptually it's more like HTML. Except you can also have your own "tags" on the client that can receive object attributes.
Not again, please.
This is more of a post on explaining the idea of React Server Components where they represent component trees as javascript objects, and then stream them on the wire with a format similar to the blog post (with similar features, though AFAIK it’s bundler/framework specific).
This allows React to have holes (that represent loading states) on the tree to display fallback states on first load, and then only display the loaded component tree afterwards when the server actually can provide the data (which means you can display the fallback spinner and the skeleton much faster, with more fine grained loading).
(This comment is probably wrong in various ways if you get pedantic, but I think I got the main idea right.)
do you think a new data serialization format built around easier generation/parseability and that also happened to be streamable because its line based like jsonld could be useful for some?
What immediately comes to mind is using a uniform recursive tree instead, where each node has the same fields. In a funny way that would mimic the DOM if you squint. Each node would encode it's type, id, name, value, parent_id and order for example. The engine in front can now generically put stuff into the right place.
I don't know whether that is feasible here. Just a thought. I've used similar structures in data driven react (and other) applications.
It's also efficient to encode in memory, because you can put this into a flat, compact array. And it fits nicely into SQL dbs as well.
As of right now, I could only replace the JSON tool calling on LLM's on something I fully control like vLLM, and the big labs probably are happy to over-charge a 20-30% tokens for each tool call, so they wouldn't really be interested on replacing json any time soon)
also it feels like battling against a giant which is already an standard, maybe there's a place for it on really specialized workflows where those savings make the difference (not only money, but you also gain a 20-30% extra token window, if you don't waste it on quotes and braces and what not
Thanks for replying!
It’s become a thing, even beyond RSCs, and has many practical uses if you stare at the client and server long enough.
I've been trying to create go/rust ones but its way harder than just json due to all the context/state they carry over
From an outsider's perspective, if you're sending around JSON documents so big that it takes so long to parse them to the point reordering the content has any measurable impact on performance, this sounds an awful lot like you are batching too much data when you should be progressively fetching child resources in separate requests, or even implementing some sort of pagination.
And the most annoying antipattern is showing empty state UI during loading phase.
Quoting the article:
> You don’t actually want the page to jump arbitrarily as the data streams in. For example, maybe you never want to show the page without the post’s content. This is why React doesn’t display “holes” for pending Promises. Instead, it displays the closest declarative loading state, indicated by <Suspense>.
> In the above example, there are no <Suspense> boundaries in the tree. This means that, although React will receive the data as a stream, it will not actually display a “jumping” page to the user. It will wait for the entire page to be ready. However, you can opt into a progressively revealed loading state by wrapping a part of the UI tree into <Suspense>. This doesn’t change how the data is sent (it’s still as “streaming” as possible), but it changes when React reveals it to the user.
[…]
> In other words, the stages in which the UI gets revealed are decoupled from how the data arrives. The data is streamed as it becomes available, but we only want to reveal things to the user according to intentionally designed loading states.
https://www.haskellpreneur.com/articles/slaying-a-ui-antipat...
Since React is functional programming it works well with parallelization so there is room for experiments.
> Especially if it involves content jumping around.
I remember this from the beginning of Android, you'll search for something and click on it and the time it takes you to click the list of results changed and you clicked on something else. Happens with adds on some websites, maybe intentionally?
> And the most annoying antipattern is showing empty state UI during loading phase.
Some low quality software even show "There are no results for your search" when the search didn't even start or complete.
If that happened maybe a programmer messed-up the green threads!
"The Smalltalk-80 system provides support for multiple independent processes with three classes named Process, ProcessorScheduler, and Semaphore. "
p251 "Smalltalk-80 The Language and it's Implementation"
https://rmod-files.lille.inria.fr/FreeBooks/BlueBook/Blueboo...
It’s been so long since I used ember that I’ve forgotten the terms, but essentially the rearranged the tree structure so that some of the children were at the end of the file. I believe it was meant to handle DAGs more efficiently but I may have hallucinated that recollection.
But if you’re using a SAX style streaming parser you can start making progress on painting and perhaps follow-up questions while the initial data is still loading.
Of course in a single threaded VM, you can snatch Defeat from the jaws of Victory if you bollocks up the order of operations through direct mistakes or code evolution over time.
In .NET land, Utf8JsonReader is essentially this idea. You can parse up until you have everything you need and then bail on the stream.
https://learn.microsoft.com/en-us/dotnet/standard/serializat...
I suppose it's because doing it breadth-first means you need to come up with a way to reference items that will arrive many lines later, whereas you don't have that need with depth-first serialisation.
I think that's a very contemporary problem and worth pursuing, but I also somehow won't see that happening in real-time (with the priority to reduce latency) without necessary metadata.
Progressively loaded JPEGs just apply some type of "selective refinement" to chunks of data, and for Progressive selective refinement to work it's necessary to "specify the location and size of the region of one or more components prior to the scan"[0][1]. If you don't know what size to allocate, then it's quite difficult(?) to optimize the execution. This doesn't seem like the kind of discussion you'd like to have.
Performance aware web developers are working with semantic awareness of their content in order to make tweaks to the sites loading time. YouTube might prefer videos (or ads) to be loaded before any comments, news sites might prioritize text over any other media, and a good dashboard might prioritize data visualizations before header and sidebar etc.
The position of the nodes in any structured tree tells you very little about the preferred loading priority, wouldn't you agree?
[0] https://jpeg.org/jpeg/workplan.html
[1] https://www.itu.int/ITU-T/recommendations/rec.aspx?id=3381 (see D.2 in the PDF)
EDIT: Btw thanks for your invaluable contributions to react (and redux back then)!
RSC streams outside-in because that's the general shape of the UI — yes, you might want to prioritize the video, but you have to display the shell around that video first. So "outside-in" is just that common sense — the shell goes first. Other than that, the server will prioritize whatever's ready to be written to the stream — if we're not blocked on IO, we're writing.
The client does some selective prioritization on its own as it receives stuff (e.g. as it loads JS, it will prioritize hydrating the part of the page that you're trying to interact with).
Personally I prefer that sort of approach - parsing a line of JSON at a time and incrementally updating state feels easier to reason and work with (at least in my mind)
[1] https://en.wikipedia.org/wiki/JSON_Patch
- Sending a request for posts, then a request for comments, resulting in multiple round trips (a.k.a. a "waterfall"), or,
- Sending a request for posts and comments, but having to wait until the commends have loaded to get the posts,
...you can instead get posts and comments available as soon as they're ready, by progressively loading information. The message, though, is that this is something a full-stack web framework should handle for you, hence the revelation at the end of the article about it being a lesson in the motivation behind React's Server Components.
What is innovative trying to build a framework that does it for you.
Progressive loading is easy, but figuring out which items to progressively load and in which order without asking the developer/user to do much extra config is hard.
Do developers even control the order in which stuff is loaded? Tha depends on factors beyond a developer's control, such as the user's network speed, the origin server's response speed, which resources are already cached, how much data each request fetches for user A or user B, etc.
It's a solved problem. Use HTTP/2 and keep the connection open. You now have effectively a stream. Get the top-level response:
Now reuse the same connection to request the nested data, which can all have more nested links in them, and so on.This still involves multiple round-trips though. The approach laid out in the article lets you request exactly the data you need up-front and the server streams it in as it becomes available, e.g. cached data first, then data from the DB, then data from other services, etc.
However, sequentially-dependent requests are about as slow with HTTP/2 as HTTP/1.1. For example, if your client side, after loading the page, requests data to fill a form component, and then that data indicates a map location, so your client side requests a map image with pins, and then the pin data has a link to site-of-interest bubble content, and you will be automatically expanding the nearest one, so your client side requests requests the bubble content, and the bubble data has a link to an image, so the client requests the image...
Then over HTTP/2 you can either have 1 x round trip time (server knows the request hierarchy all the way up to the page it sends with SSR) or 5 x round trip time (client side only).
When round trip times are on the order of 1 second or more (as they often are for me on mobile), >1s versus >5s is a very noticable difference in user experience.
With lower latency links of 100ms per RTT, the UX difference between 100ms and 500ms is not a problem but it does feel different. If you're on <10ms RTT, then 5 sequential round trips are hardly noticable, thought it depends more on client-side processing time affecting back-to-back delays.
For an already-open HTTP/2 connection? Or for a new connection for each request?
https://github.com/rgraphql/rgraphql
But it was too graphql-coupled and didn't really take off, even for my own projects.
But it might be worth revisiting this kind of protocol again someday, it can tag locations within a JSON response and send updates to specific fields (streaming changes).