Blink and you missed the interesting bit, which is how they hooked them together. Looks like it's with this project: https://napi.rs/
That project seems to be creating platform-specific DLL's, which I guess doesn't matter too much if it's wrapped up nicely in an npm package and you're deploying Docker files on Linux anyway? But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's.
Alternatively, writing very C-like code using a Uint8Array and not allocating in the loop might have fixed it too? But if you don't use JavaScript string libraries, it's not much like writing JavaScript anymore.
Yeah, I think Uint8Array might've been a strong solution. But I've got the feeling this isn't the whole story. Unless the TSV is extremely wide, this alone feels a bit insignificant. I don't know the details so this is pure speculation, but using a transform stream on each row to split it up might've been enough to reduce memory pressure.
I've done a bit of data manipulation in node and while imo js is not the ideal language for the job, it can be very fast. Strings are hard though, I'll admit that
> But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's.
This is true, but N-API will generally perform better (sometimes significantly, usually in exactly the same CPU-bound scenarios where WASM will perform better than V8). And NAPI-RS is quite good at paving the happy path for multi-target packages.
> But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's.
If the database IO happens in Rust, the option with WebAssembly is fetch, meaning HTTP and the Atlas-only Data API. If the IO is done in JS, you pay some extra CPU time transferring UTF-16 strings over the border and copy them as UTF-8 in Rust.
Batch maintenance is a drag. However even just from the snippet of the purported memory/slow issue, I think theres a smell that was not revealed.
If your constantly running out of memory you have a bad design.
This seems like a native code agenda type post which I have a problem with.
I will not code in native code ever again. Its not worth it anymore, its a horrible practice and it just enables bad code to pretend its good.
In my experience, Node.js generally performs very well for most use cases although when it comes to data manipulation (or similar use cases), performance often goes down the hill, becoming the primary bottleneck nearly every time. It's simply not the right tool for the job.
The beauty of this is that it allows teams to isolate and optimize just the problematic portion of the code, preventing them from going down a rabbit hole of extensive changes that can be more harmful than beneficial.
NAPI is such a good tool for this; that I've been advocating the use of it to my team since forever.
> In my experience, Node.js generally performs very well
Hard disagree here. Node is extremely fast to get up and running with, has as close to 0 barrier to entry as possible, and doesn't suffer from slow startup times that plague JVM languages. It can also be fast for tight loops.
But in the experience I've had with it"generally", it's orders of magnitude slower than other languages. A hello world with express.js and gin in go are a full order of magnitude apart, and the express app with default settings dies under a load test that I would describe as "very light".
Stories like this thread are common in the JS world where technically it can be rewritten to be faster, but like with Bun, esbuild, and many other examples we've seen here on HN, a better language choice makes an enormous difference. And to me, the only time node is the right tool for the job is if you have a team of JavaScript developers who can't learn another language
Depends what you're comparing it to though. That same Node.js application will likely be an order of magnitude faster than a similar one in Python or Ruby.
So, when it comes to a technology that works well, I think it should be easy to adopt for the whole team, Node.js won here even against Go. In my experience, Go seems to fit quite well there, although there are plenty of developers out there without Go skills.
But let's face it: In most of the projects we work on, clients don't need to handle millions of requests from distributed systems and geolocated servers all over the world. Generally, our clients just want a CRUD that is a little more elegant than Excel and can help them solve their real-world problems.
IMO if someone from your team tells you that you cannot do something with an average of ~10,000 req/s [1] (around that number is the consensus nowadays for Javascript whereas Bun is highly superior to everyone here), you may be solving the problem in the wrong way.
> the only time node is the right tool for the job is if you have a team of JavaScript developers who can't learn another language
I’d push back just a little on that. If you’re on a small team writing a small app, having the frontend and backend both be in js/ts, and in the same repo, can be a big help.
Yeah I made a very broad statement. I don't think the value in having your frontend and backend in the same language is there though. Your frontend _has_ to be js/ts, but your backend should be a language you know. If you've spent the last 10 years working with rails, then rails it is. If you know go, then go. If you know node, are on a small team with a tight timeline, then _maybe_, but honestly I think I'd still rather rails or go.
Leveraging Rust to speed Node.js is cool and all, but why the author didn't attempt to optimize the JS code first? If too much intermediate objects negatively impact memory consumption, surely re-using the same structure or variables to hold the parsed result would have improved the performance? The benchmark tests not only the speedup brought up by Rust, but an optimization too, so it's unclear which impacts the performance the most.
That approach may work in short term but in long term you'd likely end up with unreadable and brittle code that is often still not as fast as it would be after the rewrite. You'd introduce additional complexity into your product to overcome limitations of the platform. And tricks like reusing objects can turn into really nasty bugs, especially in languages that don't come with any tools to control aliasing.
If you want to reuse an object to avoid allocation, you better not have a reference to that object from somewhere else. In JS AFAIK there is no way to detect such situation automatically.
In languages like JS, Java, Python is trivial to accidentally make an object escape the scope by storing a reference in a field of another object that outlives the scope. And it can be another developer 3 months later that can introduce a bug, not knowing about your clever object reuse.
I mean I hear what you mean, but counterpoints; The hot path of code is allowed to be more involved than a naive implementation. You mention "unreadable", "brittle" and "complex" as if those are inevitable; I don't believe it, and again, it's worth the tradeoff, like dropping down to ASM in the hot path of C applications.
Another counterpoint: By switching to Rust, another language was added to the company's technology stack. This means that they now need to hire for Rust as well, if the author decides to leave. The simple graphics on https://boringtechnology.club/ are pretty interesting in that regard.
I don't know anything about this company, but the general advice I'd give is: be careful with adding languages, it adds an exponent to the complexity of your organization and hiring.
That's a fair point, but we really don't know if Rust was added to the company or only to this project. Big companies often use multiple tech stacks anyway.
In my experience the language was never a bottleneck at onboarding new people to the project. It is always the complexity of the project itself, all the non-standard in-house built things that engineers invented over previous X years and often didn't even take time to document properly.
I'd personally prefer a boring / standard solution in a relatively modern tech stack I have to learn from scratch than a pile of cleverness in something I know well (let's say Java). In the former case I can learn it quite quickly from the publicly available materials. In the latter case I need to reverse-engineer all the cleverness.
Side note: it might work in a refcount GC like CPython. That isn’t node though. (It only might because it doesn’t guarantee that memory is actually reused.)
You'd end up writing your own Rust-style split implementation and `&str` class and so on. You'd basically be implemented a crap version of Rust in JS. Surely it's better to just use Rust?
> Promise.all, isn't that delightful?). However, to our surprise, we soon ran into out-of-memory issues,
I think I must be misunderstanding something, but isn’t that exactly what you should expect?
We use node.js to process solar plant data. Which is massive amounts of data as each individual solar panel will log information about its power generation and status every few seconds. We can then use this to figure out which solar panels have a birds nest on them, and so on. Node performs this task well enough, but it obviously can’t be aggregated. Even if you were to spread it out on 25 node instances “concurrently” with control shift you’d run into out-of-memory issues extremely fast. I guess you could up the upper limit on when node.js stops being cool with its own memory usage, but honestly we tend to simply avoid the use promise aggregation exactly because it tends to ”break” node.
I think moving to Rust was probably a good idea for this. Go might’ve been a better option if you were going for a “modern” concurrent language, but whatever works! We tend to use C ourselves when node needs a little boost. But often Node can handle these things on its own. With the solar data we use worker threads to collect and process the data in real time, which is really parallel-execution and not concurrency, but it works well and it doesn’t get you into performance issues.
Out of curiosity, how do you use Node for this purpose? Is it building a real time dashboard?
My company has possibly similar challenges. We have lots of sensors on equipment that streams information every few ms through a time series database from the vendor. It's used to build monitoring dashboards, trigger alarms and help engineers to prioritize and optimize equipment settings, and look for trends over time.
It's such high volumes that capturing it and analyzing can be difficult, so we use a combination of vendor systems, Azure Synapse and Power BI.
There are still lots of opportunities to improve, and I'd be interested to hear how you landed on your setup and what you do with the data.
It’s mainly for data transformation and transport. Solar inverters don’t deliver data in the same formats, there are no protocols and apparently most engineers in the solar panel business still think FTP (not SFTP) is fine but some will deliver a file per reading while others will aggregate and so on.
We use node to gather the data and transform it into consumptive formats that our 3rd party AI vendor can work with, as well as store it in a range of SQL databases for our own PowerBI consumption. We do no frontend work.
We don’t use node because it’s a good idea to use node for this. We use node because we try to use TypeScript as much as possible because we’re a small team and having a single language covering as much of our stack as possible makes it easier to work together.
> Go might’ve been a better option if you were going for a “modern” concurrent language, but whatever works!
Go doesn't embed in other runtimes as nicely as Rust, nor it cannot call foreign code (including even C) as efficiently as Rust does. Rust strength is that it needs virtually no runtime and there is no magic needed to cross the ffi boundary.
The issue isn't Node. The issue is a poor understanding of memory requirements of each task and trying to do too many at once. You need to throttle your concurrency.
Why not just use the JVM and concurrency? I routinely parse 300+ GB of geospatial data without breaking a sweat using a very simple combination of concurrent linked queues and cached thread pools. I couldn't imagine diving head long into some new esoteric language just to do something as simple as what this author is describing. And things are only going to get easier with Virtual Threads...If I'm doing something boring (parsing data) I'd just want to use a boring standard language I guess (Java).
Java isn’t really something we want to work with. So I guess that’s the quick answer. There is nothing wrong with Java, I think projects like quarkus is pretty cool and the JVM is a battle tested machine. Our issue is more process related. We’re a small team so we try to use as few languages as possible, and since we can’t realistically avoid using TypeScript we tend to use it everywhere we can. Then when we need some extra juice we turn to C. We even PoC’ed Rust but decided it wasn’t “boring” enough for us yet.
Aside from that it can be hard to hire people for Java around here. It’s not a language a lot of people want to work with, and it’s common enough that the ones who do already have jobs.
> However, to our surprise, we soon ran into out-of-memory issues, even when using utilities like p-limit to limit ourselves to just two jobs in parallel.
If only two async jobs already run OOM, your issue is probably bad code, not node.js. From the code snippet posted, it looks like they do a `records.push(...)`, whereas it probably would've been more efficient to just continue using streams downstream as well. Additionally, using a transform stream to split the string rather than doing it in a loop might've yielded better results because the previous row could be cleaned up. Obviously, this is just poking in the dark, but such stark difference in numbers in the Rust implementation leads me to believe that they changed some other fundamental part of the design (e.g. streaming the results to the database rather than inserting everything in one big batch at the end).
> "The intriguing part here is that line and fields won't be cleared from memory until the garbage collector determines it's time to do so. We find ourselves with RAM filled with redundant strings and arrays that need cleaning. It's hardly surprising that our computer wasn't thrilled with this situation."
I don't have enough experience with super-optimizing Javascript, but wouldn't this be an easy solution? Instead of creating new variables in each loop and let the engine clear them, reuse the variables to avoid using more memory:
let line, fields, httpStatus;
for await (line of readlineStream) {
fields = line.split('\t');
httpStatus = Number(fields[5]);
if (httpStatus < 200 || httpStatus > 299) continue;
await dbWrite({
pathname: fields[7],
referrer: fields[8],
// ...
});
}
Edit: and as others suggest, keep streaming those records, instead of just pushing them into an array
No, this still allocates a new string array per each line.split call. It's not really a problem with variables. The variables are likely on the stack anyways, but they only store pointers to data on the heap. If you call any function that allocates data on the heap you cannot get away with that by moving the variable declarations.
> having 25 Node.js instances running for three hours to parse 200GB of data seems like an indication that something isn't quite right.
That’s about 700kb/s.
I’m all for playing with rust, but this post strikes me as engineers being distracted by shiny new things because they are only able to parse a TSV at 700kb/s?
You should be able to understand why your current system performs so absolutely terribly before you decide to rewrite it in any language.
Hand-waving at the garbage collector whilst seemingly doing very little to understand what’s happening doesn’t convince me that there isn’t another, possibly pretty obvious issue going on.
They debugged the problem to be excessive allocations in the tight loop of the TSV parser. If you allocate a separate object per each field, I can imagine how it can result in throughput counted in kB/s instead of GB/s.
We have had very similar issues with Java - excessive heap allocations can bring the system to a crawl and there is often no easy way out.
A common technique (at least in Java, never seen it in nodejs) is to have a pool of pre-allocated objects and just reuse them as they are discarded. Essentially manual GC
It is easier said than done, when the whole ecosystem is based on libraries and frameworks that allocate like crazy. Preallocating and reusing introduces complexity and increases likelihood of bugs. And at least in Java you can run into surprising GC troubles with pooling as well. Object pools tend to be promoted to a tenured generation and if you do it too much, you essentially break the generational hypothesis many GCs are optimized for. Generational GCs don't like dealing with objects with long lifetimes.
I recall hearing there's something of a parallel ecosystem of near-zero-allocation Java libraries, used in HFT among other things. I'm unsure why people use a GC'd language in the first place if they're going to go to those lengths to avoid it, but presumably there are good reasons (perhaps memory safety, before Rust came along)
You use it only for the one class you are allocating a million times per second and let the other ones be GC, often this technique is used within a single heavy function with a tight loop
Most of your application probably doesn't need it and the tight loop might not be big enough to be worth using JNI to write in a non-GC language (as well as maintaining another compiler stack for)
Ideally the runtime should allow you to manually manage memory when you need to, like some keyword that lets you manually keep track of an object lifetime. But as far as I know most GC languages don't have that
Object pooling is typically not a common pattern in Java these days outside of HFT stuff, and even in those cases it’s often not performance, but determinism that’s the issue. (Connection pooling is still common as with other expensive objects, just not plain object pooling.) Java’s garbage collectors are excellent if you cut with the grain.
That reminds me of how iOS could achieve fast scrolling of lists, which Facebook and Twitter struggled with for years because they tried to stick with HTML. Basically if you scroll down, a list element from above is moved to the bottom and has its contents replaced.
Anyway, I'm not sure if this is in Java already or is upcoming, but Java will finally add value objects, that is, stack variables instead of just making everything a heap object. For tight loops that will be much more efficient, since it can just reuse the same memory, stay within the cpu registers, etc. (I'm talking out of my ass btw from a superficial knowledge of Go and Java and half-remembered theory from college).
Virtual scrolling is a very common technique for rendering large tables or lists in HTML (and pretty much any other environment), and is by no means unique to iOS...
They didn't debug anything. They just stuck a job queue on it and then rewrote part of it in Rust. That's the point of my comment. Using their code in the post (gist here[1]):
Do we live in a world where engineers using a mature language like NodeJS with a pretty simple line-by-line file reading process and a string split think they hit a hard wall at 700kb/s, and the problem is not their bad code?
But is the code shitty? It looks like boring normal code to me. I'm not sure what point you're making here.
The point of the article is that you can write really boring normal code in NodeJS and it's really slow, or you can write really boring normal code in Rust and it's a lot faster, and it's not that hard to use the latter to replace hot paths in the former.
It's probably all the memory allocations. I know that when I do work like this, creating a huge array up front and assigning to that typically results in massive speedups.
I know basically nothing about Node though, so I could be completely wrong here.
Node.js has a decent JIT, or so I've heard. I have experience with the HotSpot JVM. For the JVM this code would absolutely stop allocating on the heap past 10k iterations of the loop. "Split a string then grab the parts in a hot loop" is ground stakes for JIT compilers.
If they are seeing any GC activity once the loop gets hot, something else is going very wrong.
Yep the Hotspot JIT will absolutely allocate on the stack and reuse memory allocations inside loops. It also has escape analysis and it will detect you don't need the array of strings after the iteration. In that case it usually just allocates the strings directly at their final locations on the record object.
> But is the code shitty? It looks like boring normal code to me. I'm not sure what point you're making here.
1. It should never be that stupidly slow
2. That should be obvious to anyone with a napkin and some math
3. Excessive allocations are highly unlikely to be the root cause of the slowness
4. Instead of debugging, they seemingly spent a lot of time putting it on Kubernetes, scaling it out to 25 instances and then finally rewriting a portion of it in Rust.
The whole thing smells of confusion and multiple issues being conflated together, with the rewrite in rust fixing these by virtue of deletion.
And if the root is was the GC, then some `--trace-gc` output in the post would be nice at the absolute minimum. Showing a for loop with a string split doesn't cut it.
Exactly. I'm not a super-expert in Node.js but worked a lot with it and also know a lot of internals.
And I can definitely say: IO speed is pretty good. To be honest, I've never come across a language with bad IO speed and I've seen quite a lot of languages.
I also did some benchmarks (like orf did) and speed is pretty good. Maybe Rust is a little bit better because it's nearer to syscalls than Node.js is but we're not talking about 760kb/s vs. 250mb/s.
200GB of logs and 25 instances means 8GB per instance.
Not sure what they're doing, but processing of 8GB taking 3h is like doing it with my 80286 processor ;-) Even with inserting stuff in MongoDB this seems way to slow. And they only replaced the file IO part with Rust...
I guess the records array is getting pretty huge which might result in swapping to disk. But then: don't let it happen, just flush it to the database or whatever. I believe the problem could have been solved with Node.js without issues.
The point the parent is making is that the boring normal code in NodeJS is not slow, at least not as presented in the article. The bottleneck is probably somewhere else. Allocation is usually not slow in a garbage collected language. Deallocation can be, but that is usually easy to check (for example with the --trace-gc flag, or running with profiling enabled and checking the results with chrome devtools). It would have been nice if they shared such analysis.
With the limited information provided in the article, there are other points that I would check before going as far as writing a module in another language. For example
> records.push({
They are collecting all the parsed rows in a single array. This is going to consume memory proportional to the size of the file, potentially in the same order of magnitude (depends on how much information is being collected and how much discarded).
In my experience, a common mistake in handling files too large for memory is to read the data faster than you process it, consequently running out of memory.
For example, maybe they have a loop reading lines and doing a bunch of async DB writes: your process will read at disk speed and write at database speed (relatively slower), and you'll run out of memory due to async tasks/context queuing up in memory.
Perhaps this is why it worked to split the file into 8GB units X 25 servers: it didn't blow their memory budget per server.
Try it with something a little bigger than what you used. Then you would have seen that the GC kicks in and performance goes down after about 2GB of processed data.
With my first try with bigger data it blew up with an out of heap memory error. The article mentions them having memory problems as well. They probably operated at the upper levels of available memory and even got into swapping and then 700kb/s is not surprising.
So yes, this absolutely looks like a memory allocation issue.
Yes, of course the unbounded array will cause you to run out of memory. That’s… not news, and it absolutely isn’t a “memory allocation issue” in the sense they are using it in the post.
Try modifying it to clear the array (a DB flush, if you will) every 100k lines.
That's what you get if you use the (simplified) example code that is in the article. It's also what you can derive from their description of the problem.
Now we don't have any Rust code to compare so we can't say how it's different. But everything points towards a memory allocation issue that is just not their in simple Rust code.
Following you in the abstract, but still a bit confused.
* "unbounded array" - are you referring to `records`?
* If so, is the memory of `records` too large? Or is it that `records[i].pathname` has a reference to `fields[7]`, AND v8 will retain all copies of `fields` until the loop is done?
2. The job splits strings into a format that is loaded into mongodb
3. There where OOM errors that went away with a smaller workload
None of this points to an inherent issue with NodeJS and the way it references memory.
The speed issue alone points to a different problem root cause, and the post is completely lacking in concrete analysis.
If v8 is being dumb and retains references to huge strings, then understanding this is the case and forcing a copy (and thus enabling the gc to reclaim data) is quicker and easier than any of their attempts at fixing the issue.
Splitting a small TSV file and loading it into mongo can be done with even the most memory inefficient language without issue.
"and the post is completely lacking in concrete analysis."
I do agree here, but I think it's telling that the rust solution is, without analysis, just better.
Is this the best developer or best case study? Perhaps not, but it is interesting in that little analysis was needed.
In the case where you have some constraint placed on you (running in the browser, company policy forbids rust e.g.), the analysis is worthwhile. Otherwise, assuming the fix didn't take long (relative to the requisite analysis and dev training), the decision to switch to a less shoot-yourself-in-the-foot language seems reasonable to me.
I think we can all agree that the blog article, while interesting and tells a story about the way the author solved a problem, does not have high enough rigor to serve as a benchmark for "don't do this, do this instead", or "this language is bad at this, and this one is better". Those types of statements would require much more rigor and A vs B comparisons where code A and code B are the same, just in different languages.
BUT, what I am interested in - if I'm using javascript to parse a large file line by line, plucking off specific fields like the example does, what can I try to see if it's more GC-friendly? Obviously the answer is always some flavor of "profile it" and "it depends". If the blog author could have gotten a big boost in performance by adding 10 lines of js code to their loop, that is something all readers of the article could benefit from.
I will play around with it and see what I can learn about the V8 GC. For reference: here is a talk about profiling the ruby gc, which helped me fix a memory leak at work (https://www.youtube.com/watch?v=kZcqyuPeDao).
There's a lot of other considerations when adding a new language. Questions come up such as:
* How long does it take to ramp up on the new code base (author notwithstanding as a user of Rust outside of the company)?
* If the author of the code is sick and there's a bug, or logic needs to be added, how hard would it be for someone else to do this work?
* What is the build story for the Rust code like? How does it integrate with existing CI?
* How do you store the Rust library? Some artifact store that Node can interact with?
Without more details, unless the company was already writing Rust, this doesn't seem like a reasonable choice at all. Doing some profiling and experimenting with allocation, along with comments as to why you're doing something that's not naive (e.g. appending to a large array) seems much cheaper than doing all this Rust work.
I've been in situations at $WORK where we were locked into using Python for certain code. Rewriting portions of it in Rust (if it needed running in the runtime) or Java (if it was across a service boundary) would have been enjoyable. But answering all those questions made us stick to Python. And surely, in most of those instances, we never really came back to those hot loops again so it never needed subsequent work. There were a few cases where we did find ourselves constantly trying to eke out more performance and in most of those cases a rewrite did end up going through, but those are rare and part of the craft of software engineering is to identify these areas and get ahead of rewrites early enough to not waste cycles.
> They just stuck a job queue on it and then rewrote part of it in Rust
IMO that's not an inappropriate response here. You can make this code in Node, but it's super-awkward trying to reduce string allocations in JavaScript. Much easier to rewrite in Rust where such things will be fast even with the naive code. For code such as this you can likely do a 1:1 port of the Node code and it will be 10x faster. Which is much better than getting yourself in twists trying to make Node fast at tasks it's not good at.
I took the code from their post, the one they highlighted as the issue, string allocations and all, and ran it. It was 1000x as fast as their self-reported runtime.
Does this really, really not scream “there is another issue” to you?
Do you really think, even with string allocations in a hot loop, NodeJS with its very optimised runtime would struggle to hit 1mb/s?
Naive Rust code is super-awkward compared to naive JS code. I doubt one can do 1:1 port of anything more complex than Hello World (even FizzBuzz may get you tripped up).
Not sure what you mean by naive rust, but rust has good webassembly integration. To give you some context, here's a post I made about rewriting 10k lines of js into rust: https://www.reddit.com/r/rust/comments/k3jy5g/i_rewrote_10k_... (I went on to rewrite the server from nodejs to rust too)
YMMV, but my Rust code ends up looking pretty much identical to my TypeScript code most of the time. And especially for code like this where there are no complex lifetimes.
I disagree. At my job I rewrote an environment variable injector module written in JavaScript into pure Rust. Not only was it straightforward, but I'm fairly certain I finished the Rust port faster than it would have taken to port it another language like Python or Go.
By the way the very same code in Java would absolutely rip through the files with no extra allocations of arrays once the JIT gets seriously involved. A big loop with temporary arrays in it that are used to populate the fields of a record every iteration is HotSpot C2's favourite thing.
Does node not similarly allocate on the stack in hot loops?
Did you actually test it? Some time ago someone at another forum made a similar claim that returning an Optional from a method in Java will be surely optimized as we well, but it turned out that it wasn't (https://pkolaczk.github.io/overhead-of-optional/).
"almost always [...] wisdom" - Is this sarcasm? Maybe you work in a domain where that is the case but in my experience it's almost always not memory allocation or GC. "Wisdom" if there is any comes from measuring things properly and finding out the hard way that it's not what you think it is.
I think it's true more often than not. Also note that I said memory and memory allocation e.g. allocating too much hurts your page tables and your L1 cache.
What field do you have in mind? If it's typically well optimized then fair enough but my bread and butter is compilers which are a pretty typical approximately zero compute, branchy pointer chasing type of workload.
I think many people on HN are used to throughput oriented horizontally scalable services, where GC is not particularly relevant because:
a) You don't care about pause times since latency isn't your key metric
b) You can almost always just throw more money at compute power to increase throughput
It's almost always the case in such a system that your bottleneck is on the databases that you're pushing data through, as your main approach of "scale horizontally" typically leads to those sorts of bottlenecks.
Forgive me for talking about single programs then.
Databases do typically have exactly the same memory problems though, again this time I mean memory locality rather than allocating it, obviously with the transactional elements on top.
> I’m all for playing with rust, but this post strikes me as engineers being distracted by shiny new things because they are only able to parse a TSV at 700kb/s?
They definitely just wanted to A.) Play with Rust, B.) Write a post about it.
This is the key line:
> As an intermediate Rustacean (I do have a fairly successful open-source project written in Rust, called fnm),
It's fine to say "I rewrote this code in Rust because I like Rust". We don't have to do a whole song and dance about why X needed a rewrite in Rust because X wasn't doing a good enough job. People do exactly this work in Node every day successfully. They just wanted to write Rust instead of optimizing some Javascript. _Which is fine_, just totally not newsworthy.
I agree, 700kb/s is just impossibly slow; I mean personally I wouldn't use node to do it, but with node you still should be orders faster. Something is not right.
Async programming is not easy. For example the following line
for await ...
will will process the whole file, then give you all lines at once.
So the problem is probably that all lines in all files are parsed at the time creating a bottleneck somewhere and ballooning memory usage. The solution is to use streams, pause the stream in order to parse the line, then continue the stream. And pipe the parsed result to stdout or a tcp stream. That will use minimal memory, and if the GC doesn't do it's job Node.js now supports manual GC. So you could probably parse the whole thing using only a few kilobytes of memory.
I don't believe that's true. Consider this infinite generator:
async function* infinite() {
while (1) {
yield 'example'
}
}
(async function () {
for await (const value of infinite()) {
console.log(value)
}
})()
If the generator was consumed all at once, this would never print (because the generator is infinite). `for await... of` should only consume a single step of the stream / generator at a time. It's just syntactic sugar for the usual calls to .next() and etc. See the docs here[0]
N.B. Assuming `readLineStream` in that for await loop is node's readline object, the code is iterating over an asyncIterator[1] which works exactly as what you describe in your solution. (AsyncIterators have a next method that returns a Promise of the next item)
Agree. Espacially, a TSV file is very easy to process in chunks, you don't need a specialized parser like with JSON where most libs just loads the whole thing.
700kb/s smell super fishy.
I can parse that with one Python VM and a few MB of RAM way faster. And node has a JIT.
Besides, let's imagine it would be effectively the case that Python and Node are too slow for this, that would be the perfect case to delegate the task to infra instead of introducing a whole new language and ecosystem plus a rewrite. E.G: a perfect use case for DuckDB.
Agreed that there's something else going on here, but it's worth noting that (C)Python uses reference counting (with periodic cycle breaking), rather than tracing GC. Not sure if that actually affects throughput in this scenario though
If you read an article with a terse description about the author struggling to ride a bike with 1 wheel missing, would you not assume a second wheel might help?
700KB/s is embarrassingly slow! Modern computers are really fast.
That's a couple of orders of magnitude slower than a WiFi transfer. 1000x slower than SSD read speeds. TSV is possible to parse at RAM speeds — 5000x faster.
I suspect that widespread use of scripting languages, and hosting on micro instances containerized like a turducken, with disks over network, set people's expectations pretty low. You don't need a cluster of 25 machines for simple tabulation of 200GB of data.
> Hand-waving at the garbage collector whilst seemingly doing very little to understand what’s happening doesn’t convince me that there isn’t another, possibly pretty obvious issue going on.
This is pure voodoo profiling. “The job is slow. I guess it’s. . . . here!”
Although the author has my sympathy - I spent a month rewriting a module, optimizing every line of code, squeezing every last bit of performance out to discover that the slowness was entirely due to a downstream SQL statement to which I had made a minor “innocuous” change.
You call it voodoo profiling, but isn't it kind of just being experienced?
Like, if you show me code I can probably tell you where the expensive bits are just by looking. I could be wrong, but it's usually not very hard. Complex runtimes make it a lot harder because then I have to reason through things like JIT and any implicit yielding, but even still.
So I'm pretty sympathetic to someone looking at a fairly small section of code, going "oh yeah, that's gonna create a lot of allocations", and then doing work to avoid those allocations. The alternative is that they write benchmarks and start profiling, which is actually quite hard to do well, and takes time. I'm not convinced that the author would have been better served by doing so here.
I'm inclined to agree - when there's a simple solution (easier more reliable language), use it, the right answer is not "oh you are doing it wrong just follow these contortions and you'll be fine".
Maybe your language that requires contortions is the problem.
I’m curious, have you done much profiling? And what tools have you used if you have? This isn’t a judgment against you, it’s a genuine question because I’m wondering if perhaps I really am just this inexperienced. But the one thing I’ve learned is I’m always very surprised by where the hotspots in my code come from. I always assume big allocations, or bad cache coherency, or O(1) loops when I could use a hash map will be the killers, but then it turns out it’s something else entirely.
Because of this, I’ve just learned to never trust my intuition when it comes to this. Compilers are crazy these days, and it’s very difficult to tell if something will be slow or if the compiler is going to clean it up behind the scenes and make it a non issue. Add to that JIT for certain languages and I don’t even know how you would begin to intuit where a hotspot would be.
Edit: And just for additional context, I’ve used Intel’s VTune profiler, Tracy, Optick, and written a few manual benchmarks. Using these tools isn’t always easy, but they give a lot of insights that I wouldn’t pick up on my own.
Oh yes, quite a bit of profiling, certainly. I've used tools like valgrind and cachegrind, perf, manual instrumentation, JMX, or even just `time`, as well as a dozen others I'm sure.
> But the one thing I’ve learned is I’m always very surprised by where the hotspots in my code come from.
Are you looking at small or large codebases? The code in the article is quite tiny. The major complexity in terms of understanding that code's performance is really just V8 specific stuff. As someone who only reads casually about V8 I'd be much less comfortable guessing about performance.
To be clear, there's plenty of times where I can't guess. Certain languages make it really hard, like if there's a lot of inheritance I can't "glance" at the code at all, I have to dig all around the hierarchies. Extremely annoying, I'll 100% have to benchmark in that sort of language.
> Compilers are crazy these days, and it’s very difficult to tell if something will be slow or if the compiler is going to clean it up behind the scenes and make it a non issue.
For sure, like I said, the big question here is V8. If this were in a system I'm more familiar with I'd have an easier time telling you which optimizations would or would not work. But even still it's not too hard to guess, sometimes. Knowledge of your runtime is the big thing, I think. For example, reusing an allocation might seem like a trivial optimization but allocations can actually have effects (like failing) so an optimization that removes one would be somewhat aggressive - the situations where that optimization will kick in is going to depend on your compiler/JIT.
As an example, it's really easy to find documentation about how escape analysis in the JVM works. There are a few rules and you can sometimes look at code and evaluate those rules in your head. You could use those similar rules for C# or Go or JS but you'd be risking that the rules are different in those languages.
> Because of this, I’ve just learned to never trust my intuition when it comes to this.
That's totally fine fwiw. I'm definitely pro "just benchmark it", but I will maintain that:
a) Benchmarking and profiling are hard. Getting a representative benchmark of your system under load is particularly difficult and a GC can have wildly different performance characteristics under load.
b) It's often not that hard to guess. The more complex the runtime, the more complex the system, the harder it is to guess.
> and I don’t even know how you would begin to intuit where a hotspot would be.
The short answer is you just learn about how a JIT works just like you learn about how a compiler works.
Just to reiterate, I'm very much pro "measure before optimizing". Like, very pro. But also, the article shows that guessing definitely works. And I think guessing is super effective and a fine tool to use at times. Perhaps I've overstated my ability to "just look at code and see the slow bits" - it's obviously very much dependent on the code itself, my familiarity with the underlying components such as the runtime or compiler, and perhaps a bit of luck.
Thanks for the clarification and detailed response! I think your final two points are probably the two things I was failing to take into consideration here. I’m used to trying to profile medium-ish systems (like 40-50k lines of code) so that was the context I was bringing into this. In regards to the example listed in the article, you are absolutely right about being able to intuit where the hot spots may come from because it’s such a small isolated piece of code.
I’ve still got a long way to go in learning more about profiling, especially when it comes to JITed stuff or things with a heavy runtime. But it’s always nice to hear about how other developers approach profiling (it can feel like black magic to me a lot of the time).
Thanks for the article on escape analysis too! I’ll have to check that out in a bit :)
The challenge with profiling is that you are getting some results but the interpretation of these results is extremely context dependent and possibly the issue is outside the context you are examining. It may be the tasks, it may be the glue between tasks and the graphs may not always tell the story. I find having some internal assumptions and being able to test them and building deeper mental model of the critical parts helps to flush out some culprits. Being able to at least eliminate some areas as "ok enough" can help to guide the search. It can be an iterative process where finding the right question and tool is more important than later looking at the data.
It's not usually that hard because the higher level the decision, the more impactful it is on performance and the less a compiler will be able to bail you out. That's why you should start optimizing from the highest level and work your way down until the performance is acceptable. Only once you're at the point of optimizing low-level decisions does profiling really start to make sense.
A silly example: it doesn't matter how much you optimize your bubble sort implementation, it'll lose out to the most naively written quicksort, which will lose out to keeping the data always sorted and inserting with binary search, which will lose out to finding out how to make the data inherently sorted, which will lose out to figuring out how to avoid the need for the data to be sorted in the first place. The fastest code is no code at all.
> You should be able to understand why your current system performs so absolutely terribly before you decide to rewrite it in any language.
This is the case if the rewrite would take more time than the time it takes to debug and fix the issue. If you already have evidence that this is just an issue with the runtime (ie: "I have a very hard time controlling allocations in this language and allocations are a significant part of the compute time") I don't think it's the wrong call to just pick a language where you have very high confidence that it'll be "just fast".
I've run into this with Python. I just didn't care to figure out what was going on that was making things slow. Rewriting the logic in Rust was not hard, so I did it, and things were fast. I look back now, in this moment, and absolutely don't give a shit about why the Python code was slow or if I could have made it fast.
I'm not a nodejs expert at all, but couldn't they just call the garbage collection manually? Also, wouldn't it be better to have the fields (and httpstatus) variable be declared above the for loop and just replace the contents of the fields variable instead of allocating a new one each time?
The thought process this blog seems to use is 'we have a lot of allocations and nodejs is garbage collected, therefore there is no alternative to using a different language'.
Calling it manually wouldn't stop it from being called on its own. Also there are two types of GC calls in node, one for the short lived memory and one for long lived memory (this is a generational gc) so you're going to have to understand where your memory allocations live if you want to do that. That's not too hard, thankfully, thanks to profilers.
But anyway, you'd have to first stop it from being called at all, then call it manually after. This seems very hard, I don't know how much Node would like this, especially as you would have concurrent calls to the GC potentially. There's a very scary difference between "turn off, turn back on" and "turn back on, turn off", which can easily happen if you have multiple concurrent calls to the GC. Imagine if you threw an exception after turning the GC off and never turned it back on? Oof.
I'd frankly prefer someone just uses Rust than starts to fiddle with GC at runtime in such a way.
> Also, wouldn't it be better to have the fields (and httpstatus) variable be declared above the for loop and just replace the contents of the fields variable instead of allocating a new one each time?
Maybe. It's possible that V8 will do that. But also you're adding extra lines to manage your buffer if you want to take control over that allocation. It's probably what I would do tbh, although the post makes it clear that just using Rust, where such management is trivial, is lower cost than expected thanks to the FFI library used.
> The thought process this blog seems to use is 'we have a lot of allocations and nodejs is garbage collected, therefore there is no alternative to using a different language'.
I think the main issue I have with the post is this line:
"Perhaps it's time to consider that JavaScript may be the problem."
Javascript isn't so much "the problem" as it is simply a tool that makes optimizing around allocations difficult. It's very hard to know when you're properly managing your memory or not, and you're subject to the whims of V8 to optimize things for you.
If the author had phrased this instead as "I'm sure there's some way that I could get V8 to optimize the allocations, or avoid them altogether, I'd have faster code - but the cost of doing that is far higher than simply using a language where these problems don't exist" I would have no real objections.
I'm convinced that most of the time when someone blames Garbage Collection, they're just not digging deeper into their code for a different strategy. It's like when you finally realize why you need database indexes, or the benefits of memoization. You will find it and it will all feel so obvious, and you will likely refactor everywhere else you made the same mistake, and be amazed how much better everything is when you take the time to learn why your performance has downgraded.
Chances are insanely low that the GC is actually your problem.
I agree in principal, but they first put it in k8s and then split it into a job queue. They even state that it took over a day to run at one point, implying a lengthy and prolonged attempt at fixing the issue.
If you’re working on something and it’s slow, and during some iteration cycle you find that changing some lines makes it fast, cool. No need to debug it or spend time digging into it.
And it’s good that it’s faster, the post screams “we don’t know how to debug these issues” rather than “we don’t care why it’s slow”.
I think we basically agree, some debugging and profiling was probably warranted here, since they were going to go through the effort of horizontal scaling.
For you perhaps but it’s a pretty massive change in the longer term, though. You now have Rust in your codebase. How many engineers on the team will be able to work on it? If you leave tomorrow will the code become a liability?
With the numbers the author is providing it sounds very much like the problem could be solved in Node, even if it means using an unconventional programming style and there would be a smaller maintenance overhead than reaching for Rust.
> How many engineers on the team will be able to work on it?
In my experience, basically all of them. Rust isn't that hard to learn on the job, it's maybe hard for some people to learn in isolation.
Obviously adding a new language requires understanding that there's going to need to be build system support, dev support, etc. I guess I assumed that was there? Unless Wiz is still in the 'wild west' phase where you can just do whatever you want.
I've drawn a line between learning Rust at a company and learning Rust in isolation. When you learn Rust at a company you have the full support of the others on your team, who already know Rust. I have experienced this at multiple companies now - picking up Rust when you are on a team of Rust developers is much easier than picking it up in isolation at home.
I learn new languages all the time. I make it a personal goal to never let not knowing a language get in the way of what I do.
Rust was one of the easiest I've ever picked up. There is a ton of good information, examples, books, and tooling out there that helps me.
On the other hand I've had to pick up Go several times in the past. It's never stuck. Basically every project I have seen written in it is convoluted and just messed up to read and interact with.
People ALWAYS overstate the effort that Rust takes. There are a lot of spots that you can get messed up on. But not all of us are out here writing code that hits these things. People act like the existence of lifetimes means you have to write them everywhere and think about it. That's not the case for almost any of us.
> Rust isn't that hard to learn on the job, it's maybe hard for some people to learn in isolation.
I have to disagree with that. I've learned Rust but it was one of the most challenging languages I've picked up. Stuff like lifetimes are very difficult to learn on the job without already doing a lot of reading.
Just to clarify, you were learning Rust while employed, working on a Rust project, with others on your team who were already familiar with the language?
In that scenario I have never seen onboarding to the language be a problem.
> You're trying very hard to claim that Rust is easy, that is very irrelevant to the topic.
I was responding to this:
> You now have Rust in your codebase. How many engineers on the team will be able to work on it? If you leave tomorrow will the code become a liability?
It's obviously relevant to that question.
> they did not say that,
I said that in my experience learning Rust at companies is easier than learning it in isolation.
They said that they had trouble.
I asked for clarification if they had trouble while learning in a work environment.
You jumped in to say that a work environment isn't relevant.
> like who cares.
Every other person in the thread who is discussing exactly that you absolute dunce
I'm all for rewriting things in Rust too but if it's a fundamental architectural problem that you just port over to the new language, you're essentially brute forcing it with an interpreted to compiled language change rather than solving the underlying issue. For example, running a O(n^2) operation rather than an O(n) one.
Strongly agree. It’s entirely possible that the problems are attributable to GC but there are many alternatives within Node-land that could help: streams? A library that stores data in a typed array rather than raw objects?
Adding an entirely new language and toolchain to your development process reeks of over engineering. Now your project needs a Rust dev on hand, what if you leave? Will there definitely be someone to pick up the slack?
No, I do claim that attitude is completely counterproductive.
This is not some case of bad pick of algorithms. The entire optimization work will happen through microoptimizations spread through the code.
If you insist on doing the microoptimizations on a language that is actively antagonistic to them, you will only create a lot of useless work, and have only mildly useful results to show for it, that will go away as soon as anything on your ecosystem changes (in JS that is measured in what? minutes?). And also, you will end up with an horrible codebase that's probably much less legible than what a lower level language will give you.
And that besides not having any good tooling or community support, because all the tools and the community are focused on developer productivity, not software performance, as that's the entire focus of the language.
I can parse a 200gb TSV with NodeJS in under a minute.
Could it be done in 1 millisecond with assembly? Probably.
But we are talking about over a days runtime here. That is not a micro optimisation, regardless of the language you’re using. You’re just using the language wrong, the same as if it took a day to parse with C.
Your attitude, that it’s pointless to even try and investigate performance issues because it’s written in JS rather than whatever language you see as optimal, is totally counterproductive.
> Could it be done in 1 millisecond with assembly? Probably.
I know you're probably just knowingly spewing hyperbole to make your point, but unless your L1 cache is 200GB memory bandwidth alone makes this impossible.
The original code should consume not much more memory than the optimized one. Yes you're going to allocate a lot of more memory, but whatever doesn't finish in the record becomes garbage after every single line, so I can expect worse performance, but not using two order of magnitude more memory.
I would see more problematic the fact that you're collecting all the records in a single array. Depending on how you use them it can be beneficial to use a generator and consume them incrementally.
Agreed in terms of live memory, but if it's purely measuring heap size then it makes more sense. So it possibly could run with a much stricter memory limit, but with the tradeoff of more GC (and for a compacting GC, this would mean lower throughout, since the GC time is proportional only to live objects. May as well let the garbage pile up as much as you can and therefore do fewer collections)
This is probably fixable in Node.js, just preallocate objects and do not excessively create new ones. But yeah using Node.js is probably not the best choice for this. As others have mentioned a JVM could easily handle this with some pre-allocation strategy.
I believe that the memory management aspect alone doesn't provide sufficient justification for rewriting the code in Rust. While it is possible to achieve reduced memory consumption through techniques like reusing a backing buffer and using allocUnsafe(), which require developers to take on memory management responsibilities, it might result in slightly more verbose code compared to a straightforward line-by-line split() approach.
I am eager to explore the possibility of enhancing the line parsing process using Node.js. If you could provide a runnable example on a GitHub repository, it would enable me to compare its memory efficiency with the Rust rewrite version.
I just skimmed through the article but it seems that they refactored from allocating and copying substring to pointing at the substrings within the original string (inside a hot path). So 25x performance boost should be expected just from that and has nothing to do with GC or the language used.
200gB in 2.5 hours is still only 22mB/s, I feel like I've seen blog posts of people going several orders of magnitude faster than that when the bottleneck is the sequential read of a big file on an SSD
I've written similar code before using absolutely bog standard idiomatic OOP Java. For a single threaded program I'd expect at least 1GB/s before any optimization. Most of the allocations in the loop that aren't the final records objects will get erased by JIT.
See in french we don't even have the possibility for that kind of confusion because we named a "byte" an "octet" which also reflects its relation with the number 8
That way they don't get gigabytes and gigabits abbreviated with the same letters and we don't have to rely on a capitalization convention
Honestly doing line.split('\t') seems a newbie mistake. I don't know how many columns each line has but it could be a bottleneck. Wouldn't manually processing tab and non-tab character help, in the same way you don't load the entire log into memory but only read by line? When you are dealing with huge amount of data, you have to be extra careful, regardless of which language you use. That plus other optimization could make Node.js run very efficiently.
Otherwise, I still need to be convinced that the "line" array is causing trouble but not anything else -- the article definitely doesn't provide detail about how they found this problem. It is almost funny that the author tried to put this on Kubernetes instead of even attempting to optimize the JS code. I wish I had that luxury and tell my boss I should use more resources instead of fixing my code.
Yep. I've written code like this before for small inputs or one-off scripts, but I always thought it was obvious that doing it like this would be slow and produce lots of garbage. It's very nice and readable code, but for the hundreds of GBs they mention, it's clearly not ideal
About Node.js running out of memory, I wonder if they tried setting the max-old-space-size flag to increase the memory limit.
Based on the descriptions, I wonder if the real problem was not Node.js but that all these solutions were implemented inside a constrained environment.
Before going to Rust, run `node --cpu-prof main.js`, and open the profile in chrome://inspect -> DevTools for Node -> Performance tab. It will tell you what's taking time and it may not be what you expect.
If they are using Docker anyways and thus able to use any command line tools they want, I can't help but wonder how fast 'awk' or even a Perl script would have given them what they need...
I have seen convoluted stuff done in JS when a few shell commands would have taken care of things, before.
167 comments
[ 4.4 ms ] story [ 255 ms ] threadThat project seems to be creating platform-specific DLL's, which I guess doesn't matter too much if it's wrapped up nicely in an npm package and you're deploying Docker files on Linux anyway? But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's.
Alternatively, writing very C-like code using a Uint8Array and not allocating in the loop might have fixed it too? But if you don't use JavaScript string libraries, it's not much like writing JavaScript anymore.
I've done a bit of data manipulation in node and while imo js is not the ideal language for the job, it can be very fast. Strings are hard though, I'll admit that
This is true, but N-API will generally perform better (sometimes significantly, usually in exactly the same CPU-bound scenarios where WASM will perform better than V8). And NAPI-RS is quite good at paving the happy path for multi-target packages.
If the database IO happens in Rust, the option with WebAssembly is fetch, meaning HTTP and the Atlas-only Data API. If the IO is done in JS, you pay some extra CPU time transferring UTF-16 strings over the border and copy them as UTF-8 in Rust.
If your constantly running out of memory you have a bad design.
This seems like a native code agenda type post which I have a problem with. I will not code in native code ever again. Its not worth it anymore, its a horrible practice and it just enables bad code to pretend its good.
The beauty of this is that it allows teams to isolate and optimize just the problematic portion of the code, preventing them from going down a rabbit hole of extensive changes that can be more harmful than beneficial.
NAPI is such a good tool for this; that I've been advocating the use of it to my team since forever.
Hard disagree here. Node is extremely fast to get up and running with, has as close to 0 barrier to entry as possible, and doesn't suffer from slow startup times that plague JVM languages. It can also be fast for tight loops.
But in the experience I've had with it"generally", it's orders of magnitude slower than other languages. A hello world with express.js and gin in go are a full order of magnitude apart, and the express app with default settings dies under a load test that I would describe as "very light".
Stories like this thread are common in the JS world where technically it can be rewritten to be faster, but like with Bun, esbuild, and many other examples we've seen here on HN, a better language choice makes an enormous difference. And to me, the only time node is the right tool for the job is if you have a team of JavaScript developers who can't learn another language
But let's face it: In most of the projects we work on, clients don't need to handle millions of requests from distributed systems and geolocated servers all over the world. Generally, our clients just want a CRUD that is a little more elegant than Excel and can help them solve their real-world problems.
IMO if someone from your team tells you that you cannot do something with an average of ~10,000 req/s [1] (around that number is the consensus nowadays for Javascript whereas Bun is highly superior to everyone here), you may be solving the problem in the wrong way.
[1] That's around 36 million requests per hour!
Java ecosystem now has AOT compilation.
Our golang app starts up, binds a port, signs the jwt and shuts down before our kotlin app even hits our code.
Then it would serve requests at the same speed, if not faster, as your Go app.
I’d push back just a little on that. If you’re on a small team writing a small app, having the frontend and backend both be in js/ts, and in the same repo, can be a big help.
Completely agree about the same repo though.
Another counterpoint: By switching to Rust, another language was added to the company's technology stack. This means that they now need to hire for Rust as well, if the author decides to leave. The simple graphics on https://boringtechnology.club/ are pretty interesting in that regard.
I don't know anything about this company, but the general advice I'd give is: be careful with adding languages, it adds an exponent to the complexity of your organization and hiring.
In my experience the language was never a bottleneck at onboarding new people to the project. It is always the complexity of the project itself, all the non-standard in-house built things that engineers invented over previous X years and often didn't even take time to document properly.
I'd personally prefer a boring / standard solution in a relatively modern tech stack I have to learn from scratch than a pile of cleverness in something I know well (let's say Java). In the former case I can learn it quite quickly from the publicly available materials. In the latter case I need to reverse-engineer all the cleverness.
I think I must be misunderstanding something, but isn’t that exactly what you should expect?
We use node.js to process solar plant data. Which is massive amounts of data as each individual solar panel will log information about its power generation and status every few seconds. We can then use this to figure out which solar panels have a birds nest on them, and so on. Node performs this task well enough, but it obviously can’t be aggregated. Even if you were to spread it out on 25 node instances “concurrently” with control shift you’d run into out-of-memory issues extremely fast. I guess you could up the upper limit on when node.js stops being cool with its own memory usage, but honestly we tend to simply avoid the use promise aggregation exactly because it tends to ”break” node.
I think moving to Rust was probably a good idea for this. Go might’ve been a better option if you were going for a “modern” concurrent language, but whatever works! We tend to use C ourselves when node needs a little boost. But often Node can handle these things on its own. With the solar data we use worker threads to collect and process the data in real time, which is really parallel-execution and not concurrency, but it works well and it doesn’t get you into performance issues.
Out of curiosity, how do you use Node for this purpose? Is it building a real time dashboard?
My company has possibly similar challenges. We have lots of sensors on equipment that streams information every few ms through a time series database from the vendor. It's used to build monitoring dashboards, trigger alarms and help engineers to prioritize and optimize equipment settings, and look for trends over time.
It's such high volumes that capturing it and analyzing can be difficult, so we use a combination of vendor systems, Azure Synapse and Power BI. There are still lots of opportunities to improve, and I'd be interested to hear how you landed on your setup and what you do with the data.
We use node to gather the data and transform it into consumptive formats that our 3rd party AI vendor can work with, as well as store it in a range of SQL databases for our own PowerBI consumption. We do no frontend work.
We don’t use node because it’s a good idea to use node for this. We use node because we try to use TypeScript as much as possible because we’re a small team and having a single language covering as much of our stack as possible makes it easier to work together.
Go doesn't embed in other runtimes as nicely as Rust, nor it cannot call foreign code (including even C) as efficiently as Rust does. Rust strength is that it needs virtually no runtime and there is no magic needed to cross the ffi boundary.
Aside from that it can be hard to hire people for Java around here. It’s not a language a lot of people want to work with, and it’s common enough that the ones who do already have jobs.
If only two async jobs already run OOM, your issue is probably bad code, not node.js. From the code snippet posted, it looks like they do a `records.push(...)`, whereas it probably would've been more efficient to just continue using streams downstream as well. Additionally, using a transform stream to split the string rather than doing it in a loop might've yielded better results because the previous row could be cleaned up. Obviously, this is just poking in the dark, but such stark difference in numbers in the Rust implementation leads me to believe that they changed some other fundamental part of the design (e.g. streaming the results to the database rather than inserting everything in one big batch at the end).
I don't have enough experience with super-optimizing Javascript, but wouldn't this be an easy solution? Instead of creating new variables in each loop and let the engine clear them, reuse the variables to avoid using more memory:
Edit: and as others suggest, keep streaming those records, instead of just pushing them into an arrayThat’s about 700kb/s.
I’m all for playing with rust, but this post strikes me as engineers being distracted by shiny new things because they are only able to parse a TSV at 700kb/s?
You should be able to understand why your current system performs so absolutely terribly before you decide to rewrite it in any language.
Hand-waving at the garbage collector whilst seemingly doing very little to understand what’s happening doesn’t convince me that there isn’t another, possibly pretty obvious issue going on.
We have had very similar issues with Java - excessive heap allocations can bring the system to a crawl and there is often no easy way out.
This channel is a real gem, show what talent really is: https://www.youtube.com/watch?v=a1L7k35EHIc
Most of your application probably doesn't need it and the tight loop might not be big enough to be worth using JNI to write in a non-GC language (as well as maintaining another compiler stack for)
Ideally the runtime should allow you to manually manage memory when you need to, like some keyword that lets you manually keep track of an object lifetime. But as far as I know most GC languages don't have that
Anyway, I'm not sure if this is in Java already or is upcoming, but Java will finally add value objects, that is, stack variables instead of just making everything a heap object. For tight loops that will be much more efficient, since it can just reuse the same memory, stay within the cpu registers, etc. (I'm talking out of my ass btw from a superficial knowledge of Go and Java and half-remembered theory from college).
1. https://gist.github.com/orf/92bc26381e3dc00c8e96af262768591b
The point of the article is that you can write really boring normal code in NodeJS and it's really slow, or you can write really boring normal code in Rust and it's a lot faster, and it's not that hard to use the latter to replace hot paths in the former.
I know basically nothing about Node though, so I could be completely wrong here.
If they are seeing any GC activity once the loop gets hot, something else is going very wrong.
1. It should never be that stupidly slow
2. That should be obvious to anyone with a napkin and some math
3. Excessive allocations are highly unlikely to be the root cause of the slowness
4. Instead of debugging, they seemingly spent a lot of time putting it on Kubernetes, scaling it out to 25 instances and then finally rewriting a portion of it in Rust.
The whole thing smells of confusion and multiple issues being conflated together, with the rewrite in rust fixing these by virtue of deletion.
And if the root is was the GC, then some `--trace-gc` output in the post would be nice at the absolute minimum. Showing a for loop with a string split doesn't cut it.
And I can definitely say: IO speed is pretty good. To be honest, I've never come across a language with bad IO speed and I've seen quite a lot of languages.
I also did some benchmarks (like orf did) and speed is pretty good. Maybe Rust is a little bit better because it's nearer to syscalls than Node.js is but we're not talking about 760kb/s vs. 250mb/s.
200GB of logs and 25 instances means 8GB per instance. Not sure what they're doing, but processing of 8GB taking 3h is like doing it with my 80286 processor ;-) Even with inserting stuff in MongoDB this seems way to slow. And they only replaced the file IO part with Rust...
I guess the records array is getting pretty huge which might result in swapping to disk. But then: don't let it happen, just flush it to the database or whatever. I believe the problem could have been solved with Node.js without issues.
With the limited information provided in the article, there are other points that I would check before going as far as writing a module in another language. For example
> records.push({
They are collecting all the parsed rows in a single array. This is going to consume memory proportional to the size of the file, potentially in the same order of magnitude (depends on how much information is being collected and how much discarded).
In my experience, a common mistake in handling files too large for memory is to read the data faster than you process it, consequently running out of memory.
For example, maybe they have a loop reading lines and doing a bunch of async DB writes: your process will read at disk speed and write at database speed (relatively slower), and you'll run out of memory due to async tasks/context queuing up in memory.
Perhaps this is why it worked to split the file into 8GB units X 25 servers: it didn't blow their memory budget per server.
With my first try with bigger data it blew up with an out of heap memory error. The article mentions them having memory problems as well. They probably operated at the upper levels of available memory and even got into swapping and then 700kb/s is not surprising.
So yes, this absolutely looks like a memory allocation issue.
Try modifying it to clear the array (a DB flush, if you will) every 100k lines.
Now we don't have any Rust code to compare so we can't say how it's different. But everything points towards a memory allocation issue that is just not their in simple Rust code.
* "unbounded array" - are you referring to `records`?
* If so, is the memory of `records` too large? Or is it that `records[i].pathname` has a reference to `fields[7]`, AND v8 will retain all copies of `fields` until the loop is done?
Do you think this code would be more GC-friendly?
```
let count = 0;
const MAX = 1000;
const flushedRecords = [];
for await (const line of readlineStream) {
}```
Or were you thinking something totally different?
1. It parses 200gb at an average rate of 700kb/s
2. The job splits strings into a format that is loaded into mongodb
3. There where OOM errors that went away with a smaller workload
None of this points to an inherent issue with NodeJS and the way it references memory.
The speed issue alone points to a different problem root cause, and the post is completely lacking in concrete analysis.
If v8 is being dumb and retains references to huge strings, then understanding this is the case and forcing a copy (and thus enabling the gc to reclaim data) is quicker and easier than any of their attempts at fixing the issue.
Splitting a small TSV file and loading it into mongo can be done with even the most memory inefficient language without issue.
I do agree here, but I think it's telling that the rust solution is, without analysis, just better.
Is this the best developer or best case study? Perhaps not, but it is interesting in that little analysis was needed.
In the case where you have some constraint placed on you (running in the browser, company policy forbids rust e.g.), the analysis is worthwhile. Otherwise, assuming the fix didn't take long (relative to the requisite analysis and dev training), the decision to switch to a less shoot-yourself-in-the-foot language seems reasonable to me.
BUT, what I am interested in - if I'm using javascript to parse a large file line by line, plucking off specific fields like the example does, what can I try to see if it's more GC-friendly? Obviously the answer is always some flavor of "profile it" and "it depends". If the blog author could have gotten a big boost in performance by adding 10 lines of js code to their loop, that is something all readers of the article could benefit from.
I will play around with it and see what I can learn about the V8 GC. For reference: here is a talk about profiling the ruby gc, which helped me fix a memory leak at work (https://www.youtube.com/watch?v=kZcqyuPeDao).
* How long does it take to ramp up on the new code base (author notwithstanding as a user of Rust outside of the company)?
* If the author of the code is sick and there's a bug, or logic needs to be added, how hard would it be for someone else to do this work?
* What is the build story for the Rust code like? How does it integrate with existing CI?
* How do you store the Rust library? Some artifact store that Node can interact with?
Without more details, unless the company was already writing Rust, this doesn't seem like a reasonable choice at all. Doing some profiling and experimenting with allocation, along with comments as to why you're doing something that's not naive (e.g. appending to a large array) seems much cheaper than doing all this Rust work.
I've been in situations at $WORK where we were locked into using Python for certain code. Rewriting portions of it in Rust (if it needed running in the runtime) or Java (if it was across a service boundary) would have been enjoyable. But answering all those questions made us stick to Python. And surely, in most of those instances, we never really came back to those hot loops again so it never needed subsequent work. There were a few cases where we did find ourselves constantly trying to eke out more performance and in most of those cases a rewrite did end up going through, but those are rare and part of the craft of software engineering is to identify these areas and get ahead of rewrites early enough to not waste cycles.
IMO that's not an inappropriate response here. You can make this code in Node, but it's super-awkward trying to reduce string allocations in JavaScript. Much easier to rewrite in Rust where such things will be fast even with the naive code. For code such as this you can likely do a 1:1 port of the Node code and it will be 10x faster. Which is much better than getting yourself in twists trying to make Node fast at tasks it's not good at.
Does this really, really not scream “there is another issue” to you?
Do you really think, even with string allocations in a hot loop, NodeJS with its very optimised runtime would struggle to hit 1mb/s?
Does node not similarly allocate on the stack in hot loops?
What field do you have in mind? If it's typically well optimized then fair enough but my bread and butter is compilers which are a pretty typical approximately zero compute, branchy pointer chasing type of workload.
a) You don't care about pause times since latency isn't your key metric
b) You can almost always just throw more money at compute power to increase throughput
It's almost always the case in such a system that your bottleneck is on the databases that you're pushing data through, as your main approach of "scale horizontally" typically leads to those sorts of bottlenecks.
Databases do typically have exactly the same memory problems though, again this time I mean memory locality rather than allocating it, obviously with the transactional elements on top.
They definitely just wanted to A.) Play with Rust, B.) Write a post about it.
This is the key line:
> As an intermediate Rustacean (I do have a fairly successful open-source project written in Rust, called fnm),
It's fine to say "I rewrote this code in Rust because I like Rust". We don't have to do a whole song and dance about why X needed a rewrite in Rust because X wasn't doing a good enough job. People do exactly this work in Node every day successfully. They just wanted to write Rust instead of optimizing some Javascript. _Which is fine_, just totally not newsworthy.
So the problem is probably that all lines in all files are parsed at the time creating a bottleneck somewhere and ballooning memory usage. The solution is to use streams, pause the stream in order to parse the line, then continue the stream. And pipe the parsed result to stdout or a tcp stream. That will use minimal memory, and if the GC doesn't do it's job Node.js now supports manual GC. So you could probably parse the whole thing using only a few kilobytes of memory.
[0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
[1] https://nodejs.org/api/readline.html#rlsymbolasynciterator
700kb/s smell super fishy.
I can parse that with one Python VM and a few MB of RAM way faster. And node has a JIT.
Besides, let's imagine it would be effectively the case that Python and Node are too slow for this, that would be the perfect case to delegate the task to infra instead of introducing a whole new language and ecosystem plus a rewrite. E.G: a perfect use case for DuckDB.
That's a couple of orders of magnitude slower than a WiFi transfer. 1000x slower than SSD read speeds. TSV is possible to parse at RAM speeds — 5000x faster.
I suspect that widespread use of scripting languages, and hosting on micro instances containerized like a turducken, with disks over network, set people's expectations pretty low. You don't need a cluster of 25 machines for simple tabulation of 200GB of data.
This is pure voodoo profiling. “The job is slow. I guess it’s. . . . here!”
Although the author has my sympathy - I spent a month rewriting a module, optimizing every line of code, squeezing every last bit of performance out to discover that the slowness was entirely due to a downstream SQL statement to which I had made a minor “innocuous” change.
Like, if you show me code I can probably tell you where the expensive bits are just by looking. I could be wrong, but it's usually not very hard. Complex runtimes make it a lot harder because then I have to reason through things like JIT and any implicit yielding, but even still.
So I'm pretty sympathetic to someone looking at a fairly small section of code, going "oh yeah, that's gonna create a lot of allocations", and then doing work to avoid those allocations. The alternative is that they write benchmarks and start profiling, which is actually quite hard to do well, and takes time. I'm not convinced that the author would have been better served by doing so here.
Maybe your language that requires contortions is the problem.
Because of this, I’ve just learned to never trust my intuition when it comes to this. Compilers are crazy these days, and it’s very difficult to tell if something will be slow or if the compiler is going to clean it up behind the scenes and make it a non issue. Add to that JIT for certain languages and I don’t even know how you would begin to intuit where a hotspot would be.
Edit: And just for additional context, I’ve used Intel’s VTune profiler, Tracy, Optick, and written a few manual benchmarks. Using these tools isn’t always easy, but they give a lot of insights that I wouldn’t pick up on my own.
> But the one thing I’ve learned is I’m always very surprised by where the hotspots in my code come from.
Are you looking at small or large codebases? The code in the article is quite tiny. The major complexity in terms of understanding that code's performance is really just V8 specific stuff. As someone who only reads casually about V8 I'd be much less comfortable guessing about performance.
To be clear, there's plenty of times where I can't guess. Certain languages make it really hard, like if there's a lot of inheritance I can't "glance" at the code at all, I have to dig all around the hierarchies. Extremely annoying, I'll 100% have to benchmark in that sort of language.
> Compilers are crazy these days, and it’s very difficult to tell if something will be slow or if the compiler is going to clean it up behind the scenes and make it a non issue.
For sure, like I said, the big question here is V8. If this were in a system I'm more familiar with I'd have an easier time telling you which optimizations would or would not work. But even still it's not too hard to guess, sometimes. Knowledge of your runtime is the big thing, I think. For example, reusing an allocation might seem like a trivial optimization but allocations can actually have effects (like failing) so an optimization that removes one would be somewhat aggressive - the situations where that optimization will kick in is going to depend on your compiler/JIT.
As an example, it's really easy to find documentation about how escape analysis in the JVM works. There are a few rules and you can sometimes look at code and evaluate those rules in your head. You could use those similar rules for C# or Go or JS but you'd be risking that the rules are different in those languages.
Example of an article that goes into some depth on JVM escape analysis and the implications: https://blogs.oracle.com/javamagazine/post/escape-analysis-i...
> Because of this, I’ve just learned to never trust my intuition when it comes to this.
That's totally fine fwiw. I'm definitely pro "just benchmark it", but I will maintain that:
a) Benchmarking and profiling are hard. Getting a representative benchmark of your system under load is particularly difficult and a GC can have wildly different performance characteristics under load.
b) It's often not that hard to guess. The more complex the runtime, the more complex the system, the harder it is to guess.
> and I don’t even know how you would begin to intuit where a hotspot would be.
The short answer is you just learn about how a JIT works just like you learn about how a compiler works.
Just to reiterate, I'm very much pro "measure before optimizing". Like, very pro. But also, the article shows that guessing definitely works. And I think guessing is super effective and a fine tool to use at times. Perhaps I've overstated my ability to "just look at code and see the slow bits" - it's obviously very much dependent on the code itself, my familiarity with the underlying components such as the runtime or compiler, and perhaps a bit of luck.
I’ve still got a long way to go in learning more about profiling, especially when it comes to JITed stuff or things with a heavy runtime. But it’s always nice to hear about how other developers approach profiling (it can feel like black magic to me a lot of the time).
Thanks for the article on escape analysis too! I’ll have to check that out in a bit :)
A silly example: it doesn't matter how much you optimize your bubble sort implementation, it'll lose out to the most naively written quicksort, which will lose out to keeping the data always sorted and inserting with binary search, which will lose out to finding out how to make the data inherently sorted, which will lose out to figuring out how to avoid the need for the data to be sorted in the first place. The fastest code is no code at all.
This is the case if the rewrite would take more time than the time it takes to debug and fix the issue. If you already have evidence that this is just an issue with the runtime (ie: "I have a very hard time controlling allocations in this language and allocations are a significant part of the compute time") I don't think it's the wrong call to just pick a language where you have very high confidence that it'll be "just fast".
I've run into this with Python. I just didn't care to figure out what was going on that was making things slow. Rewriting the logic in Rust was not hard, so I did it, and things were fast. I look back now, in this moment, and absolutely don't give a shit about why the Python code was slow or if I could have made it fast.
The thought process this blog seems to use is 'we have a lot of allocations and nodejs is garbage collected, therefore there is no alternative to using a different language'.
But anyway, you'd have to first stop it from being called at all, then call it manually after. This seems very hard, I don't know how much Node would like this, especially as you would have concurrent calls to the GC potentially. There's a very scary difference between "turn off, turn back on" and "turn back on, turn off", which can easily happen if you have multiple concurrent calls to the GC. Imagine if you threw an exception after turning the GC off and never turned it back on? Oof.
I'd frankly prefer someone just uses Rust than starts to fiddle with GC at runtime in such a way.
> Also, wouldn't it be better to have the fields (and httpstatus) variable be declared above the for loop and just replace the contents of the fields variable instead of allocating a new one each time?
Maybe. It's possible that V8 will do that. But also you're adding extra lines to manage your buffer if you want to take control over that allocation. It's probably what I would do tbh, although the post makes it clear that just using Rust, where such management is trivial, is lower cost than expected thanks to the FFI library used.
> The thought process this blog seems to use is 'we have a lot of allocations and nodejs is garbage collected, therefore there is no alternative to using a different language'.
I think the main issue I have with the post is this line:
"Perhaps it's time to consider that JavaScript may be the problem."
Javascript isn't so much "the problem" as it is simply a tool that makes optimizing around allocations difficult. It's very hard to know when you're properly managing your memory or not, and you're subject to the whims of V8 to optimize things for you.
If the author had phrased this instead as "I'm sure there's some way that I could get V8 to optimize the allocations, or avoid them altogether, I'd have faster code - but the cost of doing that is far higher than simply using a language where these problems don't exist" I would have no real objections.
Chances are insanely low that the GC is actually your problem.
If you’re working on something and it’s slow, and during some iteration cycle you find that changing some lines makes it fast, cool. No need to debug it or spend time digging into it.
And it’s good that it’s faster, the post screams “we don’t know how to debug these issues” rather than “we don’t care why it’s slow”.
For you perhaps but it’s a pretty massive change in the longer term, though. You now have Rust in your codebase. How many engineers on the team will be able to work on it? If you leave tomorrow will the code become a liability?
With the numbers the author is providing it sounds very much like the problem could be solved in Node, even if it means using an unconventional programming style and there would be a smaller maintenance overhead than reaching for Rust.
In my experience, basically all of them. Rust isn't that hard to learn on the job, it's maybe hard for some people to learn in isolation.
Obviously adding a new language requires understanding that there's going to need to be build system support, dev support, etc. I guess I assumed that was there? Unless Wiz is still in the 'wild west' phase where you can just do whatever you want.
I've learned Go in a day. Did the Go tour, and immediately felt comfortable writing it.
I've tried learning Rust multiple times, did tours, tutorials, even started reading the book - still don't feel comfortable writing it.
Rust is not a simple language. I say that, and my first language was C++.
Not guaranteed. Maybe it was that one guy who decided to introduce Rust to the codebase, then quit.
But I agree, generally, learning anything is easier when you have someone to ask.
I learn new languages all the time. I make it a personal goal to never let not knowing a language get in the way of what I do.
Rust was one of the easiest I've ever picked up. There is a ton of good information, examples, books, and tooling out there that helps me.
On the other hand I've had to pick up Go several times in the past. It's never stuck. Basically every project I have seen written in it is convoluted and just messed up to read and interact with.
People ALWAYS overstate the effort that Rust takes. There are a lot of spots that you can get messed up on. But not all of us are out here writing code that hits these things. People act like the existence of lifetimes means you have to write them everywhere and think about it. That's not the case for almost any of us.
I have to disagree with that. I've learned Rust but it was one of the most challenging languages I've picked up. Stuff like lifetimes are very difficult to learn on the job without already doing a lot of reading.
In that scenario I have never seen onboarding to the language be a problem.
> Just to clarify, you were learning Rust while employed
they did not say that, but you had to frame it that way to, again, claim that Rust is easy, like who cares.
I was responding to this:
> You now have Rust in your codebase. How many engineers on the team will be able to work on it? If you leave tomorrow will the code become a liability?
It's obviously relevant to that question.
> they did not say that,
I said that in my experience learning Rust at companies is easier than learning it in isolation.
They said that they had trouble.
I asked for clarification if they had trouble while learning in a work environment.
You jumped in to say that a work environment isn't relevant.
> like who cares.
Every other person in the thread who is discussing exactly that you absolute dunce
Adding an entirely new language and toolchain to your development process reeks of over engineering. Now your project needs a Rust dev on hand, what if you leave? Will there definitely be someone to pick up the slack?
This is not some case of bad pick of algorithms. The entire optimization work will happen through microoptimizations spread through the code.
If you insist on doing the microoptimizations on a language that is actively antagonistic to them, you will only create a lot of useless work, and have only mildly useful results to show for it, that will go away as soon as anything on your ecosystem changes (in JS that is measured in what? minutes?). And also, you will end up with an horrible codebase that's probably much less legible than what a lower level language will give you.
And that besides not having any good tooling or community support, because all the tools and the community are focused on developer productivity, not software performance, as that's the entire focus of the language.
Could it be done in 1 millisecond with assembly? Probably.
But we are talking about over a days runtime here. That is not a micro optimisation, regardless of the language you’re using. You’re just using the language wrong, the same as if it took a day to parse with C.
Your attitude, that it’s pointless to even try and investigate performance issues because it’s written in JS rather than whatever language you see as optimal, is totally counterproductive.
I know you're probably just knowingly spewing hyperbole to make your point, but unless your L1 cache is 200GB memory bandwidth alone makes this impossible.
The original code should consume not much more memory than the optimized one. Yes you're going to allocate a lot of more memory, but whatever doesn't finish in the record becomes garbage after every single line, so I can expect worse performance, but not using two order of magnitude more memory.
I would see more problematic the fact that you're collecting all the records in a single array. Depending on how you use them it can be beneficial to use a generator and consume them incrementally.
Also as others have mentioned the js code could have been optimized.
Id be very interested in a part two testing out some of the suggestions and sharing the benchmarks!
I am eager to explore the possibility of enhancing the line parsing process using Node.js. If you could provide a runnable example on a GitHub repository, it would enable me to compare its memory efficiency with the Rust rewrite version.
They are doing something very wrong.
They're doing something slightly wrong that's making a non-bottleneck part of their code less optimal than it should be.
That way they don't get gigabytes and gigabits abbreviated with the same letters and we don't have to rely on a capitalization convention
Join the enlightened gigaoctet convention
If this process was an on-demand pipeline they wouldn't have to deal with such a storm of data every day.
It would also mean people got stats 'closer-to-live' which feels like it would be useful.
Otherwise, I still need to be convinced that the "line" array is causing trouble but not anything else -- the article definitely doesn't provide detail about how they found this problem. It is almost funny that the author tried to put this on Kubernetes instead of even attempting to optimize the JS code. I wish I had that luxury and tell my boss I should use more resources instead of fixing my code.
Based on the descriptions, I wonder if the real problem was not Node.js but that all these solutions were implemented inside a constrained environment.
I have seen convoluted stuff done in JS when a few shell commands would have taken care of things, before.
How could this be written in node to use streams more effectively (at whatever stage necessary, parsing, reading, etc ..) to limit memory usage?