I haven't thought about that that much, but it looks like dsymutil is a companion command of a linker, so the answer is probably yes, but I need to investigate it further to give you a concrete answer.
The biggest reason is because it is multi-threaded. When building a program, the compilation step is parallelized (the build system invokes a compiler for each source file), but the final link step is not. So it is important to make the linker itself multi-threaded.
But even without multi-threading, mold is still faster than other linkers. I can think of various reasons why, but I don't know which attributes how much. I believe the biggest contributor is its efficient data structure -- it is hard to make program faster by writing fast code, but it can naturally be achieved by designing efficient data structures. That said, it is hard to compare two or more programs to find out why one program is faster than the others unless their designs are similar.
I wouldn't be surprised. In mold, if we have more than one choices to implement a feature, I always take the one that scales well for more cores even if it doesn't perform the best on low-core count machines.
My assumption is that future machines will have more cores than we have today on average, so I'm optimizing mold for such computers.
> My assumption is that future machines will have more cores than we have today on average, so I'm optimizing mold for such computers.
From the manpage of mold-1.3.0, I get the impression mold is designed to scale up to 32 cores, but not more.
man mold | rg -C2 32
--threads
--no-threads
Use multiple threads. By default, mold uses as many threads as the number of cores or 32, whichever is the smallest. The reason why it is capped to 32 is because mold doesn't scale well beyond that point. To use only one thread, pass --no-threads or --thread-count=1.
Is this correct or does the manpage need to be updated?
We do support ELF (Unix) already, and we are working on Mach-O (macOS/iOS/watchOS/etc) now. Once Mach-O is finished, we'll be working on PE/COFF (Windows).
Does mold support jobserver or some other mechanism to throttle threads?
We use lld currently but had to disable threading as sometimes in CI several of our test binaries would get linked at the same time. Whenever this happened the lld instances appear to have spawned enough threads to overload our Jenkins slave to the point that the master wasn't able to reach it and failed the build.
It's being discussed (https://github.com/rui314/mold/issues/117) but haven't reached any conclusion. The problem is that the jobserver protocol assumes that one process is one job, and its model doesn't fit very well to programs such as mold.
If you use Ninja, you can create a job pool for linking, separate from compiling, and restrict how many simultaneous linker jobs are run. You can even create and set job pools for Ninja through CMake, if you use that in your tooling.
Unfortunately GNU Make offers no such mechanism. And for Ninja build generation I don't think Meson does it either.
I want to write a book about linkers so that the knowledge I earned during the development of the lld and mold linkers wouldn't lost, but I don't have enough time to do that!
> I want to write a book about linkers so that the knowledge I earned during the development of the lld and mold linkers wouldn't lost, but I don't have enough time to do that!
I know this is a type, but it is interesting to see knowledge as a score or commodity you can "earn".
Your tweet shows quicker linking on a Mac yet you say down-thread that Mac executable linking is still not fully supported. So can we or can we not use it today on a Mac? (My main use-case is Rust.)
It can create Mac executables, but mold/macOS is still in pre-alpha and no one should expect it to work for their programs. Once it becomes out of beta, I'll release it as mold 2.0, so please wait for it.
That's an even crazier goal which is probably 100x harder than writing a 10x faster linker. But I believe it's technically doable. At least, the world needs more crazy people who believe it is technically doable and take it as a challenge. If I get $$$ by selling the mold project to a big tech, I might be able to create a team with that money to tackle that crazy goal...
Looking further into the future and the advent of io_uring in Linux, would you consider special-casing Linux IO ops to use io_uring or do you not expect any speedup there?
I'm not sure if io_uring can improve mold's performance, as it has to access random locations while copying file contents to apply relocations. Currently, we mmap all input files and an output file and use memcpy to copy file contents.
If it takes 20 seconds, you go for a coffee or switch the window to start web browsing, but if it takes only 2 seconds, you can wait without being distracted. So it's more than just saving 18 seconds for each build. It wouldn't interrupt your flow.
Jesus. What product? The only thing I've ever worked on that got close to that was a compiler that built LLVM and Boost. LLVM takes like 40 minutes on a laptop.
Apple and Microsoft should really throw some money at Rui; Mold is a massive improvement to developer experience, and the sooner it's ported to their platforms the better.
I find myself defaulting to Linux for Rust development these days, mostly because Mold makes the inner loop of development so much faster.
Rui has made it explicit that a commercial license is available. Hey Rui, you should replace "please call" with a dollar amount in order to increase your conversion rate ;)
I dislike "contact us for pricing" too, but pricing is hard! I don't know how much I should ask for. This is also a bit unusual business model because mold can be used for free. I'm happy if users pay me 10%-20% of the total cost they could save by using mold though.
$200K is nowadays an annual total compensation of a junior dev. I believe mold values much more than a one year outcome of a typical 5-person junior dev team.
I am sorry I was not trying to undermine the value of Mold. I was told the average / median of Junior Dev are't even $200K in FAANG. But then I dont work in the US so I dont know.
( I hope more US companies are opened to truly remote )
I didn't take it as an offense, so don't say sorry! I just wanted to say that if a senior dev wants to earn $1M in a few years, they can just find a job and do whatever assigned to them there. They didn't have to take a risk of starting a completely new ambitious open-source project to earn that much.
In my country, $200k is what a senior dev makes in 10 years. The issue is, I very much doubt people like Rui could be expected to live in places like mine. So comparing against US salaries is probably what you should do anyway.
I'm curious - is that salary difference enough to cause a significant brain drain? Can you hire really good developers for that price in your country?
I live in the UK and dev salaries are maybe half of the US, but people don't leave in droves because there are other factors - family, friends, quality of life, effort of moving your life across continents, etc.
Sure. Significant brain drain might be an understatement — I don't think there are any excellent developers left here (if there were any at all). Anyone half worth their salt either moves to Russia where there was a decent IT sector up until the recent events, or wins a green card/marriage lottery and leaves for the West. FWIW I'm not leaving because I didn't want to live in Russia even before the war started (and even less so now), and I'm not good enough to emigrate to the West.
Not even in most of the U.S. I’ve never broken that salary in the Midwest and I’ve gone from Junior to SWE to Senior to Lead to Manager to Director. Even in total compensation it took me until Director (managing managers who in turn manage individual contributors) to reach that level.
Please give a list of companies (Europe/Japan) where I can apply for such a salary. I started working recently and I'm at more than 10 times less than this...
I don’t know much about the European job market, but you can see lots of examples of US job positions and their salaries at websites such as https://www.levels.fyi.
I was joking and I don't think you need a price on there. But if you're serious, you can put out an asking price and everyone involved should just assume it's negotiable. Just use something like a couple of million USD as a starting point.
> This is also a bit unusual business model because mold can be used for free.
You're selling the copyright, right? That's ok - businesses pay for stuff like this. Although I wonder if instead you'd be better off if you created a corporation and instead offered to sell the company's IP + your commitment to support the work over ~1-2 years. This might be a more common scenario for M&A teams to work with. Especially if you are keen on supporting more target architectures/OSs. Someone like ARM or SiFive or FAANG would easily shell out that kind of money to get mold.
Build times are a huge issue impacting how fast I can develop at my job (and I'm fairly certain that's the norm for my team), and I suspect that using a faster linker would mitigate that by a significant chunk. Unfortunately I'm just an IC and don't have the authority to look into something like paying for this, and I'd likely need to actually try it out and show numbers for how much things would improve to get management interested in actually looking into this. Because the project I work on requires interfacing with some cloud services that I can't connect to locally, I would either need to actually build on one of our cloud servers or compile the entire thing locally and copy it over, but I imagine the performance gains from multithreading would not be nearly as great on the 8 cores on my laptop compared to the 64 on my cloud server. I'm a bit hesitant to take initiative and just try this out due to the AGPL license though, since IANAL and I imagine if I tried to get permission for this internally, I'd run into the circular issue that without data to show it's worth it to look into, I'd probably just be told not to do it.
Basically, I think your best chance of generating interest from this is the bottom-up (since the individual engineers are the ones who would be feeling the pain that this could help solve), but I'm not really sure ICs at the companies large enough to be potential customers for this have any likely path forward with the way you've structured things right now. I'm not sure if you're flexible on this at all, and obviously I can't guarantee we'd get anywhere, but if you're interested in hearing more details about the potential use my team (and probably a number of other teams at my company would be able to make use if we were able to work something out), feel free to email me! Any prefix @<my username>.com will forward to my gmail.
It's not really something you can "link" to tho? But the basic loop of development in most cases (in compiled languages) is making a code change, then building the software (or tests), and then running the software (or tests) to check that your change did what it was supposed to do.
Building the software is divided into two steps; compiling your source files into object files, then linking those object files together into an executable. As a project grows, the time taken by the compile step in this development loop stays roughly constant; if you only change a single source file, only that source file has to be recompiled. However, the link step has to link together all your object files, from scratch, every time, so the time taken by the link step grows roughly linearly as the project size grows.
Since mainstream linkers are fairly slows, an incremental build of huge projects is generally dominated by the link step. I've personally experienced the pain of making a small change to Chromium, waiting a second or so for the source file I changed to be recompiled, then waiting a minute or two for the linker to go through every single object file and produce the final executable. Mold would have reduced the build time in the development loop from minutes to seconds.
The Apple linker also seems to have a habit of randomly changing it's mind about certain niggly details about MachO binaries which is extremely irritating as a compiler dev
I can't remember the details (I'm stuck on my phone but I'll try and dig it up later if I remember) but we had some issues with certain sections not being to it's liking so code that was fine on one xcode version then broke on upgrade.
Similarly I had to replace how we implement C runtime destructors because Apple deprecated the way to write a destructor in a binary. This wasn't too hard to do, just call atexit from the itanium C++ ABI, but we had basically no warning as far as I'm aware.
It has an introduction section on what is a linker and why you need one. The gist is, linker is the piece of software that puts together different source file, be it at compile time or runtime. So, the faster is your linker the quicker the app compiles and opens.
To assemble executable code from separately compiled files. Or to tell your executable code from where other executable code it needs can be loaded at runtime.
John R. Levine’s book Linkers and Loaders remains the only comprehensive coverage on the topic. You can get the manuscript proofs for free on the author’s website:
Individual compilation units (ex: .o files built from .c files) hold references (ex: from .h files) to other compilation units. In order to assemble a full executable, a linker takes all these individual units and links them together.
mold/macOS aims to be a drop-in replacement for the Apple's ld. I don't think it supports all the feature you mentioned already, as it is still pre-Alpha. But we are working on it to make it "just work".
>> Why is time spent on System is so much higher than the other one even though that one takes longer?
> Good question but I don't know the answer.
Don’t know if that’s the final answer here (didn’t actually investigate), but system is the time spent in syscalls, and locks are usually provided by the OS, aka syscalls (though there might be userland components to avoid the syscall e.g. futex).
Thus the need for synchronisation of threaded program generally leads to higher system time. Though here mold is clearly a lot more efficient as well (lower user time).
I’ve always been more attracted to the higher level languages and followed llvm from a compiler interest. So I know the principles, but I would like to understand more. Especially related to cross compilation.
Does someone have a source explaining the state of the art? With all the different compilers, gcc, clang and the different flavours of linkers: ld, lld, mold, gold, ?
Well at a minimum it has to load into RAM all the built objects, which probably include debugging symbols. For something like Chrome, thats probably 20 Gigs...
Linkers can also do something called "Identical code folding", or ICF, whereby the linker notices that two pieces of code are exactly the same and can merge them. lld's sources include a little overview of how this is done, see https://github.com/llvm-mirror/lld/blob/master/ELF/ICF.cpp
ICF exists but no linker is going to silently do it behind your back without an explicit directive, because it breaks debugging in certain ways. Folded identical functions can't be disambiguated in the file/line tables, so symbolized backtraces may contain impossible calls.
> CF exists but no linker is going to silently do it behind your back without an explicit directive,
Those explicit directives might be more implicit than you think. A linker will likely fold functions declared as inline. Template functions and template classes are implicitly inline, so for example, all uses of std::vector<std::string> will (likely) be implicitly folded together.
Not if they're compiled by separate invocations of the compiler. If a.cpp and b.cpp both have a vector<string> and are compiled independently, the first tool that actually gets to see them both is likely a linker (or archiver but that's a glorified zip tool)
OK but removing functions that are identical and that have the same name is not "identical code folding". ICF is removing functions that have identical bodies and different names.
No, outputs are generally smaller than inputs, and for large programs they are much smaller than the inputs. The .o files contain all the data necessary to put the program together and after linking most of that information is no longer needed. I just built a small program I happen to have locally and the constituent .o files add up to 580KiB but the linked program is 208KiB. Another small program has 192KiB of linker inputs and 124KiB of output. This effect is larger for large programs.
I can infer from context that mold is a linker. It took searching for https://github.com/rui314/mold to learn what it actually was, though:
"mold is a faster drop-in replacement for existing Unix linkers. It is several times faster than the LLVM lld linker, the second-fastest open-source linker which I originally created a few years ago. mold is designed to increase developer productivity by reducing build time, especially in rapid debug-edit-rebuild cycles."
I don't understand how things like this still manage to front-page without a cursory explanation.
I think yn is a mature enough site that stories are placed by the site curators and twitter users with connections to the site moderators are part of the curation process. There have been a lot more twitter posts as new submissions this past year or two.
Also, thank you for finding out what the topic discussion was about and sharing. I couldn't infer the meaning.
Some posts get picked for a second-chance by moderators - see https://news.ycombinator.com/item?id=26998308 - but it doesn't have any connection to twitterers, nor to 'connections to the site moderators', except in the sense that we recognize some usernames as good submitters.
The intention is just to comb through the stacks looking for good stories that didn't get attention the first time around—especially on topics that aren't correlated with anything else. HN has a wealth of these and I'm pretty sure we still miss half of them.
119 comments
[ 2.8 ms ] story [ 173 ms ] threadIf not, any plans to support this in the future? :)
But even without multi-threading, mold is still faster than other linkers. I can think of various reasons why, but I don't know which attributes how much. I believe the biggest contributor is its efficient data structure -- it is hard to make program faster by writing fast code, but it can naturally be achieved by designing efficient data structures. That said, it is hard to compare two or more programs to find out why one program is faster than the others unless their designs are similar.
My assumption is that future machines will have more cores than we have today on average, so I'm optimizing mold for such computers.
From the manpage of mold-1.3.0, I get the impression mold is designed to scale up to 32 cores, but not more.
Is this correct or does the manpage need to be updated?Thanks for the great write up as well as mold itself!
[1] https://github.com/rui314/mold/issues/181
We use lld currently but had to disable threading as sometimes in CI several of our test binaries would get linked at the same time. Whenever this happened the lld instances appear to have spawned enough threads to overload our Jenkins slave to the point that the master wasn't able to reach it and failed the build.
Unfortunately GNU Make offers no such mechanism. And for Ninja build generation I don't think Meson does it either.
Thanks for your good work! I still remember the O(n^2) complexity of ld.bfd when linking C++ code.
I know this is a type, but it is interesting to see knowledge as a score or commodity you can "earn".
( ͡° ͜ʖ ͡°)
I find myself defaulting to Linux for Rust development these days, mostly because Mold makes the inner loop of development so much faster.
( I hope more US companies are opened to truly remote )
*in the US.
In my country, those salaries are completely unheard of.
I live in the UK and dev salaries are maybe half of the US, but people don't leave in droves because there are other factors - family, friends, quality of life, effort of moving your life across continents, etc.
Look at Sidekiq for inspiration on how to charge for open source.
> This is also a bit unusual business model because mold can be used for free.
You're selling the copyright, right? That's ok - businesses pay for stuff like this. Although I wonder if instead you'd be better off if you created a corporation and instead offered to sell the company's IP + your commitment to support the work over ~1-2 years. This might be a more common scenario for M&A teams to work with. Especially if you are keen on supporting more target architectures/OSs. Someone like ARM or SiFive or FAANG would easily shell out that kind of money to get mold.
Basically, I think your best chance of generating interest from this is the bottom-up (since the individual engineers are the ones who would be feeling the pain that this could help solve), but I'm not really sure ICs at the companies large enough to be potential customers for this have any likely path forward with the way you've structured things right now. I'm not sure if you're flexible on this at all, and obviously I can't guarantee we'd get anywhere, but if you're interested in hearing more details about the potential use my team (and probably a number of other teams at my company would be able to make use if we were able to work something out), feel free to email me! Any prefix @<my username>.com will forward to my gmail.
Building the software is divided into two steps; compiling your source files into object files, then linking those object files together into an executable. As a project grows, the time taken by the compile step in this development loop stays roughly constant; if you only change a single source file, only that source file has to be recompiled. However, the link step has to link together all your object files, from scratch, every time, so the time taken by the link step grows roughly linearly as the project size grows.
Since mainstream linkers are fairly slows, an incremental build of huge projects is generally dominated by the link step. I've personally experienced the pain of making a small change to Chromium, waiting a second or so for the source file I changed to be recompiled, then waiting a minute or two for the linker to go through every single object file and produce the final executable. Mold would have reduced the build time in the development loop from minutes to seconds.
I hope that helped.
Similarly I had to replace how we implement C runtime destructors because Apple deprecated the way to write a destructor in a binary. This wasn't too hard to do, just call atexit from the itanium C++ ABI, but we had basically no warning as far as I'm aware.
https://github.com/rui314/mold#why-does-the-speed-of-linking...
It has an introduction section on what is a linker and why you need one. The gist is, linker is the piece of software that puts together different source file, be it at compile time or runtime. So, the faster is your linker the quicker the app compiles and opens.
https://www.iecc.com/linker/
The Solaris Linkers and Libraries docs are very good and mostly relevant to Linux, with fewer distractions about Windows:
https://docs.oracle.com/cd/E37838_01/html/E36783/index.html
> Good question but I don't know the answer.
Don’t know if that’s the final answer here (didn’t actually investigate), but system is the time spent in syscalls, and locks are usually provided by the OS, aka syscalls (though there might be userland components to avoid the syscall e.g. futex).
Thus the need for synchronisation of threaded program generally leads to higher system time. Though here mold is clearly a lot more efficient as well (lower user time).
Does someone have a source explaining the state of the art? With all the different compilers, gcc, clang and the different flavours of linkers: ld, lld, mold, gold, ?
No. Linkers today even do link-time code generation. Debug info is gargantuan and can be kept as separate files too.
Those explicit directives might be more implicit than you think. A linker will likely fold functions declared as inline. Template functions and template classes are implicitly inline, so for example, all uses of std::vector<std::string> will (likely) be implicitly folded together.
(Or are you assuming that a debug flag is used so that all inlined functions are not really inlined by the compiler?)
"mold is a faster drop-in replacement for existing Unix linkers. It is several times faster than the LLVM lld linker, the second-fastest open-source linker which I originally created a few years ago. mold is designed to increase developer productivity by reducing build time, especially in rapid debug-edit-rebuild cycles."
I don't understand how things like this still manage to front-page without a cursory explanation.
Also, thank you for finding out what the topic discussion was about and sharing. I couldn't infer the meaning.
The intention is just to comb through the stacks looking for good stories that didn't get attention the first time around—especially on topics that aren't correlated with anything else. HN has a wealth of these and I'm pretty sure we still miss half of them.