> I installed Xcode 15 Beta to try out the new Apple's linker. It seems that my mold/sold linker is still a bit faster than theirs, but the gap has now reduced to ~60% when linking the sold linker itself on my Mac Studio. Apple's new linker is in the same league as mine. Great job!
At $8/user/month (while I realize that still likely be pocket change to Apple)... how much is that and what is the risk that the company changes the licensing once you're using it?
Large companies (Apple, Amazon, and the like) would likely buy the company rather than license their product (see Dark Sky (now WeatherKit), Kiva (now Amazon Robotics - https://www.fromscratchradio.org/show/mick-mountz 23:22 )) so that they can't lose control over something that they want when management for one of there tools changes.
And so why use mold (license problems and they wouldn't want to impose the AGPL on all their developers), or be a reseller of sold ($8/month/user - how many developers are in the Apple developer program) - it comes down to "it is better for us to develop our own and release it so that there is certainty about the license for any developers who use it."
Apple doesn't even touch the GPL if it can avoid it (thus why its software is mostly based on BSD licensed code).
It would even be further beyond the pale to say "here is a linker for 3rd party developers to use... oh, and its AGPL, sorry 'bout that."
So to the question from the GP:
> Why didn't Apple just use mold?
Because they won't tell 3rd party developers to use AGPl software or require a 3rd party developer to use a different 3rd party developer's licensed software that may change.
If Apple is going to release something, they're going to develop it and release it themselves with as few other encumbrances as possible.
They don’t touch GPLv3 code; they avoid GPLv2 code because it can be relicensed GPLv3. That’s why they are shipping ancient bash and switching to another shell, for instance.
There's software that could be used if public builds were used, and contributions were made in the open, but lawyers and security guys don't really like the risk that comes from using something you didn't built.
If Apple was releasing software as part of the Xcode package, would AGPL code be acceptable to include in that package as libraries for 3rd party developers to use?
Sorry, I was talking about software used by FAANG employees (sorry Meta, I hate your new name, not sorry really).
Now, thinking about that case, I guess that Apple could make it easy for people to depend on AGPL software that they download on their own, and that Apple only provides some sort of index or crawlers for it. No need for Apple to have AGPL software on their end.
And scaring those corps into buying instead of complying was explicitly the author’s reason[1] for choosing AGPL instead of GPL even though there was no expectation the difference would ever actually matter.
I remember an SFC blog post[2] expressing distaste for such scare-tactics applications of copyleft on the part of MySQL AB in the early 2000s. (As opposed to the initial goal of copyleft, that is using a legal hack to eventually force most software to become free/libre.) This case is maybe less unpleasant, but only marginally so, I think.
Hostile licensing. mold, like many other ill-informed projects lately, is pseudo-open source aimed at being locked up as a commercial product with a license that is incompatible with free use in the real world.
A related question might be why Apple didn't make a business offer to Ueyama-san to improve mold/sold further, e.g. by offering him a position or by buying his company. But my guess is Apple want all the development done in-house.
I think this "Apple's new linker is in the same league as mine. Great job!" response is full of tact and grace, considering Apple just undercut his business.
Can anyone recommend a resource on how linkers work other than the "Linkers and Loaders" book? Maybe a linker repo that is easier to read through, or a more modern book or series of articles on the topic?
No shame on the book, it is a classic, but it is certainly a bit long in the tooth.
I am very interested in linkers, but find them a bit inaccessible to learn about.
> Hopefully this will support more architectures in the future, that's pretty impressive
What architectures would it support? I assume it handles aarch64. It may already handle x86-64. macOS and iOS don't run on any others. I doubt Apple would be interested in making it support other OSs that run on more architectures.
(It's probably a faux pax of me to say this, but I wonder if the parent comment was written by a LLM...)
Interesting point. Maybe. On the other hand, afaik they're just doing risc-v on tiny things like security enclaves and they might have a right different toolchain for those.
> I wonder if the parent comment was written by a LLM.
That's pretty disrespectful, I expect an apology from you
> I doubt Apple would be interested in making it support other OSs that run on more architectures
I don't think Apple is only using ARM and their chip for their whole stack, in fact, their servers run something else, and they support software that runs on other OSes wich run plenty of different architectures, including X86_64, and they are exploring a plethora of architectures [1]
> I assume it handles
> It may already handle
"Faux pas" indeed, let's stop assuming and let's stay positive, nice and respectful to each other
I'll keep my suspicions to myself next time. Folks are increasingly going to wonder this when there's a short/vague comment that doesn't quite match their understanding of the world. I wouldn't have wondered the same with the additional context you've supplied here, e.g.:
> I don't think Apple is only using ARM and their chip for their whole stack, in fact, their servers run something else, and they support software that runs on other OSes wich run plenty of different architectures, including X86_64, and they are exploring a plethora of architectures [1]
Now I understand why you suggested it might run on other architectures / OSs. I'm skeptical though. They may develop/run software on Linux/Windows, but I haven't seen any examples of them releasing a development toolchain (e.g. any part of Xcode) for those OSs (with the obvious exception of platform-independent LLVM improvements). Likewise microprocessor-level stuff (afaict everything they do with risc-v).
In my dream world 64 bit OS, every program would be statically relocated at install time. It would permanently have assigned its own segment of the global system address space. Every library it depends on would also have a permanently assigned segment of the global address space.
Also at install time the application would run all its initialization steps including GUI object allocation, with GUI object handles permanently assigned in their own segment of a global address space. The linked and initialized state of the application would be stored. Then launching the application would be more similar to a resume-from-suspend operation: memmap the pre-initialized state into its already reserved segment of address space and reactive the frozen gui objects.
Probably I'm naive: I've never built any of this kind of low-level OS stuff. But I really do think there is room to re-envision some things which were decided in the 1970's based on constraints of 1970's hardware.
Sounds cool until you think that you have to maintain this after new things are installed, removed and upgraded.
ASLR is around for a reason, having everything statically fixed may bring security concerns.
And, how do you run multiple programs out of the same binary?
Running multiple instances: launch+fork with copy-on-write semantics?
Re: ASLR, yeah I think you would lose most of that mitigation. Would it be worth it to make launching even the biggest programs instantaneous? The usability upside could be something amazing; maybe some other security mitigations could take the place of ASLR.
Security mitigations given everything you can run has a fixed address sounds cool, but wouldn't the copy-on-write semantics kind of break that? I guess you might still be able to do something though.
For individual applications there are many reasons that isn't reasonable (performance, security, etc).
But the perf gains you're asking for are mostly supported by the shared cache. The vast majority of launch time for an application is not the dynamic linker. It's the actual "load from disk" for the text and resources, then a pile of init logic.
I'm not sure what you mean here - you can't move application launch logic to install time, it's inherently a launch time process - loading resources into memory, setting up runtime data structures, etc fundamentally requires time.
Now plenty of apps have unnecessarily large start up times, but those apps can largely be improved without just doing more work at install time - it's simply that a lot of devs don't care. There's a lot of "if it's not being measured, it's not being improved" for launch time for many - most? - projects and companies that means launch time is just slow for no reason.
you can't move application launch logic to install time, it's inherently a launch time process - loading resources into memory, setting up runtime data structures, etc
That is exactly what I want to do. Imagine your system had the full 64 bit address space populated with RAM. You launch every installed application simultaneously and let them run thru init to the point where they are ready for user interaction. Then suspend the whole system to disk. Thats what I want the "applications installed" state to look like.
In actual use you have limited RAM and launch applications one at a time. At launch time it is un-suspending the already initialized application. It has its own reserved virtual memory addresses so no need to reallocate new memory and no conflict with other running applications.
Emacs used to do something like this (maybe still does?) but in a very hacky way. I'd like it to be how the whole OS works.
Apple calls their (now old) linker, ld64, a static linker. On the other hand, dyld is their dynamic linker. This is wrong but Apple then is a $2.8T company. They should call ld64 their system linker or just their linker. They have of late altogether disabled static linking for 3rd part applications. All applications must now start with dyld. You only have dynamically linked applications. This was done for several reasons, but application start time and process security were probably tied for first on the list.
ld64 is a fork of the old and slow GCC ld linker. It has accumulated a thousand switches and features and hacks over the years. The move to only dynamic linking meant that most of them can never be used again. So now is a great good time to do a complete rewrite, especially with lld and mold as models; both were architected by the awesome Rui Ueyama. Throw away all the crap, parallelize everything and yeah, it should be fast.
What do you mean by “disabled static linking?” You can make and use static libraries if you want, but on Darwin-based platforms it’s never been supported to make direct system calls; everything has been required to link libSystem.dylib for system call access all along, and dyld has been unavoidable all along.
47 comments
[ 2.7 ms ] story [ 102 ms ] threadhttps://twitter.com/nabbisen/status/1665936938837344256
> I installed Xcode 15 Beta to try out the new Apple's linker. It seems that my mold/sold linker is still a bit faster than theirs, but the gap has now reduced to ~60% when linking the sold linker itself on my Mac Studio. Apple's new linker is in the same league as mine. Great job!
see https://github.com/bluewhalesystems/sold
Large companies (Apple, Amazon, and the like) would likely buy the company rather than license their product (see Dark Sky (now WeatherKit), Kiva (now Amazon Robotics - https://www.fromscratchradio.org/show/mick-mountz 23:22 )) so that they can't lose control over something that they want when management for one of there tools changes.
And so why use mold (license problems and they wouldn't want to impose the AGPL on all their developers), or be a reseller of sold ($8/month/user - how many developers are in the Apple developer program) - it comes down to "it is better for us to develop our own and release it so that there is certainty about the license for any developers who use it."
It would even be further beyond the pale to say "here is a linker for 3rd party developers to use... oh, and its AGPL, sorry 'bout that."
So to the question from the GP:
> Why didn't Apple just use mold?
Because they won't tell 3rd party developers to use AGPl software or require a 3rd party developer to use a different 3rd party developer's licensed software that may change.
If Apple is going to release something, they're going to develop it and release it themselves with as few other encumbrances as possible.
Now, thinking about that case, I guess that Apple could make it easy for people to depend on AGPL software that they download on their own, and that Apple only provides some sort of index or crawlers for it. No need for Apple to have AGPL software on their end.
I remember an SFC blog post[2] expressing distaste for such scare-tactics applications of copyleft on the part of MySQL AB in the early 2000s. (As opposed to the initial goal of copyleft, that is using a legal hack to eventually force most software to become free/libre.) This case is maybe less unpleasant, but only marginally so, I think.
[1] https://gist.github.com/lleyton/9c0b75d065f37333ea9851b6cad1..., discussed today at https://news.ycombinator.com/item?id=36225016
[2] https://sfconservancy.org/blog/2020/jan/06/copyleft-equality...
I think this "Apple's new linker is in the same league as mine. Great job!" response is full of tact and grace, considering Apple just undercut his business.
I dont think he was willing to sell to Apple for what Apple would consider as reasonable price anyway.
No shame on the book, it is a classic, but it is certainly a bit long in the tooth.
I am very interested in linkers, but find them a bit inaccessible to learn about.
Gold blog posts:
https://www.airs.com/blog/archives/38 and https://en.wikipedia.org/wiki/Gold_(linker)
Design and implementation of Mold:
https://github.com/rui314/mold/blob/main/docs/design.md
What architectures would it support? I assume it handles aarch64. It may already handle x86-64. macOS and iOS don't run on any others. I doubt Apple would be interested in making it support other OSs that run on more architectures.
(It's probably a faux pax of me to say this, but I wonder if the parent comment was written by a LLM...)
That's pretty disrespectful, I expect an apology from you
> I doubt Apple would be interested in making it support other OSs that run on more architectures
I don't think Apple is only using ARM and their chip for their whole stack, in fact, their servers run something else, and they support software that runs on other OSes wich run plenty of different architectures, including X86_64, and they are exploring a plethora of architectures [1]
> I assume it handles
> It may already handle
"Faux pas" indeed, let's stop assuming and let's stay positive, nice and respectful to each other
[1] - https://www.techpowerup.com/298936/report-apple-to-move-a-pa...
> I don't think Apple is only using ARM and their chip for their whole stack, in fact, their servers run something else, and they support software that runs on other OSes wich run plenty of different architectures, including X86_64, and they are exploring a plethora of architectures [1]
Now I understand why you suggested it might run on other architectures / OSs. I'm skeptical though. They may develop/run software on Linux/Windows, but I haven't seen any examples of them releasing a development toolchain (e.g. any part of Xcode) for those OSs (with the obvious exception of platform-independent LLVM improvements). Likewise microprocessor-level stuff (afaict everything they do with risc-v).
How on Earth would you know, much less prove, text originated from LLM?
Your comments come across as meaningless if you're on a mission to disrespect commentors frivolously.
Also at install time the application would run all its initialization steps including GUI object allocation, with GUI object handles permanently assigned in their own segment of a global address space. The linked and initialized state of the application would be stored. Then launching the application would be more similar to a resume-from-suspend operation: memmap the pre-initialized state into its already reserved segment of address space and reactive the frozen gui objects.
Probably I'm naive: I've never built any of this kind of low-level OS stuff. But I really do think there is room to re-envision some things which were decided in the 1970's based on constraints of 1970's hardware.
And, how do you run multiple programs out of the same binary?
Re: ASLR, yeah I think you would lose most of that mitigation. Would it be worth it to make launching even the biggest programs instantaneous? The usability upside could be something amazing; maybe some other security mitigations could take the place of ASLR.
We’ve been moving away from this for security reasons with ASLR and position independent executables.
Fixed targets are dangerous for security reasons.
For individual applications there are many reasons that isn't reasonable (performance, security, etc).
But the perf gains you're asking for are mostly supported by the shared cache. The vast majority of launch time for an application is not the dynamic linker. It's the actual "load from disk" for the text and resources, then a pile of init logic.
Now plenty of apps have unnecessarily large start up times, but those apps can largely be improved without just doing more work at install time - it's simply that a lot of devs don't care. There's a lot of "if it's not being measured, it's not being improved" for launch time for many - most? - projects and companies that means launch time is just slow for no reason.
That is exactly what I want to do. Imagine your system had the full 64 bit address space populated with RAM. You launch every installed application simultaneously and let them run thru init to the point where they are ready for user interaction. Then suspend the whole system to disk. Thats what I want the "applications installed" state to look like.
In actual use you have limited RAM and launch applications one at a time. At launch time it is un-suspending the already initialized application. It has its own reserved virtual memory addresses so no need to reallocate new memory and no conflict with other running applications.
Emacs used to do something like this (maybe still does?) but in a very hacky way. I'd like it to be how the whole OS works.
ld64 is a fork of the old and slow GCC ld linker. It has accumulated a thousand switches and features and hacks over the years. The move to only dynamic linking meant that most of them can never be used again. So now is a great good time to do a complete rewrite, especially with lld and mold as models; both were architected by the awesome Rui Ueyama. Throw away all the crap, parallelize everything and yeah, it should be fast.
ld64 was a complete rewrite, circa 2005, courtesy of Nick Kledzik. `opensource.apple.com` has still the old Changelog(s). https://github.com/rotten-apples/ld64/blob/master/ChangeLog
https://patents.justia.com/inventor/nick-kledzik