HN Explorer

Ask HN: How does Rosetta 2 actually work?

21 points by lisper ↗ HN
Apple's docs say that Rosetta 2 works by translating x86 code into ARM64 code when an application starts up and then running the resulting ARM code natively. But that can't actually be true because if I use dlopen to load a dynamic library at runtime, the architecture of the library has to match the architecture of the executable that called dlopen or it results in an error (Error opening shared library foo.dylib : dlopen(foo.dylib, 0x000A): tried: 'foo.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')

So at a minimum, dlopen can invoke Rosetta again in this case. But why thrown an error? Why not allow an x86 application to open an ARM library if it is actually ARM code at run-time?

The situation is actually even weirder than that because there is an application (Clozure Common Lisp) that produces x86 code at run time but does not go through dlopen, it just writes x86 opcodes directly into memory (AFAIK). And yet, it works when run under Rosetta. How is that possible?

All of the evidence indicates to me that Rosetta is actually an emulator and not a translator. But why would Apple misrepresent this? It makes no sense.

18 comments

[ 2.5 ms ] story [ 49.2 ms ] thread
58028641
Rosetta mainly does translation, but it does emulation for code produced at runtime.
lisper
How does it know when to start emulating? Surely there is some overlap in opcodes between the x86 and ARM64.

And why won't it let me load a dylib compiled for ARM into an x86 process?

xmodem
> How does it know when to start emulating? Surely there is some overlap in opcodes between the x86 and ARM64.

Well, it would have to keep track of which regions of memory are code it has translated and which are data that was written out by the program it's executing.

It can check if a branch or jump instruction goes to data, and then jump to its emulator instead.

EdSchouten
> Why not allow an x86 application to open an ARM library if it is actually ARM code at run-time?

Because the Application Binary Interface (ABI) of ARM and x86-64 is different. Function calling conventions don't map 1:1, structure layouts may differ, exceptions can't be propagated properly, etc. etc. etc..

That's why Rosetta only allows you to mix code of a single architecture in a given address space, so that it can be emulated as a whole. That way all code that's being executed has the same ABI.

giyanani
The memory ordering is different, too. Apple gets around it by adding support for intel ordering in the m1 cpus, and then changing to that mode on a per-process basis.

https://twitter.com/ErrataRob/status/1331735383193903104

If you mix arm and x86 code inside a process, you can't tell which ordering to use.

saagarjha
God I wish people stopped sharing this thread because it’s full of things that just cause confusion. To respond to your point though it’s a per-thread flag that the kernel can flip for you if Apple had wanted it to. It’s probably better to just run the ARM code with full TSO, though; the performance penalty is like 10-15% for most code, which is probably less than making a system call into the kernel to be rescheduled with the right ordering whenever you wanted to switch. That doesn’t change the fact that the real challenge here is ABI, not ordering.
lisper
> Because the Application Binary Interface (ABI) of ARM and x86-64 is different

How does Rosettafied x86 code call system code then?

EdSchouten
System calls tend to have a _very_ basic calling convention. Typically up to six arguments, of which all of them are integers or pointers. No floating-point values. It's fairly easy to translate such calls.

Even if translation is non-trivial, there are only a finite number of system calls (hundreds). It's not hard to provide bindings for those.

lisper
Yeah, I didn't phrase my question very well. I didn't mean "system calls" as in kernel calls. I meant system libraries, i.e. Cocoa frameworks. Quartz. Homekit. That sort of thing.
EdSchouten
I think all of those are simply built as universal libraries as well, right?
lisper
I have no idea. That's what I'm asking. But I don't think that's the case because Rosetta is a separate download. If it included universal binaries for all the system libraries it would be huge.
lisper
> fairly straightforward conversion of x86 code into ARM, albeit with a 1-to-1 mapping between registers rather than following AAPCS or any standard calling convention

OK, that explains why I can't dlopen an ARM dylib. But then how does Rosettafied code call system library routines?

toast0
> how does Rosettafied code call system library routines?

(I have no specific knowledge about how Rosetta works)

x86 code (translated or not) needs to call into system libraries with the x86 calling convention. The easiest way would be to have the x86 system libraries, and translate those too, although someone mentioned they're included as pre-translated cache files.

System calls (typically made from libc) in x86 will trap into the kernel, but I'd assume the CPU will run the kernel handler in ARM mode... probably? a different handler address for ARM and x86, but either way, some method to determine which type of cpu and use the appropriate syscall numbers and structures depending on the type. It's possible, but I'd think unlikely, that Apple made all the syscalls the same across both architectures, down to the width and alignment of all of the structs.

saagarjha
Rosetta is an emulator, that is correct. It consists of both an ahead-of-time translator that does fairly straightforward conversion of x86 code into ARM, albeit with a 1-to-1 mapping between registers rather than following AAPCS or any standard calling convention. At runtime, if new code is generated (e.g. a JIT compiler) it’s run through what’s essentially a runtime version of this process. Self-modifying code is detected on write (code page is marked as read-only if it looks like it’s code, and the write faults and indicates the page should be marked with read-write permissions) or indirect branch (code has been written out and needs to be executed, so page is translated and flipped back to read-only). Calling into arm64 code directly is not possible without thunks due to the mismatched ABI. The system libraries are actually provided as a translated shared cache, rather than being mapped in as “normal” code.
xmodem
Fascinating. Where can I read more about how it works?
saagarjha
https://ffri.github.io/ProjectChampollion/ is a good high-level resource. Random details are mostly strewn about the internet in a bunch of tweets and Discord messages that are hard to collate, so you’ll probably have better luck just asking your question than trying to piece that together.
[deleted]
(comment deleted)
matthewmacleod
The ABI used by a Rosetta-translated application is not the same as the native arm64 one - that’s why you can’t load an arm64 library. Then keeping track of memory which is executable allows trapping and translation on-the-fly.

Lots of useful information in this reverse-engineering article: https://ffri.github.io/ProjectChampollion/part1/