Sending a message is simply a function call. Whereas in Akka it's the ask pattern where you have to manually reify the function call parameters into a case class.
If you want to wait in processing a message in Akka, you either have to juggle the message queue with stashing or block a thread. In Orbit it is a `suspend` fun.
It's an interesting idea; not having to stash messages in the actor implementation when doing an async call. I did a little experiment what that could look like using Akka and kotlin suspend functions https://github.com/joost-de-vries/akka-kotlin
> Orbit is a framework to write distributed systems using virtual actors on the JVM. A virtual actor is an object that interacts with the world using asynchronous messages.
Could anybody elaborate on this? How does an actor differ from an object that uses promises to talk to a server?
To some approximation. The Actor Model formalism [0] requires certain properties, notably:
1. Communication solely by asynchronous message passing
2. A mailbox per actor that means the reception of messages and the processing of them is decoupled (i.e. an actor can receive new messages even when processing a previously-received message).
The OO model in general supports that paradigm. Pretty much all mainstream OO languages are synchronous by default though. Call a method and the caller is suspended until the method returns. Multiple clients can call methods on the same object concurrently, but doing that safely requires some form of locking/protection to be implemented in the target object. Stated alternatively: threads in mainstream OO languages run "across" objects, whereas each actor has its own thread in the Actor model.
There are certainly similarities - in the sense that both actors and objects encapsulate state, with reading/writing occurring through well-defined interfaces (messages and methods respectively). But the threading model is quite different - it's not just a matter of scale.
Thanks for making me feel old by not including agents in the list. Those seem even more similar to actors. At least until someone mentions BDI, then it's suddenly an entirely different universe.
Usually the difference is the "mailbox" an actor has, making the difference more about the level you're calling an actor more than the differences in any code you might write.
It’s not the same as async objects. There are differences between different score systems. But have a look at Elang and Akka for example.
But I’m actor systems the actors maybe more stateful that you typical object and be location transparent. Communication between actors does require knowledge of if another actor is in process or remote.
Also the actor supervising model is a completely different way of recovering from errors than in OO systems.
Actors are useful for keeping code clean/understandable/correct when dealing with heavy concurrency AND lots of mutable state.
Actors are like objects, but the only way to interact with them is sending them a message - you can’t directly access their properties, call their methods, etc. Messages are bits of immutable data sent between actors asynchronously, but those messages go into the actor’s mailbox, and then the actor processes them synchronously, one at a time. You can get parallelism by adding more actors of the same type (10 actors of the same type lets you process up to 10 messages concurrently). Because actors are totally synchronous INTERNALLY, and nobody can directly access their internal state, it’s fine for them to have mutable internal state, with no locking/synchronization needed, and much easier to reason about than mutable state normally is in highly concurrent applications.
With that being said, the actor model has a lot of inherent complexity. If you don’t need it, don’t use it. But if you really do need lots of concurrency and lots of mutable state, it’s often a great choice.
They’re also good for distributed systems. Actors only communicate by sending messages to other actors, but it doesn’t really matter if that message is sent in memory to an actor in the same process, or over the network to an actor on a different machine. So you can take a single actor system and split it across many machines pretty easily. Generally the actor framework you’re using will handle delivering messages over the network, ensuring there’s the right number of actors running across all nodes combined, etc.
> Actors are like objects, but the only way to interact with them is sending a message - you can’t access their properties, call their methods, etc.
While the implementation in many popular, e.g. C++ and family descended from it, languages muddies the waters, basically methods in OO are a convenient way to define handlers for messages sent to objects, and in pure OO you also can't interact with objects except by sending messages to them. The difference between the basic OO model and the actor model is that OO message sends are synchronous request/response and actor model message sends are asynchronous fire-and-forget.
Agreed, although it’s not always fire-and-forget (i.e. send a message and don’t wait for a response). Actors generally do support request/response style messaging, but it’s always async - it’s really only synchronous request/response communication that’s not allowed.
> Agreed, although it’s not always fire-and-forget (i.e. send a message and don’t wait for a response).
Sure, you can do async request/response in the actor model, but the low-level basic mechanism all communication is built on is send a message to an actor mailbox and keep going. Everything else is built in top of that.
Not always. Goblins[0] is an example of an actor system that supports both synchronous and asynchronous communication between actors, with synchronous communication only possible if both actors are in the same “vat” (a kind of event loop).
Yes, Goblins' "vat" model descends from E, which supports the "hybrid worldview". http://www.erights.org/
I have a tendency to call Goblins objects "actors", though some people in the community like to point out that "distributed object" is preferable since synchronous call/return is not supported in actors. But yeah.
So much this. This "actor model" thing as described by GP sounds like plain old OO and RPC with extra steps and also distributed. This may be a low-effort post on my part, but I'm 100% not impressed because it sounds like marketing fluff and an opportunity to rewrite native/built-in things using frameworks. Also justifying Yet Another Framework or tool hosted on an .IO domain (snarky I know) for promotions and endless blog posts.
Erlang is decades old, a highly successful (for a relatively niche language, anyway) proof that the actor model (or something approximating it) can be a great tool.
CSP/actors is from the 70s, like most CS. It is sort of the original OO, but that's because OO is a newer version that's both less powerful and over-complicated.
It doesn't need to be distributed, and it's better not to include that because the approach to error handling gets a lot easier. If you're distributed every call can fail, be randomly slow, has marshaling costs, is untrusted, etc.
I don't know what you mean by "rewrite native/built-in things using frameworks", the things you are talking about require using frameworks. You need a framework for RPC. You also need a framework for distributed shared memory.
I've read about actors from time to time for decades. I can't say I fully grok the concept.
Kubelet wasn't necessarily designed as an actor. I think it's a concrete thing that people have interacted with that is a pretty good example of an actor. It has it's own control loop. It sits there and actively monitors the state of the node. It constantly reports the state. If something's wrong, it can try to correct it.
Most services and objects just sort of sit around and wait to be called. Kubelet is a little different, it's got a main driving thread that's constantly looking for trouble, and acting on what it finds.
The line is still pretty fuzzy for me, but maybe this helps someone connect some conceptual dots, to distinguish between a service and an actor.
> those messages go into the actor’s mailbox, and then the actor processes them synchronously
Not trying to be pedantic, but that's technically not true of the Actor formalism [0]. Quoting wikipedia:
> an actor can designate the behavior to be used to process the next message, and then in fact begin processing another message M2 before it has finished processing M1.
Available implementations (such as Erlang, Akka, Pony) do not support that pipelining so are single threaded per actor as you state though. As an aside, the behaviour you describe is part of the CSP formalism [1] - a related but different approach to concurrent systems.
Just to be a bit pedantic (but important), Erlang is not an actor system. It's a system designed for fault tolerance and definable failure domains. Concurrency fell out of those requirements and it just happens to vaguely look like an actor system if you squint hard enough. I don't believe theories of actor systems played into the design of Erlang.
Perhaps some of the blame belongs on the creators of Erlang for kind of jumping on the "actor" (and later, "OOP") bandwagons as marketing? to try to make Erlang less scary.
Fair point: Erlang wasn't designed as an Actor system. But you don't have to squint that hard to see the similarities. Again quoting wikipedia [0]:
> An actor is a computational entity that, in response to a message it receives, can concurrently:
send a finite number of messages to other actors;
create a finite number of new actors;
designate the behavior to be used for the next message it receives.
> There is no assumed sequence to the above actions and they could be carried out in parallel.
All above are true of Erlang except intra-actor concurrency. Even the last bullet on designating behaviour: An Erlang process (actor equivalent) can decide to use a different function when receiving the next message. It just can't pipeline handling messages.
So Erlang isn't that far removed from the Actor model in its behaviour, even if it wasn't designed as an Actor system in the first place. One might say actors are an (approximate) emergent property of Erlang rather than an intentionally designed one.
Slightly off-topic but the only system I've used that was designed & built from the outset as an actor system is Rosette [1]. It was a really interesting project, and does support pipelining, but has been dormant for more than a decade.
No but also key to the definition of the actor system is that those are the only things they are allowed to do. Also, there's strange stuff like naming processes for service discovery, and iirc something is wrong with the way that Erlang does selective receives that disqualifies it from being an actor system, and ultimately more modern Erlang things like ets tables and sharing memory with nifs.
Point is all of these deviations from actor system were choices made by the Erlang team in the name of pragmatism. They all exist because there was a use case and the first teams using Erlang needed them for something real; that Erlang is not an actor system is important, because it's a highly pragmatic system -- not one that is based in theory.
> Also, there's strange stuff like naming processes for service discovery, and iirc something is wrong with the way that Erlang does selective receives that disqualifies it from being an actor system, and ultimately more modern Erlang things like ets tables and sharing memory with nifs.
Ets tables are observationally equivalent to an Erlang process per table and sending it a message for each function call. It's not actually implemented that way, but I don't think that is grounds to disqualify it. I don't remember enough details about the naming process, but I think that might be similar; you could send a message to a naming process to set and lookup the names (although you'd have a bit of trouble finding out what the process id of the naming process is, wouldn't you?), it's just not very pragmatic.
Nifs certainly have the potential to break the model of course. I'd think selective receive should be fine too, it's equivalent to reading (or peeking) and saving messages until a matching message is received, processing that message, then processing future messages from the saved queue. It's just implemented in a more pragmatic way.
If immutability is important, then the process dictionary means Erlang doesn't qualify, but again, it's pragmatic.
I'd rather have a pragmatic almost actor system than a dogmatic Actor system that's hard to use.
> designate the behavior to be used for the next message it receives.
I think that is the key insight of actor-systems.
Actors don't have state. But they can calculate their replacement based on their immutable data. Thus the way an actor at a given address evolves is described by the functions that calculate the successor actors.
Thus you get Pure Functional Programming implemented on top of a fabric of
distributed evolving entities. You can understand the behavior and evolution of such a system as a composition of function-calls, where functions always produce
the same result for the same arguments.
I mean, if it doesn't count as an implementation of an actor system, then I'm not sure what does, regardless of whether it was incidental to the design goals of Erlang as a language. It certainly walks and quacks like an actor system in my opinion, and I don't think you have to squint very hard at all to see it.
Perhaps we have different ideas of what an "actor system" is though.
Dr. Hewitt typically chimes in on these threads to emphasize some of the differences between his model and Erlang. I don’t recall his username to locate some of those.
Out of curiosity (not much experience with Actor systems), do these systems solve a different problem than having disparate processes communicate by sticking a distributed message queue between them? From my naive point of view it seems like alot of the scaling and fault tolerance concerns nowadays can be solved in any language, by having a queue be the interface between two services.
Same question applies to Erlang, which I realize is not exactly an actor system as per the below comment, and has a much more sophisticated error recovery story with supervisors, but the general question holds.
So I think the Erlang distinction is a little unrelated (I mean, it's technically accurate, but not because of the error recovery; that's an implementation detail. The original Actor formulation Carl Hewitt proposed was heavily influenced by Smalltalk semantics; whereas Smalltalk was truly OO, in that everything is an Object that passed messages, one issue it had was that objects did not equate to threads of execution. The Actor model, then, said each object executed independently. But in the same way that Smalltalk did not have primitives; even integers are always Objects, the Actor model stipulates there are no primitives; even integers are Actors). But I'll take a stab at answering what I think you're asking.
If your services don't deal with internal mutable state, nor high degrees of concurrency then there isn't much gain to be had with an actor system. That said, that begs the question of what the queue is for; just create more instances, since there's no internal state to share.
As soon as you start having internal mutable state and high levels of concurrency, that's where the actor model applies. Queues don't exist for concurrency (you don't need them; just create more executors), they exist for imposing sequence where it is needed (an obvious case; you have a DB connection, you want to only have one query at a time. So every desired query goes into a queue, and the process at the end that owns the DB connection pulls from it). Internal mutable state gets stored inside of an actor; updates and reads get serialized on that actor.
At the highest level, I would describe the actor model as taking a 'successful' model for distributed computing, and making it the only model you use, even locally.
In, let's say Java, for instance, using standard concurrency approaches, it matters where a process lives. My way of operating/communicating to another thread of execution (unit of concurrency) is very, very different than my way of operating/communicating to another machine. Locally I have threads and locks and need to be very mindful. When communicating to another machine, I send a message and that's it (maybe I expect a response, and timeout if I don't get one, but that's really just the same thing, the other machine sending a message).
I don't actually need a queue involved for theoretical correctness unless I need to process messages in sequence (after all, I could have multiple copies of the other process, and send a message to each of them). Now, in the real world I do, simply because if my concurrency gets too large it can't be handled by what the units of concurrency already available (instances, threads, whatever), and scale up takes time, but that's really just a special case of why I need to impose a sequence on messages (handle these first, then handle these, rather than handle all of them concurrently).
The actor model makes this the local model of communication (and so makes the impedance mismatch negligible between local and distributed; so much so that some languages it's actually irrelevant whether you're sending a message to a local actor, or a remote one). Scaling concurrency up internally just means spin up a new actor. When you need to serialize, you send messages to the same actor, where it ends up in a queue.
So it's not solving a different problem, exactly, if that problem is "how do we write systems that can do multiple things at once", but the specifics, complexity, etc, tend to be pretty different. The problems it's solving are a bit more subjective than simply "can we handle this problem", and more "how well do our tools and mental model lend themselves to the problem we're trying to solve".
>> With that being said, the actor model has a lot of inherent complexity. If you don’t need it, don’t use it. But if you really do need lots of concurrency and lots of mutable state, it’s often a great choice.
I kind of disagree with this. I mean, it's kind of correct, if you literally have a sequential problem with immutable state you're gaining nothing from it, but you're also losing nothing (you...will have one actor, no messages being passed, so the only cost is the syntax of the language; the actor model isn't adding any complexity because you aren't using it).
But, a lot of problems we've historically learned to view as sequential are in fact concurrent. Deeply concurrent. We've created entire concepts specifically to impose sequential processing upon innately concurrent activities.
As an example, almost everywhere we use queues we could instead model as unrelated processes able to be run concurrently. You still may have a bit of queueing/scheduling for unbounded processes, but I know from production experience that that is far, far simpler to do and get right using actors than a traditional queue/priority queue and worker(s).
And that's where it shines. It encourages you to start thinking about what in fact -should- be done concurrently, by making that easy rather than a chore as it is in most other languages, and that leads to -less- complexity.
I'm curious what you see as so different in an actor vs a queue with workers. An actor's mailbox effectively acts as a queue.
Is it the supporting structure surrounding an actor focused library/language? Lots of person-hours have gone into making them to work well. Some home grown queue + pool of threads working off that queue... not so much.
What I meant is that you can launch a co/go-routine and have a channel that only it can read. That is like 70% the power of having an actor system.
---
> an actor vs a queue with workers
See the sibling comment.
TLDR: Actors each has their own queue. The orderings in each queue are independent. This can help with parallelization if it fits the shape of the problem.
>What I meant is that you can launch a co/go-routine and have a channel that only it can read. That is like 70% the power of having an actor system.
You can also iterate over an array in parallel and only modify the current element and that is like 70% of the power of having an actor system.
"Structured concurrency is a programming paradigm aimed at improving the clarity, quality, and development time of a computer program by using a structured approach to concurrent programming." - Wikipedia
So pulling from the alluded to real life example, we wrote a job scheduling system. Now, each job could be defined by a lifecycle. It wasn't just a single fire and forget command, but multiple in time (a prep start, start, stop, clear, commands), and also tracking status of an external resource (looking for error states or ambiguities, etc), and could also be modified (change of start and stop times, including "now" for both).
Now, using actors, this was trivial to do. Each actor was basically a glorified state machine, with each command, status update, etc, a message coming in that updated the internal state, and caused it to change state, and, where appropriate, update anything it needed to (such as the internal timer for when it would stop the job). We loaded up everything that had to happen in the next 12 hours, via an actor that would wake up and load more periodically (with an actor registry to find and confirm what existed already, as well as used to route incoming commands from users to change jobs). There is no complexity in managing a global state of priority between the actors; that's a 'solved problem' by the time slicing the underlying actor engine is doing for us; it's well tested, well proven, and invisible to us. Our implementation allows us to treat each job as existing in isolation, and write our code as though it and it alone is the job that needs to be executed. There is no global synchronization state to manage (well, with the caveat of taking the infinite list of future jobs, and ensuring we have actual processes equating to those scheduled to start in the next 12 hours, but that's a comparatively simple problem, and a proper use of sequential ordering. We want to execute on these jobs first), nor should their be based on the problem we're trying to solve.
Using a queue and traditional threading model though? Well, we'd need a synchronized priority queue, as items could have their priority changed at any time, from multiple sources. We'd also still need a registry. We'd need a pool of workers, that are mostly just blocking...hope we don't have > (number of workers) things needing to execute concurrently. Our complexity to manage all of this is high; the level of testing needed to validate that there aren't emergent bugs, also high. Our semi-realtime requirements are probably out the window (high contention on the queue, or high enough concurrency to exhaust the thread pool could lead to very measurable delays in updates). And ultimately, it all comes to the fact that the queue exists to pretend that the state of the world is sequential, and that we then had to then contort ourselves around to bring back to a state that allows for suitable concurrency. Updates to a job can end up killing a worker and inserting something back onto the queue, can remove something from the queue and reinsert it at a different position, etc, all while other workers are trying to access the queue simultaneously.
Actors are the simpler solution.
The comment about an actor's mailbox being a queue is missing the point (though, amusingly, demonstrating it). Queues are sequential. Actors are capable of doing only one thing at a time, and so when they are requested to do multiple things, they will do them sequentially. A mailbox filling with things that don't demand sequential processing is a code smell in an actor system. This is useful if you have limited resource access (queries on a DB connection, say), or need things to be done sequentially, and are very much akin to threads. But if you actually want multiple things concurrently...you should have multiple actors, not send them into the mailbox of one actor (which is very much the queue situation mentioned above; you imposed a sequential ordering on things you want done concurrently. Why?). That's the key difference; in most other languages, the queue is taking multiple things you actually want concurrently, and imposing a sequence to them. In the actor model, your queu...
So let's say I implement some job scheduling system like you suggested: One job actor instance per job and one scheduler actor instance to create new job actor instances. The job can be in one of many lifecycle states like started, in progress, etc. Unfortunately, since this is a distributed system with faults there can be network partitions, crashes and such at any moment, so every time the job actor changes state it has to write it down in some highly available database so it can be restarted if it crashes and every time it wants to do anything it has to check that database to see if someone else is already doing it. At that point the actors are basically stateless, so what's the benefit of using actors in contrast to Futures or similar stream-processing frameworks, whose abstraction boundaries are not coupled directly to operational concurrancy concerns and thus much more maintainable when you have more complicated workflows with fan-out/fan-in stages? (Not to mention that Akka Streams provides this API on top of actors with the ability to distribute work across nodes in a network without users having to know anything about the actor model).
I’m curious about the following: Suppose you have an actor that is tasked to perform a long-running operation (enacted by a specific message sent to the actor). Now suppose you want to know how far the actor has already progressed with the operation, or possibly you want to pause or stop/abort the operation – basically a monitoring and controlling interface. Would that be out of scope for the actor model? Polling the progress or gracefully stopping the operation would seem to require some asynchronous responsiveness from the actor. Is that something that is commonly supported by actor frameworks? If so, how does that work? Is that a gray area where the concrete actor implementation has to deal with concurrency after all, e.g. special callback routines that will be invoked concurrently to the main operation?
Many actor frameworks (including Orbit, though I advise you don't use it for the reasons in my above messages) are strictly single threaded in that only one thread can operate on an actor at any one time, and by default they are also not concurrent in that only one message gets processed at once.
However, while the single-threadedness is a hard requirement and behavior of the system, the rule that only one message is processed at a time is not.
In Orbit (and Orleans) the ability to process multiple messages at the same time is called reentrancy. it can be specified at the actor or method level usually.
So in your example, say your initial message kicks off a write that is going to take 30 seconds. You start the write and assuming it's async, the actor then suspends while it waits for that to complete. While the actor is suspended another message asking for the progress arrives and that message is reentrant, it can process that message and respond with the progress, then the actor suspends again and at some point another reentrant message arrives or the initial write finishes and the first message finally gets a response. If a none reentrant message is received it will queue in the mailbox until after the first one is completed.
So, with reentrancy you still get the single threaded guarantee but you effectively get interleaving of messages when the actor is suspended waiting for some other async task to complete. This means that if you have an actor with reentrant messages you have to make that code safe to run during any potential interleaving point in another message (for example an await in async C#).
Other actor frameworks (particularly ones that are not based on virtual actors) may have different guarantees and behaviors but from what I've seen there is always some equivalent.
Doesn't reentrancy defeat the serial processing property that makes locks unnecessary? If a new message is accepted before the previous one has finished processing, then it can change the mutable state inside the actor. Essentially, you can never know what the state will be because at any blocking point some other message can make arbitrary changes. At that point the actor is just a crappy class with message passing instead of method calls and all the annoyances that entails.
It really has nothing to do with promises (as used colloquially) and is a way to design large distributed systems using message passing and localized state that's persisted.
“Exactly one” is a hard concept to get right. How do they ensure that exactly one instance of an actor is alive at once, without sacrificing latency of actor start and/or while tolerating network partitions?
That's still "at most one" – ie. even though state of database itself is "exactly one" (...can "hold the lock") – the fact that actor thread is remote, it means it can be partitioned/starve the rest of the system.
"Exactly once" can be achieved with "at least once" + _local_, reliable state where you can resolve/ignore duplicates. But if that "local" state is actually "remote" – well, then you loose this guarantee (because you can be arbitrarily partitioned from that state so you're in square one again).
> The last-generation of Orbit powered some of the key technology behind the Dragon Age Keep and Dragon Age: Inquisition. Our plans for the next-generation framework are even more ambitious.
So at least it was used for some of Bioware's games
> Especially that we already have a perfectly good, battle-hardened, and relatively lightweight implementation of Actor model with Erlang / Elixir.
Yeah, if you're already using Erland or Elixir, why don't you go with that instead? This seems to be for the JVM, so one could assume that the ones who want to use this, is already invested heavily in the JVM ecosystem (which as far as I know, EA is when it comes to backend servers).
Orbit is also the name of the always online DRM system that Ubisoft made many years ago.
Fun fact, the tech that was developed to power Orbit forked and evolved along largely divergent paths to be Uplay and the framework on which the Division and Division 2 online backends were made.
Vert.x uses a concept called Verticles - which are like actors..with an external message bus. Which ends up being more practical if you start leveraging kafka, etc in the architecture.
The OSS project I work on, Dapr (Distributed Application Runtime - an incubated CNCF project) implements the virtual actor pattern if anyone is interested.
I found out about this one a few years back from a former (online) friend who worked on it (although I'm not entirely certain in his specific involvement, I want to say he was the lead developer initially though), his name is Joe Hegarty[0]. There was another implementation as well from him in TypeScript for Node.js called Ratatoskr[1].
As far as I've always known him, he always works on networking projects, I believe he worked on the Fable 3 networking for coop and several other games. I can't even imagine what he works on at EA now honestly.
JoeH if you're reading this by chance: Just remember you were an inspiration back in the day (mid to late 2000s) to a lot of people who became developers and learned a ton from your work back then, myself included. To this day I remember asking you questions almost every day. Thank you for your efforts and your patience.
So, I'm the original lead developer of this project.
This was meant to be a followup to the original version of Orbit which now lives on as orbit-legacy on GitHub.
Unfortunately other priorities have meant this project hasn't proceeded in the past couple of years. I wouldn't really recommend anyone use either of them.
However, the ideas and implementation both for Orbit Legacy and the early version of Orbit 2 contain some ideas and concepts that may be useful to folks.
I still think there is immense value in virtual actors (Orleans from Microsoft is one great alternative for .NET, and Dapr also looks interesting) but I can't say if we'll ever get back to it.
While in general I agree with that philosophy, it doesn't really apply in this case.
When Orbit 1 was initially developed, Orleans was not open source, so the initial implementation was based off the Microsoft Research Whitepaper.
In our specific case, we had a lot of existing experience and services built on the JVM so adopting DotNet for Orleans (even if it had been available at the time) was not an option. The relative drop in priority for Orbit vs other projects was due to a need to dedicate more time to other parts of the platform.
As a large dev/publisher EA has a sizeable central technology group that develops everything from engines (Frostbite) to services (EA Digital Platform).
Ultimately I agree with the philosophy for smaller developers who are focused purely on making a game, but that's just not the reality at large companies and ultimately someone has to develop those services/engines.
Did you really make a throwaway just to make a comment that you knew would be downvoted, for both objective reasons as in Joe's response, and because you were unnecessarily rude?
If there's something you want to say that will be downvoted, just say it and eat the downvotes.
I'm not saying I agree with "ThE DowNVoteS SaY Im RigHt", I'm saying own your actions.
Edit: And if you don't want to own it, think twice before posting.
I'm a Dapr early adopter. It's cool and the SDKs are improving a lot, but there are some quirks. I've found that some blocks (connectors) that are listed as general availability are unstable to the point of being useless, while some blocks listed as alpha work fine.
I think the JVM and particularly Java are in desperate need of improvements to the concurrency model.
When I compare the concurrency model on the JVM (and especially Java itself) to the modern competition from the likes of GoLang and C# there are clear and obvious drawbacks. Even comparing just the Java language and some other JVM languages like Kotlin and Scala, it comes up well short.
Project Loom certainly looks to be a step in the right direction. I haven't looked at it in depth recently but Virtual Threads and Continuations will be a welcome addition and last time I did look it was promising in terms of implementation.
If I were creating a new project today I'd likely reach for GoLang or DotNet (now that Core and Framework are unified) before the JVM, and if I did decide on the JVM I'd choose Kotlin over Java.
I'm historically a huge fan of the JVM and Java, and I'd like to see it back on top, but right now I think the competition is better positioned and until the many initiatives going on (such as Loom) start to have a major impact on the ecosystem, it's no longer automatically my first choice.
I'd like to mention the native actor model implementation CAF, the C++ Actor Framework, and share some experiences. (Disclaimer: I've been developing on CAF in the past and have a good relationship with the creator.) CAF (1) provides native actors without an VM layer, (2) type-safe interfaces so that the compiler yells at you when a receiver cannot handle a message, and (3) transparent copy-on-write messaging so that you can still push stuff through pipelines and induce only copies only when a ref count is greater than one.
In our telemetry engine VAST, we've been using CAF successfully for several years for building a distributed system that always has a saturated write path. CAF provides a credit-based streaming abstraction as well, so that you can have backpressure across a chain of actors, making burst-induced OOM issues a blast from the past. You also get all the other benefits of actors, like linking and monitoring, to achieve well-defined failure semantics: either be up and running or collectively fail, but still allowing for local recovery—except for segfaults, this is where "native" has a disadvantage over VM-based actor models.
With CAF's network transparent runtime, a message ender doesn't need to know where receiver lives; the runtime either passes the message as COW pointer to the receiver or serializes it transparently. Other actor model runtimes support that as well, but I'm mentioning it because our experience showed that this is great value: we can can slice and dice our actors based on the deployment target, e.g., execute the application in one single process (e.g., for a beefy box) or wrap actors into single OS processes (e.g., when deploying on container auto-scalers).
The deep integration with the C++ type system allowed us to define very stable RPC-like interfaces. We're currently designing a pub/sub layer as alternate access path, because users are interested in tapping into streaming feeds selectively. This is not easy, because request-response and pub/sub are two ends of a spectrum, but it turns out we can support nicely with CAF.
I still don't get why you need actors. In all my years I've never come across some problem and thought "this would be a great use for actors". Is the kind of applications that I write simply not suited to it? Many other people here in the comments have mentioned how actors can be used to serialize access to mutable state. My applications almost never have mutable state, especially not non-local mutable state that would require locks, except maybe caches. All mutable data is generally in some kind of highly available database and too large to hold in memory anyway. Message queues (like in an actor's message box) can never be in memory or they would lose data when the application crashes.
The reasoning in the book that you linked does not make sense to me. Any serious language nowadays has some kind of lightweight thread abstraction which you can spawn thousands or millions of without making a dent into system resources, so this is a non-issue. It goes on to say that
> This alleviates the need for the programmer to reason about an entire system. Instead the programmer has a fixed set of concerns, meaning they can ensure behavioral correctness in isolation, rather than having to worry about an interaction they hadn’t anticipated occurring.
which is just blatantly false. You can reason about a single actor, yes, but doing so is not helpful when you are interested in the correctness of the entire system. Since actor boundaries are an operational concern, they do not align with the domain logic. To understand the domain logic you always have to look at interactions of multiple actors (a single actor would be pointless) all with their own mutable state, all which can receive messages from anywhere at any time with no guarantees of timely reply to its own messages. A functional programming approach on the other hand is much easier to reason about with a well defined denotational semantics, no peer-to-peer interactions, no mutable state (generally) and composition of components that are aligned with domain boundaries and thus meaningful in isolation.
> Without a lightweight process abstraction, users are often forced to write parts of concurrent applications in an event-driven style which obscures control flow, and increases the burden on the programmer.
This is exactly the argument _against_ actors in my mind. Because actors communicate only through messages, they enjoy high decoupling but the price for this is horrible cohesion. Any domain logic that is spread across multiple actors will quickly become incomprehensible and unmaintainable. The same is true for event-driven microservices and a great deal of architectural work goes into making the microservices as large as possible to improve cohesion while making them as small as necessary to be independently scalable.
94 comments
[ 3.1 ms ] story [ 154 ms ] threadSending a message is simply a function call. Whereas in Akka it's the ask pattern where you have to manually reify the function call parameters into a case class.
If you want to wait in processing a message in Akka, you either have to juggle the message queue with stashing or block a thread. In Orbit it is a `suspend` fun.
Could anybody elaborate on this? How does an actor differ from an object that uses promises to talk to a server?
When people talk about actor-based languages or frameworks however, theyre suggesting syntax primitives which help express this more naturally.
The simple pitch is: OO where objects could be on any machine.
1. Communication solely by asynchronous message passing
2. A mailbox per actor that means the reception of messages and the processing of them is decoupled (i.e. an actor can receive new messages even when processing a previously-received message).
The OO model in general supports that paradigm. Pretty much all mainstream OO languages are synchronous by default though. Call a method and the caller is suspended until the method returns. Multiple clients can call methods on the same object concurrently, but doing that safely requires some form of locking/protection to be implemented in the target object. Stated alternatively: threads in mainstream OO languages run "across" objects, whereas each actor has its own thread in the Actor model.
There are certainly similarities - in the sense that both actors and objects encapsulate state, with reading/writing occurring through well-defined interfaces (messages and methods respectively). But the threading model is quite different - it's not just a matter of scale.
[0] https://en.wikipedia.org/wiki/Actor_model
EDIT: corrected grammar & formatting.
But I’m actor systems the actors maybe more stateful that you typical object and be location transparent. Communication between actors does require knowledge of if another actor is in process or remote.
Also the actor supervising model is a completely different way of recovering from errors than in OO systems.
https://en.wikipedia.org/wiki/Actor_model
Actors are like objects, but the only way to interact with them is sending them a message - you can’t directly access their properties, call their methods, etc. Messages are bits of immutable data sent between actors asynchronously, but those messages go into the actor’s mailbox, and then the actor processes them synchronously, one at a time. You can get parallelism by adding more actors of the same type (10 actors of the same type lets you process up to 10 messages concurrently). Because actors are totally synchronous INTERNALLY, and nobody can directly access their internal state, it’s fine for them to have mutable internal state, with no locking/synchronization needed, and much easier to reason about than mutable state normally is in highly concurrent applications.
With that being said, the actor model has a lot of inherent complexity. If you don’t need it, don’t use it. But if you really do need lots of concurrency and lots of mutable state, it’s often a great choice.
They’re also good for distributed systems. Actors only communicate by sending messages to other actors, but it doesn’t really matter if that message is sent in memory to an actor in the same process, or over the network to an actor on a different machine. So you can take a single actor system and split it across many machines pretty easily. Generally the actor framework you’re using will handle delivering messages over the network, ensuring there’s the right number of actors running across all nodes combined, etc.
While the implementation in many popular, e.g. C++ and family descended from it, languages muddies the waters, basically methods in OO are a convenient way to define handlers for messages sent to objects, and in pure OO you also can't interact with objects except by sending messages to them. The difference between the basic OO model and the actor model is that OO message sends are synchronous request/response and actor model message sends are asynchronous fire-and-forget.
Sure, you can do async request/response in the actor model, but the low-level basic mechanism all communication is built on is send a message to an actor mailbox and keep going. Everything else is built in top of that.
[0] https://docs.racket-lang.org/goblins/
Yes, Goblins' "vat" model descends from E, which supports the "hybrid worldview". http://www.erights.org/
I have a tendency to call Goblins objects "actors", though some people in the community like to point out that "distributed object" is preferable since synchronous call/return is not supported in actors. But yeah.
It doesn't need to be distributed, and it's better not to include that because the approach to error handling gets a lot easier. If you're distributed every call can fail, be randomly slow, has marshaling costs, is untrusted, etc.
Kubelet wasn't necessarily designed as an actor. I think it's a concrete thing that people have interacted with that is a pretty good example of an actor. It has it's own control loop. It sits there and actively monitors the state of the node. It constantly reports the state. If something's wrong, it can try to correct it.
Most services and objects just sort of sit around and wait to be called. Kubelet is a little different, it's got a main driving thread that's constantly looking for trouble, and acting on what it finds.
The line is still pretty fuzzy for me, but maybe this helps someone connect some conceptual dots, to distinguish between a service and an actor.
Not trying to be pedantic, but that's technically not true of the Actor formalism [0]. Quoting wikipedia:
> an actor can designate the behavior to be used to process the next message, and then in fact begin processing another message M2 before it has finished processing M1.
Available implementations (such as Erlang, Akka, Pony) do not support that pipelining so are single threaded per actor as you state though. As an aside, the behaviour you describe is part of the CSP formalism [1] - a related but different approach to concurrent systems.
[0]: https://en.wikipedia.org/wiki/Actor_model#Inherently_concurr...
[1]: https://en.wikipedia.org/wiki/Communicating_sequential_proce...
Perhaps some of the blame belongs on the creators of Erlang for kind of jumping on the "actor" (and later, "OOP") bandwagons as marketing? to try to make Erlang less scary.
> An actor is a computational entity that, in response to a message it receives, can concurrently:
> There is no assumed sequence to the above actions and they could be carried out in parallel.All above are true of Erlang except intra-actor concurrency. Even the last bullet on designating behaviour: An Erlang process (actor equivalent) can decide to use a different function when receiving the next message. It just can't pipeline handling messages.
So Erlang isn't that far removed from the Actor model in its behaviour, even if it wasn't designed as an Actor system in the first place. One might say actors are an (approximate) emergent property of Erlang rather than an intentionally designed one.
Slightly off-topic but the only system I've used that was designed & built from the outset as an actor system is Rosette [1]. It was a really interesting project, and does support pipelining, but has been dormant for more than a decade.
[0]: https://en.wikipedia.org/wiki/Actor_model#Fundamental_concep...
[1]: https://github.com/leithaus/Rosette
--
EDIT: clarified that Rosette is the only system I've used that was explicitly based on the Actor model from the outset.
Point is all of these deviations from actor system were choices made by the Erlang team in the name of pragmatism. They all exist because there was a use case and the first teams using Erlang needed them for something real; that Erlang is not an actor system is important, because it's a highly pragmatic system -- not one that is based in theory.
Ets tables are observationally equivalent to an Erlang process per table and sending it a message for each function call. It's not actually implemented that way, but I don't think that is grounds to disqualify it. I don't remember enough details about the naming process, but I think that might be similar; you could send a message to a naming process to set and lookup the names (although you'd have a bit of trouble finding out what the process id of the naming process is, wouldn't you?), it's just not very pragmatic.
Nifs certainly have the potential to break the model of course. I'd think selective receive should be fine too, it's equivalent to reading (or peeking) and saving messages until a matching message is received, processing that message, then processing future messages from the saved queue. It's just implemented in a more pragmatic way.
If immutability is important, then the process dictionary means Erlang doesn't qualify, but again, it's pragmatic.
I'd rather have a pragmatic almost actor system than a dogmatic Actor system that's hard to use.
I think that is the key insight of actor-systems.
Actors don't have state. But they can calculate their replacement based on their immutable data. Thus the way an actor at a given address evolves is described by the functions that calculate the successor actors.
Thus you get Pure Functional Programming implemented on top of a fabric of distributed evolving entities. You can understand the behavior and evolution of such a system as a composition of function-calls, where functions always produce the same result for the same arguments.
Perhaps we have different ideas of what an "actor system" is though.
Same question applies to Erlang, which I realize is not exactly an actor system as per the below comment, and has a much more sophisticated error recovery story with supervisors, but the general question holds.
If your services don't deal with internal mutable state, nor high degrees of concurrency then there isn't much gain to be had with an actor system. That said, that begs the question of what the queue is for; just create more instances, since there's no internal state to share.
As soon as you start having internal mutable state and high levels of concurrency, that's where the actor model applies. Queues don't exist for concurrency (you don't need them; just create more executors), they exist for imposing sequence where it is needed (an obvious case; you have a DB connection, you want to only have one query at a time. So every desired query goes into a queue, and the process at the end that owns the DB connection pulls from it). Internal mutable state gets stored inside of an actor; updates and reads get serialized on that actor.
At the highest level, I would describe the actor model as taking a 'successful' model for distributed computing, and making it the only model you use, even locally.
In, let's say Java, for instance, using standard concurrency approaches, it matters where a process lives. My way of operating/communicating to another thread of execution (unit of concurrency) is very, very different than my way of operating/communicating to another machine. Locally I have threads and locks and need to be very mindful. When communicating to another machine, I send a message and that's it (maybe I expect a response, and timeout if I don't get one, but that's really just the same thing, the other machine sending a message).
I don't actually need a queue involved for theoretical correctness unless I need to process messages in sequence (after all, I could have multiple copies of the other process, and send a message to each of them). Now, in the real world I do, simply because if my concurrency gets too large it can't be handled by what the units of concurrency already available (instances, threads, whatever), and scale up takes time, but that's really just a special case of why I need to impose a sequence on messages (handle these first, then handle these, rather than handle all of them concurrently).
The actor model makes this the local model of communication (and so makes the impedance mismatch negligible between local and distributed; so much so that some languages it's actually irrelevant whether you're sending a message to a local actor, or a remote one). Scaling concurrency up internally just means spin up a new actor. When you need to serialize, you send messages to the same actor, where it ends up in a queue.
So it's not solving a different problem, exactly, if that problem is "how do we write systems that can do multiple things at once", but the specifics, complexity, etc, tend to be pretty different. The problems it's solving are a bit more subjective than simply "can we handle this problem", and more "how well do our tools and mental model lend themselves to the problem we're trying to solve".
I kind of disagree with this. I mean, it's kind of correct, if you literally have a sequential problem with immutable state you're gaining nothing from it, but you're also losing nothing (you...will have one actor, no messages being passed, so the only cost is the syntax of the language; the actor model isn't adding any complexity because you aren't using it).
But, a lot of problems we've historically learned to view as sequential are in fact concurrent. Deeply concurrent. We've created entire concepts specifically to impose sequential processing upon innately concurrent activities.
As an example, almost everywhere we use queues we could instead model as unrelated processes able to be run concurrently. You still may have a bit of queueing/scheduling for unbounded processes, but I know from production experience that that is far, far simpler to do and get right using actors than a traditional queue/priority queue and worker(s).
And that's where it shines. It encourages you to start thinking about what in fact -should- be done concurrently, by making that easy rather than a chore as it is in most other languages, and that leads to -less- complexity.
Is it the supporting structure surrounding an actor focused library/language? Lots of person-hours have gone into making them to work well. Some home grown queue + pool of threads working off that queue... not so much.
Some examples:
- Some common patterns. E.g. actor hierachy and supervision, circuit breaker.
- Persistence. The actor persists its state (changes) and can wake up from a sleep.
- Clustering. The actor can live in another node, and you interact with it using the same API.
What I meant is that you can launch a co/go-routine and have a channel that only it can read. That is like 70% the power of having an actor system.
---
> an actor vs a queue with workers
See the sibling comment.
TLDR: Actors each has their own queue. The orderings in each queue are independent. This can help with parallelization if it fits the shape of the problem.
You can also iterate over an array in parallel and only modify the current element and that is like 70% of the power of having an actor system.
"Structured concurrency is a programming paradigm aimed at improving the clarity, quality, and development time of a computer program by using a structured approach to concurrent programming." - Wikipedia
Now, using actors, this was trivial to do. Each actor was basically a glorified state machine, with each command, status update, etc, a message coming in that updated the internal state, and caused it to change state, and, where appropriate, update anything it needed to (such as the internal timer for when it would stop the job). We loaded up everything that had to happen in the next 12 hours, via an actor that would wake up and load more periodically (with an actor registry to find and confirm what existed already, as well as used to route incoming commands from users to change jobs). There is no complexity in managing a global state of priority between the actors; that's a 'solved problem' by the time slicing the underlying actor engine is doing for us; it's well tested, well proven, and invisible to us. Our implementation allows us to treat each job as existing in isolation, and write our code as though it and it alone is the job that needs to be executed. There is no global synchronization state to manage (well, with the caveat of taking the infinite list of future jobs, and ensuring we have actual processes equating to those scheduled to start in the next 12 hours, but that's a comparatively simple problem, and a proper use of sequential ordering. We want to execute on these jobs first), nor should their be based on the problem we're trying to solve.
Using a queue and traditional threading model though? Well, we'd need a synchronized priority queue, as items could have their priority changed at any time, from multiple sources. We'd also still need a registry. We'd need a pool of workers, that are mostly just blocking...hope we don't have > (number of workers) things needing to execute concurrently. Our complexity to manage all of this is high; the level of testing needed to validate that there aren't emergent bugs, also high. Our semi-realtime requirements are probably out the window (high contention on the queue, or high enough concurrency to exhaust the thread pool could lead to very measurable delays in updates). And ultimately, it all comes to the fact that the queue exists to pretend that the state of the world is sequential, and that we then had to then contort ourselves around to bring back to a state that allows for suitable concurrency. Updates to a job can end up killing a worker and inserting something back onto the queue, can remove something from the queue and reinsert it at a different position, etc, all while other workers are trying to access the queue simultaneously.
Actors are the simpler solution.
The comment about an actor's mailbox being a queue is missing the point (though, amusingly, demonstrating it). Queues are sequential. Actors are capable of doing only one thing at a time, and so when they are requested to do multiple things, they will do them sequentially. A mailbox filling with things that don't demand sequential processing is a code smell in an actor system. This is useful if you have limited resource access (queries on a DB connection, say), or need things to be done sequentially, and are very much akin to threads. But if you actually want multiple things concurrently...you should have multiple actors, not send them into the mailbox of one actor (which is very much the queue situation mentioned above; you imposed a sequential ordering on things you want done concurrently. Why?). That's the key difference; in most other languages, the queue is taking multiple things you actually want concurrently, and imposing a sequence to them. In the actor model, your queu...
In short: different computational model -- Actor model vs. Communicating sequential processes.
This really helped me: https://youtu.be/7erJ1DV_Tlo
Edit: added link to video
However, while the single-threadedness is a hard requirement and behavior of the system, the rule that only one message is processed at a time is not.
In Orbit (and Orleans) the ability to process multiple messages at the same time is called reentrancy. it can be specified at the actor or method level usually.
So in your example, say your initial message kicks off a write that is going to take 30 seconds. You start the write and assuming it's async, the actor then suspends while it waits for that to complete. While the actor is suspended another message asking for the progress arrives and that message is reentrant, it can process that message and respond with the progress, then the actor suspends again and at some point another reentrant message arrives or the initial write finishes and the first message finally gets a response. If a none reentrant message is received it will queue in the mailbox until after the first one is completed.
So, with reentrancy you still get the single threaded guarantee but you effectively get interleaving of messages when the actor is suspended waiting for some other async task to complete. This means that if you have an actor with reentrant messages you have to make that code safe to run during any potential interleaving point in another message (for example an await in async C#).
Hopefully that made sense, Orleans has some docs about reentrancy here you might find useful: https://dotnet.github.io/orleans/docs/grains/reentrancy.html
Other actor frameworks (particularly ones that are not based on virtual actors) may have different guarantees and behaviors but from what I've seen there is always some equivalent.
Then what's a virtual actor and how does it differ? You'll eventually end up at the Orleans paper from MSFT research:
https://www.microsoft.com/en-us/research/publication/orleans...
It really has nothing to do with promises (as used colloquially) and is a way to design large distributed systems using message passing and localized state that's persisted.
The repository has virtually no activity since september 2020.
> This can be prevented by persisting state in a database with some form of CAS operations, e.g. Couchbase.
"Exactly once" can be achieved with "at least once" + _local_, reliable state where you can resolve/ignore duplicates. But if that "local" state is actually "remote" – well, then you loose this guarantee (because you can be arbitrarily partitioned from that state so you're in square one again).
http://blog.bioware.com/2015/03/30/launching-into-orbit/
(HN thread: https://news.ycombinator.com/item?id=9300672)
> The last-generation of Orbit powered some of the key technology behind the Dragon Age Keep and Dragon Age: Inquisition. Our plans for the next-generation framework are even more ambitious.
So at least it was used for some of Bioware's games
Especially that we already have a perfectly good, battle-hardened, and relatively lightweight implementation of Actor model with Erlang / Elixir.
Where do you see the list of dependencies? Seems to me to be the ones defined at https://github.com/orbit/orbit/blob/233956001f1206ccbfde72ef..., is that correct? Doesn't look like "an entire container ship" but maybe the NPM madness have ruined me.
> Especially that we already have a perfectly good, battle-hardened, and relatively lightweight implementation of Actor model with Erlang / Elixir.
Yeah, if you're already using Erland or Elixir, why don't you go with that instead? This seems to be for the JVM, so one could assume that the ones who want to use this, is already invested heavily in the JVM ecosystem (which as far as I know, EA is when it comes to backend servers).
Fun fact, the tech that was developed to power Orbit forked and evolved along largely divergent paths to be Uplay and the framework on which the Division and Division 2 online backends were made.
You can still see paths referencing Orbit in uplay, especially on the downloads: http://static3.cdn.ubi.com/orbit/launcher_installer/UbisoftG...
Largely C++ on Windows.
Vert.x uses a concept called Verticles - which are like actors..with an external message bus. Which ends up being more practical if you start leveraging kafka, etc in the architecture.
The community is healthy and helpful, too. I hope to stick with Vert.x for a long time even.
https://docs.dapr.io/developing-applications/building-blocks...
https://github.com/dapr/dapr
Super cool tech! so thank you for your contributions and future work on this.
As far as I've always known him, he always works on networking projects, I believe he worked on the Fable 3 networking for coop and several other games. I can't even imagine what he works on at EA now honestly.
JoeH if you're reading this by chance: Just remember you were an inspiration back in the day (mid to late 2000s) to a lot of people who became developers and learned a ton from your work back then, myself included. To this day I remember asking you questions almost every day. Thank you for your efforts and your patience.
[0] https://www.joeh.ca/
[1] https://github.com/ratatoskr/ratatoskr
This was meant to be a followup to the original version of Orbit which now lives on as orbit-legacy on GitHub.
Unfortunately other priorities have meant this project hasn't proceeded in the past couple of years. I wouldn't really recommend anyone use either of them.
However, the ideas and implementation both for Orbit Legacy and the early version of Orbit 2 contain some ideas and concepts that may be useful to folks.
I still think there is immense value in virtual actors (Orleans from Microsoft is one great alternative for .NET, and Dapr also looks interesting) but I can't say if we'll ever get back to it.
It's always better to focus on delivering the game rather than trying to make an own engine...
When Orbit 1 was initially developed, Orleans was not open source, so the initial implementation was based off the Microsoft Research Whitepaper.
In our specific case, we had a lot of existing experience and services built on the JVM so adopting DotNet for Orleans (even if it had been available at the time) was not an option. The relative drop in priority for Orbit vs other projects was due to a need to dedicate more time to other parts of the platform.
As a large dev/publisher EA has a sizeable central technology group that develops everything from engines (Frostbite) to services (EA Digital Platform).
Ultimately I agree with the philosophy for smaller developers who are focused purely on making a game, but that's just not the reality at large companies and ultimately someone has to develop those services/engines.
If there's something you want to say that will be downvoted, just say it and eat the downvotes.
I'm not saying I agree with "ThE DowNVoteS SaY Im RigHt", I'm saying own your actions.
Edit: And if you don't want to own it, think twice before posting.
When I compare the concurrency model on the JVM (and especially Java itself) to the modern competition from the likes of GoLang and C# there are clear and obvious drawbacks. Even comparing just the Java language and some other JVM languages like Kotlin and Scala, it comes up well short.
Project Loom certainly looks to be a step in the right direction. I haven't looked at it in depth recently but Virtual Threads and Continuations will be a welcome addition and last time I did look it was promising in terms of implementation.
If I were creating a new project today I'd likely reach for GoLang or DotNet (now that Core and Framework are unified) before the JVM, and if I did decide on the JVM I'd choose Kotlin over Java.
I'm historically a huge fan of the JVM and Java, and I'd like to see it back on top, but right now I think the competition is better positioned and until the many initiatives going on (such as Loom) start to have a major impact on the ecosystem, it's no longer automatically my first choice.
In our telemetry engine VAST, we've been using CAF successfully for several years for building a distributed system that always has a saturated write path. CAF provides a credit-based streaming abstraction as well, so that you can have backpressure across a chain of actors, making burst-induced OOM issues a blast from the past. You also get all the other benefits of actors, like linking and monitoring, to achieve well-defined failure semantics: either be up and running or collectively fail, but still allowing for local recovery—except for segfaults, this is where "native" has a disadvantage over VM-based actor models.
With CAF's network transparent runtime, a message ender doesn't need to know where receiver lives; the runtime either passes the message as COW pointer to the receiver or serializes it transparently. Other actor model runtimes support that as well, but I'm mentioning it because our experience showed that this is great value: we can can slice and dice our actors based on the deployment target, e.g., execute the application in one single process (e.g., for a beefy box) or wrap actors into single OS processes (e.g., when deploying on container auto-scalers).
The deep integration with the C++ type system allowed us to define very stable RPC-like interfaces. We're currently designing a pub/sub layer as alternate access path, because users are interested in tapping into streaming feeds selectively. This is not easy, because request-response and pub/sub are two ends of a spectrum, but it turns out we can support nicely with CAF.
Resources:
- CAF: https://github.com/actor-framework/actor-framework
- VAST: https://tenzir.github.io/vast/docs/understand-vast/actor-mod... (sorry for the incompleteness, we're in migration mode from the old docs, but this page is summarizing the benefits of CAF for us best)
- Good general actor model background: http://dist-prog-book.com/chapter/3/message-passing.html#why...
The reasoning in the book that you linked does not make sense to me. Any serious language nowadays has some kind of lightweight thread abstraction which you can spawn thousands or millions of without making a dent into system resources, so this is a non-issue. It goes on to say that
> This alleviates the need for the programmer to reason about an entire system. Instead the programmer has a fixed set of concerns, meaning they can ensure behavioral correctness in isolation, rather than having to worry about an interaction they hadn’t anticipated occurring.
which is just blatantly false. You can reason about a single actor, yes, but doing so is not helpful when you are interested in the correctness of the entire system. Since actor boundaries are an operational concern, they do not align with the domain logic. To understand the domain logic you always have to look at interactions of multiple actors (a single actor would be pointless) all with their own mutable state, all which can receive messages from anywhere at any time with no guarantees of timely reply to its own messages. A functional programming approach on the other hand is much easier to reason about with a well defined denotational semantics, no peer-to-peer interactions, no mutable state (generally) and composition of components that are aligned with domain boundaries and thus meaningful in isolation.
> Without a lightweight process abstraction, users are often forced to write parts of concurrent applications in an event-driven style which obscures control flow, and increases the burden on the programmer.
This is exactly the argument _against_ actors in my mind. Because actors communicate only through messages, they enjoy high decoupling but the price for this is horrible cohesion. Any domain logic that is spread across multiple actors will quickly become incomprehensible and unmaintainable. The same is true for event-driven microservices and a great deal of architectural work goes into making the microservices as large as possible to improve cohesion while making them as small as necessary to be independently scalable.