Bridging Elixir and Python with Oban (oban.pro)

Very nice, Oban is great. I effectually found a similar approach with pgflow.dev (built around pgmq) - but the stateless deno "workers" are pretty unreliable and built an elixir worker (https://github.com/agoodway/pgflow) that can pick up and process jobs that were created by pgflow's supabase/typescript client. So maybe there's an opportunity also with Oban to have a TypeScript/Node client that can insert jobs that Elixir/Python Oban can pick up. Also, I wonder if another approach vs the python workers picking things up is to have elixir workers call/run python/lua, etc code or is that too limiting?

I feel like if you need to utilize a tool like this, odds are pretty good you may have picked the Wrong Tool For the Job, or, perhaps even worse, the wrong architecture.

This is why it's so important to do lots of engineering before writing the first line of code on a project. It helps keep you from choosing a tool set or architecture out of preference and keeps you honest about the capabilities you need and how your system should be organized.

We have a similar use case. All Elixir code base, but need to use Python for ML libraries. We decided to use IPC. Elixir will spawn a process and communicate over stdio. https://github.com/akash-akya/ex_cmd makes it a breeze to stream stdin and stdout. This also has the added benefit of keeping the Python side completely stateless and keeping all the domain logic on the Elixir side. Spawning a process might be slower compared to enqueuing a job, but in our case the job usually takes long enough to make it irrelevant.

I don't see the point of Elixir now. LLMs work better with mainstream languages which make up a bigger portion of their training set.

I don't see the point of TypeScript either, I can make the LLM output JavaScript and the tokens saved not having to add types can be used to write additional tests...

The aesthetics or safety features of the languages no longer matter IMO. Succinctness, functionality and popularity of the language are now much more important factors.

[dead]

I absolutely love Elixir, but if this is the bridge you need to cross, just write it in Python in the first place.

Oban is great!

This is a similar concept to Faktory, which uses a built in Redis server to manage shared job state.

You then implement workers in your language of choice and subscribe to queues.

Very interesting though, the article mentioned a few things I hadn't considered before like shared access to one database from multiple (different) apps.

I wonder how database schema state is handled in a case like that. And CI/CD.

When rust was still a fairly new language i remember using capn'n proto to communicate between some rust code and python as a way to experiment with handing off performance critical tasks to a compiled language.

I wonder how well a similar approach would work with elixir + python. Elixir obviously has very easy process isolation, but I think you'd be stuck using a NIF approach for Elixir, which probably removes any reason to try capn'n proto over just protobufs?