Sounds very relatable coming from c++. Managing dependencies is generally not too fun, it's very easy to waste a whole day (or more) fighting with the build system/package manager/linker. I like to think it's the tools but then you see a really smart c++ developer and that excuse starts to sound a little weak. It seems c and c++ attract a relatively high average percentage of developers who are willing to go learn the ins and outs of build systems and how compilers and libraries all interact.
I'm curious about the happy path of ocaml vs c++, I'm not sure how much of a happy path there even is in c++. I guess if every dependency uses the same build system as the main project, maybe that's a happy path? Hard to call cmake a happy path though, more so just a tolerable path. Conan sounds nice in theory but my experiences do not match the hopes I had. I remember golang dependencies not being too bad when I tried it, really need to mess around with rust a bit to see what that's like too since I hear so much good about it.
The article mentions this a bit, but the Happy Path in OCaml is pretty nice. I typically don't run into any of these issues because I'm not trying to use C/C++/Rust libraries in esoteric ways. I've successfully used binding libraries for OpenSSL, libcurl, SQLite, and Postgres. It's fairly simple to get run-of-the-mill C libraries working in OCaml.
The biggest problem in C++ always comes from the fact that the there's no underlying stable ABI to take care of everything (and there cannot be due to performance reasons), and that build options are so important and affect the ABI. Most C++ projects are inherently multi-language system and bring the complexity of all languages to their build systems.
Not really with OCaml so much as with using native libraries from OCaml. Which is an awkward thing in many languages, especially if you want it to be cross-platform as this author does.
F# tooling is converging on just using the official .NET cli these days. FAKE, Paket, etc. have some great ideas but seem to work less reliably across a variety of setups.
I program all my personal projects, and I have been able to completely ignore the existence of FAKE and Paket. The only thing missing from NuGet, for me, is the ability to install a package directly from a GitHub repository instead of being forced to publish a NuGet package. I don't know if that's still true but even Poetry in Python and Hex in Elixir can do that.
I think the author's problem was using dune at all. It's a very weird and obscure tool, and not necessary, you can just run ocamlopt commands from existing build systems (like autotools, meson etc).
I've used OCaml in production for 2 decades without ever needing to use dune, opam etc.
Whenever I ran into trouble (up until ~2019) with dune I would first get it to work with (ocamlfind) ocamlopt, look what the lowest level needed to look like, and then reverse engineer to make dune spit out the same thing. I would hope things have improved since
I cordially disagree. I do not find dune weird or obscure. It’s formally part of the OCaml platform: https://ocaml.org/platform, which I think negates at least the obscure bit. It was “new” to me for a hot moment, and it’s certainly a less common format for config files (s (expressions)). However, what I see is a tool that solves concrete problems that uses a declarative, structured format (not very weird). It also has pretty great docs!
I have to deal with dune across all the Fedora OCaml packaging (something like 200 different upstreams) and it's quite difficult to deal with. Not the worst build system out there, but makes me a bit sad every time I have to modify something in a project.
Dune is part of the platform mostly because it was there and was the tool used and developed by Ocaml biggest corporate user but it’s hard to disagree with it’s weird (s-expressions in a non LISP, really?) and a bit obscure. Esy seems to solve most of these issues at the costs of a tie-in to npm.
I don't find it particularly weird or obscure either. I've had to speak over the FFI a handful of times and it was honestly a better experience than go's story (which also wasn't terrible IMO). But I also recognize that I have a high pain tolerance for build systems.
I will conceed though, the Tarides folks know their shit when it comes to OCaml projects.
The black-boxiness of dune has frustrated me as well, but to its credit it does a lot of work for you. Its biggest problem might be documentation; at least the last time I looked at it, there were only very basic examples plus a listing of the keywords with terse descriptions.
The art of writing a reference manual that actually explains how the software works is almost lost, it seems.
> The art of writing a reference manual that actually explains how the software works is almost lost, it seems.
The original INRIA docs are also better than the new things on ocaml.org. Literally the only issue with the "old" INRIA pages was the poor quality of the mailing list archive, the rest is great!
I want my tar archives and docs from INRIA. If I was rich, I'd pay for a grant to update the "old" documentation.
I often hear this misinformed remark from people who only ever need to share software within a single company or just upload something to their one web site. For us, RPM solves a very important problem: How to develop and test the exact bits that we will ship to customers, where they will install and integrate that software on their own servers, in a huge variety of environments. For this while RPM is not completely ideal in all ways, it's vastly better than other OCaml tooling (apart from .debs I suppose).
My comment was more pointing out that the reason you didn't need to use opam is because you package your dependencies yourself, with rpm. And that most people won't be doing this because in general there is a large set of packages out there which can potentially be used in the ecosystem, and growing larger day by day. So like I said, it's not scalable for others to do what you're doing and in most cases opam is the way to go.
I am sure you have specialized needs and find that rpm is best suited for them, although I don't see how opam fails to solve this issue either (anyone can maintain their own opam repository with locked down versions of packages to match exactly what they need to ship to their customers), but that's a separate argument.
We're shipping more (by value) OCaml software to end users than probably anyone. To say this is "specialized" is absurd. Also opam doesn't do what we need, showing you don't understand the requirements.
Guilty as charged, I don't fully understand your requirements because you haven't specified exactly what they are, you've spoken in fairly abstract terms that seem like they should be achievable by a locked-down opam repository. Of course I'm sure now that you have many other requirements which are not achievable by opam, but when I speak about 'specialized' I am thinking about the vast majority of OCaml development and teams, not the 'value' (which I don't doubt, I am sure you are shipping great OCaml software).
For example, avionics systems are developed in a specialized way due to rigorous safety requirements, yet they offer critical global value. They're still specialized requirements though because most software is not developed like that.
OCaml itself is built with Makefiles, which just works.
Coq has used various new tools over the years and at times has been more difficult to build than 15 years ago. With OCaml-5 the situation seems to have gotten better: At least the infamous caml4p problem was solved, and I "only" had to fix a wrong ocamlfind config file that was installed by default.
ocamlbuild is generally good, but Dune and Opam are a black box that is simply often broken. I wish more mature projects switched to Makefiles, which despite all the hate for them are always the most reliable build way.
But OCaml-5 is so great that all of this is minor bickering! It is confusing for beginners though.
Coq doesn't seem meaningfully harder to build than in the past but maybe I'm just not in touch. It was always the Gtk bits that added issues for me, not Dune or anything.
> which despite all the hate for them are always the most reliable build way.
Based on the amount of totally broken Makefiles I find on a regular basis as a package maintainer (logic bugs, recursive make, invalid deps = no parallel builds) suggests to me this is not the case at all. Doesn't even touch on how awful these things are often written, even if they work reliably enough. Insane idioms in the name of one-liners, implicit dependencies on GNU specific features, incredibly inefficient shell pipeline commands, overly large rules that mean many intermediate steps aren't cached, subtly broken rules and invocations based on spaces or non-utf8 chars, etc.
I worked on the Glasgow Haskell Compiler and the truth is that the Make-based build process was only reliable because a massive amount of effort was spent on it. I suspect it's probably the same with OCaml, and not because Makefiles are special. There are degrees to it all, but they certainly don't have good syntax (too terse), aren't easy to read (batshit insane escape rules that were fixed by Plan 9 mk in 1991), and make expressing many types of dependencies really awkward by design (I want a dependency on an environment variable?) They're just really easy to get started with and you can write something broken that looks like it works zero effort, honestly.
OCaml might be built with makefiles, but it's also not an easy path at all. Adding a module to the compiler or stdlib is really not straightforward compared to what it'd be with dune (you have to list your new file somewhere, regenerate dependencies, order matters in obscure ways, etc.). To be clear, OCaml is a special project since it also has to solve bootstrapping issues, but still, it's not an argument for "makefiles just work". Dune works extremely well for pure OCaml projects.
I'm a long-time Ocaml developer and I ended up writing my own build system. It absolutely has flaws but for my usecase, it just works. It's called pds and older (but working) versions are in opam (I just haven't updated it because I don't think anyone uses it or is interested). It takes a TOML file and outputs a makefile.
I like it because:
1. Well, I made it because I was dissatisfied with what things were like at the time, so it does almost exactly what I want. To be fair, dune was in its infancy when I created pds but even now I think pds is easier to use for most projects.
2. It outputs makefiles so you have all the power of makefiles if you need it. pds itself is really quite small.
3. pds could have different backends, such as a dune one. This doesn't quite work in practice because complex repos that I have end up using makefile features, but it wouldn't be a huge reconceptualization to make it support other backends, but it would make it more complicated.
4. Because the pds input file is TOML, I have easily built other tooling on top of it. For example I have hll which takes a pds config and outputs an opam package. hll needs a small amount of configuration but otherwise it pretty much just works. I also created merlin-of-pds which produces a merlin config of your pds build.
4. pds can support very complex builds, for example I have kqueue bindings using ctypes which has to go from compile and run ocaml, which produces c code which then needs to be compiled and run, which then again produces ocaml code, which is then the library interface to kqueue. There is no magic to make this happen although it does twist the brain a little bit to implement.
The Ocaml thought leadership in terms of builds is focusing pretty hard on making Dune the answer. Personally, I don't like dune. I think it's complicated, it's too ocaml focused, it's too "cute". In that sense I find it similar to opam, which I do use everyday but again, I find it too cute. The dependency version solver always does "the right thing" given its input which many times is not "the thing I want to do".
My biggest complaint, as things are now, is that sometimes the thought leadership makes decisions that makes the life of people outside that specific toolset harder. It's not that often though. The biggest painpoint I had in the last few years was switching to ocamllsp, which was designed to depend heavily on a dune world. It took a lot of asking to get supporting merlin files added to ocamllsp.
And I also think this is what I enjoy about the Ocaml community. I feel fairly confident I can build my own tooling in a way that serves my needs and I can use the larger ecosystem where necessary. Ocaml has a culture of rugged indivdualism, IMO. I think it will slowly converge as new people come in, with expectations, or us people that care more about using Ocaml than about using other peoples' tools age out.
P.S. I really wish opam was built on top of Nix. But I literally wish that about every bespoke package manager, which just end up making the same mistakes over and over and solving the same problems (poorly) over and over, and nothing is compatible, and it just sucks. Nix everywhere, please.
Honestly using Rust recently has taught me that, the only reason C and C++ dependencies in Rust are remotely tolerable (unlike most other languages) is because Rust has an unusually high degree of talented developers, many former C/C++, who are able and willing to mercilessly bludgeon these libraries and their build systems into a square shape, then fit them in the circle-shaped hole we call "build.rs".
I guess my point is, yeah all of this sort of stuff sucks, and to the extent it sucks less elsewhere, it's mostly due to the sheer effort consumed by it.
27 comments
[ 3.3 ms ] story [ 64.3 ms ] threadI'm curious about the happy path of ocaml vs c++, I'm not sure how much of a happy path there even is in c++. I guess if every dependency uses the same build system as the main project, maybe that's a happy path? Hard to call cmake a happy path though, more so just a tolerable path. Conan sounds nice in theory but my experiences do not match the hopes I had. I remember golang dependencies not being too bad when I tried it, really need to mess around with rust a bit to see what that's like too since I hear so much good about it.
I've used OCaml in production for 2 decades without ever needing to use dune, opam etc.
I will conceed though, the Tarides folks know their shit when it comes to OCaml projects.
The art of writing a reference manual that actually explains how the software works is almost lost, it seems.
The original INRIA docs are also better than the new things on ocaml.org. Literally the only issue with the "old" INRIA pages was the poor quality of the mailing list archive, the rest is great!
I want my tar archives and docs from INRIA. If I was rich, I'd pay for a grant to update the "old" documentation.
I am sure you have specialized needs and find that rpm is best suited for them, although I don't see how opam fails to solve this issue either (anyone can maintain their own opam repository with locked down versions of packages to match exactly what they need to ship to their customers), but that's a separate argument.
For example, avionics systems are developed in a specialized way due to rigorous safety requirements, yet they offer critical global value. They're still specialized requirements though because most software is not developed like that.
Coq has used various new tools over the years and at times has been more difficult to build than 15 years ago. With OCaml-5 the situation seems to have gotten better: At least the infamous caml4p problem was solved, and I "only" had to fix a wrong ocamlfind config file that was installed by default.
ocamlbuild is generally good, but Dune and Opam are a black box that is simply often broken. I wish more mature projects switched to Makefiles, which despite all the hate for them are always the most reliable build way.
But OCaml-5 is so great that all of this is minor bickering! It is confusing for beginners though.
> which despite all the hate for them are always the most reliable build way.
Based on the amount of totally broken Makefiles I find on a regular basis as a package maintainer (logic bugs, recursive make, invalid deps = no parallel builds) suggests to me this is not the case at all. Doesn't even touch on how awful these things are often written, even if they work reliably enough. Insane idioms in the name of one-liners, implicit dependencies on GNU specific features, incredibly inefficient shell pipeline commands, overly large rules that mean many intermediate steps aren't cached, subtly broken rules and invocations based on spaces or non-utf8 chars, etc.
I worked on the Glasgow Haskell Compiler and the truth is that the Make-based build process was only reliable because a massive amount of effort was spent on it. I suspect it's probably the same with OCaml, and not because Makefiles are special. There are degrees to it all, but they certainly don't have good syntax (too terse), aren't easy to read (batshit insane escape rules that were fixed by Plan 9 mk in 1991), and make expressing many types of dependencies really awkward by design (I want a dependency on an environment variable?) They're just really easy to get started with and you can write something broken that looks like it works zero effort, honestly.
It should be a crime to mention "Make" and "just works" in a single sentence.
I like it because:
1. Well, I made it because I was dissatisfied with what things were like at the time, so it does almost exactly what I want. To be fair, dune was in its infancy when I created pds but even now I think pds is easier to use for most projects. 2. It outputs makefiles so you have all the power of makefiles if you need it. pds itself is really quite small. 3. pds could have different backends, such as a dune one. This doesn't quite work in practice because complex repos that I have end up using makefile features, but it wouldn't be a huge reconceptualization to make it support other backends, but it would make it more complicated. 4. Because the pds input file is TOML, I have easily built other tooling on top of it. For example I have hll which takes a pds config and outputs an opam package. hll needs a small amount of configuration but otherwise it pretty much just works. I also created merlin-of-pds which produces a merlin config of your pds build. 4. pds can support very complex builds, for example I have kqueue bindings using ctypes which has to go from compile and run ocaml, which produces c code which then needs to be compiled and run, which then again produces ocaml code, which is then the library interface to kqueue. There is no magic to make this happen although it does twist the brain a little bit to implement.
The Ocaml thought leadership in terms of builds is focusing pretty hard on making Dune the answer. Personally, I don't like dune. I think it's complicated, it's too ocaml focused, it's too "cute". In that sense I find it similar to opam, which I do use everyday but again, I find it too cute. The dependency version solver always does "the right thing" given its input which many times is not "the thing I want to do".
My biggest complaint, as things are now, is that sometimes the thought leadership makes decisions that makes the life of people outside that specific toolset harder. It's not that often though. The biggest painpoint I had in the last few years was switching to ocamllsp, which was designed to depend heavily on a dune world. It took a lot of asking to get supporting merlin files added to ocamllsp.
And I also think this is what I enjoy about the Ocaml community. I feel fairly confident I can build my own tooling in a way that serves my needs and I can use the larger ecosystem where necessary. Ocaml has a culture of rugged indivdualism, IMO. I think it will slowly converge as new people come in, with expectations, or us people that care more about using Ocaml than about using other peoples' tools age out.
P.S. I really wish opam was built on top of Nix. But I literally wish that about every bespoke package manager, which just end up making the same mistakes over and over and solving the same problems (poorly) over and over, and nothing is compatible, and it just sucks. Nix everywhere, please.
I guess my point is, yeah all of this sort of stuff sucks, and to the extent it sucks less elsewhere, it's mostly due to the sheer effort consumed by it.