Python, Go or Haskell?
I'm in the interesting position of setting up a team from scratch: hiring the first software developers in a well-funded company. There is a team of data scientists who have been creating models in Python (which end up as docker images). I have no legacy code of any importance for all the surrounding code (UI, data fetching, data storage, etc.) that I need to support or maintain.
The guidance I'm getting from the CEO is that he wants to build a cutting-edge, next-generation company. There's budget to hire 10-20 tech staff.
I'm comfortable enough in all three languages (and others besides).
I don't want this to turn into a flamewar: the main criteria I care about are:
1. Long-term supportability
2. Bugginess of typical code generated in the language.
3. Availability of developers
What are your thoughts?
68 comments
[ 2.7 ms ] story [ 120 ms ] threadGo is also nice and will bring you support for types and high er runtime performance (but slower development speed and maintenance speed).
Haskell will be hard to find developers for, but the ones you find will be very good. But getting a team of 20 to use haskell will be a great challenge, and also the library support for data science is tiny.
- Data Science: Pandas, Scikit-learn, Numpy, ... - Web (Django,..) - APIs (Flask,..)
Also many Artificial Intelligence Tools and Apis of other services have support for Python.
The size the community is also very important. When There are a big community is more easy to find developers and events/meetups where the community talks about Python and use cases.
Haskell is also a good lenguaje and easy to learn but Python and Haskell are languages with differences https://wiki.python.org/moin/PythonVsHaskell
1. Long-term supportability: I'd definitely favor Go here because of the type system. I work(ed) in large Python codebases that deteriorate because ensuring correctness requires enormous amounts of test coverage, and refactors are absolute nightmares. People tend to rewrite code instead of refactoring because it's too difficult to figure out what on earth is going on. Deployment and cross-compilation is also a breeze with Go binaries, which has been another historical painpoint with Python.
2. Bugginess of typical code generated in the language: Favor Go for the above reasons. When the code really has to work, I choose Go. If I had more Haskell experience (and if it were a nice language to deal with), I'd probably favor Haskell for its correctness, because I still run into annoying nil pointer exceptions and array out-of-bounds in Go relatively often. I use Elm (Haskell-influenced frontend language) and love it for its lack of runtime errors.
3. Availability of developers: I'm sure there are more Python developers than Go developers. I work with data scientists, and I don't know many who are comfortable in a Go ecosystem, so if you're hiring lots of scientific computing folks, I'd go with Python. In terms of software engineers, when I've hired folks, a lot have expressed particular interest in working in Go even with no prior experience, which is a good signal. Easy for Java/C-style programmers to pick up.
Note that: You can do both/all three. We extensively use Go for systems building, and Python for data science.
On a side note: might it be the case that the Go codebases you've worked with are just newer/smaller than the Python ones? Do you work with Go systems that feel as much legacy as you've encountered with Python?
In terms of legacy code, though, my metric is the following: When I haven't touched code in one year and need to go back to fix something, what language would I want that code to have been written in?
I prefer that code to be in a system that's easy and obvious to reason about, i.e. with a type system.
This, sadly, brings us back to the problem.
I'm of the opinion nowadays that these rules need to be mandatory at a group level (i.e. either enforced by the compiler or by convention at your workplace) to be useful.
When coupled with mypy (http://mypy-lang.org/) they can be used to do type checking at "compile time", and with DropBox's pyannotate (https://github.com/dropbox/pyannotate), they can even be auto generated for existing code.
I'm not saying Python with type hints is better than a statically typed language, merely suggesting a possible solution if Python ends up being the preferred language.
Personally i love Go for the things i've used it for, and i've written and maintained large code bases in Python, and hated every minute of it. The same can be said for large code bases i've written in C, or even Java.
I've written kernels for GSM mobile phones, software for sortation devices (airport baggage sorters, mail sorters, etc), banking software, and despite our teams best intentions, given enough time and customization to meet customer demands, there is always some remote corner of the code that ends up as a "steaming pile"
- Python is established for Data Science vs Haskell You could use Haskell for whatever models you need (although finding 10-20 people for that might be a challenge) and it would be really easy to refactor for any changes you might have but I dunno if data science is something you'd use Haskell for. Most data science applications I can think of are either scripts to inspect/clean data (Python works well) and the rest involve a lot of trial and error (Haskell errors aren't great).
- Haskell libraries are awesome You could use Haskell for whatever important libraries you use on a regular basis - it's incredibly easy to write APIs for different languages in Haskell (which is great if you switch languages for some particular use) and it's easy to refactor (which is important in a new company that's likely to change a lot). Also, it's a lot harder to write shitty core libraries in Haskell.
I'd honestly recommend sticking to whatever language you already have code in. If you already have python models built, stick to them!
If they are opposed to Haskell or Go, then what are you going to do - fire them? Are you budgeting time/money to retrain them?
Your customers don't care what you use, unless you are selling them the actual software. (But then you would know what your customers want, and wouldn't be asking this question.
If you use language X because of a perceived marketing advantage that your customers will see you as "a cutting-edge, next-generation company" then can you estimate how much that marketing is worth? And compare it to other marketing approaches?
My view, as a long-time Python user, is of course Python.
Having worked in an environment where we had data science people, I can tell you that we talked a lot about helping them write code, and it never really happened. Let them use what they're productive with, and let your devs use what they're productive with. If those are both the same language then that's nifty, but I wouldn't bother making it a goal.
Basically, any language / ecosystem that should meet your most basic business and technological requirements should be heavily in contention if it is among the most widespread in the community where you're situated. And similarly, if a language is significantly less common than a language that meets the requirements, you probably shouldn't consider it.
Whether that is Python, Go, Haskell or something altogether different, only you know.
You mention long-term support. I'd have to give Haskell high marks on that one. You can use Stack to get highly repeatable builds with LTS releases of Haskell and the packages you rely on. More so than any other language I'm familiar with, Haskell does a great job with long term support and builds that should be 100% reproducible in 5 years.
As far as bugginess, Haskell obviously wins here again. If you're getting runtime exceptions in Haskell then you're doing something wrong. If rock-solid stability is important to you (and you list it as #2 on your list, so I assume it is), then Haskell is hard to beat.
Availability of developers is a tough one. Python clearly has the advantage here, despite being markedly worse on the first two sections. I don't know many Haskell developers, but the Haskellers I've met have all been exceptional software developers in general.
I find it unlikely that you're going to be able to hire 20 people who already know Haskell, unless you're willing to hire remotely. However, if you're willing to teach, I think it could work out. For example, the Elm language would be a fantastic gateway to Haskell. Build your front end with Elm and make new developers work on the front end for a few weeks. Once they're comfortable with it, switch them to the back end.
Really? I've never used Stack, but the experience of building dependencies from Hackage has always been awful in my experience. Even if Stack did change the game (I think it pins down the exact versions of package dependencies, I would figure using third party Haskell libraries is still a major pain:
The reason is that the types in the APIs are very elaborate (elaborate types of course is generally a Haskell thing), which means very often they are not super stable. So upgrading libraries will be no fun. And you will want to upgrade, whether you use Stack or not.
(Note, I haven't used Haskell seriously in a long time. Maybe someone with more up-to-date experience can validate these thoughts).
or you give julia a try.
I have been struggling with my business vs pleasure languages for some time now. I work 95% in C# but whenever I find willpower to do some hobby coding I go for Haskell.
Every time I go for a new hobby project after not coding in Haskell for a while I have to go through the hurdles of remembering how to set things up, downgrading from Visual Studio to Visual Studio Code with some broken Haskell language extension and last time before I started coding my new hobby project I asked myself: What would it take for me to make C# as lovable as Haskell.
I came up with these things which annoy me most (I know everyone has their own list): 1. Lack of higher order types 2. Lack of syntactic sugar for partial function application et al. 3. Lack of GADTs 4. Still waiting for those nullable reference types.
On the other hand I would get: 1. Great language server with support for refactorings, formatting etc. 2. Test tooling and frameworks and performance analysis tooling and frameworks. 3. Way bigger ecosystem of open source libraries.
While I see a reason to go for either C# or Haskell, I really don't think it's worth going for Python or Go instead in most cases. I just need a better type system than what Go has to offer (and Python doesn't do any type checks without 3rd party support). Hell, I'd go for TypeScript or PureScript before these 2.
C# has grown a lot in the past few years. With C# 7 you have all those fancy things like local functions, pattern matching, deconstruction, async improvements etc. You have a stable language and ecosystem, open source compiler and language server (Rosyln), free editor support if you want to save (Visual Studio Code) or if you want enterprise tooling you can go up to Visual Studio Enterprise. With .Net Core going into 2.0 the coverage of .Net is really broad, and Entity Framework and Asp.Net are being rewritten from scratch for all platforms. It's easy to do foreign language interfacing if you want to use some highly performant lib like LAPAC or something and someone usually already wrapped around it.
Btw, if not Haskell, check out PureScript
The development is also quite active and not run by one person anymore.
I am quite pleased to be honest.
[1] https://github.com/haskell/haskell-ide-engine
If you do end up going the Python route, I suggest you enforce using Python 3 type annotations, and use an external validator (such as mypy) to try and capture as much of that value as possible.
If you're a pure data science startup, most of your talent pool will be most proficient in Python, and the highest quality APIs will be in Python.
If you're a web app, going for Go makes more sense, as the language was built from the ground up with these things in mind.
If you're business tooling or service oriented.. I'd go for C#.. it has a huge talent pool, super high quality libraries and ecosystem, etc
I would expose the Python code using Flask microservices and do the rest in either Go or Haskell. But I have no idea what needs to be built outside of the data science part. If you have lots of small moving parts you can get away with using Python everywhere since you'll not end up with big balls of code.
Data science stuff? Python of course. Haskell has things like repa but still very immature compared to Python. You will have to reinvent the wheel a lot if you choose Haskell.
But if your developers are going to do something very different from data science, say building a typical web app, Haskell can be a good choice. As usual refer to Gabriel Gonzalez’s State of the Haskell Ecosystem: https://github.com/Gabriel439/post-rfc/blob/master/sotu.md
Haskell especially excels in (2): bugginess of typical code is pretty low even without extensive unit testing; Go probably fares pretty well but Python is terrible for a large code base. But on the other hand (3) could also potentially be an issue for Haskell, depending on the geographical location, whereas Python would likely not.
Having worked (and still working) in a Haskell startup, though I believe ultimately technology doesn’t really matter that much. The business is more important than the technology.
If you do choose Haskell though, my advice is that don’t use too many fancy type system features. A boring business app doesn’t usually need them. Stick to basic language features and perhaps just a dozen most popular language extensions; use the rest very sparingly.
I guess there are domains where it's adequate, but what are those?
Haskell is a general purpose lang (unlike Go), and compiles to native code (unlike Python) which makes it quite fast running.
There's a list here:
https://wiki.haskell.org/Haskell_in_industry
Plus it's incredibly stable. Once you learn to avoid a few sharp edges (head, !!, etc.) you should never encounter an exception at runtime. Literally never.
Python you already have, I'd not ditch it, it can live. Main thing it lacks, imho, is a type system which relates to 1 & 2.
I'd personally not care for 3 too much. You find less Haskellers, but also less competition, and according to many they are better.
I'd say all 3 langs you named allow point 1 "well enough". Point 2, I'd say: Haskell > Go > Python, with significant margins in between.
Long-term supportability: As long as you are not choosing a proprietary tech like Microsoft Access, you should be just fine. There are a lot more code bases which suffer from bad quality than from abandoned base technologies. So its much more important that you know exactly how you want to use a language and let your team create rules (bottom up) on what they want to allow themselves to do and what they don't want to (like a semantic style guide), than to pick the golden language in the first place.
My advise would be: Start by building a team. The technologies will be the result of what your team is able and likes to use. One condition: Only allow technologies they have positive personal experience with (no "I always wanted to try XYZ").
We've spent so much time fixing obscure bugs and behaviors that betray everything we've learned in the languages we used previously in our careers that it was really not worth it. And we still do. Finding developers is also an issue, as most developers do not really want to learn a new language, and Go is still not that common in enterprises (I used to be like that myself, but I've learned long ago that this shouldn't be such a big factor when thinking about a new job).
You can make the case that we're simply not good enough developers or Go is just not a good fit for the type of developers we are. Maybe. But the choices made in designing this language, and its popularity, absolutely baffle me.
But at the same time you probably have good reason to try that, as that was the way you did it in the past (with other languages). So while Go allows you to quickly learn to write code, it is a much harder challenge to learn how to structure you code in simple small packages. In the end that brings a lot of benefits (testing is easier), but if you try to create a project with very few packages you will go through lot of pain due to the lack of structure.
Its very similar with interfaces. Programmers coming from other languages mostly ignore them in the beginning and it takes a while until they understand that exporting types from a package is a failure and that interfaces would have been a much better choice.
In my opinion, the key to successful Go development is building many small, well tested library packages while using interfaces a lot.
Your point about interfaces is mostly correct though, and we're progressively learning to utilize them better.
That sounds very frustrating; I'm sorry your day-to-day became consumed by such things. Would you mind providing an example of such an "obscure bug and behavior"?
1. We have a client component that makes HTTP requests to a server component of ours. We've had production downtime issues in the server caused by filehandle exhaustion. Turns out the client was creating a new HTTP connection for every request rather than reusing connections, and it did that because Go expects you to read the response body in its entirety even if you don't need it, otherwise connections aren't reused. And in many of the cases we thought that we were reading the body in its entirety, for example in locations where the JSON decoder was reading and decoding the response, but sometimes it simply did not really read it in full, despite definitely parsing the entire JSON structure returned from the server. A "bad" net/http client can easily bring down a net/http server.
2. We've had several instances of severe memory leaks, with the Go garbage collector not releasing or reusing memory ever. These were sometimes very difficult to track down with the available Go profiling tools (pprof for example), and even more difficult to understand. And in some of these cases we're still not even sure we understand despite fixing the issues, mostly because our code is not different than examples in the Go documentation itself. For example, we had functions that read a stream of bytes into a byte slice, then cast that byte slice into a string and return it. We would intuitively think that the memory allocated for that byte slice would be released, but apparently the casting performed constitutes an active reference of the data (rather than a copy of it), so the data is not marked for garbage collection. But the most obscure part is that it was never released, even after the cast data had no more references to it. Memory management in Go is very hard for us to reason, despite the language claiming that developers needn't worry about this subject as it is handled for them.
This is just the top my head, I can bring more.
I also read Alan Donovan and Brian Kerninghan's book "The Go Programming Language" [4]. So mostly these four, and of course simply writing code (along with the standard library documentation [5]).
Since then I've also gained a lot of info and insight from various articles and blogs, such as "Debugging Performance Issues in Go Programs" [6] from an Intel blog, various posts from Peter Bourgon's blog [7] and Dave Cheney's blog [8], both of which I highly recommend. Oh, and "The complete guide to Go net/http timeouts" from the Cloudflare blog [9] is also worth reading.
Previously I was mostly a Perl developer, but I also work(ed) a lot with JavaScript, and to a (much) less extent with Haskell, Racket and Nim.
[1] https://golang.org/doc/code.html
[2] https://golang.org/doc/effective_go.html
[3] https://golang.org/doc/faq
[4] http://www.gopl.io/
[5] https://golang.org/pkg/
[6] https://software.intel.com/en-us/blogs/2014/05/10/debugging-...
[7] https://peter.bourgon.org/articles/
[8] https://dave.cheney.net/
[9] https://blog.cloudflare.com/the-complete-guide-to-golang-net...
Mentioning here verbatim -
> The http Client and Transport guarantee that Body is always non-nil, even on responses without a body or responses with a zero-length body. It is the caller's responsibility to close Body. The default HTTP client's Transport may not reuse HTTP/1.x "keep-alive" TCP connections if the Body is not read to completion and closed.
Regarding #2, I think what you are taking about is this - https://blog.golang.org/go-slices-usage-and-internals (gotcha section). But maybe I am wrong.
To be more specific: Python 3.6, with type annotations and unit tests, and good libraries (pandas and others for the data, Tornado for Web, SQLAlchemy for databases etc).
But it's much more important to get your architecture right, to know exactly what is the data flow and what is supposed to happen when and where. No single language will give you answers to that, while the architecture might decide the choice of language(s).