HN Explorer

Ask HN: Is it worth it for me to learn Go or Rust as a Data Engineer?

5 points by astrea ↗ HN
I've been working as a data engineer now for about 4 years professionally, exclusively in Python and the various flavors of SQL. I also have some experience in C, C++, C#, and Java from university. I have seen the occasional job post that mentions Go or Rust as something that would be nice to have. I have yet to encounter any projects in my career either in-house or otherwise that uses these languages. Would it be worth my time to learn them on the side or to try and propose our next tool, pipeline, etc. be written in one of them?

5 comments

[ 2.9 ms ] story [ 18.5 ms ] thread
PaulHoule
I’d say Rust, not Go. Rust’s memory management model is a hassle but it has the potential to trounce garbage collected languages in performance. Go is more like Java—-, what you’d get if you were trying to make a language like Java but read too many blog posts about why Java sux.
captaintobs
Go and rust aren't going to help you as data engineer. Data Engineering is dominated by python, sql, java, and scala.

Writing a pipeline in those languages isn't particularly great unless you have very specific reasons for it because there's no ecosystem for them.

wdroz
It's relatively easy to extend Python with project like Py03[0] and Maturin[1]. Polars[2] is the perfect example of that.

It's not easy to push coworkers/companies to use an unfamiliar language. Rust isn't fast to learn. You need very good arguments and a good usecase to make it works.

I doubt that learning Rust will help you more that learning more about the data engineers tools, so this isn't really "worth" your time.

[0] -- https://pyo3.rs/v0.18.3/

[1] -- https://github.com/PyO3/maturin

[2] -- https://www.pola.rs/

diehunde
Not much. Go with Java or Scala instead. Those are useful when learning processing frameworks such as Spark, Beam, Flink, etc.