Rust is a very compelling environment for many kinds of scientific computing, also due to the features noted in the article:
“Unlike the many compilers and ancillary utilities that programmers use to build C code, Rustaceans can use a single tool, called Cargo, to compile Rust code, run tests, auto-generate documentation, upload a package to a repository and more. It also downloads and installs third-party packages automatically.”
Fetching required packages in this way is very convenient, something one notices rather soon when writing Rust applications. There are already many dedicated packages also for highly specialized applications, and with cargo it is easy to use them immediately.
It's really better than that. Cargo lets you specify the version of the package that you are using/used. This is a huge deal for getting something to run later and for reproducibility.
I've been learning Rust and find it not hard to learn, I think because I already know C/C++ and understand the concepts of pointers, references, stack and heap allocations.
I can confirm. I started learning Rust as a second language right after Python and it was hard to make sense of anything. Why the hell some values have the Copy trait and others don't? Then I learned C and now I have good mental model of how memory is allocated and Rust feels okay. I just don't like that it hides away the complexity because when it does not work, you don't have the means to understand why, you just know did not follow the rules that you never remember anyways because they don't make sense to you. C has quirks too and the learning can also be steep, but it forces you to understand how things are working under the hood. Will continue to learn Rust but I have high hopes for the Zig language which seems more approachable with C in mind (rather than C++ which Rust is said to be a replacement for).
I work in hci research and have never played with/ heard of Rust being used. Was quickly checking for visualisation libraries and found
https://lib.rs/visualization
Loved the Color scheme and font ... lol.
Anyways, will be using Rust for a small side project (short paper for augmented Humans ( https://augmented-humans.org/ deadline in 2 weeks :) let’s see how quickly I can learn it).
Any tutorial recommendations go using Rust for statistics/data plotting?
https://lib.rs/crates/plotlib
Or the gnuplot Wrapper looks up for the task for me.
That's.. ambitious. Rust is a nice language, but the borrow checker, for all its awesomeness, has a fairly steep learning curve. Unless you're a C++ programmer, in which case this level of thinking won't be totally alien to you at least.
For my first program (task scheduling, regexes, redis, rest), everything was simple and obvious, it was made in about 3 days.
My second program was dealing with several kinds of graphs, optimal sub string addressing, and then I hit the painful points and it took me several weeks of this side project to get my head around the complex concepts of Rust, and several months to really feel at ease.
There's a subset of problems for which Rust is "easy" and it's probably possible, with some guidance and occasional help, to organize one's work around those.
You know why I like Rust as a programmer? It's community feels a lot like how the Perl community did at the height of Perl's popularity.
This is in contrast to Python's (has a friendly community, but tends to have bikeshedding), Node's (outright toxic npm-based warfare), or Java's (corporations shit out awful code when they're done with it, ie, everything donated to Eclipse or Apache, or all the dumpster fire code Google has released, and its up to you to judge how awful it really is).
Perl's CPAN? Lots of really nice code that follows a pretty good UNIX "does one thing, and does it well" philosophy, with lots of friendly folks.
As long as Rust doesn't have a Perl 6 moment, we're good.
Well, Rust does share its first letter with the Raku Programming Language, the new name of Perl 6 (https://raku.org using the #rakulang tag on social media).
I'm a researcher and I loved prototyping in Perl once upon a time ago. It was so fast and easy to get an idea out. I've played with Rust too, and have done a project in it. I like it too, but as a necessity, not out of a strong love. I do love the package manager Rust has though. It's so nice to not have to worry about that kind of thing.
Once Raku has dataframes, and it is stable, I'll definitely be checking it out.
What other flavor is there? A dataframe is a data structure that is a 2 dimensional array, where each column has its own type. Compared to a matrix, which is also a 2d array, but each column has the same type.
I think you are wearing pink glasses for Python. The most talked about lib lately is TensorFlow, and its code is worse, than any Java code I've ever seen.
No, I positively hate Python as a language. I don't hate their community, though.
That said, TensorFlow was written by Google in the typical Google Java way. See that bit above where I shit on Google-supplied Java? This also applies to TensorFlow, even though its written in Python.
There is just something culturally wrong at Google, and I don't know why; Amazon and Microsoft and Netflix and whoever else have donated actually pretty decent code to the FOSS community, and none of it is even a fraction of the trainwreck that Google produces.
I've started learning Julia lately, and it also has an appeal, especially for fortran-ish computations. (Yup, I said fortran ... it is still used in lots of scientific computations e.g. fluid-mechanical models.)
I have a collaborator who was writing Fortran this summer (and not even in physics!).
Oddly, it didn't seem like he was particularly locked in by a vast pile of legacy code; he just found it a good match for his mental model of the project.
Yes, people still write fortran in 2020. It's not just for legacy products (as, I think, Cobol is). Fortran is a very expressive and efficient language for high-performance computation. When you start a calculation that may take weeks to complete, you want to use a tool that you can trust.
Fortran is a great language for legacy products, with code written 50 years ago still working fine. But it has also gained new features over the years, keeping it ahead of competitors in terms of its ability to harvest all a machine can deliver.
This is all for technical work like numerical modelling. Scientists are a very practical lot, switching from Fortran to C (usually C++) when that makes sense, then to python, R, etc when that makes sense, and so forth.
The original article was about scientists switching to rust. I don't know anyone who has done that. I do know many who are exploring Julia, though.
Bear in mind that scientists tend to be quite good at technical work. Learning a new language is not a significant problem for them. But scientific programming is about much, much more than programming. The essence of scientific programming is in the underlying mathematics and science, and Fortran just happens to be pretty good at those things.
Many were but many turned back to whatever they were coming from. Julia it's an interesting language with rather odd design choices though.
In my opinion, the major hurdle to a wider adoption is that it was/is oversold. They claim c-like processing speeds, and in many cases they can be reached, but not off the bat. You need lots of hacks to get to those speeds.
This seems like a stretch, or it seems very limited to the specific people mentioned in the article. I find it hard to believe that scientists who have specialised very narrowly in their domain will also want to be full-time software engineers too and choose one of the most complex languages on top of that. This just doesn't scale beyond the few people who are really into it.
> Köster, now at the University of Duisburg-Essen in Germany, was looking for a language that offered the “expressiveness” of Python but the speed of languages such as C and C++.
Rust doesn't have the expressiveness of Python (even when you put expressiveness in quotes), it's not even close. What you're looking for is probably Julia (and Julia is already quite a bit more difficult than Python even if the syntax looks superficially similar).
It's considerably more difficult than Python, people just get misled by the similar syntax. The advanced features that you're mentioning aren't optional in Julia, especially if you care about performance.
But you don't care about performance with Julia. You just write your thing. Then you complain to the community that it's slow, and they optimize it for you! Only half kidding.
For me it was understanding the type unification rules. After a long time I read the original dissertation papers, and only that was the time when I finally understood what method is being called why at what point. While method dispatch is the most interesting part of Julia, it's the most complex as well.
Compared to Python? Just about everything starting from simple things like how to create a simple data structure, how to convert a variable to a different type, how to understand any error message that you get, how to open a file or modify a visualisation and extending all the way up to overarching concepts like the typing system, importing foreign code and managing dependencies.
- I had none of those examples in mind (for example by foreign code I was referring to code written in a different project, not to using a different language). No, I don't need you to give me code snippets on how to do my actual list.
- Obviously Julia can do everything that Python can. And most of it will look ergonomic, short and simple. The difficult part isn't whether the language can do it once you know how, it's how quickly you can get to figuring it out when you don't know it.
- So what makes the small things difficult is really the mix of unhelpful error messages and the limited resources online.
This is my overall point - if you take someone who hasn't written code in either Python or Julia, give them access to online search and ask them to complete almost any task, they'll do it faster and with less effort in Python.
Most of Rust's difficulty comes from the borrow-checker. And if you're mostly doing linear work, if you're doing a lot of math, if you have a lot of pretty flat data structures, if you mostly keep things on the stack, then you may not run up against the borrow-checker that often. I would assume (I don't know for sure) that scientific computing tends to do more of this kind of stuff, and not as much heap-juggling/heterogeneous data modeling as, say, a web server.
I’ve done simulation work using Rust and in particular the Rayon data-parallelism library and this has been my experience. The kind of organization of data you need to do for good cache behavior (e.g. split data by phases of algorithm, divide work by memory destination and only randomly read when possible) meshes quite well with the borrow checker restrictions.
Really depends I guess. I can tell you that at least for HEP, especially for the LHC experiments, software is nothing like what you describe. Parts are designed and optimized in such a way but I would imagine it it to be a gigantic effort to try and wrangle with the borrow checker for the rest of the software.
I agree, a memory managed language will always beat the borrow checker in expressiveness, ergonomics, simplicity, maintainability. Unless you're writing system-level or low-latency apps I don't see the benefit to Rust. If you need better-than-python performance there are many good memory-managed LLVM or JVM/.Net options available.
>I find it hard to believe that scientists who have specialized very narrowly in their domain will also want to be full-time software engineers too and choose one of the most complex languages on top of that.
I think it's really field-dependent. I know academics/profs in the computational biology space that accumulate a looot of software over time from their colleagues/students. Also, it's not the scientists/PIs writing the software :P
Yes. And a lot of that software collects dust after the paper it produced is published :/
Academia needs to recognize and incentivize software (including maintenance) as a research output, and make funding available for more research software engineers.
Amen! More long-term staff scientist jobs of all stripes!
One lesson I hope people take away from AlphaGo/AlphaFold (and a lot of the work coming out of the Allen Institutes, Broad, etc) is that some really amazing work can be done when big groups of experienced people can work together on a project.
Seriously! My lab does plant root growth modeling, which (like protein folding) is a 3D morphology prediction problem. Tweaking crops for even a small increase in carbon uptake and sequestration in soil (e.g., via deeper or longer roots) may be able to cut a sizeable chunk out of annual CO2 emissions[0], but almost nobody's working on it because it doesn't have direct applications to human healthcare.
And that Fortran on average was pretty bad. Any language that requires certain level of discipline will have hard time to be adopted at a large scale. And this is why Python, R, and "C/C++" are still the best friends for most people in scientific software development.
It may help to remember that the previous go-to language for scientists was Fortran. Of course, not all scientists, but a very much alive subset that still use it today.
I'm a data scientist and I know Rust. Though, to be fair, I'm probably not the type the article is describing.
I'm also highly sceptical. In my experience scientific software is mostly restricted to:
C, C++, Fortran, Python, R, Matlab/Simulink, Mathematica, Julia, and Excel.
I would think VERY few doing numerical computing or statistical or other engineering computations would have the bandwidth to really dive into Rust. I also wouldn't consider it stable enough for many projects (although I'm sure it's fine for some).
Rust is easier than C++ for this kind of work because of its packages. In some ways Rust is closer to Python for prototyping an idea to see if it works. Rust gets challenging when you have to deal with large systems, which is not a problem for research.
Researchers / scientists will sometimes not even write functions. It's that small of a scope.
> Researchers / scientists will rarely even write functions. It's that small of a scope
Scientific computing can mean so many different things that if feels like some of us are talking past each other in this thread.
The person in the article talks about needing to get out every ounce of performance and writing an application that identifies genome variants at scale. You're talking code so small that it doesn't need functions. Those are completely different worlds.
Choosing Rust for something that small is a waste of time. Expecting the people who write so little code that their code doesn't require functions, to be comfortable with Rust is even a bigger stretch.
The code I'm talking about needs to run fast (or be type safe) or you wouldn't consider using Rust. But internally in a library or a function that gets ran over and over again and needs to run super fast often isn't a lot of lines of code. So what I mean by a researcher may not even write a function is, they write the meat of the algorithm, it gets handed off to software engineers, and they turn it into a library or OOP it or whatever, and then run it on servers.
My current project in Python is massive compared to a normal one atm and it's about 2-3 pages of code in Jupyter. In Rust, if the libraries are there, it would be about the same size. I have 8 functions atm. It's also the end of the project and I'm in talks with data engineers to wrap it in OOP. They'll handle the IO. Or at least, that's how I've always done it and I've done a project in Rust.
Technically they're infrastructure software engineers without the title, but most people don't know what that is, so it can be easier to call them data engineers. The company doesn't give titles like that, they're just "the cloud people" with a vanilla software engineer title.
Unless you're a BI / business analyst, there are probably software engineers that work with AWS / gCloud at your company, who setup the database(s) and do the backend. What you want to do is called an IT request, where you request they help out when you need to productionize. It's bad form for a data scientist or any other kind of researcher to have the keys to the live production servers. While it can be fun to learn that stuff, there is a barrier of entry, specially monitoring software and unit tests.
This is more on the PM (or DPM (data project manager)) side but: https://www.datascience-pm.com/domino-data-science-lifecycle... can give a good map of the process. Many who do research are on their own, not apart of larger teams and are disconnected, so they don't know what options they have. Maybe that roadmap can help, depending on your situation.
When it comes to productionization, I've had too many years of software engineers rewriting my models and adding bugs left and right. The second they want to rewrite it from a notebook into a py file (or equivalent language, usually Java), I get cautious. I find the best process skips these issues and the software engineers will thank you for it: Automate most of the productionization process. Using notebooks as an example, python has libraries that can import notebooks, so you don't have to copy paste code into a py file. Instead an OOP wrapper (which they love) can be written with the interface they like. They can do the IO like streaming data or batching or whatever. They can do the unit tests and monitoring part. What you can do is write the function calls inside of the interface. Usually you want a .predict() method, similar to how ML algos work, but .run() is popular too. Inside of that, put all of the functions from the notebook you want called and in what order. In metaphor it's like writing a header file in C or C++. This py file will then import the notebook and call those functions.
The beauty of this is at least four fold: 1) no new bugs 2) no explaining how the model works, because you're doing that part and 3) If you want to update the model, you can update the notebook and then no work needs to be done in the py file. It's importing the notebook, so it will import the changes. It's fully automated. 4) You can have EDA, plots, documentation, all that stuff that usually takes up 40-60% of the notebook, not be required to remove it, because it will not get called.
The only downside is that every cell in the notebook that the py file will call needs to be wrapped in a function. However, this is a good idea by default, because you'll get global variable type bugs when you run the cells out of order if your cells are not wrapped in a function, so to avoid that you're probably already writing functions in your notebooks.
*Sometimes code has to go fast, and notebooks tend to use dataframes, which is fine for 99% of the work out there. In that case, instead of having the software engineers rewriting my model, because of the problems mentioned above, I will instead 1) Write libraries in C/C++ that accelerate parts of the notebook. And if that can't be done I will 2) write the model directly while prototyping in a language like Rust, C, C++, or Java. I'd rather write it than have someone port it. Most will not do this, because it's beyond the average researcher's skill, but it alleviates so many problems, and I'm not afraid of statically typed languages.
One usually needs Fortran or C++ or Matlab or Python with Numpy for numerical computing that is doing matrix or sparse matrix work (a lot of numerical computing is like this). I don't know of any uses that are so small as to not need functions outside of some extremely small/simplified models for explanatory purposes.
There is also symbolic mathematics that Mathematica really excels at although it's pretty good at a lot of things these days.
If someone just needs fast code to loop through some data, they should be able to use whatever compiled or JIT language they see fit. Heck, it sounds like bash/ask might even fit this use case if the code is so small as to not need functions.
Yep, autocorrect I guess. My point being if someone is just doing a little string work than she'll scripting can be very performant and also dead simple.
I normally think of scientific programming as something involving lots of large matrices, but that isn't the case for everything.
>I normally think of scientific programming as something involving lots of large matrices, but that isn't the case for everything.
It typically is. If you have the right libraries, it's only a few lines of code to do a lot of math. C = A + B could be matrix math, not just single variables being added.
In my experience yes, but I started thinking about stuff like the human genome project which was mostly done with Perl scripts and BioPerl in the year 2000 iirc. I assume that is mostly just regex work dealing with very large strings. Is it scientific computing? Yes, just not as how I typically view scientific computing.
C++ still has a few desirable features re: value-parameterized templates and compile-time metaprogramming that currently lack an idiomatic Rust equivalent. In principle, you can replicate anything C++ can do via procedural macros, but it's not necessarily easy. Also proc macros are a nightly-only feature, so there's that.
I first thought of Rustlang when I saw the title, then I saw the nature.com domain and corrected thinking "oh interesting, what are scientists studying in iron oxide?"
The computationalists I know are maintain large sets of fortran code. When they’re not doing that, they’re porting fortran codes to julia or python or writing new codes in julia or python.
Yah, you're not going to need Rust, but some kinds of research could benefit and does use Rust. Eg, anything financial, where floating point is a no go, Python falls on its face. Anything physics like NASA type research can benefit from strict types. Robotics, and so on.. ymmv.
Rust shouldn't take away from Python or Java, but it may take away from research that uses Fortran, C, or C++.
Scientist are not turning to rust. Scientist, if they are turning, are turning to the scientific standard compute language and its plethora of platforms and libraries: Python.
Rust is fine, but it will never catch up or replace Python. Python is just good enough at what it does and will only become better.
Yeah, I agree. All the scientists that I know who are using a programming language, are using Python because it’s easy enough to learn how to use and it’s powerful for their needs.
I really don’t see Rust being used for anything other than very specific cases.
I can't help but feel that given the requirements laid out in the article Nim would be a more suitable fit for their goals over Rust. I do think Nim really nails great performance with a very clean and straightforward syntax, certainly easier for someone who only works in Python to pick up over Rust.
Kind of a chicken and egg scenario there really. A few early adopters will need to branch out and start using it to get that momentum building, which is probably a great opportunity for scientists who are looking for an alternative
Nim has plotting, dataframe, matrix libs and even a fully featured machine learning library, I don't see the reason why it can't be used, especially by scientists.
I don't think scientists should worry themselves with the cognitive burden of Rust.
It's hard enough to figure out how to do things in any language as a full-time software developer, never mind someone that just needs a program for their research, etc.
Scientists are currently writing their code in C++ in many cases. If you have needs that require the performance benefits of C++, why not do it in a language that has a cohesive design (and is actually intuitive) instead of learning the pile of layers of decades-old debt that is C++?
I question if scientists ever should have to tackle the ins and outs of software architecture and performance optimization. Never mind the daunting task of running software.
I don’t really see Rust being altogether different to someone outside the software industry. It’s yet another complex knowledge domain that can be abused and intractable like any other.
I suspect the lines are not as cleanly drawn as anyone would like them to be, no more making programming unnecessary for scientists than they make cooking unnecessary for <insert non-chef profession here>.
Hard to answer this question but I worked with scientists who were wasting an incredible amount of time debugging their badly designed fortran or c++ programs, or just hitting walls when they shouldn't.
So yes you can easily start hacking something in fortran or c++, and the cognitive load is probably much smaller, but then you have the risk of being eaten by your bugs, or of not working on some problems just because the tool you make is so badly designed you give up doing the necessary changes.
Does anyone know why scientists aren’t just using Excel? Nothing against Python/Julia but Excel seems like it would be my first choice (paired with some CSV files).
It's true. Fun and relevant fact: The job title data scientist was invented at LinkedIn when they saw data analysts that needed to know R or Python skills. These skills seemed different enough from the traditional Excel and SAS data analyst jobs at the time to warrant a new job title.
So did Excel not cut it, or was it a preference to use Python? Back then Excel supported up to 65,535 rows, and would typically crash with over 8,000 rows. I worked in that era in Excel. One model once was split across three spreadsheets. Only one spreadsheet could be open at a time, they would take 20 minutes each to load, and there was about a 50% chance they would crash while loading.
So what do you do if you need over 65,000 instances of labeled data? For a neural network it's nice to have a million instances, yes a million.
R and Python have in them what is called a dataframe. It's a spreadsheet, but in another programming language. We tend to load our data into those, which is just like Excel, but without the hardware limitations. So in many ways, today it's just like it once was, but we get to choose which programming language to use while working in a spreadsheet, and let's be fair, the Excel programming language isn't exactly great.
I think you are...dramatically underestimating the size of many "modern" datasets and what most scientists want to do with them.
We record (electrical) neural activity from behaving animals. The raw data is about 1 Gb/minute, and we collect hours of it per day, 5-7 days/week. The lab next door has microscopes that produce image sets that are in the Gb to Tb range (big swathes of human brains, imaged at micron resolution).
An RNAseq experiment is ~20 Gb range; other genomic things are similar, maybe a bit smaller.
All of this would be intractable in Excel, even over a long weekend :-) On top of that, Excel often doesn't do much of what you would need: there's basically no support for signal processing or image analysis or genomic work.
Python itself is rather slow indeed, but more often than not, it doesn't matter: for many data science projects, python is only used as a glue to control the processing and/or pass data between low-level C++ implementation that can chew through data at great speed, (e.g using parallelism).
Examples that immediately come to mind include : numpy, tensorflow, etc ...
Even heavy-duty I/O can be made to crank with python if you do it properly.
I don’t know what people think scientists are doing, but we are not just spreadsheet number crunching.
In my PhD I was analyzing petabytes of structured data from particle detectors that required relatively complex algorithms just to reconstruct, before any physics analysis is ever started, and the analyses were just as complex. My experiment’s total code base was at least 100kloc of C++ and Fortran if not more, and used a farm with over 20k cores for massive distributed processing, single jobs regularly taking hundreds or thousands of cores and terabytes of memory.
I’m currently working on distributed GPU accelerated hydrodynamic simulations where a single input is in the ~1-10Gb range of structured mesh data and the calculation requires ~200+ Gb of GPU memory.
I really don’t see how excel would be a viable option for basically anything I have worked on in my scientific career, beyond my undergraduate toy analyses.
Has someone attempted to write something like Numerical Recipes Textbook or the GSL Scientific Library using Rust a lot of people in my org use those two resources - especially the NR textbook all the time.
I've found a lot of people bash NR online as outdated but it has really clear explanation of how certain algorithms are written as well as clean to read code examples. Very helpful to follow along with an explanation of the algorithm and see it all laid out in front of you with code.
So maybe a modern version written in a newer language could be a hit.
Yep, with the exception of the computery fields of pyhsics, they cold not care less about programming languages.
I have seen physicists write entire non trivial image processing pipelines in Matlab. Where they happily processed gigabytes of images made by some weird microscope costing stupid amounts of €.
It (mostly and somehow) works but it is slow as hell (porting to c++ cut the time to run a process from 1 day to a few seconds).
They never even bothered with a "real" language because the "computer guys" can port that stuff if need be.
104 comments
[ 1.8 ms ] story [ 146 ms ] thread“Unlike the many compilers and ancillary utilities that programmers use to build C code, Rustaceans can use a single tool, called Cargo, to compile Rust code, run tests, auto-generate documentation, upload a package to a repository and more. It also downloads and installs third-party packages automatically.”
Fetching required packages in this way is very convenient, something one notices rather soon when writing Rust applications. There are already many dedicated packages also for highly specialized applications, and with cargo it is easy to use them immediately.
Loved the Color scheme and font ... lol. Anyways, will be using Rust for a small side project (short paper for augmented Humans ( https://augmented-humans.org/ deadline in 2 weeks :) let’s see how quickly I can learn it).
Any tutorial recommendations go using Rust for statistics/data plotting? https://lib.rs/crates/plotlib Or the gnuplot Wrapper looks up for the task for me.
That's.. ambitious. Rust is a nice language, but the borrow checker, for all its awesomeness, has a fairly steep learning curve. Unless you're a C++ programmer, in which case this level of thinking won't be totally alien to you at least.
I program C, C++, matlab, Python and Ruby. Just looked a bit into Julia and Haskell.
Language Lover ... also trying not so common languages (my favourite is still OpenDylan).
For my first program (task scheduling, regexes, redis, rest), everything was simple and obvious, it was made in about 3 days.
My second program was dealing with several kinds of graphs, optimal sub string addressing, and then I hit the painful points and it took me several weeks of this side project to get my head around the complex concepts of Rust, and several months to really feel at ease.
There's a subset of problems for which Rust is "easy" and it's probably possible, with some guidance and occasional help, to organize one's work around those.
This is in contrast to Python's (has a friendly community, but tends to have bikeshedding), Node's (outright toxic npm-based warfare), or Java's (corporations shit out awful code when they're done with it, ie, everything donated to Eclipse or Apache, or all the dumpster fire code Google has released, and its up to you to judge how awful it really is).
Perl's CPAN? Lots of really nice code that follows a pretty good UNIX "does one thing, and does it well" philosophy, with lots of friendly folks.
As long as Rust doesn't have a Perl 6 moment, we're good.
Once Raku has dataframes, and it is stable, I'll definitely be checking it out.
That said, TensorFlow was written by Google in the typical Google Java way. See that bit above where I shit on Google-supplied Java? This also applies to TensorFlow, even though its written in Python.
There is just something culturally wrong at Google, and I don't know why; Amazon and Microsoft and Netflix and whoever else have donated actually pretty decent code to the FOSS community, and none of it is even a fraction of the trainwreck that Google produces.
Oddly, it didn't seem like he was particularly locked in by a vast pile of legacy code; he just found it a good match for his mental model of the project.
Fortran is a great language for legacy products, with code written 50 years ago still working fine. But it has also gained new features over the years, keeping it ahead of competitors in terms of its ability to harvest all a machine can deliver.
This is all for technical work like numerical modelling. Scientists are a very practical lot, switching from Fortran to C (usually C++) when that makes sense, then to python, R, etc when that makes sense, and so forth.
The original article was about scientists switching to rust. I don't know anyone who has done that. I do know many who are exploring Julia, though.
Bear in mind that scientists tend to be quite good at technical work. Learning a new language is not a significant problem for them. But scientific programming is about much, much more than programming. The essence of scientific programming is in the underlying mathematics and science, and Fortran just happens to be pretty good at those things.
> Köster, now at the University of Duisburg-Essen in Germany, was looking for a language that offered the “expressiveness” of Python but the speed of languages such as C and C++.
Rust doesn't have the expressiveness of Python (even when you put expressiveness in quotes), it's not even close. What you're looking for is probably Julia (and Julia is already quite a bit more difficult than Python even if the syntax looks superficially similar).
- I had none of those examples in mind (for example by foreign code I was referring to code written in a different project, not to using a different language). No, I don't need you to give me code snippets on how to do my actual list.
- Obviously Julia can do everything that Python can. And most of it will look ergonomic, short and simple. The difficult part isn't whether the language can do it once you know how, it's how quickly you can get to figuring it out when you don't know it.
- So what makes the small things difficult is really the mix of unhelpful error messages and the limited resources online.
This is my overall point - if you take someone who hasn't written code in either Python or Julia, give them access to online search and ask them to complete almost any task, they'll do it faster and with less effort in Python.
I think it's really field-dependent. I know academics/profs in the computational biology space that accumulate a looot of software over time from their colleagues/students. Also, it's not the scientists/PIs writing the software :P
Academia needs to recognize and incentivize software (including maintenance) as a research output, and make funding available for more research software engineers.
ok sorry.. bad dad joke..
[1]: https://bilalhusain.com/dust/
1: https://github.com/linkedin/dustjs
One lesson I hope people take away from AlphaGo/AlphaFold (and a lot of the work coming out of the Allen Institutes, Broad, etc) is that some really amazing work can be done when big groups of experienced people can work together on a project.
[0]: http://www.nature.com/articles/nature17174
I'm a data scientist and I know Rust. Though, to be fair, I'm probably not the type the article is describing.
C, C++, Fortran, Python, R, Matlab/Simulink, Mathematica, Julia, and Excel.
I would think VERY few doing numerical computing or statistical or other engineering computations would have the bandwidth to really dive into Rust. I also wouldn't consider it stable enough for many projects (although I'm sure it's fine for some).
Researchers / scientists will sometimes not even write functions. It's that small of a scope.
Scientific computing can mean so many different things that if feels like some of us are talking past each other in this thread.
The person in the article talks about needing to get out every ounce of performance and writing an application that identifies genome variants at scale. You're talking code so small that it doesn't need functions. Those are completely different worlds.
Choosing Rust for something that small is a waste of time. Expecting the people who write so little code that their code doesn't require functions, to be comfortable with Rust is even a bigger stretch.
My current project in Python is massive compared to a normal one atm and it's about 2-3 pages of code in Jupyter. In Rust, if the libraries are there, it would be about the same size. I have 8 functions atm. It's also the end of the project and I'm in talks with data engineers to wrap it in OOP. They'll handle the IO. Or at least, that's how I've always done it and I've done a project in Rust.
Technically they're infrastructure software engineers without the title, but most people don't know what that is, so it can be easier to call them data engineers. The company doesn't give titles like that, they're just "the cloud people" with a vanilla software engineer title.
Unless you're a BI / business analyst, there are probably software engineers that work with AWS / gCloud at your company, who setup the database(s) and do the backend. What you want to do is called an IT request, where you request they help out when you need to productionize. It's bad form for a data scientist or any other kind of researcher to have the keys to the live production servers. While it can be fun to learn that stuff, there is a barrier of entry, specially monitoring software and unit tests.
This is more on the PM (or DPM (data project manager)) side but: https://www.datascience-pm.com/domino-data-science-lifecycle... can give a good map of the process. Many who do research are on their own, not apart of larger teams and are disconnected, so they don't know what options they have. Maybe that roadmap can help, depending on your situation.
When it comes to productionization, I've had too many years of software engineers rewriting my models and adding bugs left and right. The second they want to rewrite it from a notebook into a py file (or equivalent language, usually Java), I get cautious. I find the best process skips these issues and the software engineers will thank you for it: Automate most of the productionization process. Using notebooks as an example, python has libraries that can import notebooks, so you don't have to copy paste code into a py file. Instead an OOP wrapper (which they love) can be written with the interface they like. They can do the IO like streaming data or batching or whatever. They can do the unit tests and monitoring part. What you can do is write the function calls inside of the interface. Usually you want a .predict() method, similar to how ML algos work, but .run() is popular too. Inside of that, put all of the functions from the notebook you want called and in what order. In metaphor it's like writing a header file in C or C++. This py file will then import the notebook and call those functions.
The beauty of this is at least four fold: 1) no new bugs 2) no explaining how the model works, because you're doing that part and 3) If you want to update the model, you can update the notebook and then no work needs to be done in the py file. It's importing the notebook, so it will import the changes. It's fully automated. 4) You can have EDA, plots, documentation, all that stuff that usually takes up 40-60% of the notebook, not be required to remove it, because it will not get called.
The only downside is that every cell in the notebook that the py file will call needs to be wrapped in a function. However, this is a good idea by default, because you'll get global variable type bugs when you run the cells out of order if your cells are not wrapped in a function, so to avoid that you're probably already writing functions in your notebooks.
*Sometimes code has to go fast, and notebooks tend to use dataframes, which is fine for 99% of the work out there. In that case, instead of having the software engineers rewriting my model, because of the problems mentioned above, I will instead 1) Write libraries in C/C++ that accelerate parts of the notebook. And if that can't be done I will 2) write the model directly while prototyping in a language like Rust, C, C++, or Java. I'd rather write it than have someone port it. Most will not do this, because it's beyond the average researcher's skill, but it alleviates so many problems, and I'm not afraid of statically typed languages.
One usually needs Fortran or C++ or Matlab or Python with Numpy for numerical computing that is doing matrix or sparse matrix work (a lot of numerical computing is like this). I don't know of any uses that are so small as to not need functions outside of some extremely small/simplified models for explanatory purposes.
There is also symbolic mathematics that Mathematica really excels at although it's pretty good at a lot of things these days.
If someone just needs fast code to loop through some data, they should be able to use whatever compiled or JIT language they see fit. Heck, it sounds like bash/ask might even fit this use case if the code is so small as to not need functions.
Assuming that "ask" was a typo and you meant "awk", awk does support user-defined functions, as does bash.
https://www.gnu.org/software/gawk/manual/html_node/User_002d...
https://tldp.org/LDP/abs/html/functions.html
The support for that in both is somewhat limited as compared to "full" programming languages, though.
I normally think of scientific programming as something involving lots of large matrices, but that isn't the case for everything.
It typically is. If you have the right libraries, it's only a few lines of code to do a lot of math. C = A + B could be matrix math, not just single variables being added.
Rust shouldn't take away from Python or Java, but it may take away from research that uses Fortran, C, or C++.
Rust is fine, but it will never catch up or replace Python. Python is just good enough at what it does and will only become better.
Perhaps?
> and will only become better.
Very unlikely
I really don’t see Rust being used for anything other than very specific cases.
https://github.com/Vindaar/ggplotnim https://github.com/unicredit/neo https://github.com/mratsim/Arraymancer
It's hard enough to figure out how to do things in any language as a full-time software developer, never mind someone that just needs a program for their research, etc.
I don’t really see Rust being altogether different to someone outside the software industry. It’s yet another complex knowledge domain that can be abused and intractable like any other.
So yes you can easily start hacking something in fortran or c++, and the cognitive load is probably much smaller, but then you have the risk of being eaten by your bugs, or of not working on some problems just because the tool you make is so badly designed you give up doing the necessary changes.
So did Excel not cut it, or was it a preference to use Python? Back then Excel supported up to 65,535 rows, and would typically crash with over 8,000 rows. I worked in that era in Excel. One model once was split across three spreadsheets. Only one spreadsheet could be open at a time, they would take 20 minutes each to load, and there was about a 50% chance they would crash while loading.
So what do you do if you need over 65,000 instances of labeled data? For a neural network it's nice to have a million instances, yes a million.
R and Python have in them what is called a dataframe. It's a spreadsheet, but in another programming language. We tend to load our data into those, which is just like Excel, but without the hardware limitations. So in many ways, today it's just like it once was, but we get to choose which programming language to use while working in a spreadsheet, and let's be fair, the Excel programming language isn't exactly great.
We record (electrical) neural activity from behaving animals. The raw data is about 1 Gb/minute, and we collect hours of it per day, 5-7 days/week. The lab next door has microscopes that produce image sets that are in the Gb to Tb range (big swathes of human brains, imaged at micron resolution). An RNAseq experiment is ~20 Gb range; other genomic things are similar, maybe a bit smaller.
All of this would be intractable in Excel, even over a long weekend :-) On top of that, Excel often doesn't do much of what you would need: there's basically no support for signal processing or image analysis or genomic work.
Examples that immediately come to mind include : numpy, tensorflow, etc ...
Even heavy-duty I/O can be made to crank with python if you do it properly.
https://genomebiology.biomedcentral.com/articles/10.1186/s13...
https://www.theverge.com/2020/8/6/21355674/human-genes-renam...
In my PhD I was analyzing petabytes of structured data from particle detectors that required relatively complex algorithms just to reconstruct, before any physics analysis is ever started, and the analyses were just as complex. My experiment’s total code base was at least 100kloc of C++ and Fortran if not more, and used a farm with over 20k cores for massive distributed processing, single jobs regularly taking hundreds or thousands of cores and terabytes of memory.
I’m currently working on distributed GPU accelerated hydrodynamic simulations where a single input is in the ~1-10Gb range of structured mesh data and the calculation requires ~200+ Gb of GPU memory.
I really don’t see how excel would be a viable option for basically anything I have worked on in my scientific career, beyond my undergraduate toy analyses.
I haven’t worked with it extensively (I use nix on my workstation) but I’m a big fan of what the Guix folks are doing targeting the scientific community https://hpc.guix.info/blog/2019/05/gnu-guix-1.0-foundation-f...
I've found a lot of people bash NR online as outdated but it has really clear explanation of how certain algorithms are written as well as clean to read code examples. Very helpful to follow along with an explanation of the algorithm and see it all laid out in front of you with code.
So maybe a modern version written in a newer language could be a hit.
Nowadays, a similar effort would have to be community driven, and open-source, and include the deep discussions for domain specializations.
I have seen physicists write entire non trivial image processing pipelines in Matlab. Where they happily processed gigabytes of images made by some weird microscope costing stupid amounts of €.
It (mostly and somehow) works but it is slow as hell (porting to c++ cut the time to run a process from 1 day to a few seconds).
They never even bothered with a "real" language because the "computer guys" can port that stuff if need be.
If you have a hammer...
All these pro rust/mozilla journalists are all related to Mozilla in some ways
Few google search and you start to see how all these people are connected
Hmm
What about CSS developpers, they are also turning to Rust?
We'll need the Markdown writters to turn to rust too!