Perl is expressive but the code can be hard to read. In general, Python is readable because it enforces the indentation and other language design choices.
Python also got heavyweight libraries such as Numpy and Pandas which put it in the front. Perl do not have have such well known libraries as far as I know.
Perl seems to have died down just about everywhere. I've not used it professionally or seen it in use in well over a decade. It's so expressive and has so many ways to do things that reading other people's Perl can be challenging. I think this contributed to its downfall. And "other people" can often mean "yourself in six months" too.
> And "other people" can often mean "yourself in six months" too.
Amen to this point. I am currently the single engineer in a small software biz and after the last engineer left I thought to myself "well, at least I won't have to decipher his code anymore"...needless to say, I've been shocked by how much time I can spend on a system only to revisit it within a year and have little to no memory of how or why I did something a certain way!
> Python also got heavyweight libraries such as Numpy
This. For the kind of tasks data science involves, you either write C++ or you need the equivalent of Numpy. No Numpy means no Pandas, which means no sane way to wrangle datasets.
For all the hate it gets, Numpy is a world-class library.
The last time I wrote production Perl code (2012) I stressed about keeping it readable, especially by anyone who would be learning Perl. The company was mostly C/C++ focused with maybe Python as a #3. I think Perl-isms can be foreign if you haven't seen them, so they should be avoided in any non-Perl focused company.
Looking now (it's open source) there's two keywords I have forgotten. It's a Perl binding to a C library using an old version of Swig. There's lots of Perl or C helper code to smooth out the bumps. It also looks like it hasn't been updated (bummer).
perl doesn’t have the numerical chops to keep up, and if it started to fix that now, it has 20 years of headwind to fight through for probably marginal gains. Good luck.
Perl is also a nasty language to work with. Incredibly ugly. Even if perl was the standard for data science, I would be looking to escape it at every opportunity. In fact, that's what I did when I worked in a sector that had tooling and network effects for perl. I tried to escape using perl every chance I could.
There's R which is weird in a way like Perl is weird.
I think R is worse but it is bundled with a very good stats library. R users aren't so interested in "programming in the large" or even the small so it is OK.
Other than the meaningful indentation Python strikes me as a generic and normal programming language. I'm quite amused at how Python and Java got pattern matching at about the same time with a broadly similar approach to "fit this new feature in this language so that it feels like it belongs and plays well with decades of existing practice".
I got into Python because I outgrew Excel (and Origin) and started with Jupyter notebooks and Pandas. The Pandas functions DataFrame.from_excel and DataFrame.to_excel combined with how visual a notebook is made the transition very easy for me.
I do come across some perl every now and then but it looks like Bash on steriods to me (perhaps because of the $variables), and I never see any DataFrames or similar structure that looks familiar to me.
Idk, Perl never occurred to me and nobody ever recommended it. Is Perl good for data science? Do you have any examples? I never ran into "efficiency problems" with Python btw, I'm not using it at that scale, at what scale would I notice this? I have run out of ram at times, but that's usually when a dataset on disk is already larger than my ram. But then I usually find a tool to deal with the data anyway (for example pyosmium for super large osm/o5m fles).
Edit: I feel that Python also gives me other nice things to get started with, things like Django and Snakemake. This also leads me to recommend Python to other people, it's a broad basis for a lot of stuff. That's why even though some people recommended R when I got started, I choose Python anyway. I have no regrets, unless you are going to blow my mind with some examples...
Every time I look at an example I am turned off and my mind is in a way "boggled" ...
I just opened the tutorial on the Raku site I see this
say looks_like_number "foo";
I find this awful,
right from the beginning, the say turns me off, computers do not speak, at least I really hope the computer is not actually speaking when I type that. Then there are the weird rules of quoting to not quoting ... I can't quite tell the rules are
the example is unpalatable, why does the looks_like_number word have underscores? is that a odd variable name that needs underscores? but it looks like a message to the user, what a terrible choice either way!
Well, in a way this is funny. As far as I can see, that example is taken from the page describing the differences between Perl and Raku (https://docs.raku.org/language/5to6-nutshell#CPAN), and how you can use Perl inside Raku using the Inline::Perl5 module, and how you can load a Perl module that exports a "looks_like_number" subroutine, and use it from Raku.
The example for "looks_like_number" has probably be chosen, because many experienced Perl programmers are familiar with that.
In Raku, you can use kebab-case, and if there would be such a routine in Raku, it would probably be called "looks-like-number", and return a Bool, as in either True or False.
"say" is also builtin in Perl, at least for the past 12 years or so. And if you are on a Mac, running "say this is your computer talking to you', it will talk to you.
In any case, the thing you appear to refer to, is in no way a tutorial, but rather documentation intended for a specific group of people: Perl programmers.
PDL is just awesome, but I feel that Perl is a programmer's programming language. If you come from Linux, shell scripting, C/C++, etc, you'll probably handle Perl well.
However, it might be a bit too much if you are from other fields and just want to get your sums in order.
I came from Linux, shell scripting and C/C++ and still, Perl and me just didn't get along. Only after I learned Common Lisp much later in life, Perl made more sense, but by then it was almost entirely replaced by Python and Ruby.
Python is not a good language for low-level numerics like FORTRAN, but it has good facilities to make foreign language libraries (written in C, FORTRAN, CUDA, etc.) that do high-level operations. There is not just the foreign function interface but also the operator overloading that makes it possible to write something like
dataframe["A"] + dataframe["B"]
and get a Series. You can make things that look like numbers or arrays but actually do something different.
Both Python and Perl made bold transitions, Python from 2 to 3 and Perl from 5 to 6.
The Python transition was difficult but ultimately successful. On the other hand, Perl rolled the dice and lost. In some alternate universe it might have been the other way around.
> Both Python and Perl made bold transitions, Python from 2 to 3 and Perl from 5 to 6.
Perl 6 (Raku) is effectively a different language. Porting from 5 to 6 would change most lines of code, in other words, a complete rewrite. To ease transition, Perl 6 made it so that both could coexist, but if you want to go v6, then it is all or nothing.
Python 2 and 3 are not that different. You can easily write code that is compatible in both, and porting is often not much more than using 2to3.py most of it is replacing "print" with "print()" and converting between bytes and str.
Perl strategy was that to create a modern language inspired by Perl, in indeed, Perl 6 has quite a few features ahead of its time. And Raku (the new name for Perl 6) is still one of the most advanced language today.
Python strategy was to keep the same language but address a few pain points that require breaking compatibility, Unicode in particular.
In general Perl is faster than Python. But Python has had a lot of performance enhancements when it comes to crunching numbers so Python is faster for 'data science' stuff.
I love perl for regex scripts, where I need to quickly filter or transform a text file. I never liked it for other kinds of programming projects, for some reason to me it doesn’t feel as well suited to, say, numeric simulations.
> Naturally far more efficient
What does this mean exactly? One big reason for Python’s success in data science is numpy, which is far more efficient (especially on large data) than vanilla Python. I’m unaware of the state of the Perl ecosystem, does it have something similar?
> how come Perl lags far behind or gets dwarfed in data science by Python?
Perl seems to be lagging behind in general, no? Is there anything specific about data science where Perl should be shining?
Also data science requires a lot of intermediate/mockup data visualization and python puts matplotlib and scikit-learn right at your fingertips. Hard to find a match Perl side, but I might be wrong.
> I’m unaware of the state of the Perl ecosystem, does it have something similar [to numpy]?
Perl has PDL (http://pdl.perl.org), which I think predates numpy and does the same kind of array manipulation.
As for the original question, Python's dominance for "data science" projects is just a matter of momentum at this point: it's where Google and others have put their money, and that is now affecting undergraduate and graduate courses. Originally, I think it was just that some people didn't like Perl's syntax.
Personally, I prefer to use a mix of Perl and Julia, each for its respective strengths, and find Python all-around mediocre.
Perl doesn't have the tooling/library ecosystem Python does for Data Science. Additionally lot of Data Science people come from maths/stats and Python is easier to begin with. In my experience most Data Scientists aren't full blown devs since you focus on different things (business aspects of what you do vs scalability).
I'm not a programmer but I did get my start in tech because I installed Mandrake on my computer for fun in the 90s and couldn't figure out how to format the drive back so I could reinstall windows. I was accidentally forced to learn linux, and with that things like bash and Perl. I feel like people who customize their bashrcs and tweaking their IP tables and generally are comfortable scripting, are people who gravitate towards Perl, but for developers and data science folks, Python is much more accessible in terms of finding out how to learn that application of the language to those types of problem? When I thought about learning some more data science a few years ago, the vast majority of the tutorials on youtube etc are for people shifting from apps like excel to things like sql and python, I didn't see anything about Perl.
Is Perl even close to be on a top ten list of languages that make sense for data science? Not to be nasty to the OP, but the question would maybe make sense +10 years ago. Whats baffling now is seeing Perl as a real competitor to anything.
Well, for the longest time perl couldn't distinguish an int from a long when doing computation could be one reason. Perl is just not designed for any kind of math.
Python got numpy fairly early on. It is fast and powerful.
Then a whole ecosystem was built on top of numpy, with scipy, pandas, PIL, et cetera. Everything that uses n-dimensional arrays used numpy as their base and as a result all those things can be combined trivially. That's very powerful.
Then later came ipython, a much improved interactive shell, and Web based notebooks that are very useful for data science work.
That the language involved is Python isn't even important, imo. Numpy + ecosystem replaced Matlab. All Python has to be is be a better language than Matlab, and it is.
However, for data science, the biggest drawback vs Python is how you work with complex data structures. Because everything in Python is an object, (de)referencing things is generally straightforward, like:
It is not entirely about the syntax, for example, the code looks similar in Python (if I remember Perl correctly):
print(f"it turns out that {TV[family]['lead']} has ", end=‘’)
print(len(TV[family]['kids']), " kids named ", end='')
print(*[kid.name for kid in TV[family]['kids']], sep=', ')
For me, it was the culture “there is _one_ obvious (for a Dutch) way to do it” vs. “many” in Perl and most people lack discipline/good taste to handle the [too much] freedom provided by Perl.
Somehow Python took the niche of “executable pseudocode”, “glue language” for me.
Minus the sigils and crazy wrapping, like: @{ $TV{$family}{kids} } to refer to the array, but a different sigil/form if you're talking about one element, etc.
Which may not look intimidating on its own, but it's leaving out % and the backslash \$reference syntax, $object->method and other things that get confusing.
You can't, for example really make a "list of hashes/dicts". Try it, and it just turns the keys/values into array elements. You have to specifically make a "list of references to hashes/dicts". So you do something like:
For printing strings, there is more than one way to do it in Python; differently readable for.
print("""It turns out that {} has {} kids named {}""".format(
TV[family]['lead'],
len(TV[family]['kids'],
', '.join(*[kid.name for kid in TV[family]['kids']]*)
)
It doesn't use the nifty f string formatter, but the English sentence part is a bit easier to read.
I used Perl for a lot data processing, mostly text. That time, our machine learning code was mostly C/C++. Perl slowly died before the big frameworks took off. It didn't compete with Python to be honest. Of course, even if there was a direct battle between Perl and Python, Python would probably win anyway.
I do find Perl much easier for manipulating text files, file names, and the sorts of things where bash or awk are too tedious. Then there's a line where something like Python becomes more attractive. Perl is still a lot faster than Python for many tasks too...the Perl devs seems to spend more time optimizing certain paths. Or perhaps "everything is an object" in Python slows everything down?
Some people don't like sigils on variable names, and I get that. Personally I find them useful.
As far as the perldsc page example, I think that comes down to having a better personal style that makes your code readable. I would make the top level variable a hash reference rather than a plain hash, and purposefully not insist on dereferencing multiple levels all on one line.
As someone pointed out in another comment, you can make the Python version look just as lousy by virtue of a lousy style.
My version of code doing the same task would be something like this:
my $familyData = $TV->{$family};
my $lead = $familyData->{lead};
my $kidsList = $familyData->{kids};
my $kidCount = scalar( @{ $kidsList } );
my $kidListStr = join (
", ",
map { $_->{name} } @{ $kidsList }
);
print "it turns out that $lead has ";
print "$kidCount kids named $kidListStr\n";
Data scientist here. Readable code is important in data science. People rely on our products to be based on solid numbers and logic, sometimes without any form of external validation. We can't just scribble line noise in a REPL until we get some output that looks vaguely reasonable. We need to be able to actually read the code and know that what it's doing makes sense.
So readability matters. And Python is one of the most readable languages out there. It remains relatively readable even as the code and data structure get very complex. Its syntax resembles math and pseudo-code more closely than other languages. It feels more like a tool of thought, not just a bunch of alien hieroglyphics you have to write to crudely and inefficiently express how you really think about the problem.
Perl, on the other hand, is one of the least readable languages for a general audience. It is not a tool of thought for non-experts. Its syntax is ugly, obscure, overly symbol-laden, and beloved only by gurus. The supposed "feature" of TIMTOWTDI just makes it more obscure.
The thing I can never get over with Perl is that it didn't even have a sane, idiomatic notation for functions until recently (and even today I suspect it's not widely adopted). There was no such thing as just defining a function f(a,b) of multiple named formal arguments. You had to use some special magic variable and people start talking about shift operators or some such nonsense.
This is the point at which I start saying "yes, all the language choices are Turing complete, but that doesn't mean they're all equally effective choices".
Having a lot of different bad ways of dealing with something is not an excuse for not having a single good way of dealing with something, which is kind of the point about why Perl has failed as a programming language.
Readability is really a weak argument.
I've seen a lot of unreadable code written by data scientist in Python. And a lot of readable Perl code. It's not about the language, it's about the best practices.
The concept behind sigils is very simple and once you've learned it, it's not that threatening. And when talking about data structures, it is a benefit to have it more structured and know that you are dealing with an array or hash or scalar.
In my opinion, Perl is not used for data science because it lacks(or people are not aware of) libraries like numpy, pandas, etc.
I vehemently disagree. When choosing among languages today, performance is mostly good enough across all languages and readability becomes the most important consideration. It's "subjective", but human beings are subjects, and subjective things are important to them.
> I've seen a lot of unreadable code written by data scientist in Python. And a lot of readable Perl code. It's not about the language, it's about the best practices.
This is a bad and cliche argument. Yes, there is well-written code in languages that deprioritize readability and badly-written code in languages that highly prioritize readability. That doesn't mean that any differences in the inherent, built-in readability of X and Y are moot. And "best practices", which are not widely agreed upon or followed, absolutely do not suffice to paper over this.
> The concept behind sigils is very simple and once you've learned it, it's not that threatening.
"Once you've learned it" or "get used to it" is the biggest programming cliche in the world. People say this about all their favorite languages and language features. I'm sure I could complain about doing data science in Brainfuck and get someone in the comments who would say it's not threatening "once you get used to it".
Well sorry, but I don't want to get used to your favorite language's line noise. That's why I choose another language. That's why everyone else chooses another language.
> And when talking about data structures, it is a benefit to have it more structured and know that you are dealing with an array or hash or scalar.
You can keep track of types in Python with gradual typing. Most people don't bother, so evidently it's not that important in most cases, but you can do it.
Other than that, no, I don't need to fill my code with more line noise to constantly remind myself what type of container I'm dealing with. That's obvious enough from the methods I'm using to manipulate the container.
> In my opinion, Perl is not used for data science because it lacks(or people are not aware of) libraries like numpy, pandas, etc.
No, this is after the horse has already left the barn. Those libraries were developed in Python because the authors of those libraries preferred to work in Python. The authors preferred to work in Python because of the legibility considerations I've already mentioned.
Python's design and adoption were both heavily driven by a vision of readability from the very earliest stages in the 1980s/90s. People got into Python because they liked how the code looked:
> A friend of mine who knows nearly all the widely used languages uses Python for most of his projects. He says the main reason is that he likes the way source code looks. That may seem a frivolous reason to choose one language over another. But it is not so frivolous as it sounds: when you program, you spend more time reading code than writing it. You push blobs of source code around the way a sculptor does blobs of clay. So a language that makes source code ugly is maddening to an exacting programmer, as clay full of lumps would be to a sculptor.
> I vehemently disagree. Readability is the most important consideration. It's "subjective", but most human "subjects" prefer Python to Perl on this dimension, and it's very important to them. That's why we are where we are today.
I absolutely agree that readability is most important consideration. I just disagree that it depends on the language. As said before, there is a lot of write-only code written by data scientists in Python.
> Well sorry, but I don't want to get used to your favorite language's line noise. That's why I choose another language. That's why everyone else chooses another language.
You don't have to. It's called a programming language paradigm, not noise. You either learn it or not when studying a language. If you want to understand something without studying it, it's your choice.
I am proficient in both of these languages and can see the strengths and weaknesses in both of them.
> Those libraries were developed in Python because the authors of those libraries preferred to work in Python. The authors preferred to work in Python because of the legibility considerations I've already mentioned.
Perl was extensively used by scientific community, but Python was the lucky one that was backed by Google in 2005. And that's ok. It is better than Perl for data science now, but not because of its syntax :)
We'll have to agree to disagree on most of this, which is fine, better than having a stupidity-driven argument. I'm sure the aspects of Perl that are weird and gross to me are great for some other people.
> Perl was extensively used by scientific community, but Python was the lucky one that was backed by Google in 2005.
I'm curious about this. Can you point to any specific evidence that being "backed by Google in 2005" was a momentous event for Python? AFAIK Google is not a major player in the Python scientific computing / data analysis / ML community outside of TensorFlow, which was introduced after Python was already huge.
In 2005 Google hired Guido van Rossum, where he spent half of his time developing the Python language.
As you can see, before 2006, Python was little less popular than Perl(9th vs 8th).
By the time Guido left Google(December 2012), Python became 4th most popular programming language.
I don't think Guido did anything special for the scientific/technical/data computing community after he came to Google. Python was on version 2.4 as of 2005, and that version would have been fine to develop against indefinitely. In fact, it's been a huge headache getting scientists to move to Python 3.
The sibling comment mentioned that there's probably no contribution from Guido regarding the meteoric rise of Python in data science but the truth it's really the opposite [1]. Guido was really instrumental in suitability of Python for data science [2].
To quote from the magazine article on Python for Scientists and Engineers:
"During these early years, there was considerable interaction between the standard and scientific Python communities. In fact, Guido van Rossum, Python's Benevolent Dictator For Life (BDFL), was an active member of the matrix-sig. This close interaction resulted in Python gaining new features and syntax specifically needed by the scientific Python community. While there were miscellaneous changes, such as the addition of complex numbers, many changes focused on providing a more succinct and easier to read syntax for array manipulation. For instance, the parenthesis around tuples were made optional so that array elements could be accessed through, for example, a[0,1] instead of a[(0,1)]. The slice syntax gained a step argument— a[::2] instead of just a[:], for example—and an ellipsis operator, which is useful when dealing with multidimensional data structures."
Personally I'd much prefer D language to replace python for data science due to its Python like programming vibes but with more powerful real-time programming constructs, and its suitability for beginners or type A data scientists [3]. Python is already showing its limitations especially for the B type of data scientists [4]. There's recent discussions on Dlang forum for the potential improvements of D for data science and there's also recommendation for Walter to join such SIG as was Guido [5].
I'm sure Guido did things for the scientific community. My claim was that he probably didn't move the needle for them after coming to Google in 2005.
The other person claimed that Google magic somehow led to Python becoming dominant in the sci/numeric/data computing community, and that just strikes me as wildly implausible. Google has done negligible foundational work on scientific Python. No one in that community has ever cared what Google thinks.
Anyways, I'm curious why you think D has potential as a Data Science language? Why not a more popular language like Julia or Rust?
> performance is mostly good enough across all languages
Performance of applications written in Python is good enough only were there is a very low requirement for fast code (e.g. glue code / shell replacement of which there is a lot and, as computers get faster, more and more) or there are native code libraries (e.g. numpy). Of the popular languages, only Ruby is slower. I have high hopes for Julia (at least in the data science / numerical computing domain) which performs much better, while preserving readability and ease of use of Python.
As someone who had to interview for VLSI design positions (both sides of the table) back when Perl was popular, I used to take a printout of a small tool I developed in Perl with me to every interview (parse a VCD file and check the results).
This forced the interviewers into MY subset of Perl. That made a huge difference into how competent they thought I was with Perl.
Most of the interviewers gave up after we went through about 10-20 lines of code because they had never seen half of the stuff in my code--and I wasn't doing anything particularly obscure (decorate-sort-undecorate/Schwartzian transform would lose most of them).
I guess I'm showing my youth here but I never imagined that sorting on a computed key was once known by such an exalted title as Schwartzian Transform. This is basic, built-in functionality in Python.
It's not just that the key is computed, but that the key has an internal sort comparison which is way faster than a user code comparison. It was an example of the fact that "Big-Oh notation is nice, but constants sometimes matter."
Both sorts have the same O(n log n) number of operations performance. However, in Perl, using an internal comparison for integers was a significant constant faster than using a user specified comparison function--at least an order of magnitude and sometimes two. Consequently, the 2*O(n) to decorate and undecorate the list with integers to sort by got swamped by the 1/10*O(n log n) sort improvement.
I believe that this overhead eventually got reduced. However, I remember using this a LOT in Perl 4 and early Perl 5 days.
Sounds generally plausible to me. I learned all sorts of obscure tricks to make MATLAB run faster in my grad school days... I wouldn't be surprised if something like this was useful over there as well.
Unless you are hell-bent on writing obfuscated code, Python code is almost always readable. Even when I didn't know any Python or any other language, I could make an educated guess what some Python program is doing. However, I have seen some of Perl code and I was like wtf.
As for best practices, Python does insist on writing readable code a lot. Python people always nudge you to write readable code. Guido himself said he designed Python to be a readable language.In general, Python has the philosophy of one way of doing something which contrasts with Perl way where you can accomplish a task in numerous ways. You can't be too clever with Python (little exaggerated position). However, the same is not true with Perl.
Yes, the problem was that to read Perl you had to know every way to express things, and Perl was much, much too flexible there. I wrote a bunch of Perl in the early 2000s and I hated it. Both Ruby and Python were far superior. Even Java was better (albeit with other caveats). I hated Perl. List vs. scalar context is one example of a "feature" in Perl that causes confusion in readers.
In science doing things in a different way is good, not bad. If everybody uses the same tool you kill the cross-pollination between people with different skills and innovation often comes from that.
Exploring Bioperl for genetics leads to several big rewards, but you need to want to take the extra-mile and doing it. And is not easy. In the same way as exploring R in the past where none used it, was definitely benefical to me. R is awesome.
All is readable if you find the correct people able to read it. People at the university just keep using the same as their boss so you really can't always choose. Maybe Perl wizards were highly sough and didn't remained in the university. Maybe is just a question of different generations of programmers and Perl is a little older. Dunno.
119 comments
[ 0.28 ms ] story [ 185 ms ] threadPython also got heavyweight libraries such as Numpy and Pandas which put it in the front. Perl do not have have such well known libraries as far as I know.
Amen to this point. I am currently the single engineer in a small software biz and after the last engineer left I thought to myself "well, at least I won't have to decipher his code anymore"...needless to say, I've been shocked by how much time I can spend on a system only to revisit it within a year and have little to no memory of how or why I did something a certain way!
This. For the kind of tasks data science involves, you either write C++ or you need the equivalent of Numpy. No Numpy means no Pandas, which means no sane way to wrangle datasets.
For all the hate it gets, Numpy is a world-class library.
Looking now (it's open source) there's two keywords I have forgotten. It's a Perl binding to a C library using an old version of Swig. There's lots of Perl or C helper code to smooth out the bumps. It also looks like it hasn't been updated (bummer).
perl doesn’t have the numerical chops to keep up, and if it started to fix that now, it has 20 years of headwind to fight through for probably marginal gains. Good luck.
I think R is worse but it is bundled with a very good stats library. R users aren't so interested in "programming in the large" or even the small so it is OK.
Other than the meaningful indentation Python strikes me as a generic and normal programming language. I'm quite amused at how Python and Java got pattern matching at about the same time with a broadly similar approach to "fit this new feature in this language so that it feels like it belongs and plays well with decades of existing practice".
I do come across some perl every now and then but it looks like Bash on steriods to me (perhaps because of the $variables), and I never see any DataFrames or similar structure that looks familiar to me.
Idk, Perl never occurred to me and nobody ever recommended it. Is Perl good for data science? Do you have any examples? I never ran into "efficiency problems" with Python btw, I'm not using it at that scale, at what scale would I notice this? I have run out of ram at times, but that's usually when a dataset on disk is already larger than my ram. But then I usually find a tool to deal with the data anyway (for example pyosmium for super large osm/o5m fles).
Edit: I feel that Python also gives me other nice things to get started with, things like Django and Snakemake. This also leads me to recommend Python to other people, it's a broad basis for a lot of stuff. That's why even though some people recommended R when I got started, I choose Python anyway. I have no regrets, unless you are going to blow my mind with some examples...
0, https://www.raku.org/
Every time I look at an example I am turned off and my mind is in a way "boggled" ...
I just opened the tutorial on the Raku site I see this
I find this awful,right from the beginning, the say turns me off, computers do not speak, at least I really hope the computer is not actually speaking when I type that. Then there are the weird rules of quoting to not quoting ... I can't quite tell the rules are
the example is unpalatable, why does the looks_like_number word have underscores? is that a odd variable name that needs underscores? but it looks like a message to the user, what a terrible choice either way!
what does this print? I can't tell ...
I mean things like https://youtu.be/7Ivoh-fzDw8 , https://stackoverflow.com/questions/60510667/how-can-i-defin... and https://en.m.wikibooks.org/wiki/Raku_Programming/Grammars
The example for "looks_like_number" has probably be chosen, because many experienced Perl programmers are familiar with that.
In Raku, you can use kebab-case, and if there would be such a routine in Raku, it would probably be called "looks-like-number", and return a Bool, as in either True or False.
"say" is also builtin in Perl, at least for the past 12 years or so. And if you are on a Mac, running "say this is your computer talking to you', it will talk to you.
In any case, the thing you appear to refer to, is in no way a tutorial, but rather documentation intended for a specific group of people: Perl programmers.
That said, `say` is just a normal function, and you could always bind a new function and use that instead.
However, it might be a bit too much if you are from other fields and just want to get your sums in order.
Both Python and Perl made bold transitions, Python from 2 to 3 and Perl from 5 to 6.
The Python transition was difficult but ultimately successful. On the other hand, Perl rolled the dice and lost. In some alternate universe it might have been the other way around.
Perl 6 (Raku) is effectively a different language. Porting from 5 to 6 would change most lines of code, in other words, a complete rewrite. To ease transition, Perl 6 made it so that both could coexist, but if you want to go v6, then it is all or nothing.
Python 2 and 3 are not that different. You can easily write code that is compatible in both, and porting is often not much more than using 2to3.py most of it is replacing "print" with "print()" and converting between bytes and str.
Perl strategy was that to create a modern language inspired by Perl, in indeed, Perl 6 has quite a few features ahead of its time. And Raku (the new name for Perl 6) is still one of the most advanced language today.
Python strategy was to keep the same language but address a few pain points that require breaking compatibility, Unicode in particular.
Perl 6 took a much bigger gamble.
> Naturally far more efficient
What does this mean exactly? One big reason for Python’s success in data science is numpy, which is far more efficient (especially on large data) than vanilla Python. I’m unaware of the state of the Perl ecosystem, does it have something similar?
> how come Perl lags far behind or gets dwarfed in data science by Python?
Perl seems to be lagging behind in general, no? Is there anything specific about data science where Perl should be shining?
Perl has PDL (http://pdl.perl.org), which I think predates numpy and does the same kind of array manipulation.
As for the original question, Python's dominance for "data science" projects is just a matter of momentum at this point: it's where Google and others have put their money, and that is now affecting undergraduate and graduate courses. Originally, I think it was just that some people didn't like Perl's syntax.
Personally, I prefer to use a mix of Perl and Julia, each for its respective strengths, and find Python all-around mediocre.
> Naturally far more efficient
Most data science languages are interpreted with a few higher-level routines. Same with J, for instance - it has adequate performance as well.
(Only half joking, everything is dwarfed by Python in the data science community.)
Then a whole ecosystem was built on top of numpy, with scipy, pandas, PIL, et cetera. Everything that uses n-dimensional arrays used numpy as their base and as a result all those things can be combined trivially. That's very powerful.
Then later came ipython, a much improved interactive shell, and Web based notebooks that are very useful for data science work.
That the language involved is Python isn't even important, imo. Numpy + ecosystem replaced Matlab. All Python has to be is be a better language than Matlab, and it is.
However, for data science, the biggest drawback vs Python is how you work with complex data structures. Because everything in Python is an object, (de)referencing things is generally straightforward, like:
Where Perl devolves into a mess of sigils, when to use $ vs % vs @, wrapping references with {}, etc. This gets more complex as the nesting deepens, or as you add things like objects, want to iterate keys of a dict, and so on.Edit: A real example from the perl data structures (perldsc) man page:
Somehow Python took the niche of “executable pseudocode”, “glue language” for me.
Minus the sigils and crazy wrapping, like: @{ $TV{$family}{kids} } to refer to the array, but a different sigil/form if you're talking about one element, etc.
Which may not look intimidating on its own, but it's leaving out % and the backslash \$reference syntax, $object->method and other things that get confusing.
You can't, for example really make a "list of hashes/dicts". Try it, and it just turns the keys/values into array elements. You have to specifically make a "list of references to hashes/dicts". So you do something like:
Then, fun things like a double sigil: Note there are several ways to do all this, and I am highlighting some of the more visually confusing ones.As far as the perldsc page example, I think that comes down to having a better personal style that makes your code readable. I would make the top level variable a hash reference rather than a plain hash, and purposefully not insist on dereferencing multiple levels all on one line.
As someone pointed out in another comment, you can make the Python version look just as lousy by virtue of a lousy style.
My version of code doing the same task would be something like this:
So readability matters. And Python is one of the most readable languages out there. It remains relatively readable even as the code and data structure get very complex. Its syntax resembles math and pseudo-code more closely than other languages. It feels more like a tool of thought, not just a bunch of alien hieroglyphics you have to write to crudely and inefficiently express how you really think about the problem.
Perl, on the other hand, is one of the least readable languages for a general audience. It is not a tool of thought for non-experts. Its syntax is ugly, obscure, overly symbol-laden, and beloved only by gurus. The supposed "feature" of TIMTOWTDI just makes it more obscure.
The thing I can never get over with Perl is that it didn't even have a sane, idiomatic notation for functions until recently (and even today I suspect it's not widely adopted). There was no such thing as just defining a function f(a,b) of multiple named formal arguments. You had to use some special magic variable and people start talking about shift operators or some such nonsense.
This is the point at which I start saying "yes, all the language choices are Turing complete, but that doesn't mean they're all equally effective choices".
Just so it's clear, you do call a function in that way... some_function($a,$b).
It's how the function gets a hold of those arguments that you're talking about..the args are in an array called @_, like:
sub some_function { ($first_arg,$second_arg)=@_; }
And, like most things in Perl, there's lots of different ways to deal with it. Using shift(), or passing args as a hash, etc.
The concept behind sigils is very simple and once you've learned it, it's not that threatening. And when talking about data structures, it is a benefit to have it more structured and know that you are dealing with an array or hash or scalar.
In my opinion, Perl is not used for data science because it lacks(or people are not aware of) libraries like numpy, pandas, etc.
I vehemently disagree. When choosing among languages today, performance is mostly good enough across all languages and readability becomes the most important consideration. It's "subjective", but human beings are subjects, and subjective things are important to them.
> I've seen a lot of unreadable code written by data scientist in Python. And a lot of readable Perl code. It's not about the language, it's about the best practices.
This is a bad and cliche argument. Yes, there is well-written code in languages that deprioritize readability and badly-written code in languages that highly prioritize readability. That doesn't mean that any differences in the inherent, built-in readability of X and Y are moot. And "best practices", which are not widely agreed upon or followed, absolutely do not suffice to paper over this.
> The concept behind sigils is very simple and once you've learned it, it's not that threatening.
"Once you've learned it" or "get used to it" is the biggest programming cliche in the world. People say this about all their favorite languages and language features. I'm sure I could complain about doing data science in Brainfuck and get someone in the comments who would say it's not threatening "once you get used to it".
Well sorry, but I don't want to get used to your favorite language's line noise. That's why I choose another language. That's why everyone else chooses another language.
> And when talking about data structures, it is a benefit to have it more structured and know that you are dealing with an array or hash or scalar.
You can keep track of types in Python with gradual typing. Most people don't bother, so evidently it's not that important in most cases, but you can do it.
Other than that, no, I don't need to fill my code with more line noise to constantly remind myself what type of container I'm dealing with. That's obvious enough from the methods I'm using to manipulate the container.
> In my opinion, Perl is not used for data science because it lacks(or people are not aware of) libraries like numpy, pandas, etc.
No, this is after the horse has already left the barn. Those libraries were developed in Python because the authors of those libraries preferred to work in Python. The authors preferred to work in Python because of the legibility considerations I've already mentioned.
Python's design and adoption were both heavily driven by a vision of readability from the very earliest stages in the 1980s/90s. People got into Python because they liked how the code looked:
> A friend of mine who knows nearly all the widely used languages uses Python for most of his projects. He says the main reason is that he likes the way source code looks. That may seem a frivolous reason to choose one language over another. But it is not so frivolous as it sounds: when you program, you spend more time reading code than writing it. You push blobs of source code around the way a sculptor does blobs of clay. So a language that makes source code ugly is maddening to an exacting programmer, as clay full of lumps would be to a sculptor.
- Paul Graham (http://www.paulgraham.com/pypar.html, August 2004)
I absolutely agree that readability is most important consideration. I just disagree that it depends on the language. As said before, there is a lot of write-only code written by data scientists in Python.
> Well sorry, but I don't want to get used to your favorite language's line noise. That's why I choose another language. That's why everyone else chooses another language.
You don't have to. It's called a programming language paradigm, not noise. You either learn it or not when studying a language. If you want to understand something without studying it, it's your choice. I am proficient in both of these languages and can see the strengths and weaknesses in both of them.
> Those libraries were developed in Python because the authors of those libraries preferred to work in Python. The authors preferred to work in Python because of the legibility considerations I've already mentioned.
Perl was extensively used by scientific community, but Python was the lucky one that was backed by Google in 2005. And that's ok. It is better than Perl for data science now, but not because of its syntax :)
> Perl was extensively used by scientific community, but Python was the lucky one that was backed by Google in 2005.
I'm curious about this. Can you point to any specific evidence that being "backed by Google in 2005" was a momentous event for Python? AFAIK Google is not a major player in the Python scientific computing / data analysis / ML community outside of TensorFlow, which was introduced after Python was already huge.
Here are some stats with the popularity of various languages over time: https://statisticsanddata.org/data/the-most-popular-programm...
In 2005 Google hired Guido van Rossum, where he spent half of his time developing the Python language. As you can see, before 2006, Python was little less popular than Perl(9th vs 8th). By the time Guido left Google(December 2012), Python became 4th most popular programming language.
To quote from the magazine article on Python for Scientists and Engineers:
"During these early years, there was considerable interaction between the standard and scientific Python communities. In fact, Guido van Rossum, Python's Benevolent Dictator For Life (BDFL), was an active member of the matrix-sig. This close interaction resulted in Python gaining new features and syntax specifically needed by the scientific Python community. While there were miscellaneous changes, such as the addition of complex numbers, many changes focused on providing a more succinct and easier to read syntax for array manipulation. For instance, the parenthesis around tuples were made optional so that array elements could be accessed through, for example, a[0,1] instead of a[(0,1)]. The slice syntax gained a step argument— a[::2] instead of just a[:], for example—and an ellipsis operator, which is useful when dealing with multidimensional data structures."
Personally I'd much prefer D language to replace python for data science due to its Python like programming vibes but with more powerful real-time programming constructs, and its suitability for beginners or type A data scientists [3]. Python is already showing its limitations especially for the B type of data scientists [4]. There's recent discussions on Dlang forum for the potential improvements of D for data science and there's also recommendation for Walter to join such SIG as was Guido [5].
[1]Numpy History:
https://en.wikipedia.org/wiki/NumPy#History
There's even recomendation from Dlang forum for Walter to join such SIG [3].
[2]Python for Scientists and Engineers:
https://www.computer.org/csdl/magazine/cs/2011/02/mcs2011020...
[3]Teaching D from Scratch: Is it a viable first language?
https://news.ycombinator.com/item?id=29711800
[4]Type A Data Scientist vs. Type B Data Scientist:
https://www.projectpro.io/article/type-a-data-scientist-vs-t...
[5]Dlang forum: Pandas like features:
https://forum.dlang.org/thread/zuebxhkycgxsoipsgnbk@forum.dl...
The other person claimed that Google magic somehow led to Python becoming dominant in the sci/numeric/data computing community, and that just strikes me as wildly implausible. Google has done negligible foundational work on scientific Python. No one in that community has ever cared what Google thinks.
Anyways, I'm curious why you think D has potential as a Data Science language? Why not a more popular language like Julia or Rust?
Performance of applications written in Python is good enough only were there is a very low requirement for fast code (e.g. glue code / shell replacement of which there is a lot and, as computers get faster, more and more) or there are native code libraries (e.g. numpy). Of the popular languages, only Ruby is slower. I have high hopes for Julia (at least in the data science / numerical computing domain) which performs much better, while preserving readability and ease of use of Python.
I strongly disagree.
As someone who had to interview for VLSI design positions (both sides of the table) back when Perl was popular, I used to take a printout of a small tool I developed in Perl with me to every interview (parse a VCD file and check the results).
This forced the interviewers into MY subset of Perl. That made a huge difference into how competent they thought I was with Perl.
Most of the interviewers gave up after we went through about 10-20 lines of code because they had never seen half of the stuff in my code--and I wasn't doing anything particularly obscure (decorate-sort-undecorate/Schwartzian transform would lose most of them).
Both sorts have the same O(n log n) number of operations performance. However, in Perl, using an internal comparison for integers was a significant constant faster than using a user specified comparison function--at least an order of magnitude and sometimes two. Consequently, the 2*O(n) to decorate and undecorate the list with integers to sort by got swamped by the 1/10*O(n log n) sort improvement.
I believe that this overhead eventually got reduced. However, I remember using this a LOT in Perl 4 and early Perl 5 days.
As for best practices, Python does insist on writing readable code a lot. Python people always nudge you to write readable code. Guido himself said he designed Python to be a readable language.In general, Python has the philosophy of one way of doing something which contrasts with Perl way where you can accomplish a task in numerous ways. You can't be too clever with Python (little exaggerated position). However, the same is not true with Perl.
Also because Python is readable by most software engineers and data scientists, unlike R.
Exploring Bioperl for genetics leads to several big rewards, but you need to want to take the extra-mile and doing it. And is not easy. In the same way as exploring R in the past where none used it, was definitely benefical to me. R is awesome.
All is readable if you find the correct people able to read it. People at the university just keep using the same as their boss so you really can't always choose. Maybe Perl wizards were highly sough and didn't remained in the university. Maybe is just a question of different generations of programmers and Perl is a little older. Dunno.