Why not? Every science/statistics/math/finance/economics person I know uses R. Lots of slightly more 'casual' users, but it's gone completely mainstream. Universities around here have it installed on most lab computers.
I'm also pleased to see Fortran is still going strong. Great language for crunching numbers.
R may be niche as a programming language, but it's mainstream as a tool. It's a language used by many, many amateur and non-programmers. There are more people out there that have data to process than there are programmers.
"Swift’s rise is impressive, as it’s jumped five positions to 11th place since last year, when it first entered the rankings"
Swift will enter the top 10 next year after more iOS developers finally see it as stable. The big source changes in the upcoming 3.0 have caused a lot of people to wait.
There was talk about being able to run Swift apps on Android in the future. If it becomes official i think the community around swift would be pretty large.
Apple has developed a bad habit of abandoning compatibility with their developer tools. While this is somewhat more excusable with Swift, given that it's fairly new, it's pretty annoying to have to (e.g.) basically relearn Xcode from scratch with every major release.
I admit I have never truly learned awk outside of the most dead simple stuff, but one of the most useful python utilities I have ever written is below. It allows you to use python lambdas on lines of stdin.
Example usage is:
lambda "x0 + x1 * x2" int int int
Code:
import sys
# libs I use commonly...
import random, itertools, re
# parse the types of the columns
types = map(eval, sys.argv[2:])
# craft a lambda with the args x0, x1, ... xN
f = eval("lambda " + ','.join("x"+str(i) for i in xrange(len(types))) + ":" + sys.argv[1])
# apply lambda to stdin, don't print results of None
for line in sys.stdin:
args = []
for t, e in zip(types, line.strip().split()):
args.append(t(e))
result = print f(*args)
if result != None:
print result
Examples:
Where data.txt is
john doe 37
jane doe 35
jack bob 20
bill bob 40
And I do
cat data.txt | lambda "x0 if x2 < 36 else x1" str str int
It will output
doe
jane
jack
bob
You can use this sort of tool for a million things. e.g. sample 1 out of every 1000 lines:
lambda "x0 if random.random() <= .001 else None" str
It's probably the same power and whatnot as awk, but I know and am much more familiar with python, so it's useful for me.
$ awk '{if ($3 < 36) print $1; else print $2}' data.txt
doe
jane
jack
bob
Python has a lot of strengths and is better than awk at a lot of things, but one-liner column based text processing on the command line is literally awk's bread and butter.
I made a similar tool in Perl. It eval's a Perl command passed as string, on each line of stdin. In the middle I can go wild with regexes and hash/dictionaries and whatnot. It's one of my most used tools.
Instead of writing scripts for each little task, I just write one-liners. When they become more than 2 lines long it becomes unwieldy and I switch it to a regular script.
I grew up data munging with awk/sed/tail/head manipulations, avoiding Perl 5 (not out of any antipathy, but I just worked off of what my fathers bookshelf had and there was no Camel book on there). Back in the 90s we'd publicly post our dotfiles to our Apache 1.3 servers amongst peers, but (perhaps this was just a component of the IRC community I was a part of) we didn't share much of scripts-sets we built up overtime. The furthest we'd go is "I want to do foo", someone with more knowledge than you would give you a series of invocation parameters and over time you'd osmotically acquire enough knowledge to be the one dispensing knowledge.
From what I hear, Perl 5 is prime for the problem set you defined, but I've never seen any aggregated resource of people's Perl5 munging scripts. Do us all a favor and post a Github Gist of that tool (along with common invocations of you going wild with regexes and hashmaps). If you're feeling overly generous, post the source of the commonly used regular scripts as well.
nice one. small, sharp tools in the unix toolbox. given that this is the type of processing one might routinely over a file with tens of millions of lines, what's your guess on the difference in performance between the python snippet above your your awk one-liner? (my guess is about 100x in favor of awk)
I honestly have no idea. Given that there are different versions of awk and different versions of python, I'm not even sure there is an answer.
Given awk's age (1977-when computers were much slower and memory much more expensive) and pedigree (Aho, Weinberger, and Kernighan), I wouldn't bet against it for a task like you describe, but that's just a feeling. Again, I don't have any numbers to support that.
My gut-ballpark was going to be around an order of magnitude, not two. Here's are two naive comparisons (granted, from the late aughts and not a direct comparison but a more general one) that show ~7-8x[0,1]. The overhead of PyStringObject is not trivial[2] (though the implementation details likely have changed between Py2.x and Python 3).
For things like building accumulators a set of data/log parsing rather than data munging (hits per hour or enumerative tasks), I'd imagine (g|n)?awk might hit your 100x since you'd just grab the fd and traverse being IO bound. I'm not sure how awk does it, but if it's just saving an accumulator value (or 10) in a register. Assuming x64-64 treats, say, an r3 fetch analogously to a fetch to ecx (err..rcx now I guess), rather than having to keep a full object in L1 cache, awk has a huge advantage.
---
N.b., if you're benching tasks like this, don't use 'time' and STDOUT and think you're getting real performance numbers. Your bottleneck (terminals can only render $x lines a minute, so the kernel call to write(STDOUT, ....) will be where you choke, not at the language. Also disk fragmentation would be another issue. Put the both the test file and the output file on a RAMdisk.) Cache flush with Something like sync; echo 3 > /proc/sys/vm/drop_caches (on Linux, I forget the BSD way of doing it) then `time benchmark.py /mnt/ramdisk1/file > /mnt/ramdisk2' over multiple runs, under various loads, with different data sets, etc
Another interesting thing to note is that comp arch is so advanced (I was reading a paper on formal verification of ISAs, and apparently even 12 dollar ARMs now have out-of-order instruction execution type stuff) that between the kernel scheduler and CPU optimizations, Python will likely benefit much more from disk-seek latency (effectively allowing PyStringObject allocation to occur while you're waiting for /dev/sd$n to return).
I'm certainly not an authority on rigorous benchmarks though - someone like Brendan Gregg please jump in!
Go has replaced all my 'scripting' needs. Way easier to deploy static binaries and I get static typing for my scripts. Maintaining a python install on multiple servers and OSes is a nightmare
And that's probably why JS isn't higher... even for internal applications a LOT of UIs are web based these days, which means JS is involved to some extent... and a lot of those frameworks and tool sets are all JS. These days, I'd go so far as to say, if you're doing web anything, you should have some node too, as a significant amount of web tooling is node/js based.
Still absolutely no way people talk about D here more than JavaScript. And I imagine the reason assembly scores so high is treating "x86" and "ARM" as keywords, so that raises the question of why "Node React Angular Ember Meteor" etc aren't treated as keywords for JS.
I wonder if JavaScript just doesn't show up as much in their sources because it's implied?
At least when I've been job hunting for web-related work, your typical job posting lists the back-end language and the front-end is just assumed. (Things might have changed with all the heavy-duty front-end JS frameworks out there now.)
I have fondness on the time when HP was American... and Motorola... and calculators were meant to last... mechanical keyboards... when things were simple...
I tell people that the keyboard... the original mechanical one, just God knows how much money took to be designed... how many thousand hours of trial and error it took to be designed... just to get to hp and their new layouts... (don't make me talk about new thinkpads).
Dennis Ritchie was the IBM of programming languages... when fad after fad of JS framework pass, C stands...
It is about as expressive as a "portable assembly language" can be expected to be, it has pretty weird syntax [1], it has neither hygienic macros nor a module system, using "include files" instead. And a bunch of undefined behaviors on top of that.
Well, it was the unifying language despite all these shortcomings, because Unix. This is very much like JS is an ugly unifying language of sorts today, because the browser.
I very much hope that maybe something like Rust will take the place of the close-to-metal portable language. C made some sense when you had a 16-bit CPU @ 10 MHz and 128 KB of RAM for a development machine, and ed on a teletype as the IDE. Now we can do better.
I don't buy it. Choose any one of the weights and select only web; it'll put either Python or Java at #1 and PHP and JavaScript way below. That just doesn't seem realistic...
I think Java as #1 is realistic. Maybe within the Bay Area it is not #1, but for the past twenty years companies have been putting business logic and code into Java application servers, a lot of which is connected to intranets or the web.
While PHP is very widely used, it depends on the company. We're almost exclusively Java based web environments, outside of a single contracted web server where the dev chose PHP as a location-specific tool.
I'm not discounting the validity of anything here, just stating an example.
Not necessarily; looking at it another way, having lots of programmers who are interested in R might mean that the job market for them is fairly saturated, meaning recruiters don't have to advertise as much or as long
It's worth clicking through to the interactive page that lets you pick different criteria.
R is at #6 measured by "trending", but only #12 measured by "jobs" (below assembly).
Some of their methodology seems a little handwavy (which to be fair is no different than I've seen with other similar rankings).
For instance, the number of questions asked on StackOverflow confounds the popularity of the language with the difficulty/ambiguity/poor documentation of the language (e.g. a poorly-documented language might well result in a disproportionate number of SO questions).
Straight Google search results (as opposed to Google Trends, which they also use) would only seem to prove that a language was popular in the past. There are lots of Google search results for the Roman Empire, but that doesn't mean it's still a going concern. :-)
CareerBuilder and Dice were headhunter cesspools, the last time I looked at them. I would be very wary of drawing any conclusions about popularity from those (or similar) sites.
It could be that most R programmers have the kind of jobs where the language used is flexible and so might not be mentioned in a job listing. The jobs could be asking for statistical analysts or data scientists, and leave it up to the employee to decide if they want R or SciPy or some C++ toolkit or whatever.
Compare to someone looking to hire a web developer. There you are much more likely to need the developer to use specific languages, and so those should be mentioned in the job listing.
No, hipster is wanting to be seen as being all about niche unknown things, but it's really about jumping on the bandwagon before it overflows.
Elixir is pseudo-hipster, LFE (Lisp Flavored Erlang)is the true hipster language ;)
It is impossible for me to believe that C is the top programming language for mobile development. Are they double counting obj-c or something?
Same thing for web although I guess it could be the case that people are using C on a lot of old apps that add up to a significant marketshare? I would still be pretty shocked if that were the case.
It's unfortunate that Elixir didn't even make it on the list, but then again it's only a few years old and just starting to make a splash, especially with the popularity of the Phoenix framework.
i think a small addition to this chart (but not requiring any data not already gathered) would significantly increase its value and interest.
in particular, adding one small column (about the same width as the "Types" column) with a sparkline having five points, connected by a single line and which represents the Spectrum Ranking for the years 2012 - 2016
with this, the reader could see for each language the five-year trend in their Spectrum Index.
two languages might be next to each other on this chart, yet one might be trending monotonically downward, while the other might be trending sharply and consistently upward.
so for instance, look at places 26 - 29: Rust, Delphi, Fortran, and D. Those are contiguous, hence have similar Spectrum index values, but the shapes of their last-five-year time series are probably quite different (eg, Rust, sharply upward trend, only perhaps slightly flatter for D, essentially flat for fortran, and perhaps downward for delphi).
I hope to see ruby enter the top ten next year.
It is still circling around the top ten, being listed in top 20. After using Python, java, bash, lisp, javascripts. I have begun to like ruby. I am sure ruby lovers will grow in future as well, because ruby is so addictive.
Java Developers when they learn ruby, I am sure the will switch to JRuby, coding every thing in ruby and compiling them with JRuby to run them in JVM.
So, I am hopeful that ruby though not listed in top 10 this year, it will be listed next year.
I refactored the Ruby code write by other experienced Ruby developers before. And guess why they ask me to do the refactor job for them? Because I know how to write clean and deciplined code learn in Java and C lands. :-)
First, the article says "The 2016 Top Programming Languages" (emphasis mine), and yet HTML is on the list -- which is not a programming language, but a markup language.
Second, if you decide to ignore this, and see how HTML ranks, and turn off everything but the "Web" types, HTML ranks #8 (behind Java, Python, C#, PHP, JavaScript, Ruby, and Go).
I agree with many of the comments on that video. By that line of reasoning almost any kind of formatted data could be considered a "programming language".
I'm not buying it. I think there is a valuable distinction between a programming language and a markup language. HTML is not a programming language, it's a markup language (hence the last two letters of its name).
84 comments
[ 3.4 ms ] story [ 101 ms ] threadI'm also pleased to see Fortran is still going strong. Great language for crunching numbers.
Swift will enter the top 10 next year after more iOS developers finally see it as stable. The big source changes in the upcoming 3.0 have caused a lot of people to wait.
Nativescript, titanium, xamarin, or qt is where react native would be categorized with
Apple has developed a bad habit of abandoning compatibility with their developer tools. While this is somewhat more excusable with Swift, given that it's fairly new, it's pretty annoying to have to (e.g.) basically relearn Xcode from scratch with every major release.
I wouldn't mind if I never had to use awk again...
Example usage is:
Code: Examples:Where data.txt is
And I do It will output You can use this sort of tool for a million things. e.g. sample 1 out of every 1000 lines: It's probably the same power and whatnot as awk, but I know and am much more familiar with python, so it's useful for me.I don't think of it as a better awk, just a drop-in to minimize my cognitive overhead. It's basically Python-flavored awk.
Instead of writing scripts for each little task, I just write one-liners. When they become more than 2 lines long it becomes unwieldy and I switch it to a regular script.
From what I hear, Perl 5 is prime for the problem set you defined, but I've never seen any aggregated resource of people's Perl5 munging scripts. Do us all a favor and post a Github Gist of that tool (along with common invocations of you going wild with regexes and hashmaps). If you're feeling overly generous, post the source of the commonly used regular scripts as well.
Given awk's age (1977-when computers were much slower and memory much more expensive) and pedigree (Aho, Weinberger, and Kernighan), I wouldn't bet against it for a task like you describe, but that's just a feeling. Again, I don't have any numbers to support that.
For things like building accumulators a set of data/log parsing rather than data munging (hits per hour or enumerative tasks), I'd imagine (g|n)?awk might hit your 100x since you'd just grab the fd and traverse being IO bound. I'm not sure how awk does it, but if it's just saving an accumulator value (or 10) in a register. Assuming x64-64 treats, say, an r3 fetch analogously to a fetch to ecx (err..rcx now I guess), rather than having to keep a full object in L1 cache, awk has a huge advantage.
---
N.b., if you're benching tasks like this, don't use 'time' and STDOUT and think you're getting real performance numbers. Your bottleneck (terminals can only render $x lines a minute, so the kernel call to write(STDOUT, ....) will be where you choke, not at the language. Also disk fragmentation would be another issue. Put the both the test file and the output file on a RAMdisk.) Cache flush with Something like sync; echo 3 > /proc/sys/vm/drop_caches (on Linux, I forget the BSD way of doing it) then `time benchmark.py /mnt/ramdisk1/file > /mnt/ramdisk2' over multiple runs, under various loads, with different data sets, etc
Another interesting thing to note is that comp arch is so advanced (I was reading a paper on formal verification of ISAs, and apparently even 12 dollar ARMs now have out-of-order instruction execution type stuff) that between the kernel scheduler and CPU optimizations, Python will likely benefit much more from disk-seek latency (effectively allowing PyStringObject allocation to occur while you're waiting for /dev/sd$n to return).
I'm certainly not an authority on rigorous benchmarks though - someone like Brendan Gregg please jump in!
[0] https://diamondinheritance.blogspot.com/2008/04/awk-vs-pytho...
[1] https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-...
[2] http://www.laurentluce.com/posts/python-string-objects-imple...
The top ten languages on HN are C, C++, Assembly, Python, Go, Java, Rust, Swift, D and Lisp.
http://www.tiobe.com/tiobe_index
RedMonk with its more web-centric focus places it at #1:
http://redmonk.com/sogrady/2015/01/14/language-rankings-1-15...
This matches my experience I think JavaScript is popular on the internet but within businesses it is much less common.
Within my world (Industrial manufacturing) the most frequently used languages at least in my orginization...
Control Systems (HMI's, PLCs, Operator Guidance etc): C, C++, A lot of domain specific stuff for PLC's
Process Modelling / Engineering Simulation: Fortran, C, C++, some python, some matlab.
Scrpting: Perl, VBS, .BAT
Analytics / Reporting: SAS, COGNOS, ACCESS (older mostly replaced)
Web Apps: Java, ASP (older), Oracle Forms (older)
Legacy/Mainframe: PL/1 (rarely touched mostly just runs)
In House (business) Apps: Java, C++ (older), VBA (i.e. excel spreadsheets)
Intranet: Sharepoint
At least when I've been job hunting for web-related work, your typical job posting lists the back-end language and the front-end is just assumed. (Things might have changed with all the heavy-duty front-end JS frameworks out there now.)
Let the karma bleed =P
I have fondness on the time when HP was American... and Motorola... and calculators were meant to last... mechanical keyboards... when things were simple...
I tell people that the keyboard... the original mechanical one, just God knows how much money took to be designed... how many thousand hours of trial and error it took to be designed... just to get to hp and their new layouts... (don't make me talk about new thinkpads).
Dennis Ritchie was the IBM of programming languages... when fad after fad of JS framework pass, C stands...
Oranges vs apples
It is about as expressive as a "portable assembly language" can be expected to be, it has pretty weird syntax [1], it has neither hygienic macros nor a module system, using "include files" instead. And a bunch of undefined behaviors on top of that.
Well, it was the unifying language despite all these shortcomings, because Unix. This is very much like JS is an ugly unifying language of sorts today, because the browser.
I very much hope that maybe something like Rust will take the place of the close-to-metal portable language. C made some sense when you had a 16-bit CPU @ 10 MHz and 128 KB of RAM for a development machine, and ed on a teletype as the IDE. Now we can do better.
[1]: http://c-faq.com/decl/spiral.anderson.html
I'm not discounting the validity of anything here, just stating an example.
R is at #6 measured by "trending", but only #12 measured by "jobs" (below assembly).
Some of their methodology seems a little handwavy (which to be fair is no different than I've seen with other similar rankings).
For instance, the number of questions asked on StackOverflow confounds the popularity of the language with the difficulty/ambiguity/poor documentation of the language (e.g. a poorly-documented language might well result in a disproportionate number of SO questions).
Straight Google search results (as opposed to Google Trends, which they also use) would only seem to prove that a language was popular in the past. There are lots of Google search results for the Roman Empire, but that doesn't mean it's still a going concern. :-)
CareerBuilder and Dice were headhunter cesspools, the last time I looked at them. I would be very wary of drawing any conclusions about popularity from those (or similar) sites.
And so on.
Compare to someone looking to hire a web developer. There you are much more likely to need the developer to use specific languages, and so those should be mentioned in the job listing.
The only reason R is there is because of mentions in research papers which is weighted at 100%.
Java is #1 for jobs and StackOverflow.
Python and Swift is trending + hackernews mentions.
C/C++ are always up there.
Javascript is 5 or 6 always.
Shouldn't surprise anyone.
I will have to turn in my Hipster Membership Cards :-)
Same thing for web although I guess it could be the case that people are using C on a lot of old apps that add up to a significant marketshare? I would still be pretty shocked if that were the case.
Web and app developers code apps connecting net front ends to databases. JS front end and maybe obj-c/swift/c#/python/ruby/perl for the middle tier.
R is domain specific to stats/econometrics/engineering/data analysis. It competes with matlab, c++, maybe fortran and recently python.
in particular, adding one small column (about the same width as the "Types" column) with a sparkline having five points, connected by a single line and which represents the Spectrum Ranking for the years 2012 - 2016
with this, the reader could see for each language the five-year trend in their Spectrum Index.
two languages might be next to each other on this chart, yet one might be trending monotonically downward, while the other might be trending sharply and consistently upward.
so for instance, look at places 26 - 29: Rust, Delphi, Fortran, and D. Those are contiguous, hence have similar Spectrum index values, but the shapes of their last-five-year time series are probably quite different (eg, Rust, sharply upward trend, only perhaps slightly flatter for D, essentially flat for fortran, and perhaps downward for delphi).
Java Developers when they learn ruby, I am sure the will switch to JRuby, coding every thing in ruby and compiling them with JRuby to run them in JVM.
So, I am hopeful that ruby though not listed in top 10 this year, it will be listed next year.
I have faced similar problems like you. Its because ruby is easy to learn for everyone including morons. :D
First, the article says "The 2016 Top Programming Languages" (emphasis mine), and yet HTML is on the list -- which is not a programming language, but a markup language.
Second, if you decide to ignore this, and see how HTML ranks, and turn off everything but the "Web" types, HTML ranks #8 (behind Java, Python, C#, PHP, JavaScript, Ruby, and Go).
OK, I know, I'm easily amused :-D
I'm not buying it. I think there is a valuable distinction between a programming language and a markup language. HTML is not a programming language, it's a markup language (hence the last two letters of its name).