I write code in both daily. I don't think there is a burning need to know both, but there are definitely tasks each is good at. The RStudio IDE is really quite wonderful for interactive stuff. The pipe operator (allowing left to right evaluation of function chains) makes for extremely literate code. Most of the basic statistical functions are substantially better than their Python equivalents. Many of the ways in which you use Python for data science stuff are just poor imitations of R (in particular, pandas is a take on R's data frames that is imo not as productive). In some conditions R can be faster [in others, slower]. The R package ecosystem for more off the beaten path statistical stuff is better than the Python package ecosystem (if you're doing more intense ML and CNNs, the opposite is true). If you're doing dashboards, Shiny is great. But in each case, you can basically work in either language and you'll be fine.
I think it makes more sense to think of R as replacing Stata, SPSS, SAS, and to a lesser extent Matlab, rather than replacing Python.
Python vs R is the new Matlab vs R (and soon it may be Julia vs R).
My answer is always the same: there are a few excellent and quasi-standard packages/libraries that exist in one of them without fully equivalent analogues in the others, so if you need any of those then go with that system (eg Simulink in Matlab, tons of classical statistical models in R, some ML/DL libraries in Python). In fact, you might have to use multiple systems if your problems are complex enough and you don't want to compromise. If you don't fall in that category, then just stick with what you have, any of those languages are decent enough general purpose tools and the differences don't justify a codebase migration.
Python is great for plumbing, R is great for new analyses. I use Python to knit Rmarkdown documents from premade templates.
Of course you can achieve something similar with Jupiter notebooks. The real benefit is the perceived ability to get going with a new dataset.
With respect to massaging and visualizing rectangular data, I can get much further much faster in R tidyverse than python. Just as we see python as a language that allows for rapid development with its straightforward syntax - I see R tidyverse providing a faster data analysis exploration “experience”. Tidyverse also introduced me to functional programming which I have found to be very useful for data analysis; you can very strictly separate procedures in a way I have found beneficial. (Not to say you cannot acquire this knowledge another way, however.)
I remain an avid user of pandas data frames inside the python ecosystem. On more than one occasion, I have translated rapidly written tidyverse data wrangling routines into more maintainable functions in python with pandas.
Base R only? tapply? Ick, no thanks. New R programmers should learn tidyverse https://www.tidyverse.org/learn/, not just base R.
There are a lot of R programmers (including the author of the OP, apparently https://github.com/matloff/TidyverseSkeptic) who are used to the old way of doing things and allergic to tidyverse. But the base-R only, anti-Tidyverse attitude is going the way of COBOL.
I have worked full time in R for many years and it is no contest.
I agree with the author that one must learn vectors and subscripts, but that is not an excuse to shove dplyr under the rug. It is an easier way to do lots of things.
I agree that new R programmers should start with the tidyverse, though actually the first element of the tidyverse I'd teach is the pipe and that's now part of base R; the second element I'd teach is using the readr stuff mainly to not have to worry about stringsAsFactors and stringsAsFactors is now default off.
Still, I think ggplot is a better way of thinking about plotting than base R's multiple and not very coherent plotting systems, dplyr beats the pants off any kind of base tools for manipulations, a lot of the tibble/pillar display stuff is great, and personally I disagree with Norm and think functional programming is as accessible as loops are to novice programmers.
I've taught a lot of people programming, and frequently work with novices. Functional concepts are nearly universally a sticking point for learners. For loops haven't caused any more trouble than the other basic concepts in my experience. It's an order of magnitude more difficult for learners.
Loops are very easy to debug in R. When they break and they're not inside a function, your REPL state is set to exactly what the environment was before the loop broke.
It's not as convenient because now you have to rerun the code and get in and out of a debugging environment (which can be slow).
I think FP is good for more concisely expressing simple stuff that's unlikely to break. Don't really believe in using it as a complete replacement for loops.
How about a conditional breakpoint? Though this doesn’t solve the “issue” of rerunning through the whole list.
Or - once you’ve found what element in your list is causing the issue- extract it and feed it to your function with your debugger on.
Admittedly I have often had a more challenging time fixing FP code than if I had used a for loop. This might be because I am not as familiar/comfortable with FP than procedural flow.
There has to be a for loop eventually. But crawling through a for loop within a procedure can also be painful if there are many different case scenarios. You could counter that good code wouldn’t suffer such a problem, but if the code was good it might not have been broken!
> I've taught a lot of people programming, and frequently work with novices. Functional concepts are nearly universally a sticking point for learners. For loops haven't caused any more trouble than the other basic concepts in my experience. It's an order of magnitude more difficult for learners.
The problem is that is teaches people the "wrong" * way to use R. I agree for loops are easier to understand, but when nearly the entire R universe operates on vectors, you end up with students who will do R the hard way their entire career.
If you surveyed most R users (verses R programmers) I would say the majority would be unaware that the base type is a vector.
* I know there is no wrong way to do anything, but I have seen plenty of times for loops simply applying an already vectorized function to each element of the vector.
I think R users should know how to use vectors, vectorized functions, loops, and core tidyverse (e.g. basic dplyr and ggplot). FP stuff like purrr::map isn't essential.
My impression was that the last time I was using R intensively there was a movement back from tidyverse to base R. But that's a few years ago, so you say that's reversed again?
I'm not against it at all, in fact I was doing nearly all of my work in dplyr, reshape, ggplot,... and I second that it's much more convenient and intuitive than the various apply's,...
My impression was that it was quite political and Hadley Wickham had antagonized some people in the core R community (and tbf, it was a bit annoying with him introducing redundant packages, reinventing and relabelling multiple ways of doing the same things, and presenting each new iteration as the end-all-be-all).
I'd say there was been a backlash against tidyverse, but tidyverse is beloved by most new R programmers, and fighting it is a one-way ticket to irrelevance.
Re: antagonistic behavior, I would just recommend that all participants consider the old glass houses proverb as applicable to themselves, and try to do better going forward.
In contrast, in my experience students relying heavily on tidyverse love it - but are also poorly positioned for doing anything that's not contained in an easily found template.
I don't think that is the tidyverse's fault. Learning the tidyverse doesn't corrupt your brain and make it impossible to learn about vectors and subscripts.
You’re being downvoted but it’s true - R would be dead today if not for the tidyverse (both because of ease-of-use and fostered community).
The experience between using the two is night-and-day. I would rather use the cli than base R at this point, and am convinced that the only proponents of base R over the tidyverse haven’t tried the latter for themselves.
Thank you. In reading the author's tidyverse critique, I think he has some valid points. It is wonderful that you can do so much with just vectors in base R, and this may be better for true beginners with no coding background. I do not agree with hiding vectors and subscripts and attempting to indoctrinate students with very pure and abstract coding approaches - as it seems to be alleged that some tidyverse advocates do. But "dplyr is bad" is just an awful take, and "dplyr is bad for beginners" is no better. These are just about the worst takes you could have about modern R programming.
I strongly recommend data.table R. Tidyverse is an improvement on base R, no question. Data.table has less intuitive syntax and can be harder to learn, but is lightning fast and memory efficient. If you're working with more than 1M rows, you should be using data.table.
I think it's great people like data.table's performance but I don't think performance is a compelling reason to be prescriptive about data.table. It's a reason data.table may be appealing to some. People can try them and decide for themselves if the performance is worth the trouble. In many cases the performance difference with dplyr is hardly noticeable and IMO readability of dplyr becomes the more meaningful advantage.
I have plenty of projects with tables with millions of rows, and I rarely find dplyr to be slow enough to get frustrated about. And even then I'd rather endure, or maybe use dtplyr, or maybe use data.table for a small performance-critical part of the code, than make a wholesale switch.
I find data.table code as obscure as base R if not more so. Purported brevity (always illustrated only on very simple problems) doesn't compensate me enough for the obscurity.
I could get used to it, I'm sure. I could also get used to doing everything in base R, pandas, or Clojure. Doesn't mean it's a good use of my time.
I get the sense from how some people blog that basic "filter + group by + summarize" type pipelines on giant datasets are all they really do. data.table may be a great choice for that kind of thing. But I use dplyr as a tool of thought. I can solve complex, novel data processing and analytics problems in dplyr more precisely and efficiently than I can explain in words. Instead of having an idea and then having to think about how to code it, the ideas translate into dplyr in an obvious and automatic way.
All that said, data.table's authors have worked much harder to provide advanced functionality, most notably non-equijoins, that it looks like the tidyverse may never catch up to. So if you need that kind of thing a lot, you need data.table or maybe DuckDB.
If the goal here is becoming productive quickly, try https://exploratory.io/ which is a sort of WYSIWYG environment for R that will still let you code by hand if needed.
No affiliation, just a happy customer for 2 years.
32 comments
[ 3.4 ms ] story [ 89.9 ms ] threadI think it makes more sense to think of R as replacing Stata, SPSS, SAS, and to a lesser extent Matlab, rather than replacing Python.
Julia is also fun.
You might want to check out Quarto [1], which i recently discovered on here
[1] https://quarto.org/
My answer is always the same: there are a few excellent and quasi-standard packages/libraries that exist in one of them without fully equivalent analogues in the others, so if you need any of those then go with that system (eg Simulink in Matlab, tons of classical statistical models in R, some ML/DL libraries in Python). In fact, you might have to use multiple systems if your problems are complex enough and you don't want to compromise. If you don't fall in that category, then just stick with what you have, any of those languages are decent enough general purpose tools and the differences don't justify a codebase migration.
- R Shiny
- RStudio Jobs
- RStudio Server
- RMarkdown, Blogdown, Bookdown, etc.
Of course you can achieve something similar with Jupiter notebooks. The real benefit is the perceived ability to get going with a new dataset.
With respect to massaging and visualizing rectangular data, I can get much further much faster in R tidyverse than python. Just as we see python as a language that allows for rapid development with its straightforward syntax - I see R tidyverse providing a faster data analysis exploration “experience”. Tidyverse also introduced me to functional programming which I have found to be very useful for data analysis; you can very strictly separate procedures in a way I have found beneficial. (Not to say you cannot acquire this knowledge another way, however.)
I remain an avid user of pandas data frames inside the python ecosystem. On more than one occasion, I have translated rapidly written tidyverse data wrangling routines into more maintainable functions in python with pandas.
YMMV
There are a lot of R programmers (including the author of the OP, apparently https://github.com/matloff/TidyverseSkeptic) who are used to the old way of doing things and allergic to tidyverse. But the base-R only, anti-Tidyverse attitude is going the way of COBOL.
I have worked full time in R for many years and it is no contest.
I agree with the author that one must learn vectors and subscripts, but that is not an excuse to shove dplyr under the rug. It is an easier way to do lots of things.
Still, I think ggplot is a better way of thinking about plotting than base R's multiple and not very coherent plotting systems, dplyr beats the pants off any kind of base tools for manipulations, a lot of the tibble/pillar display stuff is great, and personally I disagree with Norm and think functional programming is as accessible as loops are to novice programmers.
I think FP is good for more concisely expressing simple stuff that's unlikely to break. Don't really believe in using it as a complete replacement for loops.
Or - once you’ve found what element in your list is causing the issue- extract it and feed it to your function with your debugger on.
Admittedly I have often had a more challenging time fixing FP code than if I had used a for loop. This might be because I am not as familiar/comfortable with FP than procedural flow.
There has to be a for loop eventually. But crawling through a for loop within a procedure can also be painful if there are many different case scenarios. You could counter that good code wouldn’t suffer such a problem, but if the code was good it might not have been broken!
The problem is that is teaches people the "wrong" * way to use R. I agree for loops are easier to understand, but when nearly the entire R universe operates on vectors, you end up with students who will do R the hard way their entire career.
If you surveyed most R users (verses R programmers) I would say the majority would be unaware that the base type is a vector.
* I know there is no wrong way to do anything, but I have seen plenty of times for loops simply applying an already vectorized function to each element of the vector.
I'm not against it at all, in fact I was doing nearly all of my work in dplyr, reshape, ggplot,... and I second that it's much more convenient and intuitive than the various apply's,...
My impression was that it was quite political and Hadley Wickham had antagonized some people in the core R community (and tbf, it was a bit annoying with him introducing redundant packages, reinventing and relabelling multiple ways of doing the same things, and presenting each new iteration as the end-all-be-all).
Re: antagonistic behavior, I would just recommend that all participants consider the old glass houses proverb as applicable to themselves, and try to do better going forward.
The experience between using the two is night-and-day. I would rather use the cli than base R at this point, and am convinced that the only proponents of base R over the tidyverse haven’t tried the latter for themselves.
Here are some benchmarks: https://h2oai.github.io/db-benchmark/
I have plenty of projects with tables with millions of rows, and I rarely find dplyr to be slow enough to get frustrated about. And even then I'd rather endure, or maybe use dtplyr, or maybe use data.table for a small performance-critical part of the code, than make a wholesale switch.
I just enjoy having to read much less code when compared to tidyverse and SQL.
Plus the syntax difference between tidyverse and dt are overstated.
https://atrebas.github.io/post/2019-03-03-datatable-dplyr/
I could get used to it, I'm sure. I could also get used to doing everything in base R, pandas, or Clojure. Doesn't mean it's a good use of my time.
I get the sense from how some people blog that basic "filter + group by + summarize" type pipelines on giant datasets are all they really do. data.table may be a great choice for that kind of thing. But I use dplyr as a tool of thought. I can solve complex, novel data processing and analytics problems in dplyr more precisely and efficiently than I can explain in words. Instead of having an idea and then having to think about how to code it, the ideas translate into dplyr in an obvious and automatic way.
All that said, data.table's authors have worked much harder to provide advanced functionality, most notably non-equijoins, that it looks like the tidyverse may never catch up to. So if you need that kind of thing a lot, you need data.table or maybe DuckDB.
No affiliation, just a happy customer for 2 years.