There's a bit of circular logic in here. Can somebody make a short explanation on what's this is about?
Overview
The tidyverse is a set of packages that work in harmony because they share common data representations and API design. The tidyverse package is designed to make it easy to install and load core packages from the tidyverse in a single command.
It's a popular set of packages for data analysis in R. Similar to Python, it's an interpreted language with compiled libraries that do most of the heavy lifting.
Thanks; that's helpful. "Data analysis in R" belongs somewhere in the quoted intro paragraph. Otherwise that paragraph could describe almost anything: game engine, graphics library, string processing, HTML rendering, embedded systems...
There is "the tidyverse" which is a collection of packages like dplyr and tidyr for data manipulation and ggplot2 for visualization.
Then there's "the tidyverse package" which allows you to import them all at once with "library(tidyverse)". You don't have to use the tidyverse package, you can load individual libraries as you need them.
Also worth noting that one of the common organizing principles between all packages in the tidyverse is the concept of "tidy data" (see specifically H Wickham's paper), and the use of intercompatible combinators.
Under the hood, the tidyverse is comprised of many packages, some very large and sprawling, like ggplot, and others that do one single thing (readr). Here’s a list: https://www.tidyverse.org/packages/
Individual packages can be used in isolation, if that’s what you need. However, they complement each other nicely, in that they share data representations and similar conventions while handling different facets of data analysis. Thus, people often import the whole thing in one shot as the tidyverse and go from there. For example, you might read the data in with readr, clean it up with tidyr or dplyr, analyze it with tidymodels, and plot the results with ggplot.
The python equivalent would be a hypothetical meta-package that tightly integrates pandas, seaborn, and sklearn.
I have some concerns about the direction that certain packages within tidyverse, particularly dplyr, are heading.
One issue I've noticed is the increasing frequency of warnings, such as when performing left or inner joins. These warnings can be a bit overwhelming at times.
However, it's worth noting that you can suppress these warnings by using the "multiple='all'" parameter, which is easy to fumble when you’re exploring data, why not simply "all=TRUE”? It seems like a lack of ergonomics and considering user experience.
Yeah maybe, but chaining a bunch of joins together when exploring some new data is a thing I do all the time. Then, 4-5 warnings push my results off the screen. It’s infuriating.
I avoid joins in R like a plague. Been burned. If I am working with a finite number of colummns, I just use base R match and add column by column. yes it breaks DRY but I don't get weird surprises. and I can re-run any section of the code at any time from any point without breaking things or joining the same and creating unintended duplicate columns
Take a look at the powerjoin package. It let's you specify strict rules for your joins, for example requiring unique keys in the right-side table when doing a left join. It's got some other neat features as well.
The Tidyverse is a true gem in modern programming that deals with data.
This release feels like a nice incremental improvement rather than a revolution. Adding lubridate as a core package will save me a few keystrokes. I also did not know about the conflicts package, but will certainly be using it in the future.
I'm a bit surprised that other "better" programming languages haven't done more to replicate the Tidyverse. The Tidyverse is a masterclass in metaprogramming, and has a pretty big following. While R has some warts as a language, I'm not sure I've seen anything comparable to Tidyverse in terms of scope and influence. I see Julia trying with the Dataframes ecosystem, but it's got a ways to go before that feels cohesive like Tidyverse.
If I could wave a magic wand, I'd get Hadley and the RStudio crew really excited about Julia or Racket. R does have warts, and I dream of the day when something better is as good.
> This release feels like a nice incremental improvement rather than a revolution.
Look at the big picture, not simply the delta from the prior .x.y release.
The Tidyverse feels like a product that evolves over time, rather than a traditional approach that jumps versions. Major changes sometimes come in seemingly minor releases. I feel that 2.0 simply means the evolution has gone on enough to put a retrospective marker in the sand.
For something geared more to data science and exploration, rather than production use, I’m good with the evolutionary approach.
I agree 100% that this is a good thing. I had intended this as a compliment that Tidyverse had achieved this level of maturity, but failed in my under-caffeinated state. Had Tidyverse 2.0.0. been a revolution, I'd have been worried.
This illustrates the point perfectly. Julia is attempting this and has a beachhead with Dataframes.jl. Confusingly though, Tidier.jl isn't really analogous to R's Tidyverse. It's more like one of a handful of meta-packages around Dataframes.jl.
Then there are Grammar of Graphics (ggplot was Tidyverse's first star) style plotting libraries that Julia has been building. I'm probably most excited about Algebra of Graphics (https://github.com/MakieOrg/AlgebraOfGraphics.jl/) as part of the Makie Plots ecosystem. It does still feel a bit like Julia community can't decide between following Matplotlib or R's Grid/Ggplot approach.
The seeds of a Tidyverse for Julia are there, but it'll take some time to achieve the consistency and maturity of the original Tidyverse.
I think DF.jl works remarkably well as a "fits in RAM" dataframe backend, but I think it just lacks in usability and integratedness with the wider Julia ecosystem. Or rather, the wider ecosystem isn't as mature in key data analysis areas.
In particular, as you mention, plotting is one of the evolving parts of the ecosystem. Plots.jl is fine, Makie is powerful but very DIY, and AoG is slick but unwieldy. ggplot2 is far from perfect, but it works so well due to its maturity and integration with the rest of the Tidyverse.
In my ideal world, there would be a DataFrames.jl wrapper to provide nice (not just nicer like the two DFM.jl packages) syntax, and a powerful high-level plotting package (Makie is powerful but syntax is low level, Plots is mid on both) which is heavily integrated with the wrapper package.
Admittedly, I'm not a data scientist (anymore) so I don't follow the new developments in the dataviz scene much. If something like this exists then I would love to find it.
I wonder what my ideal syntax would look like anyway. Maybe something close to Tidyverse but with symbols as column names `:col_name` for ambiguity reasons.
I'd agree with all of this. It's not so much the presence of ecosystem, it's the maturity of the ecosystem and knowing how to navigate it. Lots of competing approaches in a relatively small community makes me a little nervous.
If people are curious, DataFrames.jl has done a fair bit of work in consolidating a list of other packages that complement DataFrames.jl:
I do have to say that DataFrames.jl itself is a pretty impressive bit of work, and Bogumił Kamiński deserves no small amount of credit. If Julia ends up stealing significant mindshare from R / Python data science communities it'll be because of of this work.
Also for the Julia curious, I'll say that I find the ergonomics and sensibilities of the DataFrames.jl ecosystem much more in line with R and the Tidyverse than Python/Pandas.
Bogumil is a truly outstanding member of the community and DataFrames.jl is an impressive, versatile package.
From my perspective, however, DataFrames.jl's power is what makes it quite unergonomic for me. As an example, take the `args => transformations => result` syntax for doing pretty much anything in DataFrames. It versatile, but the lack of rank polymorphism in Julia i.e. broadcasting/mapping has to be explicit (which is usually a good thing given that type polymorphism is Julia's whole schtick) means that the transformation syntax feels cumbersome.
It's not that I want everything rowwise by default, an option provided by DataFramesMacros.jl, it's that I want things to be rank polymorphic when it makes sense. Base R got this right, hell S got this right, and so the Tidyverse inherited it and it makes the package so much more ergonomic than it would otherwise be.
I cannot overstate how impressive DataFrames.jl is, but I have to caveat this with "but I really try to avoid using it if possible". It's a shame, but I just think R's laissez-faire hackability, which in many cases results in spaghetti code, works really well in the tabular programming world where ergonomics are king and performance is easy.
> In my ideal world, there would be a DataFrames.jl wrapper to provide nice (not just nicer like the two DFM.jl packages) syntax
Have you come across [Tidier.jl](https://github.com/TidierOrg/Tidier.jl) yet? It's a relatively recent package (to my understanding), but developing at a pretty rapid pace, and tries to be more ergonomic with its syntax.
I don't know if it's impressive or worrying regarding how much Posit/RStudio as a company have completely taken over a programming language.
I have colleagues who write R code all day, and they don't know the difference between RStudio the IDE and R the language. They can't perform basic tasks such as defining a function, or adding a column without the tidyverse.
I see that same thing all the time. RStudio = R, and they have no concept of the language without the IDE. Which is fine, I guess? They are not professional programmers, but people with a job to do.
It is just frustrating to me when they RShiny all the things, and I have to pickup the pieces because the spaghetti has collapsed under its own weight.
I see it a little bit more like a meritocracy in that space that RStudio has largely won. Microsoft has certainly made an attempt to become the face of R. Remember when they purchased Revolution Analytics and started baking it into everything? Or how about Oracle making a play on Graalvm with "FastR"?
RStudio has somewhat indefatigably looked for pain points that R users had back in the olden days. And yes, I remember R back when lattice plots were the hottest game in town. Not only has the Tidyverse made R infinitely more pleasant to work with, the community around Tidyverse has made the R community infinitely less insufferable for newcomers. The obtuse responses to beginners on the old R mailing lists were legendary.
My preferred R IDE is emacs, and I'm not upset with what RStudio has done. When I interact with devtools, I don't feel that I'm wanting for functionality. This is in spite of the fact that RStudio has heavily integrated devtools functionality into their IDE, and could arguably have made devtools less easy to work with using generic tooling.
I understand the mediocrity of many RShiny applications in terms of how they are written. But, another way of looking at all those dashboards and interactive applications, is that they are AMAZING prototypes written by scientists, grad students, journalists, etc., who would have never been able to build them some other way. If we forced everyone to write these apps in Rust, of course the average quality would be higher, but we'd also have way fewer of them. Democratization of technology means that its users will be more... medium.
>...who would have never been able to build them some other way...
It has definitely become the best available MVP generator. I just wish I liked R more. Python still have nothing even close to how easily a novice can spin up a Shiny application.
I think that package is still pretty early days, but does show promise. I was speaking less to the code and more to the ecosystem.
If I tell someone they should use Streamlit/Dash/Shiny for Python, how do they get started? The beginner onboarding experience is horrible for someone without an existing programming background. Download Python, maybe instruct them to setup a "virtual environment", use this "pip" thing to install these packages, point them at a way to edit code, and try to give them a way to launch it. God help them if want to share their work. Huge amount of existing resources assume Linux by default or that people have any idea how to use Docker.
While I hate...basically everything about the R ecosystem, the beginner experience is leagues better. Download RStudio. It will setup R, install packages, and has a big button that will launch the Shiny application. Straightforward guidance on how to get it hosted on an existing provider.
I write in R for much of my work days, and perhaps because I learned when the tidyverse was in its infancy (or before) I don't use much at all of its packages besides ggplot2. With the base r pipe operator I use zero dplyr and find most of its functionality has long existed in base R with functions like transform(). lubridate has a few good convenience functions (that ultimately I could do without), and I do like plotting in ggplot2 - in fact the latter is the only place I reckon there has been too much of a takeover because most plotting development effort I imagine is based on the grid and ggplot2 framework now. I do like tidyr::pivot_longer(), although it wouldn't change much if I had to use the less intuitive base R versions for dataframe manipulation.
Your colleagues are probably a symptom of how low the barrier to entry has been made by the tidyverse moreso than indicative of a total capture of the language.
I like the tidyverse in general, but I have to admit I've never gotten the hang of tibbles. I use dataframes because I like row names and pretty much every non-tidyverse package uses dataframes and row names too. I really don't understand why row names (a basic feature of R) are viewed with such disdain by the ecosystem and can't be set using tidyverse functions like read_tsv. I generally end up converting them to dataframes after using them.
Most R programmers are scientists or statisticians though. They aren't programmers in a professional sense. They are using a tool to get things done for their real work, and if RStudio helps them, so much the better.
I don't think it really matters. Tidyverse is fine for datasets of size 2 * (1...20?) . Data table buys you maybe 21..23? Then you're off to spark for 24+ anyway. So you'd only be covering a pretty narrow slice of the space.
I have been using R almost daily for years now and its no hyperbole when I say that if ggplot2/dplyr wasn't a thing I would have never bothered with R.
Adding to that, it is what makes me miss R when using other languages. It just isn't very much fun to me to do discovery and building in python, but ultimately where things get operationalized.
I completely agree with you. I think about what I want and R code comes out of my fingers. By comparison, pandas requires more thought to do intermediate-to-advanced things.
"There are multiple OOP systems to choose from. In this book, I’ll focus on the three that I believe are most important: S3, R6, and S4. S3 and S4 are provided by base R. R6 is provided by the R6 package, and is similar to the Reference Classes, or RC for short, from base R.
"There is disagreement about the relative importance of the OOP systems. I think S3 is most important, followed by R6, then S4. Others believe that S4 is most important, followed by RC, and that S3 should be avoided. This means that different R communities use different systems."
My 2¢ on the tidyverse is that it is a bit misleading to call it "a collection of packages". I think it would be more accurate to call it a "framework". Though I suppose "opinionated collection of packages" is perhaps a fuzzy synonym for framework.
The difference between "collection of packages" and "framework" is that "collection of packages" implies interoperable via API conventions, but otherwise standalone packages, which could easily be used in isolation alongside base R.
Whereas "framework" generally means tightly coupled components, in a "do things my way or the highway" fashion.
Technically speaking, it's not "impossible" to mix tidyverse with base R, but in general if you wanted to mix and match tidy vs base R syntax, you'd find yourself working "against" the tidy api rather than alongside it most of the time.
R is not my main language, but every time I come across a task and I need to make a decision between R vs tidyR, the two feel like completely different languages.
I also have some beef against some of the decisions/conventions used (e.g. abusing metaprogramming to pass variables and use the variable's label as a value, completely ignoring the variable's actual value, instead of passing it as a string or as an actual variable ... seriously confusing when you first come across it).
> Whereas "framework" generally means tightly coupled components, in a "do things my way or the highway" fashion.
Using that definition how would lubridate be part of the tidyverse "framework"?
> Technically speaking, it's not "impossible" to mix tidyverse with base R, but in general if you wanted to mix and match tidy vs base R syntax, you'd find yourself working "against" the tidy api rather than alongside it most of the time.
I don't know, wrapping readxl::read_excel with a as.data.frame or as.data.table call works beautifully for me.
I think he's almost certainly speaking of the %>% group_by convention that is basically demanded of within the tidyworld.
I load magrittr and ggplot and toss the rest out. Right to left reading order is all I really want from that world. Data tables are such a superior convention to tibbles that it's not even funny (I need a tibbles like I need another hole in the head).
I was (wrongly) guessing performance. Tibbles are ok for smaller data sets where data transformations don’t need to be in place and there’s no risk of running out of RAM. Data tables perform better, at the cost of what I think is a less ergonomic interface. See benchmarks here (compare data.table with dplyr):
But base r already has split() which does the same thing, turns a dataframe into a list of dataframe divided by values of a column. With the base r pipe operator and base functions like subset, aggregate, transform, split, you can basically replicate that convention while loading exactly no tidyverse packages. You can also use group_by(data, column) without a pipe if wanting to smuggle in a specific dplyr function.
I think this is a great framing. The tidy verse framework is straight up antagonistic towards base R and any non Hadleyverse packages wwith minor exception.
The other glaring rhino in the room is the relationship between RStudio and the Hadleyverse.
I find the relationship and degree of influence the tidy verse and Hadleyite way of doing things to be a dark cloud over the R community.
For the most part that community is still around and still uses quite a bit of base R, you will see them overrepresented answering curly questions in help forums
Eh, framework is a bit strong (if we're thinking about something like Ruby on Rails). You can easily mix dplyr and lubricate with base R things like for loops, subscripted assignments, low-level functions like grep, etc. I do it all the time.
I agree that nonstandard evaluation can be tricky, but the nice notation you get is worth it IMHO, and at this point you can switch to standard evaluation in a number of convenient ways. My favorite way is to just use base R temporarily, since outside of modeling functions it mostly works by standard eval.
And I think that's good advice in general. The best way to use the Tidyverse is to use it for what it's good at. And for things it's not so good at, use something else for a bit before coming back to it.
I find it entirely possible and it's my normal work flow to mix a little tidyverse in with base R without going against the grain, the worst it gets its probably converting data from long to wide format now and again but the functionality to do so is intuitive and simple. Bigger differences between graphics but there wouldn't be a need to mix ggplot grid graphics and base r graphics in the same plot (although panels in the same multi figure plot for example is trivial).
> You’ll notice one other small change to the tidyverse message: we now advertise the conflicted package.
Not a fan of the nature in which tidyverse promotes itself. They used to be more aggressive - i.e. advertising their books in package load messages [1]. But these new changes still feel like an ad.
To me, as an outsider, tidyverse seems to mainly be about "dplyr". The main thing holding all of it together otherwise is just the fact that majority of their functions return a "tibble" instead of a data.frame, forcing you to load all the jungle of the rest of their stuff whenever you use one of their packages.
Has anybody switched from R (tidyverse and ggplot2) to python and found they can be equally or more productive in tabular data wrangling, data exploration and visualisations?
Still using R because I find python cannot do exploratory data analysis of visualisation as well as R.
What R has going for it is ggplot2, which is heavily inspired by a formally validated "Grammar of Graphics" (see the L Wilkinson book). One of the main reasons why the tidyverse works so well is because it tends to follow the organizing principle of "tidy data" (see the H Wickham paper) with some combinator interoperability between packages. R itself is pretty insane ("bad") in terms of language design, even though it has some neat tricks and features.
Python is a much saner language than R (I am not talking of packages here), but it's still super clunky by design (less so than R). It can most certainly do all the things that R does, although its plotting libraries are either lacking in their design (matplotlib) or maturity (the other GoG-inspired libraries).
When you continue on your journey to learn more languages, at some point you realize that when you have a well-designed, simple and consistent language, you really don't need very many packages to get the job done fast. I had this a-ha moment with Clojure (although I find working with the JVM annoying).
Haha I wouldn’t say it like that, but I’m pretty sure we feel the same way deep down.
Regardless of how true your comment might be, this can seem a bit hostile or condescending to people who don’t have a “cleaner” reference point for language design.
Or, in the “fog of war” of competing alternatives, it may even appear unreasonable to experienced programmers: we’re so used to thinking that everything is a tradeoff that over time we can forget that some things really can be “just better” than other things. Most of the time when we hear such claims, it comes from a place of subjective preference or over-generalization, so our brains pattern match strongly against that.
And at the end of the day, as a practical matter, the best camera is the one you carry with you, and the best tool is the one you can use.
No, I’m in the same boat. If I’m only interested in data wrangling, visualization, or analysis, R blows Python out of the water. With all the time and effort that have gone into Python data libraries, it is basically on par with base R. With close to a decade of tidyverse development, it really isn’t even close.
But the second I’m doing anything outside of those specific tasks above, Python outclasses R to an equal degree.
I have used much more Python than R in my career and I still find I'm able to do most analytics work about 2x faster in dplyr/tidyverse + data.table than in the pandas ecosystem. Polars is better but still not comparable to dplyr.
I am an unapologetic Hadley fanboy. He is a genius. Never in my working life has anyone made so many good decisions that directly benefitted me. My R productivity is so incredibly much higher than it would have been without him.
The worst thing I can say about tidyverse is that I find the stack traces increasingly gnarly. The information you need to debug is there, there's just a lot of filler. I have some faith that they know and they're working on it.
Same here! Hadley is incredible. He can think, code, and write amazingly well. Lots of people can do two of the three. It's rare to find someone who does all three.
78 comments
[ 2.8 ms ] story [ 151 ms ] threadThere is "the tidyverse" which is a collection of packages like dplyr and tidyr for data manipulation and ggplot2 for visualization.
Then there's "the tidyverse package" which allows you to import them all at once with "library(tidyverse)". You don't have to use the tidyverse package, you can load individual libraries as you need them.
Also worth noting that one of the common organizing principles between all packages in the tidyverse is the concept of "tidy data" (see specifically H Wickham's paper), and the use of intercompatible combinators.
Individual packages can be used in isolation, if that’s what you need. However, they complement each other nicely, in that they share data representations and similar conventions while handling different facets of data analysis. Thus, people often import the whole thing in one shot as the tidyverse and go from there. For example, you might read the data in with readr, clean it up with tidyr or dplyr, analyze it with tidymodels, and plot the results with ggplot.
The python equivalent would be a hypothetical meta-package that tightly integrates pandas, seaborn, and sklearn.
I don't think that lubridate has any relationship with anything else, for example, apart from being created by the same people.
One issue I've noticed is the increasing frequency of warnings, such as when performing left or inner joins. These warnings can be a bit overwhelming at times.
However, it's worth noting that you can suppress these warnings by using the "multiple='all'" parameter, which is easy to fumble when you’re exploring data, why not simply "all=TRUE”? It seems like a lack of ergonomics and considering user experience.
This release feels like a nice incremental improvement rather than a revolution. Adding lubridate as a core package will save me a few keystrokes. I also did not know about the conflicts package, but will certainly be using it in the future.
I'm a bit surprised that other "better" programming languages haven't done more to replicate the Tidyverse. The Tidyverse is a masterclass in metaprogramming, and has a pretty big following. While R has some warts as a language, I'm not sure I've seen anything comparable to Tidyverse in terms of scope and influence. I see Julia trying with the Dataframes ecosystem, but it's got a ways to go before that feels cohesive like Tidyverse.
If I could wave a magic wand, I'd get Hadley and the RStudio crew really excited about Julia or Racket. R does have warts, and I dream of the day when something better is as good.
Look at the big picture, not simply the delta from the prior .x.y release.
The Tidyverse feels like a product that evolves over time, rather than a traditional approach that jumps versions. Major changes sometimes come in seemingly minor releases. I feel that 2.0 simply means the evolution has gone on enough to put a retrospective marker in the sand.
For something geared more to data science and exploration, rather than production use, I’m good with the evolutionary approach.
https://github.com/TidierOrg/Tidier.jl
Then there are Grammar of Graphics (ggplot was Tidyverse's first star) style plotting libraries that Julia has been building. I'm probably most excited about Algebra of Graphics (https://github.com/MakieOrg/AlgebraOfGraphics.jl/) as part of the Makie Plots ecosystem. It does still feel a bit like Julia community can't decide between following Matplotlib or R's Grid/Ggplot approach.
The seeds of a Tidyverse for Julia are there, but it'll take some time to achieve the consistency and maturity of the original Tidyverse.
In particular, as you mention, plotting is one of the evolving parts of the ecosystem. Plots.jl is fine, Makie is powerful but very DIY, and AoG is slick but unwieldy. ggplot2 is far from perfect, but it works so well due to its maturity and integration with the rest of the Tidyverse.
In my ideal world, there would be a DataFrames.jl wrapper to provide nice (not just nicer like the two DFM.jl packages) syntax, and a powerful high-level plotting package (Makie is powerful but syntax is low level, Plots is mid on both) which is heavily integrated with the wrapper package.
Admittedly, I'm not a data scientist (anymore) so I don't follow the new developments in the dataviz scene much. If something like this exists then I would love to find it.
I wonder what my ideal syntax would look like anyway. Maybe something close to Tidyverse but with symbols as column names `:col_name` for ambiguity reasons.
If people are curious, DataFrames.jl has done a fair bit of work in consolidating a list of other packages that complement DataFrames.jl:
https://dataframes.juliadata.org/latest/#DataFrames.jl-and-t...
I do have to say that DataFrames.jl itself is a pretty impressive bit of work, and Bogumił Kamiński deserves no small amount of credit. If Julia ends up stealing significant mindshare from R / Python data science communities it'll be because of of this work.
Also for the Julia curious, I'll say that I find the ergonomics and sensibilities of the DataFrames.jl ecosystem much more in line with R and the Tidyverse than Python/Pandas.
From my perspective, however, DataFrames.jl's power is what makes it quite unergonomic for me. As an example, take the `args => transformations => result` syntax for doing pretty much anything in DataFrames. It versatile, but the lack of rank polymorphism in Julia i.e. broadcasting/mapping has to be explicit (which is usually a good thing given that type polymorphism is Julia's whole schtick) means that the transformation syntax feels cumbersome.
It's not that I want everything rowwise by default, an option provided by DataFramesMacros.jl, it's that I want things to be rank polymorphic when it makes sense. Base R got this right, hell S got this right, and so the Tidyverse inherited it and it makes the package so much more ergonomic than it would otherwise be.
I cannot overstate how impressive DataFrames.jl is, but I have to caveat this with "but I really try to avoid using it if possible". It's a shame, but I just think R's laissez-faire hackability, which in many cases results in spaghetti code, works really well in the tabular programming world where ergonomics are king and performance is easy.
Have you come across [Tidier.jl](https://github.com/TidierOrg/Tidier.jl) yet? It's a relatively recent package (to my understanding), but developing at a pretty rapid pace, and tries to be more ergonomic with its syntax.
Python has a very weak metaprogramming story so it is hard if not impossible to replicate something like Tidyverse in that.
If Ruby had succeeded in data science instead of Python it would almost certainly be with Tidyverse-like abstractions.
I have colleagues who write R code all day, and they don't know the difference between RStudio the IDE and R the language. They can't perform basic tasks such as defining a function, or adding a column without the tidyverse.
It is just frustrating to me when they RShiny all the things, and I have to pickup the pieces because the spaghetti has collapsed under its own weight.
RStudio has somewhat indefatigably looked for pain points that R users had back in the olden days. And yes, I remember R back when lattice plots were the hottest game in town. Not only has the Tidyverse made R infinitely more pleasant to work with, the community around Tidyverse has made the R community infinitely less insufferable for newcomers. The obtuse responses to beginners on the old R mailing lists were legendary.
My preferred R IDE is emacs, and I'm not upset with what RStudio has done. When I interact with devtools, I don't feel that I'm wanting for functionality. This is in spite of the fact that RStudio has heavily integrated devtools functionality into their IDE, and could arguably have made devtools less easy to work with using generic tooling.
I understand the mediocrity of many RShiny applications in terms of how they are written. But, another way of looking at all those dashboards and interactive applications, is that they are AMAZING prototypes written by scientists, grad students, journalists, etc., who would have never been able to build them some other way. If we forced everyone to write these apps in Rust, of course the average quality would be higher, but we'd also have way fewer of them. Democratization of technology means that its users will be more... medium.
It has definitely become the best available MVP generator. I just wish I liked R more. Python still have nothing even close to how easily a novice can spin up a Shiny application.
If I tell someone they should use Streamlit/Dash/Shiny for Python, how do they get started? The beginner onboarding experience is horrible for someone without an existing programming background. Download Python, maybe instruct them to setup a "virtual environment", use this "pip" thing to install these packages, point them at a way to edit code, and try to give them a way to launch it. God help them if want to share their work. Huge amount of existing resources assume Linux by default or that people have any idea how to use Docker.
While I hate...basically everything about the R ecosystem, the beginner experience is leagues better. Download RStudio. It will setup R, install packages, and has a big button that will launch the Shiny application. Straightforward guidance on how to get it hosted on an existing provider.
Your colleagues are probably a symptom of how low the barrier to entry has been made by the tidyverse moreso than indicative of a total capture of the language.
https://dtplyr.tidyverse.org/
maybe this?
repo: https://github.com/RConsortium/OOP-WG
talk: https://www.youtube.com/watch?v=P3FxCvSueag&t=258s
"There are multiple OOP systems to choose from. In this book, I’ll focus on the three that I believe are most important: S3, R6, and S4. S3 and S4 are provided by base R. R6 is provided by the R6 package, and is similar to the Reference Classes, or RC for short, from base R.
"There is disagreement about the relative importance of the OOP systems. I think S3 is most important, followed by R6, then S4. Others believe that S4 is most important, followed by RC, and that S3 should be avoided. This means that different R communities use different systems."
https://rconsortium.github.io/OOP-WG/
"The S7 package is a new OOP system designed to be a successor to S3 and S4."
(https://xkcd.com/927/)
S7 does look nice though, from the README overview at least.
The difference between "collection of packages" and "framework" is that "collection of packages" implies interoperable via API conventions, but otherwise standalone packages, which could easily be used in isolation alongside base R.
Whereas "framework" generally means tightly coupled components, in a "do things my way or the highway" fashion.
Technically speaking, it's not "impossible" to mix tidyverse with base R, but in general if you wanted to mix and match tidy vs base R syntax, you'd find yourself working "against" the tidy api rather than alongside it most of the time.
R is not my main language, but every time I come across a task and I need to make a decision between R vs tidyR, the two feel like completely different languages.
I also have some beef against some of the decisions/conventions used (e.g. abusing metaprogramming to pass variables and use the variable's label as a value, completely ignoring the variable's actual value, instead of passing it as a string or as an actual variable ... seriously confusing when you first come across it).
Using that definition how would lubridate be part of the tidyverse "framework"?
> Technically speaking, it's not "impossible" to mix tidyverse with base R, but in general if you wanted to mix and match tidy vs base R syntax, you'd find yourself working "against" the tidy api rather than alongside it most of the time.
I don't know, wrapping readxl::read_excel with a as.data.frame or as.data.table call works beautifully for me.
I load magrittr and ggplot and toss the rest out. Right to left reading order is all I really want from that world. Data tables are such a superior convention to tibbles that it's not even funny (I need a tibbles like I need another hole in the head).
https://h2oai.github.io/db-benchmark/
The other glaring rhino in the room is the relationship between RStudio and the Hadleyverse.
I find the relationship and degree of influence the tidy verse and Hadleyite way of doing things to be a dark cloud over the R community.
I believe that Hadley's work is the driving reason R remains a competitive alternative to Python for data analysis.
I agree that nonstandard evaluation can be tricky, but the nice notation you get is worth it IMHO, and at this point you can switch to standard evaluation in a number of convenient ways. My favorite way is to just use base R temporarily, since outside of modeling functions it mostly works by standard eval.
And I think that's good advice in general. The best way to use the Tidyverse is to use it for what it's good at. And for things it's not so good at, use something else for a bit before coming back to it.
The pipe operator as well as treating csv tables as SQL was intuitive for me as a software engineer.
Not a fan of the nature in which tidyverse promotes itself. They used to be more aggressive - i.e. advertising their books in package load messages [1]. But these new changes still feel like an ad.
To me, as an outsider, tidyverse seems to mainly be about "dplyr". The main thing holding all of it together otherwise is just the fact that majority of their functions return a "tibble" instead of a data.frame, forcing you to load all the jungle of the rest of their stuff whenever you use one of their packages.
[1]: https://win-vector.com/2019/08/30/it-is-time-for-cran-to-ban...
Still using R because I find python cannot do exploratory data analysis of visualisation as well as R.
However, there's a ggplot clone called plotnine in Python which seems to be reasonably comprehensive.
Python is a much saner language than R (I am not talking of packages here), but it's still super clunky by design (less so than R). It can most certainly do all the things that R does, although its plotting libraries are either lacking in their design (matplotlib) or maturity (the other GoG-inspired libraries).
When you continue on your journey to learn more languages, at some point you realize that when you have a well-designed, simple and consistent language, you really don't need very many packages to get the job done fast. I had this a-ha moment with Clojure (although I find working with the JVM annoying).
That's like saying that cat turds smell much better than dog turds.
Maybe it's technically true, but let's not split hairs here.
Regardless of how true your comment might be, this can seem a bit hostile or condescending to people who don’t have a “cleaner” reference point for language design.
Or, in the “fog of war” of competing alternatives, it may even appear unreasonable to experienced programmers: we’re so used to thinking that everything is a tradeoff that over time we can forget that some things really can be “just better” than other things. Most of the time when we hear such claims, it comes from a place of subjective preference or over-generalization, so our brains pattern match strongly against that.
And at the end of the day, as a practical matter, the best camera is the one you carry with you, and the best tool is the one you can use.
But the second I’m doing anything outside of those specific tasks above, Python outclasses R to an equal degree.
The worst thing I can say about tidyverse is that I find the stack traces increasingly gnarly. The information you need to debug is there, there's just a lot of filler. I have some faith that they know and they're working on it.