If it means we now get a term that, at least for a couple of years, filters out all the garbage roles recruiters throw at me then I'm on board with adopting this terminology.
Hah! At my company a decent proportion of engineers spend their lives scrambling to productionalize and operate the Lovecraftian concoctions of R and Python that our data scientists cook up on their laptops.
Even the Stanford NLP Java implementations don’t always match the code they’re meant to be a translation of. I think R is a worse offender than Python.
Why do you compare point to point, it is meaningless? The result for 0.1 + 0.2 shows 0.3. It doesn't show you wrong result. If you print it prints 0.3, nothing else.
Why do you compare point to point? I have never once in my Statistics education compared point to point. You always need to see the probability of the result if it is within 2 points.
But if you want to compare use all.equal(0.1 + 0.2, 0.3)
I thought that's what you mean by "equal". It didn't appear to me that you were merely talking about the default formatting of numbers printed on the REPL, which is utterly inconsequential to developing and porting applications.
This entire subthread is under a comment talking about "productionalizing" code developed in R. If you want to make an argument that R should not be used to make products, but only as an interactive notebook and to make nice plots, maybe make that instead of just mentioning small UI details.
Why do you compare point to point? I have never once in my Statistics education compared point to point. You always need to see the probability of the result if it is within 2 points.
But if you want to compare use all.equal(0.1 + 0.2, 0.3)
In R 0.1 + 0.2 returns 0.3. In Python 0.1 + 0.2 returns 0.30000000000000004
Why do you compare point to point? I have never once in my Statistics education compared point to point. You always need to see the probability of the result if it is within 2 points.
But if you want to compare use all.equal(0.1 + 0.2, 0.3)
Not only is this technically wrong, but Python has a decimal module for performing these kinds of calculations. It uses floats/doubles for the native float type which produces exactly the type of results you see above.
Where I work, the data scientists are more educated and experienced on containerization, CI tooling, unit testing, profiling tools, web service prototyping including API validation tools, caching layers, queues, GPU systems programming, etc. etc.
We are constantly thwarted by infrastructure teams that use superficial policy basically so they can whine and complain that they don’t want to have to provide support for the extremely heavily researched and tested implementation we create.
They don’t care that different technologies, database systems, whatever are chosen to solve customer use cases and that growing our business means supporting “Lovecraftian concoctions of R and Python” — they just don’t want to do their jobs (which indeed requires providing infrastructure support for crazy screwball data services that repeatedly break all the assumptions)...
If the data scientists know more about CI, unit testing, profiling, and caching than the engineers then they are better engineers and I'd wonder a bit about their math/stats chops and whether their role was just re-branded "data scientist" to keep up with trends.
It’s quite common to start out with a PhD / masters in math / stats, with deep specialization in fields like NLP, computer vision, MCMC sampling, and then to become an experienced expert in GPU computing, containerization, web service layers, etc., while working on implementations of ML models.
This was true for me anyway. The main thing I do is deep learning for computer vision and image search, but I think it’s fair to say I have significant experience with Docker, GPU architectures, various CI tooling, linux system programming, deep internals of CPython, internals of MySQL and Postgres, lots of frustrating performance tradeoffs with py4j in the pyspark world, as well as all the usual crap with pandas, sklearn, data visualization tools, and a lot more.
I’d say almost all data scientists I’ve worked with are just like me, just with maybe different specialization areas, except possibly for very young data scientists right out of undergrad.
If you don't mind me asking, where do you work at? (or even a family of companies that have this type of roles)
What you describe sounds like where I want to be, and it would help to where I could go looking. (Deep Learning / Vision specialization with a math focused CS background but want to learn more core software skills going ahead)
I work in a fairly mature startup that’s been around for over 10 years. It began primarily with an app, but shifted focus to other business areas. The image processing products are mostly related to offering information retrieval and search services for app users that have curated personal image collections.
I would say your description is accurate but only accidentally. The reason we have to learn more core engineering skills is that infrastructure will not take responsibility of bringing our solutions to production, and seek to limit the tools in our toolbox with policy.
It’s not fun when your everyday life is constant impedance mismatch against tools that infra will allow you to use. You can constantly see better / faster / safer / cheaper ways to solve problems, which have no downsides at all relative to the bad, slow, insecure, expensive ways infra currently makes everyone solve problems, but you just feel constantly sad that you are superficially prevented from the autonomy to select the efficient solution accordung to your creativity and skill.
I feel like you are roughly describing research programmers versus system administrators or operators in academic computing environments.
I think a big difference between research programmers and production/ops people is that as researchers we often chase a transient goal. Build some complex and horrible integration to compute a result or put something in a paper. We used to call these Rube Goldberg machines rather than Lovecraftian horrors, but we mean the same thing. Something that belongs on a movie set, with some Jacobs ladders arcing in the background. In some circles, it is called the heroic demo.
In the past, we might substitute other fads for your CI tooling or API validation tools. I remember when some research programmers were all-in on enterprise junk like J2EE/managed code, SOAP/WSDL, and other stovepipe tooling. There is a lot of cargo culting of such tools. When you have furnished your lab with rapid prototyping tools and focused on crazy integration stunts, you are almost always deluding yourself to think these tools are also giving you "production" system qualities.
Building something at the hairy edge of possibility is inherently about experimentation and risk-taking. Building reliable, production operations is inherently about conservative design and risk-mitigation. There seems to be a new cargo cult of devops which believes you somehow mash these together and the conflict disappears. You don't have to have to map the negotiation onto two teams with opposing objectives, but the negotiation has to live somewhere.
Magically erasing the negotiation just means that you have chosen to default on the optimization task and jettison concern for at least one of functionality, cost, or risk. Startups commonly do this because the VC funding has mitigated the risk elsewhere: you can fail because they've also funded your competitor who may succeed...
I’m not referring to transient research prototypes, but to robust long-lived systems needed for experimentation and reproducible results tracking, and services that are directly customer facing.
We are often required to create new services and functionality because it is how our company can grow, and we have to have ease of access to experimental working space, with freedom to do things like custom compilations of ML frameworks, using programming languages that haven’t been widely used in the company yet to gain access to an important library, define complex assumption-breaking deployment constraints relative to GPU runtimes or containerized notebook servers, etc.
I think people who see how these things grow out of prototypes and wrongly conclude it was designed with transient concerns and thus isn’t robust in some way, they are rushing to judgment and discounting the fact that that ML expert who also wrote the web service layer and who also wrote the Jenkinsfile and who also wrote the container definition abd who also knows how to tune indices in the database, etc., really made their choices for serious, pragmatic engineering reasons that solve the business problem efficiently, and that they already anticipated and accounted for the shallow tradeoffs and caveats that IT will use as potshots to try to circumvent the responsibility to help maintain it.
We may be talking past each other. I am on the research side of academic computing/informatics and have faced these struggles my whole career, encountering some very reluctant IT divisions.
We have had to bite the bullet and use colo facilities to self-host internet-facing deployments that the overhead-funded IT groups would not touch with a ten foot pole. From these experiences, I also acquired a more nuanced perspective on the IT division perspective and constraints, and how they derive from overall organizational policy and economics. We also had funny situations where we tried to help other PIs benefit from our new-found independence, and immediately regretted it. They did not understand what self-hosting means. I think anybody trying to toss integrations over the fence to an ops team needs to have an extended tour of duty trying to operate their own solutions in production WITHOUT assistance before they form bold opinions about operations constraints.
When there are strong time-to-market constraints (which includes publishing papers in academics), you are forced to find solution points that are different than if you are planning to run something for long periods at low overhead and low accumulative risk. These solution points also have to take into account the staffing and resources for that ongoing production.
Those things like bleeding edge libraries and assumption-breaking deployment constraints are the headache for ongoing operations and maintenance. It's not enough to have an existence proof that some complex integration can be built and passes its tests. You need a plan for how all the components will be maintained, patched, and upgraded. You need contingency planning when some of those bleeding edge components are going to become deprecated. You need to consider what staff capabilities are assigned to do that maintenance work or what will happen when the institutional knowledge used to form the original integration is not on-call to reintegrate it in the face of unexpected events.
> “I think anybody trying to toss integrations over the fence to an ops team needs to have an extended tour of duty trying to operate their own solutions in production WITHOUT assistance before they form bold opinions about operations constraints.”
I think this is one of the worst possible attitudes to have. It’s a petty way to feel, desiring some type of “I’ve seen some shit” tough guy credential more than supporting the stuff needed to actually solve business problems.
If you hire people whose value add to your company is inventing completely new things, including deployment, ops, scaling, etc., that goes along with that, then it is the job of infrastructure on the other side of that fence to happily and eagerly accept whatever is tossed over the fence, to understand why developer teams made the choices they made, and to take an attitude of supporting as much as possible.
> “You need a plan for how all the components will be maintained, patched, and upgraded. You need contingency planning when some of those bleeding edge components are going to become deprecated. You need to consider what staff capabilities are assigned to do that maintenance work or what will happen when the institutional knowledge used to form the original integration is not on-call to reintegrate it in the face of unexpected events.”
Yes, of course. But all this is already what dev teams are doing. Ops / infra is not taking a hare-brained plan and adding these robustness aspects into it. Not at all. Instead they take plans from application teams and try to use policy to minimize their own maintenance burden, even when that optimization is antithetical to what the company requires at a more fundamental level.
A lot of companies languish and die because of sociological dysfunction in the policy interface between dev teams and infrastructure. The more that infrastructure has political control of that interface, the closer to death is that company.
It’s like a body that is disallowed from generating white blood cells in response to a new immune challenge. Even if the bleeding edge integrations are really hard, the alternative world where you slow them down with policy is death and attrition.
It's scary how much this comment applies to my current job. Literally just spent today discussing with entire engineering org how to steer away from this behavior.
As a data scientists I view other data scientists who need engineers to productionize their code with contempt. Perhaps (probably?) their data pipelines and systems are sufficiently more complicated than mine, but I'd feel embarrassed if I couldn't write production Python code
They are different skills. Not saying that it's hard to learn both, but there are standardized career paths that will lead you to be good at the modeling / techniques side of "data science" without learning much about software engineering. For example, studying math in undergrad. And there are certainly lots of people capable of productionizing messy R scripts without fully understanding the statistical ideas behind them. So I think, as a team leader, you are restricting yourself to some degree if you only hire people who can do both.
We don’t allow (new) Python code in production, which has something to do with it. Feature engineering is also a very different game with streaming systems and online datastores than with CSVs.
Just what the data science needed more of: gatekeeping.
Data science is an extremely broad, vague buzzword encompassing a variety of jobs and skillsets, most of which have existed for decades under different names. You do work that involves putting models into production in Python, congrats. The insistence that all data scientists must also do so is silly, especially considering that there are surely many skills used by many data scientists that you are incapable of.
As a data analyst who is pushing towards data science, do you have any resources or advice on how to steer away from these "Lovecraftian concoctions" you're speaking of?
Well, say that to an actual computer scientist. They will argue without data science/informatics there wouldn't be any computers in the first place. Steven Levy's Hackers is a showcase that it's true. Coders/Programmers/Hackers emerged from that open computer science lab culture. And so on.
> Consider that the “data scientist” uses compact languages such as SQL and R. An entire interesting application may fit in one file. There is an input, some processing, and an output answer.
The argument this post is making is reductive. Yes, sometimes data science is simple. Sometimes it isn't, and that's when you really need someone with the appropriate skillset.
systems programming=irrelevant bloat and abstraction
while
data reduction=definite purpose and utility
People writing python notebooks to do data analysis are probably fairly comparable to the scientific computing programmers of the past, but I feel like this picture tends to dismiss the computer science side of systems programming:things like GUIs, network code, processes and virtual memory, all the architectural aspects of computing.
One might prefer APL or Forth for writing one-page programs, and it's probably true that systems now are bloated relative to what they could be. Still, there is much of interest going on in a typical operating system, compiler or video game, while a typical data analysis notebook is IMO fairly dull and even basic, from a software angle.
Yeah, the author's take is myopic. What they call bloat, people from the 70s would call wondrous: ubiquitous networking with and without wires, beautiful graphical interfaces, encryption everywhere (and expanding), far more open systems than proprietary re-engineered ones, the list goes on and on.
It's fair to mention that he is well known, though in fact I'm one of the old guard that remembers when he had a higher profile.
But, as with a new Paul Graham essay, surely we can critique the blog post on its merits instead of falling back on an assessment based on some kind of appeal to authority/"expertise by association". Philip Greenspan doesn't need to be treated with kid gloves as if he was the pope.
John Ousterhout made comments that touch on some similar (though not identical) distinctions in programming practices. That was years ago, and he was then a much more credible figure in software than Greenspun. All the same, his essay was heavily criticised. That's what serious intellectual discussion should involve.
I wonder why "data engineer" isn't one of the suggested terms. Scientists do not really program science, nor do programmers research programs, as their respective fields of expertise.
To be nitpicky, in the US, engineer means you graduated from an ABET accredited program in something like: Chemical engineering, mechanical engineering, civil engineering, electrical engineering, industrial engineering, computer engineering....etc.
That is not to say programming isn't a difficult job that requires a lot of analytical and creative thinking similar to an engineer. The difference is in getting a degree in something that has 4-5 years of calculus based math, physics...etc classes. There is also a rigorous 8 hour test to get a license after 4 years on the job.
I guess the broad term of building something and doing analysis fits here, but I don't see any Data Engineers in practice. What I see are Data Scientists and Data Analysts. Of course I'm arguing over semantics here, but it is important to get the distinction correct.
> To be nitpicky, in the US, engineer means you graduated from an ABET accredited program in something like: Chemical engineering, mechanical engineering, civil engineering, electrical engineering, industrial engineering, computer engineering....etc.
Do you happen to have a reference for this? At first glance, it seems to be incorrect rather than nitpicky.
Anecdotally, I know plenty of people who do not have ABET accredited degrees and have "engineer" in their title in the US.
Unless you have a professional engineers license, you can't testify in court as an engineer. To get a professional engineer's license, you must graduate from an ABET accredited program.
Since we're being nitpicky, neither of those claims are true.
>To get a professional engineer's license, you must graduate from an ABET accredited program.
In some states you can take the FE and PE exam with a related non-ABET accredited degree and work experience based, and in some states you can take the exams with no degree based on experience only. For example, NY let's you substitute work experience for a degree. There's a table where you can see how much experience you need based on your degree (or lack of one).
>Unless you have a professional engineers license, you can't testify in court as an engineer
Courts determine what credentials qualify someone to testify as an expert witness, not state licensing boards.
In some states, testifying in court as an engineer does qualify as practicing engineering without a license, and the state licensing board could fine you after the fact. However, in other states testifying in court doesn’t necessarily qualify as practicing engineering.
Someone can certainly call themselves an "engineer", but it doesn't have the same consequences as someone who has received an ABET accredited engineering degree... for example, if you've graduated from an ABET accredited engineering degree, you are able to become a licensed and registered professional engineer - https://en.wikipedia.org/wiki/Regulation_and_licensure_in_en....
The OP says "the US". That article is one state in the US. Most states don't have restrictions on the use of Engineer in job titles. Canada does though.
A Computer Engineering degree does not require ABET accreditation - it didn't when I went to UCSD... the difference between Computer Science and Computer Engineering was just 4 classes. Exactly the same calc/physics classes between the two, which were engineering level.
There are plenty of engineering programs without ABET accreditation. If you get a degree from a non accredited program, you'll likely have a different title than engineers who graduated from an ABET school and will make less. This is because there has to be some standardization in what the people who build bridges learn in school. All companies are different, but my company won't even look at an engineer without ABET accreditation. Even a Ph.D. from China, which is just really dumb. It is reality though.
Interesting. Here's a list of mid career pay by major. Fields that tend to be accredited and value accreditation are certainly present in high places on the list, but variants of mathematics are in comparable places. My guess is that "economics and mathematics" or "computer science and mathematics" are probably examples of fields that require a lot of math but aren't part of an ABET accredited degree. Doesn't look like there's much of a pattern here.
Now, if you plan to work in structural or civil engineering or another field where accreditation is important, than yes, I would tend to agree that having an ABET accredited degree is important (as well as taking the PE exam for that field).
Fields where ABET accreditation isn't terribly important (computational finance, google, facebook, and so forth) often pay more than the fields where it is (mechanical, civil). My guess is that passing the PE for computer engineering wouldn't make a big difference in your compensation at a top tech company, I certainly don't know anyone at those companies who has bothered - though I certainly am willing to evidence to the contrary!
I don't think your nitpick is correct. To be a licensed engineer within some fields, you do need ABET but there are many people who are "engineers" (their job title) but aren't required to be licensed.
Indeed, in the US, most engineers work under an "industrial exemption," for instance if they are employed within a company that makes a product, and not providing engineering services directly to the public. Most of the engineers at my workplace do not have licenses. On the other hand, our products get checked out by a certification lab, and the people who sign off on the test reports are in fact licensed. The work they do is phenomenally dull and bureaucratic.
Of course there are also fields where everybody pursues a license such as civil.
If I were to call myself an engineer, it would be in a field that doesn't have a discipline-specific license in my state. I don't know if it means that I don't need a license, or what. So far it has never been an issue.
My current job title is "Data Engineer", before this role I was a "Process Engineer". In my opinion those two jobs are actually pretty similar.
When I was working in process engineering I was trying to optimize the outputs from our industrial process on a day to day basis in this role broadly speaking I try to optimize the data extracted from the same industrial process.
Mostly I'm concerned with how can we extract data out of our plant, how do we represent and present that data (particularly to operators and technicians) and how can we better recognize and respond to underlying trends in the data.
Before I assumed the role (in 2011) my predecessor, who had a background as a statistician, was called a 'Process Statistician' so I assume my Manager changed the job title to reflect my background as an Engineer (Materials Engineering in my case).
What I would consider the difference between an engineer and not engineer (I am not an engineer) is delivering a qos or sla driven by measurement of tolerances and either empirical or imputed information from existing qos or tolerance information... Not necessarily the optimization part. Everyone does optimization to some degree (possibly negative) but not everyone is an engineer.
I think of engineering as the practical counterpart to science. Science is finding patterns/uncovering truths/building and testing models; engineering is the deployment of technology to fulfill an objective.
Yes this is my view as well Engineering is applying theoretical knowledge to achieve practical solutions.
In my world, which is industrial manufacturing there are scientific theories - fluid dynamics, thermodynamics, kinetics etc. which govern the fundamentals and limits of the process.
As an engineer your take this knowledge along with your own intuitive experience and work to ensure the reactor is operating at peak efficiency.
Engineering is more than just application. In some places you need to be certified as an engineer, for better or worse, which says that there is an understanding of how to calculate and communicate product tolerances and service level guarantees that go beyond just applying science. Basically be the bare minimum of applying science I would call hacking - which is a great thing, but there is value in the distinction between hacking a solution and engineering
Someone said any field with "Science" in the name isn't really a science. Computer science, data science, political science, social science, etc. Physics, chemistry, biology don't have science in their name.
>> Someone said any field with "Science" in the name isn't really a science. Computer science, data science, political science, social science, etc
The etc. would also include cognitive science/neuroscience, medical science, earth science, material science, agricultural science, veterinary science, geoscience, food science, etc.
And of course as we all know climate science is fake./s
Generally when I hear a field with the word "Science" in the name I think of it as a more interdisciplinary field. Take Earth Science it draws on different areas of physics (ie wave physics), biology (ie ecology) and chemistry (ie kinetics). Earth science is still very much science it is just doesn't fit perfectly into the more foundational fields.
That's an expression, not an equation. And the space between data and scientist quite obviously indicates that the two combine multiplicatively, not linearly as you have mistakenly written.
Hmmm! Funny, but I'll bet you could learn a lot from research simulations. It is, after all, a field that does seem to benefit from the big-picture review...
(I was just talking to a client about this today, at a micro-level: What is your personal research model, for important career-related, yet non-work-related projects? Does it exist, or do you just perform research as the intuition prompts? What kind of structure can be leveraged to achieve a quality outcome? So maybe that's why this seemed interesting to me.)
Why is software engineering not a valid engineering? I worked on both software and hardware engineering and general principles seem to be the same. You deal with complexity and simplify it by making abstractions. You make calculations to make sure your project is feasible. It's not like EE and Aerospace engineering are literally the same field but there are some principles shared in those fields, and with software engineering. Am I missing something?
Education is one particular difference. With few exceptions, engineering jobs require an engineering degree, and consequently an ability to basic engineering math and understand basic physics. The same cannot be said of software
When software engineers stop disclaiming all liability for their products failing, we can talk about them like we talk about engineers (sidenote: some are already take responsibility)
It's foolish for anyone to take responsibility for any software written under the current prevailing industry practices.
It would be funny if real engineers were able to get away with making crumbling messes that can't hold their own weight because their middle managers don't believe in concepts like stress and strain like ours don't believe in refactoring or abstractions.
Yes, while in many countries Software Engineering is actually a degree that needs accreditation and in some cases even a professional exam if you need to sign off projects as the responsible person, on US apparently you can call yourself software engineer after a three week bootcamp.
My degree was called computer science, and run by the school of maths rather than the school of engineering, but I’m eligible to apply to be accredited as a chartered engineer (the regulated professional title for engineers in this country), though people who graduated after 2012 aren’t, due to course changes. The whole thing is a bit of a mess.
Just to add to the confusion, my degree was a BA(Mod) rather than a BSc or BEng, for obscure historical reasons.
True - Superficially SQL may appear to be simple, old fashioned and a bit verbose, but once you are expert with it (takes at least 5 years) is amazingly powerful.
Operates at a much higher abstraction level than Java, Python etc, so is I would guess 25 times more expressive.
Postgresql pure SQL CTE’s give you variables and recursion PLpgsql gives you dynamic sql for macro/meta programming.
If you use immutable tables can be purely functional.
SSDs and now even faster Optane memory have resolved the IO problem which handicapped RDBMs until recently.
They’re all Turing complete inc SQL with case and recursion.
I meant density, 1 line of a code, a SQL window function with a filter clause would probably take a page of Java to achieve the same result.
SQL window functions aren't rocket science(not that I've used them much, cause ORMs and popular stripped down DBs like MySQL tend to not support them very well), but they do a lot more than you think if you're comparing them to trivial map/filter operations.
Map and filter together in Java reflects why you get from SELECT and FROM without aggregate, much less analytical (window) functions. Aggregate and analytical functions correspond to reduction operations, which Java supports but doesn't come with canned equivalents to common analytical methods, just aggregates, AFAIK.
This is such a bizarre post. The reason why people use a language like R is because it is easy to learn and use (and install, via RStudio) for data analysis without having to be a well-trained programmer. I can’t recall ever hearing from anyone who has relied on R, doing so because it was computationally efficient. The point of the language is convenience — particularly with how easy it is to create attractive graphics using ggplot2’a defaults.
It’s a testament to the R library’s developers (particularly Hadley Wickhan) for making APIs that do so well in streamlining data work. But I’m willing to bet a majority of R users, particularly in academia, could not load a simple delimited data file without a high-level call such as read.csv.
(By “simple”, I mean a delimited text file that could be parsed with regex or even split. I don’t expect the average person to be able to write a parser that dealt with CSV’s actual complexity)
The fact that R has such buy-in despite being a rather awful programming language (a friend of mine worked on the next Lisp-like version of R under Ross Ihaka, and the next version is based on the fact that current R is a bit awful) is precisely because it offers such convenience to non-programmers.
In my sister company, they have data scientists, and data engineers. The data scientists write their algorithms in the language they're most comfortable with (typically JS), and the data engineers rewrite to perform efficiently in the application that's applying them.
Data scientist and programmer are two very different specialisations.
It’s a great set of DSLs that show off some of the pretty decent meta programming facilities of R. I think those DSLs could be created elsewhere, but I also don’t think it's an accident that it happened in R.
I don't want to pile on but that sentence strongly reminded me of the oldish saw about "a data scientist is a programmer who lives in SF". I've never heard of anyone using JS for data science. What might it have that's anywhere comparable to the tidyverse or numpy/scipy/sklearn?
They don't use JS to investigate the data, they use SQL (Hive, Impala etc.) as their lingua franca for exploring datasets. But they then write their algorithms based on that analysis in JS.
Again, I don't know why, it struck me as odd also, but hey, whatever they're doing, it works for them.
I have no idea why they use JS - it's a Berlin start-up so maybe trendiness is involved? All I know is that their data engineers have to make those algorithms work in production.
> ... despite being a rather awful programming language (...) it offers such convenience to non-programmers
I've heard people say similar things about MATLAB - that it's a poorly designed language, but many that people (mostly non-CS folk) use it out of convenience.
Can someone with experience using R explain what makes it so appealing to non-programmers? It seems like these two factors, "poorly designed" and "easy to use", should be at odds with each other.
Its all about availability of libraries. I did control systems in undergrad, and despite being a shitty language, being able to describe and manipulate dynamic systems (ODEs) was very useful. Doing numerical integration by hand for the nonlinear systems was horrible, though better than simulink (which is about as much fun as using LabVIEW or sculling H2SO4).
What makes R appealing is basically all statistical methods are available in it -- and it is often the initial implementation language of new methods in statistics. Often an R program involves very little programming as such other than to read in data, run some existing statistical methods on it and print or plot the results. I'm not a particular fan of the language itself (I kind of wish XLispStat hadn't died), but every time I feel like checking out Python or Julia I find things I need that haven't been implemented yet in those languages that are in R.
Eh, it’s not as bad as people like to whinge that it is. There are indeed warts, but they’re pretty overblown. If you are comfortable with functional idioms R mostly does what you want without a great deal of fuss. If you’re predisposed to procedural idioms, then you’re going to be fighting the language.
I started learning R about the time I started reading How to Design Computer Programs, and I found it pretty easy to transfer that model of thinking to R. And I find Clojure, Racket, and Scheme to also be somewhat comfortable after a short reacclimation period.
Some of the convenience bits have to do with most functions working on vectors without needing to explicitly iterate most of the time. Also libraries. If you want to estimate a linear regression, or make some exploratory plots, or try some rando statistical method that your graduate advisor suggests, you don’t have to worry about whether it’s already been implemented for you in R.
You can do a lot of heavy lifting by cribbing off of example code because most code is short. You just get heaps of leverage by using R.
Look, I like to do things the hard way a lot. My whole life is pretty much a string of highest friction path choices. For data science R is easy because all the work has been done for you. It's the difference between writing GUI apps against Cocoa APIs vs, I dunno, XLib or Motif.
Problem is that we are coming from completely different perspectives. When you say "programmer", you are likely referring to someone from a CS background, likely with software engineering experience, who has spent their lives working in C++, Java, Python, etc.
By that definition, I would be a non-programmer, as I come from a statistics background, and though I have lots of experience in C++ and Python, most of my experience and work is in R. But that is by choice.
If I'm trying to create an application or build a website, I wouldn't use R. But when it comes to ingesting data, transforming and cleaning data, and modeling data, R is second to none. Yes, its syntax looks ugly and bizarre if you are used to object-oriented programming, software development, etc. In the context of working with data, I have never found anything in R to be even remotely confusing or strange.
On the contrary, the next best option to R would almost certainly be Python, and the gulf between the two is massive in my opinion. Python is a great general purpose programming language, but its data analysis capabilities, using packages like Pandas and sci-kit learn, feel like poorly designed, bolted on, and unwieldy. R is better for virtually every aspect of data analysis than Python.
So it isn't that R is poorly designed. Conversely, its very well designed, for its purpose as a data analysis-focused programming language. It only seems to be poorly designed to "programmers" because programmers work on problems that R isn't meant for. But that is like complaining that a screwdriver looks poorly designed for hammering nails.
R is cumbersome, but then you look at Matlab and Stata and try to explain that to social science graduate students that just managed to grok LaTex, and R starts looking like a streamlined vision of the future.
Totally. I’ve taught intro R to mostly social scientists, and intro Python to mostly physical scientists. Every workshop, the R group is coding circles around the Python group with the same number of hours of instruction. And the intro R curriculum that we use was largely a ported version of the Python curriculum.
Having seen brand new scientific programmers tackle two “beginner friendly” languages, man the differences after a day or two are stark.
A trend I've been noticing (especially as ML/AI tooling becomes more accessible) is that people believe the quality of data science code and workflows is proportionate to its complexity/LOC (since complex problems require complex code, right?).
It's a toxic perspective that ignores recent and pragmatic innovations in the field.
Lol, reminds me of the nightmarishly complicated first NLP model I wrote. I would classify this under "resume driven development" which you see a lot on the software side too with fancy new frameworks.
Funnily enough, although the first place that let me work with Hadoop and Spark didn't need to be using Hadoop and Spark, I probably wouldn't have worked there if they didn't let me learn them, so maybe this isn't as wasteful as it seems at first glance
Something about this article strikes me as a thinly-veiled complaint about poorly designed object-oriented systems. Take, for example this comment by the author:
>Even if the money were half of what today’s coder gets paid it might still be a better job because one is spared the tedium of looking at millions of lines of Java that do almost nothing!
What all those millions of lines of code are is abstractions, decoupling, and modularization of logic/responsibilty. This is hard-won knowledge from the field of software engineering. Granted, a lot of it is probably very poorly designed or organized. But the problem is the design and not the philosophy.
Because scientists all use the same basic rules of math, but each business will each have it's own special rules (i.e. not all payroll software implements the same policy/axioms), this makes it really easy for the hard logic of scientific work to be in a general-purpose library. "Normal" developers need to customize their own rules, or in other words, develop their own services unlike the data scientists.
Now if every data-scientist had to roll his/her own version of numpy, pandas, sci-kit learn, tensorflow, etc. the author would probably be decrying the deluge of procedural spaghetti produced by data scientists. The data scientists' notebooks look simple because of all that indirection is hidden away in the many libraries.
It's "Something about poorly designed, object-oriented systems."
1. compound adjectives are hyphenated, except when the first world ends in 'ly.'
2. two adjectives, should have comma, similar to 'big, green systems.'
All of today's software is built on millions of lines of code. This isn't much of a problem when it's hidden away behind a good abstraction. Being written in another language forces the API to be documented well enough so you don't need to go deeper: "native code", "kernel code", "part of the browser".
Crappy million-line Java apps are generally crappy not due to raw line count but rather due to leaky abstractions and badly designed APIs, so you do have to navigate through a lot of that code.
I sometimes wonder if programmers instinctively overcomplicate things in interest of collective job security. Some of the stuff I've seen in (particularly awful) Java code bases is perplexing to the point where it seems intentional.
It's more likely the natural entropy of code - it's easy to add stuff to a system in a way that makes it more messy; and if the system already is a big mess, then it's much harder to do non-messy additions and the bigger mess it is, the harder it is to start cleaning it up.
I refer to this as the "spaghetti law of attraction". The burden of refactoring things gets higher and higher and no one wants to touch it. So they just add another try-catch block and do some side effect and get the PR merged.
It is also a question of management, it is easier to motivate adding new features instead of keeping thing nice and tidy. Especially as finding good abstractions are very hard and time-consuming.
Writing clear and as-simple-as-possible code is one of the most difficult parts of programming, one of the hardest skills to learn. In fact, many programmers don't ever learn it. Some do not even realize it's a problem.
I could talk about people unprepared for the jobs they do, and companies and managers setting unrealistic deadlines, having unrealistic expectations, etc. But I think most often than not, the issue is that to really build good code, you need to have a lot of experience and plan and design really well from the start. And simply, most companies and products start as they can. If you start with a deficient codebase, trying to fix that later under heavy budget and time constraints is pretty much impossible, and it's so accepted that some people believe it's the normal way to work.
And even with proper design, clear perspective, and good programmers, you still have to be lucky that no one higher up the ladder imposes decisions that f*ck up all the good work done. There are a few factors more.
What I mean is that you might find all kinds of people, but in general, for those of us who care about what we are doing it's the exact opposite. We consciously try really hard to keep code as simple as possible.
It would be far from the first HN article (or comment) that failed to make a distinction between "poorly designed object-oriented systems" and the very idea of OOP, design patterns, etc
Personally, I am increasingly convinced that a lot of this hate comes from programmers with weak abstract thinking who simply cant do it. Instead of admitting that there is learning cure involved, they will claim the system is bad and everyone else is bad. Compounding factor is difficulty dealing with system that was written by different people who holded different opinions.
Yes, there are badly designed large system. No large system is perfect. However, there are also reasonably designed large systems, including in Java and Java is used in such system for a reason. It is more challenging to write large system. Yes, it is harder when parts of system are written in style that was considered best practice few years ago but was abandoned since then.
If you are spending a lot of time looking at millions of lines of Java that do almost nothing, them you likely dont really know what it what and need to read up more. At least that is my experience.
Badly designed systems aside, for most systems that currently exist or have existed in the past, there is little or no documentation that is worthy of existing. Most comments in code and the associated documentation in manuals fails to provide the reasoning as to why the code exists, why it is written that way, what underlying assumptions have been made, etc, etc, etc.
I am going through a process at the moment of documenting all of my local codebase. It will, in turn, be turned into a literate programming base. The problem I am finding is understanding all of the assumptions that underlay the original code. Why was it written this way or that, what is it trying to do, is the code actually doing what it is supposed to do?
There are, at present, some questions that I am having difficulty answering and this is my code. How much more difficult is it for someone to come in and look at a historical piece of code and follow what the original authors and designers were trying to achieve and what were the changes that have been made over time trying to achieve.
Documentation at the level we need to be able to adequately maintain any code base is just not done - it is very hard to do and to do so in a way that will help future people manage and maintain that codebase. On of my projects involves restructuring the code base. However, I need to understand the history of that codebase and that means talking with those people who are still living who knew the original authors and give an oral insight as to why things have been done the way they are. This oral history has to be written down and the codebase documented with it. When that information is in place, why the code is written the way and how we can now rewrite that code to be more effective is now achievable.
If we then put on top of missing history and documentation all of the bad designs, well, we are then facing even bigger problems. Then we put on top of that all the egos and politics involved, we get an even bigger mess.
So just reading up more doesn't actually help, because that which is needed was never written in the first place.
I've been hearing the same complaint between Cobol and Java for years : it was simpler before, more efficient, etc.
Of course it was, but you were tied to one system (no application server), security was login/pw, database had no constraint, typing systems were ultra limited, everybody has its own way of writing batches (no Spring), business code was mixed with tons of technical code (no JPA).
Now, sure, if you glue some R, some SQL, etc. you can extract insights worht millions of dollars. But all of that exist just because we have digitalized all of the processes, data collection, etc. And the rise of data scientists will continue only if there are more stuff put in the databases, thanks to you plain, regular, normal programmers...
> But all of that exist just because we have digitalized all of the processes, data collection, etc. And the rise of data scientists will continue only if there are more stuff put in the databases, thanks to you plain, regular, normal programmers...
IMHO there's far to little attention paid to how data might be valuable in an economic sense when storage strategies are being designed by database designers. I recently gave a talk at a developer conference and was really surprised at the level of pushback to adding more data elements or higher precision data "just in case it might be useful".
The preconception that you have to be maximally efficient with storage has led to huge quantities of valuable data being lost.
>> The preconception that you have to be maximally efficient with storage has led to huge quantities of valuable data being lost.
I 100% agree. Many programmers are trained as if memory, CPU are finite resources. Although it's true, in many cases, that reality may be safely ignored, opening tons of opportunities (to store data, to develop faster because you don't optimize, etc.).
At a time where my phone has gigabytes of memory, I'm always surprised that some people ask me to put a limit on a text field. I understand the technicalities behind the question, but from a conceptual point of view, that's often pointless.
The net value of nearly all data is negative. It’s a liability not an asset. Unless you are actually selling data to customers or solving a specific problem that requires collection of specific data, it’s just wasting the company money. You would never hear a manufacturer say “ let’s stock pile aluminum in case we need it”. But that line of thinking comes up all the time with respect to data.
The problem isn't the abstractions, it's the sheer size of the codebase. The author - and I think most people - prefer a codebase they can grok. Nobody can grok millions of LOC, at best they can have a high level overview of what does what.
At that point you need the abstractions and practices that make code boring.
>But the problem is the design and not the philosophy.
If the philosophy prescribes dozens of tools for managing complexity and no tools at all for reducing it than it is the problem.
"Abstractions, decoupling, and modularization of logic/responsibilty" are not some kind of universal good. They are only useful within specific contexts. A lot of software engineers do not understand this and routinely engage in premature abstraction. As a result they produce systems that are 10 times more complicated than they need to be for absolutely no reason.
Java definitely encourages this kind of mentality, because the language itself and its standard library lack in some fundamental areas. Introduction of lambdas and streams helped, a lot, but the overall mentality is still well-entrenched.
No, at least in my understanding data scientists specialize in the analysis of data rather than the development of software. You'd hire a data scientist to look for interesting patterns in data, or create machine learning models, and other data analysis tasks. These tasks may involve writing code, but it's usually specific to data analysis, often in R or Matlab or similar. A lot like how many people in the natural sciences pick up coding to enhance their capability, but the software writing is a means to an end.
I wouldn't hire a data scientist to build a web app (well, I would if he or she had the necessary knowledge and skills - the job title wouldn't be "data scientist" though). "Software developer" is much closer to "programmer".
I think the point of the article was that it used to be more common back in the 60’s and 70’s for programmers to work on data problems. From basic stuff like census tabulation or designing file systems, to creating trigonometry or t-statistic tables, to AI.
There was less specialisation, less of a divorce between programmers and users.
There also seemed to be a conflation of computing and AI back then. Lisp was considered AI. And the early computing pioneers and theorists were strongly interested in AI, logic, and mathematics.
In my company I am a software engineer and my colleague is a data scientist, our current project that we work together on does a lot of NLU and NLP type work (think bots) and our skillsets often don't overlap and are both equally valuable to the projects success. That is, I tend to write the infrastructure and platform code that ties everything together and deal with all the software engineering type work, while my data scientist feeds in trained models and the likes.
Both are necessary to handle contractual requests/responses as per our scope design.
Well I am "lucky" enough that I can work on a legacy application written in OpenACS. It wasn't written in the 70's but it's definitely old and outdated. So this kind of narrative that everything was better back then, simply does not convince me at all. I might be wrong of course, but the author tells anecdotes which is not a real argument, so there is that.
The output of a data scientist's work includes plenty of things that aren't code. Yes, the code that I write tends to be very short, but if it represented everything I had to do to get there it would be quite a bit longer.
This post states that a data scientist uses compact languages such as SQL and R.
Genuine question - do people really believe that being able to write and understand complex SQL makes you a data scientist?
I ask because, I've been writing some of the nastiest, most difficult looking SQL around for probably at least 15 years. And yet, I would NOT call myself a data scientist because I know and can work with data and use SQL. It might make me a data engineer.
What would make me a scientist is the process, method and rigor I apply to data-driven research and in practice. It's not about what tool I use or how complicated that tool is.
I often get a whiff of imposter syndrome over this because, if being "great at SQL and R" is enough to get the big bucks as a data scientist, then I'm clearly doing it wrong. But, then again, maybe I'm being too literal thinking that a scientist means something different.
I've been working as a data scientist for several years and have written some pretty gnarly looking SQL myself. I have a background in math and hard science so I have some understanding of the scientific method as well. While I respect our DBAs I wouldn't call any of them qualified to be data scientists.
While I have been able to hold my own in this job I went back to school to pursue a graduate degree (partly) because being in the field has shown me how much more there is to know. While it's easy enough to train a simple model in R there are so many ways to fool yourself and produce an invalid analysis and so many variations on otherwise-simple problems.
It seems this field has a lot of variation. A glorified report writer might get the DS title but they're not going to get the really cool jobs.
If you're interested in data science try out a kaggle competition and try to place high. The variety of methods and tricks people try to improve their entries can be illuminating, I think.
I'll preface this with I've not had a look at any Kaggle competition, but I always assumed Kaggle competitions was on par with programming competitions in terms of how the skills transfer professionally. A great programmer is not necessarily great at programming competitions after all.
No, there's way more to data science than competitions. But for someone who is already a data engineer more or less, I think it could be a good window into the complexity of modeling.
That covers the "data" aspect, for my work however, the "scientist" aspect is just as important. While I'm expected to use SQL and R to generate reports, I need the thought process of an epidemiologist to construct my analytic samples. I also require the scientific knowledge and background to interface with MDs and clinical PhDs, who need me to bridge the gap between data and science.
Firstly, it states that "a data scientist uses compact languages such as SQL and R". It doesn't state "everyone who uses SQL is a data scientist".
That said, the term data scientist itself is a bit frustrating. It gets thrown around a lot as if it is a well-defined role, and it is anything but. In my experience, the role of a "data scientist" is about as well defined as the role of an "engineer": it has connotations about the type of work and maybe a few shared skills, but the specifics of what an "engineer" does and their skillset varies widely depending on if they are a software engineer, an electrical engineer, or a civil engineer.
So while I think that most data scientists know SQL or use SQL frequently, I don't think that all data scientists use it, nor do I think that everyone who uses SQL works in a role that would probably be considered that of a data scientist.
This doesn't make any sense. My job title is "software engineer" I never do any system administration. I produce code in python, javascript, C and SQL; never do any sort of administration. Sure, I occasionally deal with linux since our servers are linux and so some knowledge of it is useful, and I use unix tools pretty extensively (in OSX) since I prefer to write code this way. All the "software engineer"s I know have similar experience to mine with varying languages, so please suggest some evidence.
I agree that it doesn't make any sense, however I recently interviewed at two different companies for a software engineering role and both had requirements/expectations for sys admin experience. I will refrain from going off on a rant and just remain hopeful this is yet another short-lived trend.
242 comments
[ 1.7 ms ] story [ 265 ms ] threadIn R 0.1 + 0.2 is equal to 0.3.
Why do you compare point to point? I have never once in my Statistics education compared point to point. You always need to see the probability of the result if it is within 2 points.
But if you want to compare use all.equal(0.1 + 0.2, 0.3)
Why do you compare point to point? I have never once in my Statistics education compared point to point. You always need to see the probability of the result if it is within 2 points.
But if you want to compare use all.equal(0.1 + 0.2, 0.3)
> print(.1+.2, digits=18) [1] 0.300000000000000044
Why do you compare point to point? I have never once in my Statistics education compared point to point. You always need to see the probability of the result if it is within 2 points.
But if you want to compare use all.equal(0.1 + 0.2, 0.3)
If R is giving you 0.5, you should find another language (I assume you meant 1.5?)
We are constantly thwarted by infrastructure teams that use superficial policy basically so they can whine and complain that they don’t want to have to provide support for the extremely heavily researched and tested implementation we create.
They don’t care that different technologies, database systems, whatever are chosen to solve customer use cases and that growing our business means supporting “Lovecraftian concoctions of R and Python” — they just don’t want to do their jobs (which indeed requires providing infrastructure support for crazy screwball data services that repeatedly break all the assumptions)...
This was true for me anyway. The main thing I do is deep learning for computer vision and image search, but I think it’s fair to say I have significant experience with Docker, GPU architectures, various CI tooling, linux system programming, deep internals of CPython, internals of MySQL and Postgres, lots of frustrating performance tradeoffs with py4j in the pyspark world, as well as all the usual crap with pandas, sklearn, data visualization tools, and a lot more.
I’d say almost all data scientists I’ve worked with are just like me, just with maybe different specialization areas, except possibly for very young data scientists right out of undergrad.
What you describe sounds like where I want to be, and it would help to where I could go looking. (Deep Learning / Vision specialization with a math focused CS background but want to learn more core software skills going ahead)
I would say your description is accurate but only accidentally. The reason we have to learn more core engineering skills is that infrastructure will not take responsibility of bringing our solutions to production, and seek to limit the tools in our toolbox with policy.
It’s not fun when your everyday life is constant impedance mismatch against tools that infra will allow you to use. You can constantly see better / faster / safer / cheaper ways to solve problems, which have no downsides at all relative to the bad, slow, insecure, expensive ways infra currently makes everyone solve problems, but you just feel constantly sad that you are superficially prevented from the autonomy to select the efficient solution accordung to your creativity and skill.
I think a big difference between research programmers and production/ops people is that as researchers we often chase a transient goal. Build some complex and horrible integration to compute a result or put something in a paper. We used to call these Rube Goldberg machines rather than Lovecraftian horrors, but we mean the same thing. Something that belongs on a movie set, with some Jacobs ladders arcing in the background. In some circles, it is called the heroic demo.
In the past, we might substitute other fads for your CI tooling or API validation tools. I remember when some research programmers were all-in on enterprise junk like J2EE/managed code, SOAP/WSDL, and other stovepipe tooling. There is a lot of cargo culting of such tools. When you have furnished your lab with rapid prototyping tools and focused on crazy integration stunts, you are almost always deluding yourself to think these tools are also giving you "production" system qualities.
Building something at the hairy edge of possibility is inherently about experimentation and risk-taking. Building reliable, production operations is inherently about conservative design and risk-mitigation. There seems to be a new cargo cult of devops which believes you somehow mash these together and the conflict disappears. You don't have to have to map the negotiation onto two teams with opposing objectives, but the negotiation has to live somewhere.
Magically erasing the negotiation just means that you have chosen to default on the optimization task and jettison concern for at least one of functionality, cost, or risk. Startups commonly do this because the VC funding has mitigated the risk elsewhere: you can fail because they've also funded your competitor who may succeed...
We are often required to create new services and functionality because it is how our company can grow, and we have to have ease of access to experimental working space, with freedom to do things like custom compilations of ML frameworks, using programming languages that haven’t been widely used in the company yet to gain access to an important library, define complex assumption-breaking deployment constraints relative to GPU runtimes or containerized notebook servers, etc.
I think people who see how these things grow out of prototypes and wrongly conclude it was designed with transient concerns and thus isn’t robust in some way, they are rushing to judgment and discounting the fact that that ML expert who also wrote the web service layer and who also wrote the Jenkinsfile and who also wrote the container definition abd who also knows how to tune indices in the database, etc., really made their choices for serious, pragmatic engineering reasons that solve the business problem efficiently, and that they already anticipated and accounted for the shallow tradeoffs and caveats that IT will use as potshots to try to circumvent the responsibility to help maintain it.
We have had to bite the bullet and use colo facilities to self-host internet-facing deployments that the overhead-funded IT groups would not touch with a ten foot pole. From these experiences, I also acquired a more nuanced perspective on the IT division perspective and constraints, and how they derive from overall organizational policy and economics. We also had funny situations where we tried to help other PIs benefit from our new-found independence, and immediately regretted it. They did not understand what self-hosting means. I think anybody trying to toss integrations over the fence to an ops team needs to have an extended tour of duty trying to operate their own solutions in production WITHOUT assistance before they form bold opinions about operations constraints.
When there are strong time-to-market constraints (which includes publishing papers in academics), you are forced to find solution points that are different than if you are planning to run something for long periods at low overhead and low accumulative risk. These solution points also have to take into account the staffing and resources for that ongoing production.
Those things like bleeding edge libraries and assumption-breaking deployment constraints are the headache for ongoing operations and maintenance. It's not enough to have an existence proof that some complex integration can be built and passes its tests. You need a plan for how all the components will be maintained, patched, and upgraded. You need contingency planning when some of those bleeding edge components are going to become deprecated. You need to consider what staff capabilities are assigned to do that maintenance work or what will happen when the institutional knowledge used to form the original integration is not on-call to reintegrate it in the face of unexpected events.
I think this is one of the worst possible attitudes to have. It’s a petty way to feel, desiring some type of “I’ve seen some shit” tough guy credential more than supporting the stuff needed to actually solve business problems.
If you hire people whose value add to your company is inventing completely new things, including deployment, ops, scaling, etc., that goes along with that, then it is the job of infrastructure on the other side of that fence to happily and eagerly accept whatever is tossed over the fence, to understand why developer teams made the choices they made, and to take an attitude of supporting as much as possible.
> “You need a plan for how all the components will be maintained, patched, and upgraded. You need contingency planning when some of those bleeding edge components are going to become deprecated. You need to consider what staff capabilities are assigned to do that maintenance work or what will happen when the institutional knowledge used to form the original integration is not on-call to reintegrate it in the face of unexpected events.”
Yes, of course. But all this is already what dev teams are doing. Ops / infra is not taking a hare-brained plan and adding these robustness aspects into it. Not at all. Instead they take plans from application teams and try to use policy to minimize their own maintenance burden, even when that optimization is antithetical to what the company requires at a more fundamental level.
A lot of companies languish and die because of sociological dysfunction in the policy interface between dev teams and infrastructure. The more that infrastructure has political control of that interface, the closer to death is that company.
It’s like a body that is disallowed from generating white blood cells in response to a new immune challenge. Even if the bleeding edge integrations are really hard, the alternative world where you slow them down with policy is death and attrition.
Data science is an extremely broad, vague buzzword encompassing a variety of jobs and skillsets, most of which have existed for decades under different names. You do work that involves putting models into production in Python, congrats. The insistence that all data scientists must also do so is silly, especially considering that there are surely many skills used by many data scientists that you are incapable of.
http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf
https://en.wikipedia.org/wiki/Functional_programming
The argument this post is making is reductive. Yes, sometimes data science is simple. Sometimes it isn't, and that's when you really need someone with the appropriate skillset.
systems programming=irrelevant bloat and abstraction
while
data reduction=definite purpose and utility
People writing python notebooks to do data analysis are probably fairly comparable to the scientific computing programmers of the past, but I feel like this picture tends to dismiss the computer science side of systems programming:things like GUIs, network code, processes and virtual memory, all the architectural aspects of computing.
One might prefer APL or Forth for writing one-page programs, and it's probably true that systems now are bloated relative to what they could be. Still, there is much of interest going on in a typical operating system, compiler or video game, while a typical data analysis notebook is IMO fairly dull and even basic, from a software angle.
There is nothing myopic about his perspective.
But, as with a new Paul Graham essay, surely we can critique the blog post on its merits instead of falling back on an assessment based on some kind of appeal to authority/"expertise by association". Philip Greenspan doesn't need to be treated with kid gloves as if he was the pope.
John Ousterhout made comments that touch on some similar (though not identical) distinctions in programming practices. That was years ago, and he was then a much more credible figure in software than Greenspun. All the same, his essay was heavily criticised. That's what serious intellectual discussion should involve.
https://en.wikipedia.org/wiki/Ousterhout%27s_dichotomy
http://www.tcl.tk/doc/scripting.html
That is not to say programming isn't a difficult job that requires a lot of analytical and creative thinking similar to an engineer. The difference is in getting a degree in something that has 4-5 years of calculus based math, physics...etc classes. There is also a rigorous 8 hour test to get a license after 4 years on the job.
I guess the broad term of building something and doing analysis fits here, but I don't see any Data Engineers in practice. What I see are Data Scientists and Data Analysts. Of course I'm arguing over semantics here, but it is important to get the distinction correct.
Do you happen to have a reference for this? At first glance, it seems to be incorrect rather than nitpicky.
Anecdotally, I know plenty of people who do not have ABET accredited degrees and have "engineer" in their title in the US.
>To get a professional engineer's license, you must graduate from an ABET accredited program.
In some states you can take the FE and PE exam with a related non-ABET accredited degree and work experience based, and in some states you can take the exams with no degree based on experience only. For example, NY let's you substitute work experience for a degree. There's a table where you can see how much experience you need based on your degree (or lack of one).
>Unless you have a professional engineers license, you can't testify in court as an engineer
Courts determine what credentials qualify someone to testify as an expert witness, not state licensing boards.
In some states, testifying in court as an engineer does qualify as practicing engineering without a license, and the state licensing board could fine you after the fact. However, in other states testifying in court doesn’t necessarily qualify as practicing engineering.
(There are lots of other articles about that case, that one sums it up mostly in the url)
https://www.payscale.com/college-salary-report/majors-that-p...
Now, if you plan to work in structural or civil engineering or another field where accreditation is important, than yes, I would tend to agree that having an ABET accredited degree is important (as well as taking the PE exam for that field).
Fields where ABET accreditation isn't terribly important (computational finance, google, facebook, and so forth) often pay more than the fields where it is (mechanical, civil). My guess is that passing the PE for computer engineering wouldn't make a big difference in your compensation at a top tech company, I certainly don't know anyone at those companies who has bothered - though I certainly am willing to evidence to the contrary!
Of course there are also fields where everybody pursues a license such as civil.
If I were to call myself an engineer, it would be in a field that doesn't have a discipline-specific license in my state. I don't know if it means that I don't need a license, or what. So far it has never been an issue.
When I was working in process engineering I was trying to optimize the outputs from our industrial process on a day to day basis in this role broadly speaking I try to optimize the data extracted from the same industrial process.
Mostly I'm concerned with how can we extract data out of our plant, how do we represent and present that data (particularly to operators and technicians) and how can we better recognize and respond to underlying trends in the data.
Before I assumed the role (in 2011) my predecessor, who had a background as a statistician, was called a 'Process Statistician' so I assume my Manager changed the job title to reflect my background as an Engineer (Materials Engineering in my case).
In my world, which is industrial manufacturing there are scientific theories - fluid dynamics, thermodynamics, kinetics etc. which govern the fundamentals and limits of the process.
As an engineer your take this knowledge along with your own intuitive experience and work to ensure the reactor is operating at peak efficiency.
That's the most unscientific thing I've heard in a while ;)
The etc. would also include cognitive science/neuroscience, medical science, earth science, material science, agricultural science, veterinary science, geoscience, food science, etc.
And of course as we all know climate science is fake./s
Generally when I hear a field with the word "Science" in the name I think of it as a more interdisciplinary field. Take Earth Science it draws on different areas of physics (ie wave physics), biology (ie ecology) and chemistry (ie kinetics). Earth science is still very much science it is just doesn't fit perfectly into the more foundational fields.
(I was just talking to a client about this today, at a micro-level: What is your personal research model, for important career-related, yet non-work-related projects? Does it exist, or do you just perform research as the intuition prompts? What kind of structure can be leveraged to achieve a quality outcome? So maybe that's why this seemed interesting to me.)
https://medium.com/@donhopkins/designing-user-interfaces-to-...
As for the liability, I fully agree.
It would be funny if real engineers were able to get away with making crumbling messes that can't hold their own weight because their middle managers don't believe in concepts like stress and strain like ours don't believe in refactoring or abstractions.
Just to add to the confusion, my degree was a BA(Mod) rather than a BSc or BEng, for obscure historical reasons.
``` Words.stream() .map(word -> String.toUpperCase(word)) .filter(word -> word.startsWith('A')) ```
It’s a testament to the R library’s developers (particularly Hadley Wickhan) for making APIs that do so well in streamlining data work. But I’m willing to bet a majority of R users, particularly in academia, could not load a simple delimited data file without a high-level call such as read.csv.
(By “simple”, I mean a delimited text file that could be parsed with regex or even split. I don’t expect the average person to be able to write a parser that dealt with CSV’s actual complexity)
In my sister company, they have data scientists, and data engineers. The data scientists write their algorithms in the language they're most comfortable with (typically JS), and the data engineers rewrite to perform efficiently in the application that's applying them.
Data scientist and programmer are two very different specialisations.
Again, I don't know why, it struck me as odd also, but hey, whatever they're doing, it works for them.
I've heard people say similar things about MATLAB - that it's a poorly designed language, but many that people (mostly non-CS folk) use it out of convenience.
Can someone with experience using R explain what makes it so appealing to non-programmers? It seems like these two factors, "poorly designed" and "easy to use", should be at odds with each other.
Its all about availability of libraries. I did control systems in undergrad, and despite being a shitty language, being able to describe and manipulate dynamic systems (ODEs) was very useful. Doing numerical integration by hand for the nonlinear systems was horrible, though better than simulink (which is about as much fun as using LabVIEW or sculling H2SO4).
I started learning R about the time I started reading How to Design Computer Programs, and I found it pretty easy to transfer that model of thinking to R. And I find Clojure, Racket, and Scheme to also be somewhat comfortable after a short reacclimation period.
Some of the convenience bits have to do with most functions working on vectors without needing to explicitly iterate most of the time. Also libraries. If you want to estimate a linear regression, or make some exploratory plots, or try some rando statistical method that your graduate advisor suggests, you don’t have to worry about whether it’s already been implemented for you in R.
You can do a lot of heavy lifting by cribbing off of example code because most code is short. You just get heaps of leverage by using R.
Look, I like to do things the hard way a lot. My whole life is pretty much a string of highest friction path choices. For data science R is easy because all the work has been done for you. It's the difference between writing GUI apps against Cocoa APIs vs, I dunno, XLib or Motif.
By that definition, I would be a non-programmer, as I come from a statistics background, and though I have lots of experience in C++ and Python, most of my experience and work is in R. But that is by choice.
If I'm trying to create an application or build a website, I wouldn't use R. But when it comes to ingesting data, transforming and cleaning data, and modeling data, R is second to none. Yes, its syntax looks ugly and bizarre if you are used to object-oriented programming, software development, etc. In the context of working with data, I have never found anything in R to be even remotely confusing or strange.
On the contrary, the next best option to R would almost certainly be Python, and the gulf between the two is massive in my opinion. Python is a great general purpose programming language, but its data analysis capabilities, using packages like Pandas and sci-kit learn, feel like poorly designed, bolted on, and unwieldy. R is better for virtually every aspect of data analysis than Python.
So it isn't that R is poorly designed. Conversely, its very well designed, for its purpose as a data analysis-focused programming language. It only seems to be poorly designed to "programmers" because programmers work on problems that R isn't meant for. But that is like complaining that a screwdriver looks poorly designed for hammering nails.
Having seen brand new scientific programmers tackle two “beginner friendly” languages, man the differences after a day or two are stark.
This seems weird to me, I associate JS with web dev (and even more with the front end side of it)
Did anything come of this? I seem to remember seeing a paper or article where he proposed doing this, but I’ve never seen an implementation.
Edit: paper I was thinking of is https://www.stat.auckland.ac.nz/%7Eihaka/downloads/Compstat-...
I'm sorry, what? You know data scientists who use JavaScript to implement their algorithms?
It's a toxic perspective that ignores recent and pragmatic innovations in the field.
Funnily enough, although the first place that let me work with Hadoop and Spark didn't need to be using Hadoop and Spark, I probably wouldn't have worked there if they didn't let me learn them, so maybe this isn't as wasteful as it seems at first glance
Meanwhile, both demand that the employee spend all day telling a computer what to do.
>Even if the money were half of what today’s coder gets paid it might still be a better job because one is spared the tedium of looking at millions of lines of Java that do almost nothing!
What all those millions of lines of code are is abstractions, decoupling, and modularization of logic/responsibilty. This is hard-won knowledge from the field of software engineering. Granted, a lot of it is probably very poorly designed or organized. But the problem is the design and not the philosophy.
Because scientists all use the same basic rules of math, but each business will each have it's own special rules (i.e. not all payroll software implements the same policy/axioms), this makes it really easy for the hard logic of scientific work to be in a general-purpose library. "Normal" developers need to customize their own rules, or in other words, develop their own services unlike the data scientists.
Now if every data-scientist had to roll his/her own version of numpy, pandas, sci-kit learn, tensorflow, etc. the author would probably be decrying the deluge of procedural spaghetti produced by data scientists. The data scientists' notebooks look simple because of all that indirection is hidden away in the many libraries.
1. compound adjectives are hyphenated, except when the first world ends in 'ly.' 2. two adjectives, should have comma, similar to 'big, green systems.'
Crappy million-line Java apps are generally crappy not due to raw line count but rather due to leaky abstractions and badly designed APIs, so you do have to navigate through a lot of that code.
/mylife
It’s more likely boredom. Cranking out yet another CRUD app is more fun if you do it in a weird way. And a new “skill” for your resume.
I could talk about people unprepared for the jobs they do, and companies and managers setting unrealistic deadlines, having unrealistic expectations, etc. But I think most often than not, the issue is that to really build good code, you need to have a lot of experience and plan and design really well from the start. And simply, most companies and products start as they can. If you start with a deficient codebase, trying to fix that later under heavy budget and time constraints is pretty much impossible, and it's so accepted that some people believe it's the normal way to work.
And even with proper design, clear perspective, and good programmers, you still have to be lucky that no one higher up the ladder imposes decisions that f*ck up all the good work done. There are a few factors more.
What I mean is that you might find all kinds of people, but in general, for those of us who care about what we are doing it's the exact opposite. We consciously try really hard to keep code as simple as possible.
Yes, there are badly designed large system. No large system is perfect. However, there are also reasonably designed large systems, including in Java and Java is used in such system for a reason. It is more challenging to write large system. Yes, it is harder when parts of system are written in style that was considered best practice few years ago but was abandoned since then.
If you are spending a lot of time looking at millions of lines of Java that do almost nothing, them you likely dont really know what it what and need to read up more. At least that is my experience.
I am going through a process at the moment of documenting all of my local codebase. It will, in turn, be turned into a literate programming base. The problem I am finding is understanding all of the assumptions that underlay the original code. Why was it written this way or that, what is it trying to do, is the code actually doing what it is supposed to do?
There are, at present, some questions that I am having difficulty answering and this is my code. How much more difficult is it for someone to come in and look at a historical piece of code and follow what the original authors and designers were trying to achieve and what were the changes that have been made over time trying to achieve.
Documentation at the level we need to be able to adequately maintain any code base is just not done - it is very hard to do and to do so in a way that will help future people manage and maintain that codebase. On of my projects involves restructuring the code base. However, I need to understand the history of that codebase and that means talking with those people who are still living who knew the original authors and give an oral insight as to why things have been done the way they are. This oral history has to be written down and the codebase documented with it. When that information is in place, why the code is written the way and how we can now rewrite that code to be more effective is now achievable.
If we then put on top of missing history and documentation all of the bad designs, well, we are then facing even bigger problems. Then we put on top of that all the egos and politics involved, we get an even bigger mess.
So just reading up more doesn't actually help, because that which is needed was never written in the first place.
Of course it was, but you were tied to one system (no application server), security was login/pw, database had no constraint, typing systems were ultra limited, everybody has its own way of writing batches (no Spring), business code was mixed with tons of technical code (no JPA).
Now, sure, if you glue some R, some SQL, etc. you can extract insights worht millions of dollars. But all of that exist just because we have digitalized all of the processes, data collection, etc. And the rise of data scientists will continue only if there are more stuff put in the databases, thanks to you plain, regular, normal programmers...
IMHO there's far to little attention paid to how data might be valuable in an economic sense when storage strategies are being designed by database designers. I recently gave a talk at a developer conference and was really surprised at the level of pushback to adding more data elements or higher precision data "just in case it might be useful".
The preconception that you have to be maximally efficient with storage has led to huge quantities of valuable data being lost.
I 100% agree. Many programmers are trained as if memory, CPU are finite resources. Although it's true, in many cases, that reality may be safely ignored, opening tons of opportunities (to store data, to develop faster because you don't optimize, etc.).
At a time where my phone has gigabytes of memory, I'm always surprised that some people ask me to put a limit on a text field. I understand the technicalities behind the question, but from a conceptual point of view, that's often pointless.
At that point you need the abstractions and practices that make code boring.
If the philosophy prescribes dozens of tools for managing complexity and no tools at all for reducing it than it is the problem.
"Abstractions, decoupling, and modularization of logic/responsibilty" are not some kind of universal good. They are only useful within specific contexts. A lot of software engineers do not understand this and routinely engage in premature abstraction. As a result they produce systems that are 10 times more complicated than they need to be for absolutely no reason.
Java definitely encourages this kind of mentality, because the language itself and its standard library lack in some fundamental areas. Introduction of lambdas and streams helped, a lot, but the overall mentality is still well-entrenched.
I wouldn't hire a data scientist to build a web app (well, I would if he or she had the necessary knowledge and skills - the job title wouldn't be "data scientist" though). "Software developer" is much closer to "programmer".
There was less specialisation, less of a divorce between programmers and users.
There also seemed to be a conflation of computing and AI back then. Lisp was considered AI. And the early computing pioneers and theorists were strongly interested in AI, logic, and mathematics.
Genuine question - do people really believe that being able to write and understand complex SQL makes you a data scientist?
I ask because, I've been writing some of the nastiest, most difficult looking SQL around for probably at least 15 years. And yet, I would NOT call myself a data scientist because I know and can work with data and use SQL. It might make me a data engineer.
What would make me a scientist is the process, method and rigor I apply to data-driven research and in practice. It's not about what tool I use or how complicated that tool is.
I often get a whiff of imposter syndrome over this because, if being "great at SQL and R" is enough to get the big bucks as a data scientist, then I'm clearly doing it wrong. But, then again, maybe I'm being too literal thinking that a scientist means something different.
While I have been able to hold my own in this job I went back to school to pursue a graduate degree (partly) because being in the field has shown me how much more there is to know. While it's easy enough to train a simple model in R there are so many ways to fool yourself and produce an invalid analysis and so many variations on otherwise-simple problems.
It seems this field has a lot of variation. A glorified report writer might get the DS title but they're not going to get the really cool jobs.
If you're interested in data science try out a kaggle competition and try to place high. The variety of methods and tricks people try to improve their entries can be illuminating, I think.
Am I way off here?
Many data scientists use R and SQL, that does not mean that many of those who use R and or SQL are data scientists.
Many lawyers use word. Yes I’m not a lawyer just because I use word.
That said, the term data scientist itself is a bit frustrating. It gets thrown around a lot as if it is a well-defined role, and it is anything but. In my experience, the role of a "data scientist" is about as well defined as the role of an "engineer": it has connotations about the type of work and maybe a few shared skills, but the specifics of what an "engineer" does and their skillset varies widely depending on if they are a software engineer, an electrical engineer, or a civil engineer.
So while I think that most data scientists know SQL or use SQL frequently, I don't think that all data scientists use it, nor do I think that everyone who uses SQL works in a role that would probably be considered that of a data scientist.