Development - Jupyter IDE for Python
Text Editor - Sublime Text
Knowledge Base - Stackoverflow
Visualization - Matplotlib + Excel Charts
Presentation - PowerPoint
The software tools: For me, PyCharm and Jupyter for development, Dask for data preprocessing, Tensorflow, Theano, and various scipy libraries for ML, as well as helpers such as PyMC3 and Edward for probabilistic programming.
The important tool: a PhD in applied mathematics that taught me how to think originally and creatively about problems, to abstract and codify them with mathematics, and to approach them numerically and computationally.
I suspect that the answer you were looking for is the first set of tools. This set is extremely nice to have; the second set, at least for me, is essential.
Nevertheless, 80/20 (80% solution in 20% time) does wonders with the right set of tools and a rudimentary appreciation of applied maths, 80/20 being often what businesses really need. Unpopular as it may be, the latest tools come pretty good as black boxes, so that creativity is shifted from maths to application domains. No need to scare the OP, then, he is not a direct threat to your job.
> No need to scare the OP, then, he is not a direct threat to your job.
I find it a bit disheartening that this is how you read my response to OP's question. The denizens of HN may in general disapprove, but I believe my education to be one of the most important factors in doing my job. I don't think it's fair to respond with thinly-veiled accusations of elitism, especially when I explicitly tried to also give him or her a "practical" answer.
> Nevertheless, 80/20 (80% solution in 20% time) does wonders with the right set of tools and a rudimentary appreciation of applied maths, 80/20 being often what businesses really need.
In some cases, you're right, but more and more often I see people assuming that they've found the "80% solution", when in reality it's something like a 30 or 40% solution. My personal experience is that with many problems, a bit of thought shows you that many seemingly simple problems are hiding both a lot of complexity and a lot of potential - and I've seen real money left on the table because of a failure to recognise this.
My opinion is that conceptualising the problem in the first place is what allows you to see really what an "80/20" solution looks like, and in my case I don't think I would be able to do that without my most potent weapon: my mathematical education.
Not every problem is like that: sometimes a simple, out-of-the-box solution does work really well, and solves a concrete business need. My point is that when this is about the limit of what you can do, you don't really know whether your problem is in this class or not.
It was not elitism my constructive point but this: "Unpopular as it may be, the latest tools come pretty good as black boxes, so that creativity is shifted from maths to application domains." There are several tech fields and all the human sciences that are directly benefitting from an 80/20 data science approach and these are the new domains being added, they also have higher ceiling if you want.
Thank you for a very interesting and informative answer.
When you say "PhD in applied mathematics taught me how to abstract and codify with math, and to approach numerically and computationally", do you feel you already had good mathematical intuition earlier too and the PhD made it better? Or did the PhD by itself help you build such skills from scratch?
I too would like to have such skills one day, but I don't have the opportunities or time for a PhD. Do you feel it can be self taught to some extent?
> do you feel you already had good mathematical intuition earlier too and the PhD made it better? Or did the PhD by itself help you build such skills from scratch?
I really think that the PhD experience was the major factor in me developing this, though I think I showed a certain promise earlier on. I think it's certainly true that, without the PhD, I wouldn't have known how to go about the process of learning and thinking in the way that I do now.
> Do you feel it can be self taught to some extent?
I think you'd have to be a much smarter and more dedicated person than I am to achieve the same results without the environment, and as I mentioned in a cousin comment, I think that environment might be tough to replicate. I am a big proponent of formal education for exactly this reason. Academia certainly has many flaws, and I'm a harsh critic of them, but I personally experienced what I consider to be the best side of it, and it's this side that formed the opinion that I'm expressing here.
EDIT0: Repeating what I posted in a cousin comment: Also worth noting that by far the smartest person I know - a deep thinker, also nominally a data scientist, but foremost a truly original and creative mathematical thinker - didn't complete his PhD and left with a Masters. Intellectually, this guy makes me look like a geriatric Corgi on opium, and he didn't need a PhD to make him that way.
Thank you, laingc. Due to my circumstances, I wish the answer was not formal education. But I'll remember your experience and advice, and see if I can go that route atleast in the future. Thank you very much for replying, have a great day!
The important tool: a PhD in applied mathematics that taught me how to think originally and creatively about problems, to abstract and codify them with mathematics, and to approach them numerically and computationally.
What is a first approximation to this experience, particularly the creative thinking portion? As someone with a MS but without a PhD, I suspect you get to flex this ability during the research phase of your studies.
I'm note really sure what a good first approximation would be, but maybe I can break down the elements of it a bit, and we can see whether there is some other experience that mimics it.
* Spending several years, uninterrupted and undistracted, thinking deeply about a mathematical topic.
* Being advised and supported, but not led or coddled, by an experienced researcher.
* Being surrounded by clever people, who do research full-time.
* Working in an environment where not knowing the answer. immediately, or even quickly, is not just accepted by expected.
* Being "left in the dark", with the understanding that the process of being in that position and finding your way out is the very thing that you are trying to learn.
* Having rigorous expectations for the "final result" (to the extent that one can be found).
I can't think of a great approximation to that, but perhaps someone else can chime in.
EDIT0: Formatting
EDIT1: Also worth noting that by far the smartest person I know - a deep thinker, also nominally a data scientist, but foremost a truly original and creative mathematical thinker - didn't complete his PhD and left with a Masters. Intellectually, this guy makes me look like a geriatric Corgi on opium, and he didn't need a PhD to make him that way.
Out of interest, is everyone using Spark/Hadoop/etc because you need to and/or you chose to or another reason? IMO, legit use cases seem to me relatively sparse.
Let's say you're developing your analysis script, and it takes 45 minutes to run. That doesn't sound like much of a problem, but it means you only really get to try 10 things a day. You can kick something off and then start trying something different, but that's actually quite a burden. It's cognitively easier to run a series of parallel tasks instead of parallel serial tasks. Even if a job is taking 15-20 minutes, that can really wear you down, and give you a pretty unproductive week. It's hard to keep your place.
That said: I think often these 45 minute tasks should be much faster, and folks would actually benefit from optimising their code a bit more. If I have to do a word count in 30gb of text, I'd rather implement carefully or use Hyper Log Log. I find it weird that people would rather use Spark, which I always find quite painful. I guess it comes down to familiarity.
I run models on large scale user data (for recommendations). A couple of hundred gigs is what "cleaned" data looks like. This might need to be joined with metadata. This cleaning, joining is easier to do on Map Reduce than wait a few days on a beefy computational machine.
Much of the data I analyse has a spatial/geometric/geographic component and for that I wouldn't want to do my job without FME. In fact it's probably may one of my all time favorite pieces of software.
Otherwise jupyter with numpy and all the other great python libraries is where I spend much of my time.
17 comments
[ 3.4 ms ] story [ 71.7 ms ] threadDaily I'd say RStudio, Excel, Tableau, PowerPoint. Either I'm coding in R or I'm presenting.
The software tools: For me, PyCharm and Jupyter for development, Dask for data preprocessing, Tensorflow, Theano, and various scipy libraries for ML, as well as helpers such as PyMC3 and Edward for probabilistic programming.
The important tool: a PhD in applied mathematics that taught me how to think originally and creatively about problems, to abstract and codify them with mathematics, and to approach them numerically and computationally.
I suspect that the answer you were looking for is the first set of tools. This set is extremely nice to have; the second set, at least for me, is essential.
I find it a bit disheartening that this is how you read my response to OP's question. The denizens of HN may in general disapprove, but I believe my education to be one of the most important factors in doing my job. I don't think it's fair to respond with thinly-veiled accusations of elitism, especially when I explicitly tried to also give him or her a "practical" answer.
> Nevertheless, 80/20 (80% solution in 20% time) does wonders with the right set of tools and a rudimentary appreciation of applied maths, 80/20 being often what businesses really need.
In some cases, you're right, but more and more often I see people assuming that they've found the "80% solution", when in reality it's something like a 30 or 40% solution. My personal experience is that with many problems, a bit of thought shows you that many seemingly simple problems are hiding both a lot of complexity and a lot of potential - and I've seen real money left on the table because of a failure to recognise this.
My opinion is that conceptualising the problem in the first place is what allows you to see really what an "80/20" solution looks like, and in my case I don't think I would be able to do that without my most potent weapon: my mathematical education.
Not every problem is like that: sometimes a simple, out-of-the-box solution does work really well, and solves a concrete business need. My point is that when this is about the limit of what you can do, you don't really know whether your problem is in this class or not.
When you say "PhD in applied mathematics taught me how to abstract and codify with math, and to approach numerically and computationally", do you feel you already had good mathematical intuition earlier too and the PhD made it better? Or did the PhD by itself help you build such skills from scratch?
I too would like to have such skills one day, but I don't have the opportunities or time for a PhD. Do you feel it can be self taught to some extent?
I really think that the PhD experience was the major factor in me developing this, though I think I showed a certain promise earlier on. I think it's certainly true that, without the PhD, I wouldn't have known how to go about the process of learning and thinking in the way that I do now.
> Do you feel it can be self taught to some extent?
I think you'd have to be a much smarter and more dedicated person than I am to achieve the same results without the environment, and as I mentioned in a cousin comment, I think that environment might be tough to replicate. I am a big proponent of formal education for exactly this reason. Academia certainly has many flaws, and I'm a harsh critic of them, but I personally experienced what I consider to be the best side of it, and it's this side that formed the opinion that I'm expressing here.
EDIT0: Repeating what I posted in a cousin comment: Also worth noting that by far the smartest person I know - a deep thinker, also nominally a data scientist, but foremost a truly original and creative mathematical thinker - didn't complete his PhD and left with a Masters. Intellectually, this guy makes me look like a geriatric Corgi on opium, and he didn't need a PhD to make him that way.
What is a first approximation to this experience, particularly the creative thinking portion? As someone with a MS but without a PhD, I suspect you get to flex this ability during the research phase of your studies.
* Spending several years, uninterrupted and undistracted, thinking deeply about a mathematical topic.
* Being advised and supported, but not led or coddled, by an experienced researcher.
* Being surrounded by clever people, who do research full-time.
* Working in an environment where not knowing the answer. immediately, or even quickly, is not just accepted by expected.
* Being "left in the dark", with the understanding that the process of being in that position and finding your way out is the very thing that you are trying to learn.
* Having rigorous expectations for the "final result" (to the extent that one can be found).
I can't think of a great approximation to that, but perhaps someone else can chime in.
EDIT0: Formatting
EDIT1: Also worth noting that by far the smartest person I know - a deep thinker, also nominally a data scientist, but foremost a truly original and creative mathematical thinker - didn't complete his PhD and left with a Masters. Intellectually, this guy makes me look like a geriatric Corgi on opium, and he didn't need a PhD to make him that way.
That said: I think often these 45 minute tasks should be much faster, and folks would actually benefit from optimising their code a bit more. If I have to do a word count in 30gb of text, I'd rather implement carefully or use Hyper Log Log. I find it weird that people would rather use Spark, which I always find quite painful. I guess it comes down to familiarity.
Otherwise jupyter with numpy and all the other great python libraries is where I spend much of my time.