Ask HN: Where are the hard programming challenges/jobs?
I got started programming a couple of years ago because I like problem solving. The harder, the better. I know I'm not alone in feeling that way. I've gotten a bachelor's degree and learned a bit of ruby, rails, and other frameworks/languages, but I'm running into a problem. It seems like all jobs are just simple CRUD apps. I look at 99 percent of the software out there and know that I could do it if I had the time.
My question is how do you get to work on the hard/fun stuff? A quintessential example for me would be self driving cars. It's challenging and would have a real impact on the world.
I'm looking into becoming a data scientist because it's newer and seems to have more challenges. I also have a theory that any challenging computer science problem requires a lot of math. Do you need a PhD to work on these hard problems? Can anyone give advice on how to avoid a career of working on simple CRUD apps? (CRUD is a metaphor for simple problems in this case)
70 comments
[ 2.7 ms ] story [ 138 ms ] threadYou can work on hard projects on your own. Once you have made substantial contributions, you will have credentials to interest the employers. Most of the open, hard projects have very few full time developers. PyPy has only 3 I believe. If you have something to contribute, people will be more than happy to have you on-board.
If you really want to challenge yourself give yourself a goal like:
1. Decode the genome 2. Find the correlation between all aspects of someones lifestyle, diet, and their health. 3. Solve poverty - maybe there is an economic solution.
The world is full of hard problems, but you have to take the lead if you want to work on them.
It will involve acquiring some domain specific knowledge to be truly successful (what hard problem doesn't?), but you will be able to dive in straight away and make huge contributions just by making current analysis more efficient!
I would love to know as well.
In short, getting domain knowledge in the above are essential to solve the tough problems at the big-pharma or national-bio-labs level.
(PS to the other poster looking for bioinformatics programs: I suggest you look in Boston - Tufts, MIT, Harvard etc)
Billions of people around the globe haven't even noticed many a problems, yet a handful of entrepreneurs did, wasn't that the first part of problem solving, and if you haven't probably noticed, you are here after all, so you might unconsciously know the answer lays somewhere around here.
So from my eyes, whether its programming or maths, the hardest problems that exist are the problems no one has noticed yet.
For your case, you might want to find a problem of programming, if you have programmed for that many years, you will surely run into one, that no one has noticed. If you haven't, as Steve Jobs also said "Keep Looking"
If you want to tackle tough challenges may be ruby/rails is not the ideal place. Perhaps learn C,C++ and implement faster ruby or javascript interpreter or implement a distributed filesystem in ruby or java. You can also contribute to the rails framework.
This is a rather crude generalization, but PhDs are often demanded at high end research positions not to prove you know your stuff, but to guarantee you have the mettle to do research and bounce back from failure after failure.. a must if you're working on someone else's dime. Of course, if you can find another way to support yourself, you can do (almost) any research you like in your own time without strings attached ;-)
You'll need two things to get started doing this sort of thing: standing out as a member of your community so people come to you, and an understanding of business needs.
If you really want to make the switch, shoot me an email. I'll give you some tips on making the transition.
The way I see it people choose what they want to spend their time doing. Sometimes they choose research, sometimes they choose game playing, sometimes they choose reading Hacker News :-) but there is generally enough time for folks to do what they want to do.
Research is especially easy if you know what you want to research. But it's not a simple matter of "Hey I'm in research, lets blue sky something!" People in research have some passion that they are following, like parsers, or graph theory, or AI, or network coherence, or something. Some area where they have lots of questions and no answers.
Keep a notebook. One for each topic if you want, they can be the cheap composition ones that go on sale during 'back to school week' or really nice lab notebooks. But they are like pitons and rope to a climber, they anchor where you have been and where you are going. Write the questions you investigating into the first few pages after the index. Start writing down what you know, what you can prove, and what you don't know. Invent experiments or ways to discover the answers to the questions you don't know.
I've got a notebook with a couple hundred pages devoted to making carbon nanotubes. I know lots of ways you can't make them and several ways you can. Sometimes you can scrounge equipment for stuff, sometimes you have to invent (did you know that you could use a bottle of propane and a chunk of copper pipe as a way to do vapor deposition of carbon? You can! And if that copper tube rolls off the stand onto your wooden deck when its hot it leaves a big black mark? :-)
"Research" doesn't need a government funded lab, it needs good laboratory technique. Careful note taking, documentation, exposition and analysis. Software research is even easier these days because of how cheap computers have become.
In general, having a job doesn't prevent you from asking questions or doing experiments (aka "research") but if you're tired after working all day and just don't want to think any more, that is a different problem. Some folks I know get under demanding jobs just so they can do it easily without straining too hard during the 'day.'
One of the things I did as a kid that drove my Mom nuts was read every scientist biography I could, I tried to re-create their experiments because anything an adult could build in the 1800's I figured I could build as a kid in the 1900's. But what it really taught me was how these people pursued the questions they had.
Bottom line, "money and free time" isn't an excuse, its a rationalization. Why are you denying yourself your own research?
Maybe you won't be that lucky. But if you don't have the required skills, then you definitely won't be that lucky.
Incidentally a fun place to test the intersection between math and programming is http://projecteuler.net/. Besides, it is fun.
http://arxiv.org/abs/1302.3860
But my advice is to pick hard problems yourself and try to solve them. You need to start somewhere! Sure its expensive to get your hands on driving car hardware, but there are plenty of hard problems out there that need solving and can be done using opensource and your personal computer.
If you are able to get yourself through a really tough problem, you will come out better on the other side - even if it takes you years to solve.
Consumer-facing interactive native apps are typically low on CRUD and some of them require non-obvious solutions. You could try to get a job (or just create a product on your own) on mobile or in games.
Another option is working on tools instead of apps. If you manage to get some non-trivial gcc/clang/kernel patches accepted you can try to find a company that works on those.
(I've worked on Google Maps for Mobile and am now building a robotic waste recycler - no CRUD for me since 2002 unless I've decided to build some for my own needs).
We have to invent new things everyday, and due to our focus on hacking the digital and physical worlds there's so many possibilities.
Sometimes those problems are with product design and UX. Sometimes those problems are mathematical. If you are successful, often they have to do with scalability and reliability. If you work with great people there are always hard problems in enabling productivity and building great tools. Many companies have serious and significant challenges around security and may not even know it. If most companies are solving simple problems, then why aren't most companies automatically successful?
If you broaden your definitions of "hard" you might find that there are many things that qualify. If what you mean is "hard mathematical problems" then yes, you will have to learn some math.
The pay sucks compared to other development, and work hours are crap, but you will be able to work on AI, gameplay, Graphics, physics (whatever your experience is).
But in all honesty, if you started programming a couple years ago, and learned a bit of rails/ruby, you have a lot to learn in various languages and systems (try C, embedded, iOS/Android, learn Haskell, big data, etc)
Games are hard in that you're always trying to squeeze more into severely constrained resources, so you end up having to produce convincing but fake approximations of algorithms that would be far too expensive to run in their "real" forms.
Travel search was algorithmically very hard (see the few papers by de Marcken about this for a glimpse into this world).
Email is hard because it combines many disparate skills (design, UX, back-end scaling, protocol details) and because the MVP is incredibly complex.
All involved tremendous amounts of slogging.
If you think you can build 99% of the apps out there, go out and try to build a moderately complex app. Make it better. You'll see there are all types of challenges from the technical, to the social to the political in small and large companies. Being a software developer is about solving all sorts of challenges, and not all of them are technical.
The OP seems to consider getting a PhD an acceptable route. He's asking for directions, not a shortcut.
There are infinitely many hard problems out there. If you want to stay motivated, work on stuff that matters.
http://radar.oreilly.com/2009/01/work-on-stuff-that-matters-...