Becoming disillusioned in the programming profession – please help
In short, after 12 years in the industry, I know all too well what it takes to build a software system. I now know that bringing a project to a reasonable degree of completeness requires crazy amounts of work most of which is not only not sexy, but close to downright menial. Startups' products are worst in this regard as 90% of them are some variation of CRUD web systems, but even if you try to dig deeper, the ratio of intellectual reward to sheer amount of work is terribly low. For example, want to experiment with programming languages? You must plough your way through parsing (a solved problem) and maybe some variety of VM to emit code for before you can even start actual work.
What counts as a personal project on sites like HN tend to focus on solved problems as well. People talk about developing web frameworks (a mostly solved problem) or similar things that would be interesting for a sophomore CS student but not for me right now. Problems similar to what Project Euler is about seem too limited, lacking any chance of impact.
This notion of impact is the key in how I feel. Contrary to what they write in Wired sometimes, it just feels like one programmer can't make any impact any more, that you're always a cog in the machine. In a big project you always work on a very limited subset of functionality, away from the essential parts of the product, just because there's no way one person can pull out the weight of the entire real-world production-grade system. And short of working alone there's no way to get yourself closer to those essential parts.
Please help me. I'm running out of motivation and I need to believe that it can be elsewhere somewhere around.
32 comments
[ 4.2 ms ] story [ 87.4 ms ] threadI'm roughly your age, and started programming around 10, doing my first paid work around 18 on a part-time-while-studying basis. I truly miss the days where one person could know a lot about a lot. Nowadays, it's a struggle to either know a reasonable amount (not even a lot) about some niche area, or even just be barely competent in a wider spectrum of things.
Where are you based? I was starting to think the problem might be limited to areas or regions (since almost everyone I know elsewhere seems so content/fulfilled).
A friend of mine used to refer to it as cowboy engineering - where a problem needs to be solved yesterday, and everyone does anything necessary to solve it. Nowadays, even on side projects, the overhead kills you - issue tracking, version management, module management, change tracking, support. Precious little time ends up being spent on the core problem in most jobs.
Small popular programs get extended, grow, are improved by large teams of paid programmers, and then eventually the category becomes so advanced that you need to have that large team to make progress. I can (and have) written Tetris in 3-4 hours, but there's no way I could write Starcraft 2. But that doesn't mean that small programs have become impossible to write; it just means that you have to start looking where other people are not. Look at the success of 2048 or Flappy Bird. There's probably another category-killer out there for small-business apps, but if someone's found it they aren't going to advertise it.
> it just feels like one programmer can't make any impact any more
Can you explain in more detail what makes you believe this?
Some of the most creative solutions redefine the problem. What you need to do to make an impact can often be less work than you imagine, like work that removes unnecessary layers.
If you can name a few project ideas where you feel it'd be impossible to make an impact, we can try to provide examples of how to attack them.
http://en.wikipedia.org/wiki/List_of_unsolved_problems_in_co...
Working on these problems could be done in parallel with a traditional coding job, and would give you no end of intellectual exercise, even if you didn't get very far with any of them.
Another option would be to pursue teaching. Introducing other people to coding & building things is incredibly rewarding, and will also possibly help your career.
And depending on where he resides, I've found that barriers to entry into teaching jobs can actually be substantial (assuming you're doing more than teaching older people how to use a mouse).
Also, you can teach as part of many day jobs. Everything from mentoring new devs to running Rails Girls tutorials on company hack days.
If you're in a position where you can apply your skills to help others in your work environment (or at least with their blessing outside that environment), that is gold! Sadly, such opportunities do tend to be few and far between, from what I've seen :-/
If you feel passionate about social networking, work on facebook or twitter. Hell, twitter's existence was almost solely responsible for enabling the Arab Spring uprisings in 2011 and 2012, resulting in some pretty widespread social changes.
If you feel passionate about Open Source, try working for RedHat. If you think AI is going to be the next big advancement towards human progress, try to get on at IBM or another research time devoted to it.
Are you really saying that there aren't any problems out there that you feel a need to solve, that there aren't any companies solving them that you'd like to work for?
I don't want to say anything is wrong with Facebook in this situation, it's just a company with a huge product and 7000+ employees. You are almost guaranteed to be re-inventing the wheel "Facebook way" there.
"In programming, the hard part isn’t solving problems, but deciding what problems to solve." — Paul Graham
Please read this blog post. Does it get at some of what you're talking about? http://erlebacher.org/2011/03/22/time-scales/
If so, I get it. Let's talk.
You could consider going back to graduate school, getting a PhD in CS and work on, say, algorithms or machine learning or something. That way you'd be discovering new technology. The odds of getting a job in academia are very low just because of overproduction of PhD's, but the degree is not likely to hurt your employment possibilities, and might even give you some new and interesting opportunities.
"It would be great if programming were a proper profession, but it isn’t. A profession has a structured model for advancing through levels of skill and ability, be it studying for a law degree and articles (working for a legal practise) or the years of undergraduate and medical training a doctor undertakes before specialising. The latter has clearly-delineated ranks, from junior doctor, via a brutal regime of 80-hour weeks, to consultant." http://dannorth.net/2011/01/11/programming-is-not-a-craft/
Why? If you're just trying to experiment with language, why not write a simple syntax tree interpreter? Save the full, production-grade implementation for when you've experimented enough to know more or less what you want to implement.
You do not need an advanced degree to work in technical support roles.
I would imagine that the programmer who works on the analysis of protein synthesis that in turn provides a cancer cure would get a buzz out of their contribution.
1. Making an impact
2. Solving challenging technical problems
These seem like separate problems that have little to do with one another (there may actually be an inverse correlation, see below).
If you want to make an impact then create a product that solves a big problem for humanity. Programming has essentially nothing to do with this, other than the fact software happens to be at the core of most, but not all, products these days. You can make an impact by understanding people's needs, coming up with a great solution, marketing it, convincing others to join you, and creating it. Programming will likely be a part of the above, but not the most important. And its importance will decrease with your product/company getting bigger (as will the impact any single programmer makes).
If you want to solve difficult technical problems then you can find lots of those in Computer Science research, either academic or industrial research labs. These are seriously challenging and can be fun to work on. However, I believe there is an inverse correlation between things that make an impact and things are are challenging research projects. Research projects are often so narrow that not enough people care about the results, and implementation in the industry is usually delayed by 5-10 years (less for industry research within the company).
To be honest however, I never really bought in the whole "IT/software is eating the world" mantra and was tirelessly looking for an exit plan throughout my whole time working in corporate IT.
However what fixed me wasn't the glory of moving to an exciting project or getting more pay, but taking time off and finding joy in what I'm working on in the moment. I know it sounds dense but I was getting depressed because I was getting older and didn't see myself progressing along like the dreams that people advertise here, like becoming a financially independent entrepreneur who influences the blogosphere or an influential programmer who has famous open-source project who is revered here.
I equated professional success in the form of my salary, prestige associated with the project that I'm working on, the intellectual rigor associated with the tech stack that I'm working with - with my self-esteem. And I realized that life is too short to chase after these things instead of choosing to work what you want to work on but not being so hard on yourself to stop you from enjoying the process.
I know that every time I switched jobs, I was angry and frustrated with my current one and aimed for higher salary and title to compensate for what perceived slights I've experienced. But ultimately it was self-sabotaging as I was responding to what others deemed to be successful instead of what I wanted to do. And beforehand, I was too scared to do what I wanted to do because I always thought I needed more money, more time and have other parts of my life ready to tackle the problem; but really, I was just scared of failure.
But I think failure is the common lot of most folks, despite of what the VC promotion is; but if you can accept that and be happy with you just having the opportunity to work on your own thing and follow through, I personally found that more rewarding over money and arbitrary prestige.
Long version: Same situation as you 6 years ago. Quit "safe", salaried job to work on freelance projects. But eventually realized that most freelance projects too are the CRUD type. Quit that too.
Then I found that fields like computer vision, speech recognition, space exploration and embedded systems still have many many unsolved problems, and require proficiency in theoretical subjects like linear algebra and algorithms, hands on challenges involving electronics and mechanical engineering, and practical software development problems like performance optimization. Every time I learn something in these fields, my mind bursts with new ideas and makes me explore and do more. It's an expanding spiral of unsolved problems. I don't think one lifetime is enough to solve my own ideas, let alone rest of the world's. It's a complete turnaround from how I felt 6 years ago.
Now I enjoy exploring and building my own projects and ideas, and have redeveloped my love for all kinds of programming - low level systems programming , desktop app programming, mobile app programming, even web programming.
I also learnt electronics on my own and thoroughly enjoy my hobby electronics projects.
Also how do you manage to find jobs like this? I found that hiring managers tend to, like, lie about the jobs. What I mean is there's no shortage of interestingly-sounding job descriptions out there that have nothing in common with what the hired person actually supposed to do. And you can't know the day-to-day reality of any job before you'll actually get it and then it's too late. How do you work around this situation?
My solution to that might seem extreme to many, but here goes: I simply stopped applying for any salaried jobs and ignored almost all the freelance gigs that came my way! The answer to your question is that I'm not getting paid by anybody. I don't even want to get paid by anybody. Working for others on _their_ ideas is something I find downright unpleasant.
When I worked for others, many tasks seemed "menial" / "time consuming" / "boring" to my mind. In the past, I did not realize that this unpleasantness is simply because I'm working on others ideas, instead of my own. Instead, I blamed it on the field of software engineering. I even remember confessing to a friend that I'd started to find programming boring. That was some 4 years ago. As I said, things have totally turned around for me since then. Now when I work for myself, everything becomes a new adventure from which I learn something more. I feel that the only way to get work satisfaction and make any impact is to explore the frontiers in different fields, work on my own, indulge my hobbies and build my own innovative products. It's definitely a risk, but luckily I had built up a nice nest egg during my salaried and freelancing days.
If all this sounds too radical or impractical to you, then you can gently start off by offering to develop a full fledged software solution for a small business you know about via your connections (somebody mentioned veterinarians above...I agree with that kind of thinking). I'd avoid the freelancing sites though - they are a whole different set of unpleasantness. In small business freelancing projects, you are usually the only developer doing everything, and so you will make an impact without any kind of restraints on your technical creativity. Depending on the luck and nature of that business, that impact may even be widespread in some way.
Of course, like me, eventually you may get bored of this too, since most such freelance projects too start off as CRUD type applications. But even when implementing such boilerplate, I kept my mind open for alternate solutions that involved unconventional fields like computer vision and speech recognition.
Many business processes today involve humans in the chain only because vision, cognitive thinking, reasoning and speech do not have practical software solutions. Imagine implementing a computer vision based system that is able to keep accounts automatically from handwritten bills, without requiring users to enter details. Such problems are most definitely not solved. I'm ok if all I end up changing is making one local business richer and happier...it's still better than zero.
I'm kind of doing the same thing. I quit a corporate job a few months back to work on stuff I'm really interested in. Doing a bit of blogging and hoping that maybe working on things I'm interested in and then blogging about it will get me a gig getting paid to work on things I'm interested in. But I suspect it's going to be a long slog getting to that money part. I'm fortunate, in that I've been frugal over the years and saved up so that I could continue on this path for at least another six months to maybe a year (well, insurance for myself and my wife is almost the biggest expense, almost as much as the mortgage). I'm just afraid that after six months or a year or whenever I can't afford to keep doing this that in the end nobody will have taken notice and I'll just have to go back to a soul crushingly boring corporate job again.
My earnings from previous jobs and contracts, and the returns on my financial investments, will keep me going for the next 5 years. I'm single, so it's relatively easy for me.
I'm currently writing an ebook and working on 2-3 software project ideas which I hope to sell to local businesses and earn revenue. I'm also preparing some training materials, since trainings pay a decent dime here.
My long term plan too is the same - software products, some electronics products, ebooks, and trainings.
In addition to waiting for a gig, perhaps you can also think of monetizing your ideas into a business to sell to customers.
There are diminishing returns to programming skill. Instead of focusing on becoming a better programmer (learning more languages, frameworks etc) you should focus on developing domain knowledge. Most of the time, programming is only useful in conjunction with some other skill. Find some time to develop domain knowledge in a fertile area.
Random example to illustrate. How do vetrinarians work? Maybe their day-to-day productivity would be enhanced by better software. If you have worked for 12 years, chances are you are better set with cash and can take greater risk. Why not get an idiosyncratic job like working in a vet's office to learn a new domain, or work in an IT job that would at least be partially exposed to a new domain?
Get off the web/orm/whatever framework carousel, work on a farm and build a carrot picking robot, become a secretary at a law firm, get a truck driving license. Stop coding for coding's sake and get in touch with the real economy. Then start coding.
Currently, enabling conventional processes through human network linkage via internet etc. is the big thing - think Amazon, Twitter, Facebook etc. Even this wave will end sooner or later
If I, you or anyone of our age group wants a real challenge, we need to create things which provide value above the already integrated human network(via electronic medium). I mean creating next Amazon etc. is not fun - the real deal lies in thinking what people will next need once they have such services available
PS: 1. Thanks for raising this question, I too have similar doubts - writing this response helped me frame things for myself better :-) 2. I don't think making impact through programming is viable anymore - my response hinges on that being assumed and understood
Also taking a step back and seeing how things could work differently is a nice idea. One of the most recent examples is Bret Victor, just his demos gave a rise to a lot of new cool stuff. Or take LightTable, it has quite interesting ideas for interactive programming.
Regarding start-ups, take for example TransferWise, it solved a business problem, and it started as a two man band. ZeroTurnaround with it's JRebel, solved a technical problem, but was conceived by one person.
So, yes, one person can make an impact, but it takes solving unsolved problems or noticing problems needing to be solved. Of course one person usually can't be a business, which is somewhat different from making an impact.
As a start I would suggest watching Alan Kay and Bret Victor videos.
The coding work is so much more enjoyable, only about 5% of the it is UI, the rest is all design and working the data. I absolutely hated how long it took to do the UI part of a crud app, so much that I steered my career around it.
Data work is so challenging because you truly have to understand every process you are taking data from and come up with creative ways to make that data useful to the business. The BI team at my work may not be the most advanced java guru's but they are some of the most creative solution designers you can find in our company. They are also the team with the most experience and the most knowledge about our super complex business processes because you have to understand them all to produce useful analytics.
I also can't stress how nice it is to have SQL as the most used tool in your toolbox. I scoffed at it as a simple data query tool before joining a data team. I now know it as the most efficient programming language I have ever used. Sure it can take a lot of mental effort to develop a complex query but it is so much faster than putting that logic in any other language. You simply ask a detailed question of the database and it returns an answer set to you. You do not need to tell it how to get he answer, the optimizer takes care of that for you and it does a much better job than you could ever do (99.9999% of the time). No more coding endless series of loops. A couple dozen lines of sql could take thousands of lines to code in java/c#/ruby/python etc.
There is also a whole new world of data science opening up right now and corporations are cashing in. I will be learning hadoop and map reduce very soon so we can position our team to offer a competitive advantage to our business through big data. I will have to learn R and python because they are used to gather and work all that data. The future is looking bright for data workers.
Yes, real-world software projects that make money for a business (and pay you a salary) are larger than one man can handle. There's a lot of grunt work, and maybe only a handful of new and interesting things to do. At some point, you're going to have to accept this, and just get on with it. If you work for a business, the goal is to make money, not to work on new, kewl, glamorous software projects all the time. Sometimes those goals do intersect, but it's rare. It's called 'work' for a good reason.
It also sounds like you're suffering from another common malady, and that is the idea that programming is all about writing code, and programming languages, and frameworks, and fast turnaround results. I think you're starting to realize that this is not really the case, but you can't figure out why, so you suppose that maybe it's due to the insurmountable grandioseness that seems to surround even the smallest programming project. I think you've grown up under the spell of "programming as an end in itself", and now the illusion is showing some cracks.
Look, programming is not an end in itself. It's a way to express the solution to a computational problem. Think about that for a minute, or, preferably, the next week.
Programming languages are basically the same idea as mathematical notation - it's a way to express an idea about how to solve a problem. Sure, there's also the coolness of having a machine take your expression of how to solve a problem and actually solve it for you, but programming is still just a way to express the solution to a problem.
Look at all the posts we see everyday on HN and proggit about programming languages and frameworks and whatever the latest hare-brained language idea that the functional programming circle jerk fest has come up with. Man, it's like going to a math conference and all everyone talks about is the latest ways to write down algebra! It's no wonder that there've been no real fundamental advancements in computer science in the last 30 years.
I think you need to rewire your brain about the philosophy of programming, what it is, what it's for, and what it's not. You are at a crucial point where you need to decide if you're ready to go up an abstraction layer, and start thinking more deeply about the various problems whose solutions may be expressible in a computation form, and stop worrying about programming languages, frameworks, and "impact". If you don't want to do that, then you might as well just be happy laying bricks and comparing trowels, and leave the hard thinking to more interested and capable people.