Ask HN: How to transition from academic programming to software engineering?
I taught myself how to code and ended up doing a PhD in a computational discipline. Programming has been a big part of my life for at least the last decade, during which I've written code almost every day, but always by myself. After graduating I joined a medium-sized company (~10^2 developers) as a machine learning engineer and realized how much I don't know about software engineering. I feel very comfortable with programming in the small, but programming in the large still feels mostly opaque to me. Issues like testing / mocking, code review, management of dev / stage / prod workflows and, most importantly, the judgment / taste required to make maintainable changes to a million LOC repository, are areas where I can tell I need to improve.
Former academics who moved into software engineering, which resources did you find most useful as you made the transition? Python-specific or language-agnostic books would be most helpful, but all advice would be welcome.
103 comments
[ 2.5 ms ] story [ 162 ms ] threadThe number one thing you can do is read through other people's code. If your colleagues are very good then you will learn a lot and pick up good habits, if they are so-so then you will build your self-esteem and sharpen your critical thinking skills. Some developers are shifty, and others love to talk about what they are doing and share insights. Spend time with the latter type.
Don't try to be an expert at everything, most teams should have self-selected individuals that choose to specialize in different areas that the team depends on.
Simply by asking relevant questions, practicing some things a bit, you'll get used to it.
And consider that it's different in every organization, and nobody does it really well frankly.
Given the pace of change, the varying technologies and flavour-of-the-month processes, it may always seem a little unwieldy an opaque: the feeling of 'not knowing everything' never goes away.
And I concur with the itronitron: read other people's code on the team, who are known to 'code well'.
It does not mean that algorithmically it's genius or even good - it just means that those are the styles/patterns that may be expected of you. It's like learning to say certain words a certain way. You'll get it soon and then forget you're doing it.
Don't fret you'll get all of this quickly.
One practical tip - take a look at Dan Bader's book if you are deep in Python, it has a lot of good stuff.
One philosophical tip - depending on your organization, remember that slow is fast in engineering. This is somewhat different from more academic computing environments (at least that I know of). So take the time to get it right and really deeply understand your solution.
Coming from an academic field myself... be careful with this one. Depending on your personal habits in academia, you might have to learn the opposite - your code doesn't have to be perfect, it has to work.
Be careful not to end up overthinking the code/design and under-delivering on the timeline. Missing time estimate once in a while is usually OK, missing it consistently and by a lot might become a problem.
It took me 10 years to be able to skip all the technicals. And another 10+ years to understand why u may ever need the rest..
for judgment etc... Maybe pick some big-enough open-source project in a domain u know well and follow it - how and when they do change what. Dont worry, it does take years to really form your own judgment.
btw u will need some philosophy/methodology/human-side too.. there's not many of it in the above book.
For more, see the recommended readings on www.svilendobrev.com/rabota/
have fun
* testing - write your functions small enough to be readable, but not so small their abstractions are meaningless (because you have to test them all)
* testing - don't reach into your code's modules and mock. Instead use dependency injection with non-testing defaults
* code review - It shouldn't be personal, if it is are you reading it wrong or are they attacking you personally?
* code review - when referencing style complaints ask for reference material. Don't get caught in cyclic-pedantic style war between lead devs.
* code - your code should be environment agnostic, if you have environment/context specific things to do, pass along a environment/configuration dict or make a global config singleton. As long as your code depends on that you can write code more discretely.
* code - personal preference but try to not nest your loops too deeply, when you can use itertools.
* code - if you can help it, try not to mutate dicts/objects in place while in a loop. Makes testing a difficult.
* code - exit early if possible, test for failures instead of nesting your entire function inside a single `if`. Helps identify the bad inputs faster as well.
Above all, remember code isn't perfect. It's a tool to get to an end goal. If you aren't solving for the end goal you aren't solving the right problem. At the end of the day, you are employed to build a product and that product needs to perform it's job. (that isn't a pass to write super shitty code)
edit: formatting
I would also go a bit further and put nesting if statements. Sometimes it's really required but other times nesting can be avoided. I try to avoid nesting as much as possible.
This is also true of branches. If you have an if statement (or other branch), consider extracting the contents of each half out. It allows you to test each half independently. If you have nested branches, you need to have 2^n tests (where n is the number of nested branches). If you extract the contents, then you need 2n tests.
This is one of the reason for unit rather than integration tests: you can dramatically reduce the number of tests while still getting full test coverage. Of course the downside is decomposing the structure of the code more than you might be used to. It's always a judgement call.
Could you please go into more depth with this?
Why external dependencies are a good thing to mock is not just related to testing, it's related to code rot: external dependencies tend to change under your feet, so isolating them from your code base is a good thing, especially if you want to minimise vendor lock in. And for tests, it helps you simplify the test environment.
Internal dependencies aren't like that: they only change if you change them, and they're already part of your environment. Mocking them just complicates your code without simplifying your tests.
---
Now there is this idea that modules should be tested "in isolation". Imagine you have three classes, A, B and C, such that A uses B, and B uses C (and C is self contained). Without mocking, testing A will automatically involve B's and C's code as well. Mocking lets you test A alone, without the distraction of B and C.
This is mostly bullshit. If A depends on B all the time, there's no point into making B a parameter (dependency injection). It's simpler to just hard code the dependency. As for the tests, who cares about isolation? If A is only used when it depends on B, there's no point in testing it under other conditions. More generally, it is okay to pull in the transitive closure of dependencies when you test a component.
Of course, you should have tested each dependency (C, then B) before you test your module (A). That way you can mostly ensure the absence of bugs in the dependencies. For instance, if the tests for C and B succeed, and the test for A fails, the bug is probably in A. The order of the tests also matters for the test suite: if the tests for the dependencies are ran first, the first failed test should point to the failed module (instead of a user of the failed module).
Now if your program is highly stateful, you might have to mock. Testing a highly stateful program without mocking requires setting lots of bits of states, and making sure all the pieces of runtime state interact with each other as they are supposed to. But then the problem isn't the lack of mocking, it's the fact your program is highly stateful in the first place. Make your program less stateful, first, then it will no longer need mocking.
---
There is this mistaken notion that we should mock because if we don't, we're no longer making unit tests, we're making integration tests. The answer to that is simple: unit tests aren't the holy grail they are touted to be. Don't force them. If the dependencies are well tested (which they should be), then you can trust them like you would the standard library.
Besides, the integration tests will catch more bugs for less effort than pure unit tests (with mocking) would have done. Don't waste your time with mocking, just go catch the bugs.
In your main or context, you setup your application with the needed pieces.
More precisely, you want your functions to be deep. The interface to implementation ratio must be as low as is reasonable: you want to hide implementation behind interfaces that are much smaller than them.
This goes for functions (something with 5 arguments that takes 2 lines is too shallow, something with 2 arguments that spans 20 lines is deep), classes (something that is mostly getters and setters is too shalow, something with 3 methods full of business logic is deep), or anything else.
This is not just for testing, this is to make sure you can understand the program at all. Deep functions (and deep classes, and deep modules…), are also about good old decoupling: the less you have to understand about a piece of code to be able to use it, the better.
Hitchhiker's Guide to Python is a very good book (freely available online, or get a copy from O'Reilly); some of it may be obvious but some might not be.
It is true IMO that making your code testable will also make it better designed. It might even be worthwhile to do completely dogmatic test-driven development (i.e. always write tests first, then stub out everything with NotImplementedError, then write actual code until all tests pass) for a while to get used to it, and force yourself to become familiar with tools for dependency injection/mocks/etc.
This is complicated by the fact that unit-testing machine learning code can be unusually tricky; normal unit-testing practices and metrics (e.g. code coverage) may not be very effective.
There are, say, physics PhDs who only write numerical Fortran or C++ routines (in one big file, sometimes even in one big function), who really might want to attend a boot camp or something but it doesn't sound like you're in that boat.
Other than that - there is not that much theoretical basis. The core principle is that software engineering is about dealing with a situation where you have far too many variables to fit into a single persons working memory, and how to organize a group of people in a way that they can co-operate without turning the thing into a mess, really fast.
It's more about having a set of understood processes, rather than what those processes really are, so that people can communicate about and co-ordinate their work effectively. Of course the processes need to make sense, but there's no "silver bullet" process that either would fix everything, or, conversely not following would lead to the end of the world.
"Issues like testing / mocking"
Testing has sadly developed bookish dogma around itself. But it's extremely practical. The most important automated test is the high-level integration testing - will this and that work when the customer uses it.
Unit test are about creating enforceable rules to the production system, which makes thing break faster, and, hence, faster to fix.
You don't want someone else to accidentally to break your code - especially that kinda weird cornercase? Write a test - now the rule becomes enforced as a part of your domain model.
" code review,"
Same principle as in writing text. Having someone proofread the things you write generally improves the quality. No need to be dogmatic.
The second aspect is the zenlike increase in code quality. People know that their work will be looked at by someone, hence they have a higher intrinsic motivation not to fudge things.
"management of dev / stage / prod workflows"
The only thing that matters there is that there is one agreed process inhouse. Otherwise things turn really messy, real fast. It's kinda tricky to wrap ones head the first time around the ramifications of the chosen rules, so that's why there are lot of published ways of working .
"the judgment / taste required to make maintainable changes to a million LOC repository,"
"Working effectively with legacy code" by Michael C. Feathers is a good start. Now, if the corpus of code has a thorough integration and unit test suite, you can change things, and if you accidentally break something, the tests will tell you.
If there are no tests, then, better start writing them. You can't do any large scale modifications - especially to production code - without them.
Have some tool that automatically tells you the test coverage.
* "Refactoring" by Martin Fowler would probably help with writing good quality code and doing code reviews (or understanding the reasons for changes requested in others' code reviews).
* In my experience, "academic" code tends to be far more prone to very long functions. Understanding the Single Responsibility principle was a very important part of the transition from academic scripts to software engineering for me. If you regularly write functions of 30+ lines, start looking at breaking these down better -- what are you actually doing in each chunk of code? Can it be broken down further.
* In Academia, building software is 90% coding. Now, reading code will be far more important, and you might even spend more time reading code than writing code. Relatedly, the readability of your code is now the most important thing to optimize for (sometimes even at the expense of computational efficiency, you should aim to reduce the number of developer-hours spent wherever possible). This means writing more readable code (good variable names, learning and following style guides and other conventions, following a process even if it feels like a waste of time), using better tooling, more focus on testing, more time documenting, and more time communicating with programmers than actual programming.
* At times this will be frustrating because you'll remember when you could just go into the zone for several days straight and produce something fairly significant. Remember that a lot of the stuff that feels like a waste of time isn't - it's necessary to get out of the local maximum of what was possible when you did everything yourself and could keep the entire project scope in your mental model at all times.
* Whenever you have written code and are 99% sure that it will run fine and not break any other parts of the system ("it was such a small change"), rather assume that it is very likely to break something else. I still find I constantly need to recalibrate my confidence that what I'm changing has a limited area of affect.
* Spend a decent amount of time getting more familiar with things like source control. Assuming you use git, go down this rabbit hole to read about the different git workflows and get as comfortable as possible with flows that your team uses, dealing with conflicts, reversing mistakes, etc.
* Most of what you specifically asked for can only be learned through experience. Experience is gained the most quickly by doing, even if you make mistakes. For software quality, the best (only?) way to learn is to have someone more experienced review your code. The more pedantic they are, the more you will learn. If it takes you 3-4 attempts to get a code review through, then you have team that will accelerate your learning. If you're getting mainly LGTMs, you might be super competent and had no need to ask this question, but more likely you need to try find people to push you harder to learn new standards.
So try to write code for an audience. This has been the trigger for me. Also I encourage code reviews and TDD.
The main learning resources for me have been, Clean Coder videos by Robert C Martin aka Uncle Bob. They are pure gold. They can feel awkward, but after a while they make sense.
Also DDD domain driven design is a key topic to tackle.
Books: - Clean code - DDD by Vaughn Vernon
Videos: - Clean coder E1-E52
With these two books and videos you are on a good track! These worked for me.
We don't always hold ourselves to it, sometimes 5-6 line functions make sense, but we strive toward 4. Sometimes it's as easy as breaking code out into a new function, but sometimes you just simply have to create a class for it. That way a lot of complicated code suddenly becomes very easy without much effort.
Limiting function length to < 10 lines is like limiting your walking stride to 30cm. You'll spend most of your time chasing useless functions when trying to understand the system.
Of course we don't count every bracket or blank lines. Only the rows with any logic or assignment.
And yes, I agree, there are occasions where the complexity goes up. If there is a good argument for that, then we of course go with it.
But so far, almost everything anyone in our team has made, has been improved by rewriting it to something that works well in 4 lines, be it using polymorphism, object oriented, or functional.
test third-party libraries. it's uncommon to find bugs, but it's not so rare that it happens only to others.
don't forget to leave comments. a lot of my code review questions could be answered (hence could be not asked in the first place) by a well placed comment.
sometimes people say that code is documentation or code should read like documentation. this is false. code can explain (usually poorly) what it does but it can't give a rationale why it does it the way it's been written, can't say what it doesn't do, etc. always write some documentation - comments and commit messages at least. this should be enforced in code review.
i'd say engineering is about not writing code unless absolutely necessary. code is an asset, but it's also a liability. you really don't want more than you need.
My observation is that you're halfway there when you realize that you need to improve, of the folks I saw who did poorly it was because they didn't realize that you could be both the smartest person in the room and the least capable at the same time.
Right now, on your first job experience, even a kid who never went to college is better at programming than you are because they've been experiencing all the pitfalls that can happen and have built up a knowledge from that which is perhaps more intuitive than formal but serves them well. What you have over that person is that you've trained yourself to consume, classify, organize, and distill massive amounts of information in a short amount of time.
Use that training to consume everything you can on the art of writing programs. Read "Test Driven Development" read "Code Complete", read "Design Patterns", read "The Design of the UNIX Operating System", read "TCP/IP Illustrated Volume 1", and a book or two on databases. With your training you should be able to skim through those to identify the various "tricky bits" and get a feel for what is and what is not important in this new field of yours.
Soak in as much as you can and ask a lot of questions. Pretty soon no one will know you haven't been doing this your whole life.
this level of the craft is subtle, intuitive, and often inaccessible to the conscious intellect.
I would not give it to a beginner as it would corrupt their mind with useless drivel .
It's authors exhibit themselves as morons who celebrate renaming existing computer science concepts while occasionally mixing and matching them.
I would not be this uncharitable towards it if it was not such a famous (and hence harmfull) book.
It's harmfull because software engineering is really hard, and they just add up to the load by trying to have the reader memoize their junk instead of doing something that would actually make them a better programmer. And, if you try to use their concepts while programming - shudders, oh god help you.
I'm not going to iterate over every pattern. Here's an example:
Flywheight? You assholes, just tell memoization for what it is. Why don't you rename existing data structures as well? Good thing those come out of the box, otherwise you would have began the book by renaming array, linked list and dictionary. Maybe you would have called linked list "the chaingang" or something.
You don't simplify things by giving things cuddlier names. You stunt peoples growth that way.
Gang of four book is a malignant offshoot of the practice of attempting to make software engineering better by legalizing and dogmatizing it by decomposition into trivial details that hurt your brain. It's a branch of 'consultancy driven software development' where people attempt to aquire a halo of professionalism by calling things by fancy names and over-complex descriptions (while skipping the practical things with equally complex names but at least those weren't made up by a bunch of idiots).
What I understand from your comment is that you dislike the Gang of four book because it renames concepts that don't need the cutesy names that they give them. Do you have a problem with the _concept_ of design patterns? Or just the names they are given? Are the concepts themselves sound and worth paying attention to?
"It reads like it was written by a clever, verbose, and 'over-eager novice."
Design patterns are a really usefull concept. The GoF book just totally botches up that concept by dwelving deep in trivial language details while missing the big picture.
Christopher Alexander's "A timeless way of building", and "Notes on the Synthesis of Form" are the books in architecture that prompted a lot of dicussion in software design circles, and from which I presume GoF got their idea of software design patterns.
What are good design patterns in software? IMO they are composed from the programming models exposed in a basic CS book like Aho and Ullman's "Foundations of Computer Science" and further developed in a books like Structure And Intrepretation of Computer Programs.
GoF is an ok anecdotal reference after those, but it really is not suitable as a didactic resource.
Peter Norvig wryly commented that 16 of the 23 pattern are either invisible or non-existent in lisp Lisp[0].
[0] http://www.norvig.com/design-patterns/design-patterns.pdf
Among the few patterns that are more generally useful and have stood the test of time, we'd find Composite, Command and of course Observer.
"Composite, Command and of course Observer"
Each of which is trivially obvious way to structure your program when that need arises. You said it the best - GoF tried to attack a very real problem - the unwieldiness of C++ and Java - but they did really not come up with any actual solutions. And they extend this non-value-adding gibberish into a tome, that somehow has become a cultural icon of some sort.
The basic premise of the book is this: "We looked out how people use object oriented languages and found this and this and this". Congratulations. By that approach they've enumerated a bunch of gobbledygook. They've not invented anything new. Nor have they done any deep analysis of the fundamental underlying concepts, and by their chosen approach I'm taking a guess they would lack the capacity to add anything of value to that discussion if they tried. Which is totally ok, most people, like me, are mediocrities.
Reading gang of four is painfull, because it reads like the work of a very clever first year CS student who is over eager to create contrived examples of these nuggets of things he has discovered among the work of his fellow first year CS students. Without actually bothering to do a literature survey, or - most critically writing any massive programs or doing any complex abstract work of any worth.
In other words, it reads like it was written by a clever, verbose, and over-eager novice.
As an excuse, it was written in the dark ages in the 90's - a period of large scale communal lobotomization and attempts to forget the lessons of history. As such I would not judge it too harshly, if not for the fact that people keep bringing it up as some sort of book of actual wisdom like Structure and Interpretation of Computer Programs - which it really, really, really is not.
The last reason why I don't like GoF book:
In their title they steal Christopher Alexander's notion of Pattern Language in architecture, and totally and completely misapproriate it. Christopher Alexander's books are totally awesome. "Notes on the Synthesis of Form" and "Timeless way of building" are really beautiful books for anyone to read who has to design complex systems.
Christopher Alexander's patterns are not piecemeal lego parts you can put together - they describe parts of a whole and each pattern is a composition of elements that - and this is the key part - fits to it's surroundings and the design constraints of the original problem to be solved.
Gof book is borderline clueless, it confuses people, and it mis-introduced a totally awesome concept from another domain, thus effectively squatting on the concept and hindering any attempts to use those concepts in discussion of systems design since when you say "patterns" in CS context people automatically context switch to the GoF bastardized definition of the concept.
Design Patterns are very useful, but they are a tool like anything else.
They're not just a bunch of things with cutesy names, they are the most common and core patterns that we can use in OO.
Yes, they are way overused, particularly in Java, but they are useful.
The book could be summarized however.
Patterns can be useful but only in moderation and only with reason. They're spice.
A contrived example, but:
https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...
FizzBuzz Enterprise Edition. It's enterprisey!
If you do
* 100% pairing,
* buttloads of unit tests,
* lots of microservices,
* use an autoformatter,
* in a Node codebase,
to order-of-magnitude approximations you're probably going to be about as effective a team as if you
* always code alone,
* mostly write integration tests,
* write a big old monolith,
* mostly ignore code style (except for egregious cases),
* in C.
These variables do make a difference, but those differences are context-dependent and can be swamped by other factors. Use version control. Do at least some testing and some code review. Make sure engineers can actually run their code before submitting it (on their laptops or in special staging environments or whatever, it doesn't matter much.) Help coworkers when they're stuck, and ask for help when you're stuck. And try to hire Harrison Bergeron onto your team :-).
In code bases you have written yourself, often you remember enough about it that you can mitigate the effects of complexity. Possibly you already had a plan for how to implement some functionality. Or maybe you know how to avoid certain problems.
As programs become more complex, you will find that "legacy team members" are disproportionately productive. Learning the code base becomes more important than learning the problem domain.
Teams that expend effort on maintaining a status quo where the exploring the problem domain dominates the cost of development (even with new developers), will usually lag behind teams that do not. Mostly, because it is difficult to do.
Some of the practices you mention can be used to help in this, but for a "legacy team member" it might not be attractive. If you have a more typical situation, knowing the code base better means you are much more productive and therefore more valuable. This secures a senior position on the team. Turnover is usually high, so most people leave before it gets to the point where even "legacy team members" become unproductive. When you go to a new shop, you can always complain about the "crappy" code and suggest a complete rewrite (at which point, you secure your senior status again).
While your list does not necessarily comprise what I think of a "productive practices", I think it is incorrect to imply that "productive practices" can not have a profound impact on the outcome of a project. The productive practices that work best for each team will likely be different. It's important to pick practices that work for your team and produce code that is simple to read and extend. Not thinking about it is a good way to ensure that you will end up with a complex system.
[0]: https://en.wikipedia.org/wiki/Harrison_Bergeron
https://www.youtube.com/watch?v=EZ05e7EMOLM
What? Those are unnecessary details as far as software engineering is concerned.
What I like about them are that they describe large complex systems that have to work with a bunch of people plugging in their own implementation of the parts. For me, both books give a good sense of what you need to do to enable the product you are working on to evolve successfully, and highlight areas where a lack of flexibility impacted that evolution.
So often in programming you look at your code and have to evaluate not only what features you are delivering but also what paths you are closing off with your design. In my experience the folks who excel at programming hold both the model of the system in their mind while they execute some part of it.
I think both books give a flavor of that.
To this list, I would also suggest compiler construction or database systems as different topics exhibiting the same thing.
A more basic challenge for a scientific programmer might be absorbing the mindset of tool building rather than problem solving. It's a different mental perspective to think of your code as a product for other developers rather than as a means to producing a data artifact.
On a related note, I just this week stumbled upon a post / essay, which was probably the best piece of writing I've read in computer-aided engineering, and working in a disciplined manner. I'd recommend reading it right now, as the very first thing you do: https://queue.acm.org/detail.cfm?id=3197520
- Ask them to review your code and suggest changes
- Look for questions of taste and ask more. It may feel intuitive to them, but if you dig in you can often find a good reason/principle behind it.
- Read your coworkers' code
- Read the comments people leave on others' code
- Each function should do just one thing.
- Don't reuse a variable if making a new variable with a new name would describe the value better.
- Give functions verb names that describe what they do. If that's hard, they may be doing too many things.
- A function should either change something or return a value (command-query separation)
- Any data should have a single, canonical source of truth. https://en.wikipedia.org/wiki/Single_source_of_truth
- When deciding between making code DRY https://en.wikipedia.org/wiki/Don%27t_repeat_yourself or not, decide if future changes should affect both places at the same time (use DRY) or not (probably doesn't need DRY, maybe shouldn't have it.)
- Avoid spooky action at a distance https://en.wikipedia.org/wiki/Action_at_a_distance_(computer... and if you can't, refactor or at least add comments.
- Write modular functions that can be used without understanding much about the function beside what the name/arguments tell you.
One measure of your success in this area is how quickly someone who'd never seen your code could describe what it does.
Try The Art of Readable Code (a pair of google authors, IIRC), and Ousterhout's A Philosophy of Software Design.
https://rse.ac.uk/
- Go to hackathons to learn to ship code fast and get used to building "skateboards". Learn how to make tradeoffs that optimize for development speed. It's not about writing crappy code, it's about optimizing for the right variables at the right time. There are now a lot of real world variables to consider.
- Practice Kanban. Divide and conquer your projects. Make small and focused pull requests. You will naturally start strategizing on how to do things right while you're doing things quickly.
- Using category theory and functional programming in your code, but being practical about it so others can read it, will really help when it comes to writing unit tests. Unguided polymorphism is from the devil.
As a PhD student, my code and habits do not meet the standard of industry. This is because I'm constantly changing the whole architecture to try new ideas, so I I optimize for small and simple code at the expense of testability, modularization, robustness, etc.
It's important to recognize this. You will need to change your style.
I can't recommend any one book. I feel like I mostly learned these lessons through random articles, lecture videos, conversations, and personal experience.
IMO, some of the most important principles:
* Implement as much as possible with pure functions (but don't contort the code to achieve this).
* Make your commits as small as possible. Well structured version control history is valuable.
* Spend lots of time time thinking about how data flows through your program, more than how the code is organized.
* Strongly prefer DAG dependency structure. Write a set of libraries and then a top level program that uses them.
* Follow whatever formatting and style rules your workplace uses. It's religion and not worth getting into, as long as everyone uses the same style its a win.
* Dev/stage/prod is also workplace specific. Just go with the flow and avoid time wasting arguments on these topics, it's not usually worth it.
* Try to break your work into small commits. This is both easier to review and easier to estimate time on.
* Architect your code so that you can add unit tests. Make sure all your commits have this.
* Prefer longer simpler code to clever code, you're optimizing for newcomers to your code reading it.
* When a one line comment explains it to you, you'll probably need a paragraph at least for someone outside the field to get started understanding it.
* Think about how you'll respond to someone coming to you and saying "something something prod something something your code is buggy." How will you get enough information to determine if this is true, and to debug it when it is? Logging is one good tool here, so consider what you log carefully.
Finally, don't be too surprised if you find people talk down to you. Unless you are in a FAANG company, which it sounds like you are not, developers can be very condescending towards academics (and people from other fields).
1st decide on what area you want to specialize in and then look at a reputable boot camp that fits your goals. You can do it on your own but it's going to take you a lot longer and it's hard to focus on what you need to learn. Also, if you could do it, you would have done it already. The advantage is that you're already used to the scholastic environment and you'll be able to do very well and be even well ahead of everyone if you challenge yourself rather than strictly following the curriculum.
Good testing practices:
1. Minimize mocking as much as you can -- as a rule of thumb, mocking is inversely proportional to test confidence.
2. Don't test implementation details, test public-facing APIs. This way, your implementation can change. Mocking makes this harder. Don't test how you get things done -- test that they are done.
3. Make sure your API is well defined before you start writing tests, or you will waste time.
You can find loads of Python testing guides on Google on the first two points. There will be times when you have to break some of those rules, but knowing when will come with experience.
Robert C. Martin, Clean code: https://www.amazon.com/Clean-Code-Handbook-Software-Craftsma...
Vaughn Vernon, various: https://vaughnvernon.co/?page_id=168
Steve McConnell, Code Complete: https://www.amazon.com/Code-Complete-Practical-Handbook-Cons... 2
Clean coder: https://cleancoders.com/ videos
Hunt and Thomas, The Pragmatic Programmer: https://www.amazon.com/Pragmatic-Programmer-Journeyman-Maste...
Hitchhiker's Guide to Python: https://docs.python-guide.org/
Dustin Boswell The Art of Readable Code: https://www.amazon.com/Art-Readable-Code-Practical-Technique...
John Ousterhout, A Philosophy of Software Design: https://www.amazon.com/Philosophy-Software-Design-John-Ouste... This one looks particularly interesting, thanks AlexCoventry!
Kent Beck, Test Driven Development: https://www.amazon.com/Test-Driven-Development-Kent-Beck/dp/...
Dan Bader, Python Tricks: The Book: https://dbader.org/
Ian Sommerville, Software Engineering: https://www.amazon.com/Software-Engineering-10th-Ian-Sommerv...
Svilen Dobrev, various: http://www.svilendobrev.com/rabota/
The crucial point is that most of us a doing programming, and not software engineering. Learning from others is hit or miss. One can certainly learn to program from others, but that's not enough to be able to do software engineering.
I think this is a pretty interesting question. Personally, when I was young, I would have said what you said: it's engineering, specifications go in and properly engineered finished product come out. I was proud that my CS program was contained in an engineering school and that all my friends were in other engineering disciplines. The longer I do this though, the more I think it's a craft: fuzzy idea is put to a skilled practitioner and through discussion and creativity, one or more artifacts are created to satisfy that idea. You could argue this is true of more traditional engineering disciplines as well, and I would probably agree. I'm not totally sure what the distinction between the two things is. So tell me what the distinction is and convince me what we do is more the one than the other.
For what it's worth though, apprenticeship is also a very important part of engineering education. At some point, you have to see people do the thing, regardless of whether it's engineering or a craft.
Can the immense complexity of today and tomorrow's software be tamed by applying the same principles of building a cupboard? No, it requires an engineering mindset.
We're now limping along as an industry and it's not obvious because SW is bringing in massive amounts of money and we can basically get away with a lack of quality.
The point being, there are intricate details that are very hard to deliver in the traditional class-room oriented school environment with well defined requirements.
Those do not state anything about scalability. Crafts can scale - like for example how the old giant cathedrals and castles were built in middle-age europe.
They don't say anything about mindsets either. You need an engineers mindset to buid a cathedral or a castle.
The specific problem with crafts is that adapting to new requirements is a complete hit and miss process. The reason for this problem is the lack or proper theoretical framework in which to pose ones work and into which embed the requirements themselves.
The program verification people are working towards solving this problem, see for example Leslie Lamport's work in TLA+.
But until we have a general, mathematical proof backed compiler for requirements, as well as for the program implementation, we are pretty much stuck with craftsmen.
(Well, we have proof compilers but those are at the moment completely unusable for general programming since they are so complex to use.)
Sorry, but I have to strongly disagree. In it's purest form the core of software engineering - i.e. programming is a craft. The other parts are mostly about creating processes so that craftsmen can create something together without stumbling into eachother.
The difference between a craft and engineering are numerous.
- engineers generally need a license
- engineering is about repeatability and creating dependable cost estimates
- engineers are required to study for years for a very good reason. You can be a rockstar programmer out of highschool.
Just having a bunch of cargo cult gibberish bound into a book does not make a craft into an engineering discipline.
It's harmfull to call programming engineering. Engineers have curriculums that can teach them pretty well what is expected of them once employed.
Not for programmers - or, well, software engineers. If there was even one curriculum that could churn out good programmers dependably, don't you think this model wouldn't be copied instantly elsewhere? If such a curruculum existed, do you think think software interviews would be filled with whiteboarding just to check out that the candidates understand even the basics?
I think this incapability to create a curriculum for actually creating good programmers is the best evidence that programming is a craft. It's such a complex topic that you can't create a mass curriculum that would serve all equally. Not with our current understanding, anyway. Maybe if we could teach everyone assembly, and Haskell , and have them implement compilers and languages as a standard things would be different.
The second best way to learn programming without being born a programmer savant is to learn from others while doing. Apprenticeship is the traditional way to train craftsmen.
Programming is so much more like a craft than engineering that it's best to call it a craft.
Craft is not a deragatory term. It just means we don't understand it theoretically well enough to teach it properly.
You list three distinguishing features of an engineering discipline. The first and third swap cause and effect; a field doesn't become engineering because once it requires licensure and years of study. Surely you wouldn't agree that a successful campaign to require those things for software developers would bestow the status of engineering upon the work.
Your second point seems closer to the truth, but I'm not so sure it's true. If someone comes to a civil engineering firm and says, "design us a road between this city and that city", that is often a unique challenge because the terrain between those two cities is likely unique, maybe it requires a big bridge or a tunnel, which will be akin to but not identical to other bridges and tunnels you've built before.
Sorry, I wrote that in a hurry. I wasn't claiming either was a cause or effect. It was more of finding characteristics that we can use to identify one from another. I.e. following the argumentation "If it quacks like a duck and walks likes a duck it's likely a duck, and if it doesn't, we don't really have much evidence of the duckish quality of the observed thing".
So, I was not aiming to claim that licensure would turn software engineering into actual engineering. Rather, that the requirements of the field are so poorly understood in the general context that there would be very little to agree on the specific requirements. Poorly understood -> not engineering, really.
I totally agree with what you wrote above.
On the third point: I'm not claiming 100% truthiness to my argument, but it's pretty close. Software engineering projects are still among the riskiest ventures where you can think of investing capital in. If you want to build a road:
1. The language of the requirements are pretty well understood, from point A to B, this many lanes 2. Unless some unforeseen calamity arises, and you have the capital to pump to the project, eventually you will get the road
I think we can agree that these define any engineering project. Of course, engineeering is not cut and dried either - that's why you need to have actually trained professionals who can react to the events that come along as the project progresses.
I don't think 1. or 2. can hold for a software project in the general sense. Furthermore, you can end up accidentally, without anyones fault, wasting an arbitrary amount of capital on a feature that could, in the worst case, be replaced by a few lines of Python.
This poor quality of our general understanding of software development and lack of common language to describe anything means that most of the time software develoment is closer to R&D than engineering.
Generally, you can get better estimates when you are implementing a similar project the nth time. Like some general website, or a server backend. And in these instances you have the language to describe features and requirements. But in the general sense, software development isn't anything like this.
Crafts don't scale and are a poor fit for highly complex domains.
The curse of software development is its huge financial success, anemic legislative specification and the observed reality that customers will still buy poor quality software.
These are preventing the craft-like programming from turning into software engineering, but the craft is already failing to reach expectations: countless security disasters, unethical programmers enabling spying on millions, software literally killing users. This stuff will only get worse.
And finally, we do understand software engineering well enough to teach it properly. It's just not done, because it's not considered necessary when one can get by with a computer science degree, no degree or a bootcamp certificate.
This is news to me. I would very much like a citation, please. Or do you mean applying formal proof verification to everything?
The above is neither widely known (I only found out about it after many years of doing professional programming), nor is it necessary in order to be successful in the profession and/or make a lot of money.
Commercial software development is mostly a wild west and we're calling that craftsmanship.
Martin is well intentioned, but very dogmatic about some things like TDD, functions size, personal responsibility, etc. You need to already have some decent engineering experience to be able to detect and ignore the harmful stuff from his books.