That makes a lot more sense. I was trying to figure out how you take the time to write an entire book but then throw together a short blog full of grammar mistakes to promote it.
> When I first picked up Software Engineering at Google I thought it was another one of those FAANG books full of lessons that make no sense at human scale. I was surprised, the lessons apply to teams as small as 5.
> This is a "good shit stays" recap. The lessons that stick with you a few weeks after reading.
Starts with a misleading title, goes into content-free blogspam, and ends with a desperate salesman pitch for their services. To top it off, the article randomly throws in "I love and appreciate you [a random visitor to his website] for who you are :heart:"
Your tests are only as good as your mocks. They hide the true behavior of your system, drift away from reality, and take a lot of effort to maintain."
I've often felt guilty for leaning toward broader-scope integration tests, but I've long felt this too— that they're less effort to write and maintain and are much better value in terms of actually catching issues.
Not that unit tests don't have their place, but they're best for mostly-pure functions or classes where you're intentionally trying to cover a bunch of branches or corner cases.
I've felt the same way - as though not being able to shoehorn everything into pure unit tests somehow means that I'm not doing testing right. In most cases though, what we really care about is verifying the full functionality of an app. Maybe integration & e2e tests should be considered as the primary form of testing, with unit tests applying only in the specific cases where they make sense, like you mentioned.
Maybe the ulimate form of testing is something along the lines of Netflix's chaos monkey (now apparently an entire "simian army" suite of tools). Continuously test the real application, live in production, where it's not allowed to fail.
I think good unit test coverage is essential for good maintenance, but you can't cover _everything_ with unit tests. You definitely need integration and e2e tests to ensure bigger picture correctness. I have found that value one gets out of the different kinds of tests as you proceed up the pyramid get higher the closer to release you are, and are most valuable post release. During development they become obsolete too quickly due to changes in how things work. But locking in an already-released version to avoid regressions is great.
I think the real question for me is, suppose you come into maintainership of a legacy application with 100k lines of untested code, where do you start? I think the classical TDD view is "every new change, you add a test for, and so test coverage grows over time."
And that's fine, but I think it's much more valuable before making any functional changes at all, to impose some high level smoke tests that run the entire thing, and do a handful of common, end to end user flows.
Yeah, I'm a big fan of the Outside-In approach (I think it's from Michael Feathers's work).
The idea is that you test something at the very end of whatever you're working on, and ensure that it's identical to the original. You can then make larger changes to actually test the code properly while confirming that the overall system still works.
Super, super useful with ML things where running stuff can take a very looooonnnnnngggg time.
Yes, that gels with what I was saying - use e2e tests for "big picture" assurance that the system is correct. Use unit tests for super fain grained assurance of individual business rules, etc. All of the legacy projects I've ever inherited have lacked unit tests, and are theoretically functioning correctly, so the e2e tests make sense there.
I personally like and get value out of stubs and mocks in my unit tests, but I think they suffer some of the same issues as code without tests - it's hard to know the original intent and it's hard to constrain later contributors into doing the correct thing. I can mock up an external part of the system for my unit tests and be confident I'm only testing the behavior in my module, but someone who comes along later won't have the same picture in their head and will come to different assumptions. Integration tests are definitely the way to go here, when you are wanting to ensure your unit works correctly in conjunction with external systems. But that doesn't make the unit tests bad, just insufficient.
I'd probably phrase it as "Stubs and Mocks do not make sufficient tests", not that they necessarily make bad tests.
Mocks (and fakes) have their place; verifying the behavior in exceptional circumstances in a deterministic way (avoiding flaky tests) and unit-testing code exhaustively that would otherwise be too slow if written as full integration tests. Filesystem, network, and database dependencies are good places to mock/fake because errors are rare (and would be flaky) in integration tests and maintaining backend state for a lot of unit tests can slow the development cycle.
Yes, keeping mocks in sync with reality is hard but it should be up to the library owner to maintain either a good set of mocks or provide a fast enough library that integration tests are the better choice. This works in a company with uniform testing goals as a core value, not so we'll when dependencies and users have different testing philosophies.
I think the distinction I'm seeing here is that a mock is a pretend implementation of whatever the interface is, either at a code or service level, whereas as a fake is that actual service in whatever its minimally-viable state is.
So like— not mocking out the database interface with cooked responses, but instead using an actual instance of the database.
My corner of the world has ~100 people working on it and we do a mix of unit tests and integration tests. We definitely use mocks, but prefer fakes where possible. I really like our unit tests to poke and prod the corners of a given small section of code.
Our integration tests are just that, and they serve their purpose as well. But being able to stress all underlying parts of the system with all possible inputs at the integration test level is really really hard (if not impossible).
Many times, the way a system currently is can't push the corners of some helper class deep down, and you won't find those until someone makes some change in an intermediary layer. Making sure things are working at both the micro and macro level seems like the best way to go. But there is the obvious tradeoff of time it takes to do this.
Other teams we work with are write only integration tests. While it mostly works, I would say that it's harder for someone else to come into the code base and make changes. Unit tests help people understand how the code is expected to work today and how their changes may impact that. Integration tests won't always catch those kinds of things (plus tend to be way longer to run, which makes iteration time longer).
Yeah, I hear that. I wonder if there's a cart/horse issue here where the greatest value for low level unit tests is when you have a well-documented, thoughtfully architected system. Then the tests are verifying intentional behaviours and helping ensure the longevity of the system as designed.
I don't think I have this experience first-hand, but I could certainly imagine a scenario where a well-intentioned person added a bunch of unit tests to a badly designed system, and they ultimately ended up just getting in the way and causing confusion when it was time to rethink it.
> QA takes a few hours or days to ensure everything works together as expected.
Quality Assurance starts at the front of the process: what do we do to ensure (i.e. how do we accomplish) that product does what it says, as reliably as we can reasonably make it? This could include processes like design reviews, code reviews, writing tests, coding conventions, even business processes.
Quality Control is a process of running tests and identifying things that don't do what they are supposed to (or weren't done as they were supposed to be, e.g. pushing straight to production).
As I was told years ago, and took to heart: "you can't test quality into your product." This doesn't mean "don't test", it means quality comes from the start, and tests identify failures.
>>That's engineering considering the long-term effects of your code. Both direct and indirect.
No that is over engineering, and in violation of YAGNI (you ain't going to need it) an KISS (keep it simple stupid) principles. A lot of code my teams develops gets thrown away over time, as it is either no longer needed or replaced with an off the shelf tool. It is really hard to predict what code will survive AND becomes a performance bottleneck.
There is no point optimising prematurely. It is often best to tackle the problem when it occurs.
I agree in an ideal world where it's easy to get time to prioritize optimizing something (whether it's performance tuning or refactoring). Even in the most mature organizations, getting that time to do non value-producing work is challenging and so seemingly premature optimization is a pragmatic tradeoff.
The body of the article gives some good tips for organizing and maintaining code, but it frames it in a way that would suggest these are the best practices, and that's it. The points given are just the tip of the iceberg taught in a typical Software Engineering 101 course in university. There's no reason to reference Google here, none of these points are original. SE is an academic field with 50 years of publishing, and while most of it is pretty stale today, it is weird to lead a discussion on it completely disregarding the body of knowledge we already have.
The introduction is also weirdly phrased, as the process of SE doesn't start with writing code - that's putting the cart before the horse. SE starts with thinking about what you want to solve, and how you can and want to solve it. Coding should probably come after these processes.
Bear in mind that the blog is about a specific book titled "Software Engineering at Google." It's not an attempt to survey all of SE, and that explains why it refers to Google... because the book is specifically about SE _at_ Google.
Google developers are often so lost in the weeds of shipping fast and optimizing for rapid releases that they frustrate users with rapid UI changes, deprecation of major functionality, and not building the things that people often want in favor of just shipping fast.
That's part of the reason why this post doesn't talk at all about user requirements, needs definitions, or user stories or anything like that. If this is what is "Software Engineering at Google" then maybe this shouldn't be seen as "best practice".
I have found the Google experience, as a user, to be decaying steadily over the past 5-6 years.
I wholeheartedly agree that the experience of being a Google customer / user has degraded significantly. I'm not sure the omission of the topics you mentioned can be corelated exactly to that degradation, but it's an interesting idea. My assumption is that all the changes do go through proper UX and user testing, but the reason for them is questionable. I certainly haven't asked for hardly any of the changes. I also assume that the book focuses on the purely technical side intentionally.
Yes. Despite the "Move fast and break things" or rapid releases cycle being attributed to Facebook, it was Google who did it first and amplified by Mark Zuckerberg which becomes de facto standard in Silicon Valley and Tech industry worldwide. The cult of move fast and break things meant doing it the old fashioned way were wrong. For nearly 20 years, Twenty Years.
It is sin that is even worst than IE stagnation. ( Well I say that because IE is now irrelevant. If IE is still here my rage would have been even worst. )
The cult of move fast and break things meant doing it the old fashioned way were wrong.
The likes of Google and Facebook, as well as many startups inspired by them, have been an excellent demonstration of what commercial success looks like in a modern software business. In particular, they have (unfortunately, perhaps) demonstrated that quality really doesn't matter all that much in their target markets. In the early days, it turns out that users will forgive almost anything breaking if you have an attractive product. Later on, you can still break things often and almost completely ignore customer service as long as your lock-in effects are strong enough to keep most of your users anyway.
Even if you do need higher quality standards in your market, the lean startup view is to build anything you can and ship your MVP as quickly as possible. If and when you find product-market fit then you can pay off your "debts", potentially including big changes like rewriting your whole application or firing groups of early customers, because by that time you have the money and resources and user base to do these things without it hurting your bottom line too much. Again, from a business point of view, this strategy makes a lot of sense if you're operating in markets where these rules apply.
What I don't understand though is why so many people who are operating in markets where quality does matter seem to idolise big names like Facebook and Google and the people who work there. They have a high profile and they pay high salaries and their hiring processes are well known for being challenging, but I see little evidence that they effectively translate any of those apparent advantages into producing either better quality or more innovation than anyone else in the industry. If anything, given the enormous resources they have at their disposal, the opposite seems to be true.
30 comments
[ 1.9 ms ] story [ 72.1 ms ] thread> When I first picked up Software Engineering at Google I thought it was another one of those FAANG books full of lessons that make no sense at human scale. I was surprised, the lessons apply to teams as small as 5.
> This is a "good shit stays" recap. The lessons that stick with you a few weeks after reading.
I'd be curious to hear from a real Google engineer.
Unseemly and oily.
I find these statements really weird when they're made by total strangers. They strike me as shallow.
Your tests are only as good as your mocks. They hide the true behavior of your system, drift away from reality, and take a lot of effort to maintain."
I've often felt guilty for leaning toward broader-scope integration tests, but I've long felt this too— that they're less effort to write and maintain and are much better value in terms of actually catching issues.
Not that unit tests don't have their place, but they're best for mostly-pure functions or classes where you're intentionally trying to cover a bunch of branches or corner cases.
Maybe the ulimate form of testing is something along the lines of Netflix's chaos monkey (now apparently an entire "simian army" suite of tools). Continuously test the real application, live in production, where it's not allowed to fail.
I think good unit test coverage is essential for good maintenance, but you can't cover _everything_ with unit tests. You definitely need integration and e2e tests to ensure bigger picture correctness. I have found that value one gets out of the different kinds of tests as you proceed up the pyramid get higher the closer to release you are, and are most valuable post release. During development they become obsolete too quickly due to changes in how things work. But locking in an already-released version to avoid regressions is great.
And that's fine, but I think it's much more valuable before making any functional changes at all, to impose some high level smoke tests that run the entire thing, and do a handful of common, end to end user flows.
The idea is that you test something at the very end of whatever you're working on, and ensure that it's identical to the original. You can then make larger changes to actually test the code properly while confirming that the overall system still works.
Super, super useful with ML things where running stuff can take a very looooonnnnnngggg time.
I'd probably phrase it as "Stubs and Mocks do not make sufficient tests", not that they necessarily make bad tests.
Yes, keeping mocks in sync with reality is hard but it should be up to the library owner to maintain either a good set of mocks or provide a fast enough library that integration tests are the better choice. This works in a company with uniform testing goals as a core value, not so we'll when dependencies and users have different testing philosophies.
So like— not mocking out the database interface with cooked responses, but instead using an actual instance of the database.
My corner of the world has ~100 people working on it and we do a mix of unit tests and integration tests. We definitely use mocks, but prefer fakes where possible. I really like our unit tests to poke and prod the corners of a given small section of code.
Our integration tests are just that, and they serve their purpose as well. But being able to stress all underlying parts of the system with all possible inputs at the integration test level is really really hard (if not impossible).
Many times, the way a system currently is can't push the corners of some helper class deep down, and you won't find those until someone makes some change in an intermediary layer. Making sure things are working at both the micro and macro level seems like the best way to go. But there is the obvious tradeoff of time it takes to do this.
Other teams we work with are write only integration tests. While it mostly works, I would say that it's harder for someone else to come into the code base and make changes. Unit tests help people understand how the code is expected to work today and how their changes may impact that. Integration tests won't always catch those kinds of things (plus tend to be way longer to run, which makes iteration time longer).
I don't think I have this experience first-hand, but I could certainly imagine a scenario where a well-intentioned person added a bunch of unit tests to a badly designed system, and they ultimately ended up just getting in the way and causing confusion when it was time to rethink it.
Repeats a common error:
> QA takes a few hours or days to ensure everything works together as expected.
Quality Assurance starts at the front of the process: what do we do to ensure (i.e. how do we accomplish) that product does what it says, as reliably as we can reasonably make it? This could include processes like design reviews, code reviews, writing tests, coding conventions, even business processes.
Quality Control is a process of running tests and identifying things that don't do what they are supposed to (or weren't done as they were supposed to be, e.g. pushing straight to production).
As I was told years ago, and took to heart: "you can't test quality into your product." This doesn't mean "don't test", it means quality comes from the start, and tests identify failures.
No that is over engineering, and in violation of YAGNI (you ain't going to need it) an KISS (keep it simple stupid) principles. A lot of code my teams develops gets thrown away over time, as it is either no longer needed or replaced with an off the shelf tool. It is really hard to predict what code will survive AND becomes a performance bottleneck.
There is no point optimising prematurely. It is often best to tackle the problem when it occurs.
The introduction is also weirdly phrased, as the process of SE doesn't start with writing code - that's putting the cart before the horse. SE starts with thinking about what you want to solve, and how you can and want to solve it. Coding should probably come after these processes.
That's part of the reason why this post doesn't talk at all about user requirements, needs definitions, or user stories or anything like that. If this is what is "Software Engineering at Google" then maybe this shouldn't be seen as "best practice".
I have found the Google experience, as a user, to be decaying steadily over the past 5-6 years.
It is sin that is even worst than IE stagnation. ( Well I say that because IE is now irrelevant. If IE is still here my rage would have been even worst. )
The likes of Google and Facebook, as well as many startups inspired by them, have been an excellent demonstration of what commercial success looks like in a modern software business. In particular, they have (unfortunately, perhaps) demonstrated that quality really doesn't matter all that much in their target markets. In the early days, it turns out that users will forgive almost anything breaking if you have an attractive product. Later on, you can still break things often and almost completely ignore customer service as long as your lock-in effects are strong enough to keep most of your users anyway.
Even if you do need higher quality standards in your market, the lean startup view is to build anything you can and ship your MVP as quickly as possible. If and when you find product-market fit then you can pay off your "debts", potentially including big changes like rewriting your whole application or firing groups of early customers, because by that time you have the money and resources and user base to do these things without it hurting your bottom line too much. Again, from a business point of view, this strategy makes a lot of sense if you're operating in markets where these rules apply.
What I don't understand though is why so many people who are operating in markets where quality does matter seem to idolise big names like Facebook and Google and the people who work there. They have a high profile and they pay high salaries and their hiring processes are well known for being challenging, but I see little evidence that they effectively translate any of those apparent advantages into producing either better quality or more innovation than anyone else in the industry. If anything, given the enormous resources they have at their disposal, the opposite seems to be true.