TL;DR Use cloud based development environments to avoid machine (developer) specific, non reproducible problems, and increase productivity (speculatively)
Probably ok for certain needs, but not something I would use. My productivity comes from a localised and personalised setup. Issues with the local or remote environments IMHO, indicate a problems that
1) others may also encounter.
2) should at least, if possible be understood/ documented.
3) represent opportunities for greater understanding of the environments in question (assuming you have time to dig).
Yeah but what is the debugging experience like? Does that work? I think it adds a layer of complication. Also images can take a while to build. Although most would be cached hopefully with just copying your code changing. Maybe a snappy startup can fix?
> Yeah but what is the debugging experience like? Does that work?
Sort of. In certain stacks, you essentially set up remote debugging like you would for an app running in a remote environment (which is your local container with an exposed port) and your IDE just works.
It's relatively carefree when it works as expected, but a bit of a pain to set up sometimes. Admittedly, something like CPU flame graphs or tools like VisualVM that let you easily select from locally running Java processes to instrument might be harder to work with.
But even then, you can have issues with file system permissions and any bind mounts that you might need (e.g. files in a PHP container, where you want to keep developing and testing your app after page reloads, without rebuilding the entire container).
I'm still a proponent of using containers for making applications more consistently managed (configuration, resource limits, port bindings, storage), self-contained (dependencies, running different/multiple versions in parallel) and easier to launch (e.g. Docker Compose file or a fancier variation of YAML instead of Ansible + systemd services), but they definitely can be a leaky abstraction if you don't have *nix as your development machine OS.
I agree and it's why I personally invest so much time with things like Nix.
Nix offers a solution to being able to do a "cloud development environment", but on your local computer. It also allows you to easily setup the cloud development environment if you need one. While Docker works for setting up a new instance in a cloud provider, on a mac is a major pain so I don't really see it as a proper solution to the things Nix does for me.
But I can't help shake the feeling that Nix and Docker are just the best we have for now. I think the real solution to this involves far more changes before it's not just a pile of hacks. Eventually things will get bad enough that something will appear and it'll be adopted.
I don't think a VM per developer is the final version of this idea.
Nah the final version is chips with a massive data model and algorithmic magic embedded in silicon magic to bootstrap an environment that samples the data model to answer user questions.
A ways off, but some colleagues in the hardware world are working out the requirements on paper. Maybe right as I type this, not really tracking their calendar.
There will be some wiggle room for general programming, patches, hands on customization by the user.
Software is a decades long bubble to help us discover optimal machine state models we can etch into chips.
So long as we can avoid social collapse or species extinction between now than then anyway :)
Rigidly defined development environments breed fragile systems. They are enticing, for often rapid onboarding of new developers. However, not having the churn of letting new people onboard their setup to the system loses on an anti fragile mechanism.
This is ignoring the removal of control from your developers. Autonomy of tool choice is an amazing boon to job satisfaction.
Which is not to say that a cloud environment should not be used. Just don't homogenize the setup.
> Rigidly defined development environments breed fragile systems. They are enticing, for often rapid onboarding of new developers. However, not having the churn of letting new people onboard their setup to the system loses on an anti fragile mechanism.
> This is ignoring the removal of control from your developers. Autonomy of tool choice is an amazing boon to job satisfaction.
> Which is not to say that a cloud environment should not be used. Just don't homogenize the setup.
IMO, there is a good middle ground.
I agree with you on stuff like editor/ide and base OS. But I think there can be great advantages to having everyone do development in docker/VM such that the environment matches prod.
I think letting people do this is awesome. I think standardizing on it would be a huge mistake.
In particular, I think local deployment is important. And having freedom in all local deploys leads to finding the small details that matter.
Especially since you probably do not own the full prod tech stack, you are behooved by having proof that you can change it.
And, by all means, make the best attempt you can. I am arguing that somewhat purposely breaking it in a regular basis with each new developer makes you more resilient if you have to do a change in prod someday. And you will, someday.
> I think letting people do this is awesome. I think standardizing on it would be a huge mistake.
There is a hidden issue with not standardizing. Most people either don't care about local vs container (matching prod) development OR are terrible not equipped to make an informed choice. Thus, as a devops, I have to support both methods. This is a huge drain on my resources, especially since devopsing is just part of what I need to do.
The root cause of this is the myth that all software engineers will understand source control, will understand databases, will understand how systems interact with each other and on top of it, will understand the business so they can actually do the work required. This is how you end up with "it works on my machine". This is how you end up with "but it works in repl, therefore your prod environment is broken".
Standardizing doesn't mean this is the only way of doing things. It means, this is the standard way of doing things and if you get stuck, you get support. It means this is the way we consider to be most efficient of doing this stuff. You are still free to do things your way if you chose to, but if you get stuck, it's on you to get unstuck.
I don't view that as a hidden issue. I view that as the cost.
And agreed that conforming to what others are doing will, by definition, get you more support. I just think there is hidden value in that diversity. It sucks, as it can't be forced/codified.
I will further agree that too many will chase the "long game" of "if we support diverse environments, we will have a more robust product," at the expense of short term health. And often that is a false dichotomy, in that often the best long term choice for a team is also the best short term tradeoff. Surviving is important.
When I used Cloud 9 I didn’t find them rigid at all. You could put practically anything in the workspace as long as it would run in a container.
Containerised workspaces can be version controlled, which gives you the flexibility to fork a copy mess around but still go back to a known good state. And the new branch can easily be cloned for others to use.
No offense, but it's not my responsibility to make sure you have job satisfaction. It's my responsibility to keep the lights on and your paychecks coming in.
This is a public company and I have compliance requirements to meet or I am personally liable or face jail time. And with thousands/tens-of-thousands of engineers, your tool choice and lack of streamlined repeatable systems is costing us millions of dollars in profits every year and keeping your pay and the value of your equity down.
If you're going to depend on collectivist systems to provide you a stable livelihood, don't be surprised that the collective doesn't reward your individuality and entreprenureal spirit. There are other organizations that value that. Go work for them.
Millions of dollars is worth sacrificing the job satisfaction of tens of thousands of engineers? Just because you can't measure job satisfaction and report on the value that it provides to your company doesn't mean that it doesn't exist, or isn't doing that. If you divide the two rough numbers that you provided, you get hundreds of dollars per engineer. This is not very much money.
It sounds like you think that because you can't measure it, you are legally obligated to assert that it is zero.
As a matter of fact, many listed companies reward an entrepreneurial spirit and value employee satisfaction, because that’s usually linked to sustainable growth.
I think you seriously underestimate the value of job satisfaction and its effects on productivity.
I would also say that dev environments can be “streamlined” and “repeatable” while allowing for personal choice of editor and whatever other ancillary tools the developer is productive with. They’re not mutually exclusive.
I'm not sure why you think I am questioning your responsibility?
My observation has been that teams that don't have tool choice almost always lost it because they had an overly specified build environment. Often under the guise of "streamlining" the setup. And this is why they have that particular build server still sitting around that only the head developer is allowed to modify.
In what should be a straw man argument, I've seen this go so far as folks not knowing how to even try to build their software on anything other than the sanctioned IDE with them having to use the mouse to click the "now build" button.
Which is all to say, my argument is intended to be that you do need a repeatable and streamlined build system that is tested and run in a variety of environments. The more diversity in where/how it is run, the more trust I have that the team actually understands how it works. Especially as a whole.
For the personal control, letting the devs accomplish onboarding in a way that they control empowers them to make the changes to the setup that help make that happen, and gives more autonomy to them. Giving them autonomy means they can fix things they see. And you want that, as they will see far far more of what is happening than board level management will.
Does this mean you can't/don't meet compliance? Of course not. Most compliance can be tested. And it should be tested. Again, this testing benefits from diversity of execution, as well, such that a test that can pass on a variety of environments and toolsets is far more trustworthy than one that can only pass on a particularly sanctioned setup.
Now, pulling this back from a straw man argument. I accept there are situations where this level of freedom is harder to achieve. If you take my argument as absolute requiring that the entire team supports every possible variation of tool that could be thrown at them, please walk it back some. That isn't my intent. I just get scared by efforts to unify it all down to one, as well.
Extremely biased here. I'm one of the founders of Nimbus (www.usenimbus.com) and we help companies move to cloud dev environments.
Cloud dev environments are a tool like any other. How they're designed matters. A lot of home grown tooling ends up being limited for ease of build out and maintenance cost/effort.
I've found myself constantly using my /tmp directory on my machine for this reason. I really like that it's ephemeral and it forces our projects to be turnkey.
> Every few days, my development environment got borked. Often it was a quick fix. Sometimes not. But always time taken from one of the most productive programmers (Alex, not me.)
No, Alex has made this bed, and now he's lying in it.
There comes a point where you have to spend some of that 'productivity' on things that keep you from being preempted in the middle of something else. Things like a stable development environment, self-cleaning scripts, etc. It's not necessary to make things bulletproof. It is necessary to make things self-service.
Ideally we'd have reliable local developer experiences. An unreliable cloud-based developer experience is certainly worse than an unreliable local developer experience, and IME cloud-based developer experiences tend to be unreliable and extremely tiresome to debug when they break.
I pretty much run every project now through a docker compose file and use vscode remote containers with a devcontainer.json file.
This has been immense not only for my own development (nothing on my local machine, everything can be destroyed and rebuilt in a few minutes) but also for onboarding people onto projects. No mucking about with IDE settings, they are free to use whatever of course but having the basics all there makes for a great system all around.
Solve local development first and then maybe cloud dev. Reasons, just off the top of my head:
- always-on internet connection is not always viable
- intermittent and bad connections are a thing
- which local environment do you support for accessing your cloud? An ever changing combination of "latest only" browsers, apps, and select IDE integrations?
- how stable is your cloud environment? which features and versions of which tools will get deprecated tomorrow?
I've also seen "hybrid" setups in certain stacks, where the applications run locally but the database instance is shared amongst multiple developers for the same app version/branch.
In my experience, that can be a way to deal with some resource requirement and licensing constraints, but can also be pretty horrible due to either bandwidth requirements or latency that adds up for many smaller queries that follow one another (e.g. when your app needs to shuffle around lots of data to load a page).
Therefore, I'm tempted to agree that you should be able to relatively easily run as much as possible locally: especially a database with automated schema migrations and either data import or seeding (generating believable test data).
Then again, one can see why something like that would be problematic in situations where you can have dozens of microservices, running all of which locally isn't entirely viable, yet re-architecting everything isn't either. It'd still be nice to be able to launch and test everything you need for a given scenario locally, even while using certain dependencies from a cloud dev environment.
In the case when there's a staging environment, it would be nice to easily run the microservice you're working on locally. And/or mock some of the external stuff like BigQueries, Dataflows etc.
You can't even test serverless functions properly because... you need the Amazon/Google server to run them on :)
> You can't even test serverless functions properly because... you need the Amazon/Google server to run them on :)
This is unfortunate. I know that OpenFaaS and IronFunctions attempted to create a more vendor neutral offering that can run anywhere and still be usable, but neither really got the critical mass of attention/adoption for the large vendors to look in their direction.
I'd say it's rather unfortunate, when you compare it with the success of S3, which has many self-hostable (and locally launchable) compatible solutions like MinIO and Zenko.
Even managed databases are still often MySQL or PostgreSQL compatible, whereas FaaS doesn't seem to have quite gotten the sort of standardization.
“You have to delete this file & re-run that script. No problem.”
That's a problem actually, And doesn't matter where you run your program - locally or on remote host. Looks like there are issues with programming approach and stability.
At (former employer) were did a good a pretty good job at solving this. We were deployed to AWS, using EKS, SQS, Aurora, S3 and a small amount of lambda. When a new developer got onboarded, they were given their own slice of the dev environment: ingress, EKS namespace, S3 buckets, database and SQS queues. But it was still running within the same VPC and k8s cluster, with the same IAM (all those roles and policies that matched production and we're an important piece of testing). There was little additional infrastructure complexity, and no extra cost since the partitioning of usage-priced services and EKS namespaces is free. Developers also had access to shared infrastructure and the latest changes made by their colleagues. But everyone also had their own local microk8s + minio + mysql, so as long as they were happy to deploy the latest full testing release of everybody else's stuff, they could use that. When using their local cluster, they still had access to some cloud stuff like SQS and S3 (but not Aurora, they had to reply on a k8s deployed mysql).
We saw the cloud environment as an essential tool to provide the developers with, but not a tool that it was essential for developers to use. In fact, we had a preference for keeping dev working, so initial work on large changes should be done locally and deployed to dev once they're working. Dev was there to protect the testing environment and make sure that developers had a "real" cloud environment.
In other words, you don't have to deploy to dev, but if you break the testing environment because you didn't deploy to dev, there would be raised eyebrows.
41 comments
[ 3.2 ms ] story [ 117 ms ] threadThe essay is probably interesting, but I'll never know.
I opened it with an incognito window, no problem.
I opened it with a logged in window, no problem.
Some authors try to monetize their posts. Not this one.
I'd love to know if your experience was somehow different.
https://gitlab.com/magnolia1234/bypass-paywalls-chrome-clean
Probably ok for certain needs, but not something I would use. My productivity comes from a localised and personalised setup. Issues with the local or remote environments IMHO, indicate a problems that
1) others may also encounter.
2) should at least, if possible be understood/ documented.
3) represent opportunities for greater understanding of the environments in question (assuming you have time to dig).
Sort of. In certain stacks, you essentially set up remote debugging like you would for an app running in a remote environment (which is your local container with an exposed port) and your IDE just works.
It's relatively carefree when it works as expected, but a bit of a pain to set up sometimes. Admittedly, something like CPU flame graphs or tools like VisualVM that let you easily select from locally running Java processes to instrument might be harder to work with.
But even then, you can have issues with file system permissions and any bind mounts that you might need (e.g. files in a PHP container, where you want to keep developing and testing your app after page reloads, without rebuilding the entire container).
I wrote a bit more about it here: https://blog.kronis.dev/everything%20is%20broken/containers-...
I'm still a proponent of using containers for making applications more consistently managed (configuration, resource limits, port bindings, storage), self-contained (dependencies, running different/multiple versions in parallel) and easier to launch (e.g. Docker Compose file or a fancier variation of YAML instead of Ansible + systemd services), but they definitely can be a leaky abstraction if you don't have *nix as your development machine OS.
Nix offers a solution to being able to do a "cloud development environment", but on your local computer. It also allows you to easily setup the cloud development environment if you need one. While Docker works for setting up a new instance in a cloud provider, on a mac is a major pain so I don't really see it as a proper solution to the things Nix does for me.
But I can't help shake the feeling that Nix and Docker are just the best we have for now. I think the real solution to this involves far more changes before it's not just a pile of hacks. Eventually things will get bad enough that something will appear and it'll be adopted.
I don't think a VM per developer is the final version of this idea.
A ways off, but some colleagues in the hardware world are working out the requirements on paper. Maybe right as I type this, not really tracking their calendar.
There will be some wiggle room for general programming, patches, hands on customization by the user.
Software is a decades long bubble to help us discover optimal machine state models we can etch into chips.
So long as we can avoid social collapse or species extinction between now than then anyway :)
Someone has to bootstrap a DC of machines before you can run that containerized business logic.
All the old barnacles are there it’s just hidden behind the wizards curtain.
This is ignoring the removal of control from your developers. Autonomy of tool choice is an amazing boon to job satisfaction.
Which is not to say that a cloud environment should not be used. Just don't homogenize the setup.
> This is ignoring the removal of control from your developers. Autonomy of tool choice is an amazing boon to job satisfaction.
> Which is not to say that a cloud environment should not be used. Just don't homogenize the setup.
IMO, there is a good middle ground.
I agree with you on stuff like editor/ide and base OS. But I think there can be great advantages to having everyone do development in docker/VM such that the environment matches prod.
In particular, I think local deployment is important. And having freedom in all local deploys leads to finding the small details that matter.
Especially since you probably do not own the full prod tech stack, you are behooved by having proof that you can change it.
And, by all means, make the best attempt you can. I am arguing that somewhat purposely breaking it in a regular basis with each new developer makes you more resilient if you have to do a change in prod someday. And you will, someday.
There is a hidden issue with not standardizing. Most people either don't care about local vs container (matching prod) development OR are terrible not equipped to make an informed choice. Thus, as a devops, I have to support both methods. This is a huge drain on my resources, especially since devopsing is just part of what I need to do.
The root cause of this is the myth that all software engineers will understand source control, will understand databases, will understand how systems interact with each other and on top of it, will understand the business so they can actually do the work required. This is how you end up with "it works on my machine". This is how you end up with "but it works in repl, therefore your prod environment is broken".
Standardizing doesn't mean this is the only way of doing things. It means, this is the standard way of doing things and if you get stuck, you get support. It means this is the way we consider to be most efficient of doing this stuff. You are still free to do things your way if you chose to, but if you get stuck, it's on you to get unstuck.
And agreed that conforming to what others are doing will, by definition, get you more support. I just think there is hidden value in that diversity. It sucks, as it can't be forced/codified.
I will further agree that too many will chase the "long game" of "if we support diverse environments, we will have a more robust product," at the expense of short term health. And often that is a false dichotomy, in that often the best long term choice for a team is also the best short term tradeoff. Surviving is important.
Containerised workspaces can be version controlled, which gives you the flexibility to fork a copy mess around but still go back to a known good state. And the new branch can easily be cloned for others to use.
This is a public company and I have compliance requirements to meet or I am personally liable or face jail time. And with thousands/tens-of-thousands of engineers, your tool choice and lack of streamlined repeatable systems is costing us millions of dollars in profits every year and keeping your pay and the value of your equity down.
If you're going to depend on collectivist systems to provide you a stable livelihood, don't be surprised that the collective doesn't reward your individuality and entreprenureal spirit. There are other organizations that value that. Go work for them.
It sounds like you think that because you can't measure it, you are legally obligated to assert that it is zero.
You are bad at your job.
As a matter of fact, many listed companies reward an entrepreneurial spirit and value employee satisfaction, because that’s usually linked to sustainable growth.
I would also say that dev environments can be “streamlined” and “repeatable” while allowing for personal choice of editor and whatever other ancillary tools the developer is productive with. They’re not mutually exclusive.
My observation has been that teams that don't have tool choice almost always lost it because they had an overly specified build environment. Often under the guise of "streamlining" the setup. And this is why they have that particular build server still sitting around that only the head developer is allowed to modify.
In what should be a straw man argument, I've seen this go so far as folks not knowing how to even try to build their software on anything other than the sanctioned IDE with them having to use the mouse to click the "now build" button.
Which is all to say, my argument is intended to be that you do need a repeatable and streamlined build system that is tested and run in a variety of environments. The more diversity in where/how it is run, the more trust I have that the team actually understands how it works. Especially as a whole.
For the personal control, letting the devs accomplish onboarding in a way that they control empowers them to make the changes to the setup that help make that happen, and gives more autonomy to them. Giving them autonomy means they can fix things they see. And you want that, as they will see far far more of what is happening than board level management will.
Does this mean you can't/don't meet compliance? Of course not. Most compliance can be tested. And it should be tested. Again, this testing benefits from diversity of execution, as well, such that a test that can pass on a variety of environments and toolsets is far more trustworthy than one that can only pass on a particularly sanctioned setup.
Now, pulling this back from a straw man argument. I accept there are situations where this level of freedom is harder to achieve. If you take my argument as absolute requiring that the entire team supports every possible variation of tool that could be thrown at them, please walk it back some. That isn't my intent. I just get scared by efforts to unify it all down to one, as well.
Cloud dev environments are a tool like any other. How they're designed matters. A lot of home grown tooling ends up being limited for ease of build out and maintenance cost/effort.
No, Alex has made this bed, and now he's lying in it.
There comes a point where you have to spend some of that 'productivity' on things that keep you from being preempted in the middle of something else. Things like a stable development environment, self-cleaning scripts, etc. It's not necessary to make things bulletproof. It is necessary to make things self-service.
This has been immense not only for my own development (nothing on my local machine, everything can be destroyed and rebuilt in a few minutes) but also for onboarding people onto projects. No mucking about with IDE settings, they are free to use whatever of course but having the basics all there makes for a great system all around.
Solve local development first and then maybe cloud dev. Reasons, just off the top of my head:
- always-on internet connection is not always viable
- intermittent and bad connections are a thing
- which local environment do you support for accessing your cloud? An ever changing combination of "latest only" browsers, apps, and select IDE integrations?
- how stable is your cloud environment? which features and versions of which tools will get deprecated tomorrow?
I think this is it. Progressive enhancement beats graceful degradation.
In my experience, that can be a way to deal with some resource requirement and licensing constraints, but can also be pretty horrible due to either bandwidth requirements or latency that adds up for many smaller queries that follow one another (e.g. when your app needs to shuffle around lots of data to load a page).
Therefore, I'm tempted to agree that you should be able to relatively easily run as much as possible locally: especially a database with automated schema migrations and either data import or seeding (generating believable test data).
Then again, one can see why something like that would be problematic in situations where you can have dozens of microservices, running all of which locally isn't entirely viable, yet re-architecting everything isn't either. It'd still be nice to be able to launch and test everything you need for a given scenario locally, even while using certain dependencies from a cloud dev environment.
You can't even test serverless functions properly because... you need the Amazon/Google server to run them on :)
This is unfortunate. I know that OpenFaaS and IronFunctions attempted to create a more vendor neutral offering that can run anywhere and still be usable, but neither really got the critical mass of attention/adoption for the large vendors to look in their direction.
I'd say it's rather unfortunate, when you compare it with the success of S3, which has many self-hostable (and locally launchable) compatible solutions like MinIO and Zenko.
Even managed databases are still often MySQL or PostgreSQL compatible, whereas FaaS doesn't seem to have quite gotten the sort of standardization.
We saw the cloud environment as an essential tool to provide the developers with, but not a tool that it was essential for developers to use. In fact, we had a preference for keeping dev working, so initial work on large changes should be done locally and deployed to dev once they're working. Dev was there to protect the testing environment and make sure that developers had a "real" cloud environment.
In other words, you don't have to deploy to dev, but if you break the testing environment because you didn't deploy to dev, there would be raised eyebrows.