> I'm an avid supporter of Kubernetes (see About me), but I would never suggest that a small team adopt it from the start. Even with expert knowledge of Kubernetes, taking it on as a dependency still creates loads of maintenance debt.
Somewhere along the way, it became fashionable to build out your devops on day one as though you’ll be Google tomorrow, but to write your code without any architectural foresight as though the only thing that matters is tomorrow’s deployment.
It’s always nice to see people advocating towards a thoughtful balance.
I'm only a light k8s user, but isn't part of the pitch that it actually gets you out of having to run parts of your infrastructure? Like, there's no point spinning up EC2 nodes just to run microk8s or k0s on them— have ansible scripts or whatever that just run your application directly.
But the calculus changes if instead of EC2 nodes you can have a helm chart that deploys directly to a managed kubernetes services like EKS or GKE?
That's still a bunch more complicated than other hosted solutions. I once worked for a 4 year old company whose entire infrastructure consisted of a managed postgres instance + a moderately sized single VM (and a duplicated setup as a staging environment).
Eh, I've found kubernetes more complicated than the interface of a cloud provider, you always have to do the cloud provider as well for some things, and the benefits of the pre-baked helm charts or whatever else just aren't there until the relatively small difference in efficiency in each way of doing things makes a real difference as a result of scale.
If you have a lot of experience with any interface, you can do it really efficiently, but if you don't, kubernetes can make your experience worse than anything you'd come up with on a cloud provider.
Fun with k8s starts when you realize it has no LTS and gets updated on a fixed schedule. Any given version is only officialy supported for about 1 year and a half.
Then you'll need non trivial amount of knowledge of how it works to avoid many pitfalls (the helm chart will probably only go so far, it will still need small tweaks for your specific use case)
So it's a combination of upfront time investment, ongoing tuning, and recurring maintenance events.
Honestly, I find Helm to be a trap that is going to reduce your capability to run service reliably - or understand it. It's probably one of the most annoying bits in kubernetes ecosystem
I should specify I was thinking about VPS. Coming from AWS to Linode VPS with Caddy + Docker felt great, Im wondering at what point (N requests per second ?) it becomes tech dept
Yeah that's definitely something that needs to be specified.
> Coming from AWS to Linode VPS with Caddy + Docker felt great
I guess that depends what you were using with AWS, but Fargate + k8s is a pretty amazing workflow.
> Im wondering at what point (N requests per second ?) it becomes tech dept
I make these calls for our org, and I wouldn't call it tech debt based on the performance or scale, but because of maintenance. As an example, if you needed to hire a new person/replacement person for your team, would it be easier to find someone who can maintain k8s or docker compose?
The other area it shines for is the ecosystem. If you need to add monitoring to your app (e.g. with Prometheus), adding it to a k8s cluster is `helm install prometheus`
thanks for detailed response! how big is your org? My concern is that there is a cost to magic of adding deps like that. Complexity goes up and it is harder to understand what went wrong and where are the performance bottlenecks. I guess my question is at what org scale the tradeoffs are right
My last team was the same, there was ~100 developers on the main app, ~50 on the ancillary services, and a ballpark of ~10 on the "platform". I was just a team member on that team. Current org is 15 devs, 30 people total.
> My concern is that there is a cost to magic of adding deps like that.
Of course there is, there's such thing as a free lunch. It's up to you to decide which is right for your team/org. The cost of doing it manually is developer time, tech debt and the bus factor. The cost of using helm is introducing an abstraction. Profesionally, the bus factor and developer time of home rolling a solution make relying on existing solutions a very appealing proposition. Having to maintain a set of half baked shell scripts is exactly what tools exist to solve, and I've had more issues with deployments where someone has bunged together a bash script that makes varios assumptions about tooling, versions, installation paths, working directories, path formats, etc. than I have with problems related to our actual deployment infrastructure.
> I guess my question is at what org scale the tradeoffs are right
The cost of switching is huge, and likely not worth it unless you have genuine problems that would be solved by moving. For a greenfield project, frankly I'd think you're wrong to use docker-compose on any new project. Either you're managing the underlying infrastructure (ec2/vps style) or you're not (fargate/DO App Platform style), and in either case docker compose requires roughly the same amount of investment for a smaller return on a less actively developed product objectively solves less problems and requires more work to integrate.
While the specifications are more complex, I find much more sensible in management than docker-compose, and allowing easier expansion (not scaling the service, expansion of underlying hardware) which means keeping costs lower can be easier.
One of the systems I set up recently I decided to go the docker-compose way. Quite quickly I found out that either the tool doesn't support (or at least isn't exactly interested in supporting) multiple compose files, or I am too stupid to find out the right combo of options, which resulted in a growing yaml file as the setup of dockerized components rose on that specific server.
Whatever claims about user friendliness, I didn't figure a stable way to do deploy separate docker-compose.yml's from same user at the same time.
Not using Compose in production, but our development setup is split across multiple files - you can include them by passing -f flag to Docker Compose (I wrote a small wrapper script that does that automatically)
> isn't part of the pitch that it actually gets you out of having to run parts of your infrastructure?
That's a big part of many k8s pitches I've seen but in my experience it hasn't been representative of reality (or at least for every part of infrastructure you no longer need to run there's a new part of k8s architecture/config to learn & manage).
For me the selling points are:
1. For the time-cost of upfront complexity and setup, you get extremely easy and quick subsequent addition of infra/tooling & scaling thereof.
2. There are certain patterns of tooling that k8s enables (relatively easily), that would have been difficult/impossible with more traditional setups.
I don't think it makes infra management overhead less, it just reorganises it so it's more upfront and then less later while scaling.
My boss was trying to push me to use some new CICD pipeline the company built, just so he could understand what CICD meant. I kept ignoring him, because it just didn't seem needed and saw no need to complicate things.
To build the servers I was running on I wrote an Ansible job to install everything and copy the code down. All I had to do was add a couple flags to that to allow it to just deploy new changes. Instead of getting into the CICD pipeline I just took a few minute to make a little web front end for myself to kick off the job and watch the job status (we have Tower, but it's painfully slow). I can now deploy in 18 seconds with 1 click. Seems good enough. I'm sure I could set up a web hook to eliminate the click, but it's not a big deal.
Ive been banging this drum on here for a while, but we use AWS with ECS. I can spin up an ecs cluster with one runner in less than 10 minutes, and setting up a CICD pipeline is a handful of clicks on GitHub actions, or about 10 lines of pipeline .yml
I went through a React course that used Github and some online service to setup CICD in a few clicks with some service. It was painless.
At my company, it would involve going to weekly meetings to get all the updates and changes happening as they keep developing it, one-off meetings to get onboarded and setup... I'm pretty sure they also designed the system for Kubernetes, so I figured it would also mean containerizing the site, as it has been running on VMs for almost 10 years. Those weekly meetings also mean constant changes so things keep working (which I'm not dealing with on a different site that is going though CICD). All of this in the midst of a lot of bureaucracy. It's also a custom home grown solution, so it's not like AWS knowledge I can take with me or google.
It likely would have taken several months for little to no tangible benefit. No one on my team knows how CICD works. I'm the only one who has actually used it, and even my use has been pretty limited. It took 8 months for someone to bother to tell me where I could see build status/errors, at first they just told me to "wait about 10 minutes". There is no documentation anywhere. It's all tribal knowledge and no one seems to have the time to share it.
The only way to maintain my sanity is to not get too involved in it all. It will likely all change and go away within the next 2 years. Nothing lasts lone enough to get too invested in.
> I'm pretty sure they also designed the system for Kubernetes, so I figured it would also mean containerizing the site, as it has been running on VMs for almost 10 years.
Yeah - this is key. If you're already set up to _not_ be containerised, then it's likely not going to be worthwhile unless you are actively feeling some of the pain points around running on VMs.
> It's also a custom home grown solution, so it's not like AWS knowledge I can take with me or google.
One saving grace is that a huge amount of what k8s does is abstracts the aws-ness from underneath. Running a live application on k8s on aws is three separate skills - running a live app, running apps on k8s and running k8s on aws. Two of three of those still transfer!
> fit. No one on my team knows how CICD works. I'm the only one who has actually used it, and even my use has been pretty limited.
That's fair, and it's up to you to make the call as to whether it's worth it or not. If your team aren't willing then the use cases are very limited, and that's a political fight!
> The only way to maintain my sanity is to not get too involved in it all. It will likely all change and go away within the next 2 years. Nothing lasts lone enough to get too invested in.
I've been a driver for improving developer best practices on every team I've worked on. Not everything has always stuck, but I've found that processes that can be automated tend to have high rates of success, and processes that rely on humans tend to be sidestepped. I spoke to a coworker from a previous job last night who told me that some of the process fixes I made as a cog in the wheel have survived 2 years without me there. All is not lost!
>I've been a driver for improving developer best practices on every team I've worked on. Not everything has always stuck, but I've found that processes that can be automated tend to have high rates of success, and processes that rely on humans tend to be sidestepped. I spoke to a coworker from a previous job last night who told me that some of the process fixes I made as a cog in the wheel have survived 2 years without me there. All is not lost!
Yeah, it's just a matter of trying to pick those horses that seem likely to stick around, or that actually make like easier. Ansible is one of those; the CICD process and everything around it is not. We've changed out container platform 3 times in 3 years, and just kicked off a project to change it a 4th time, while also opening up some public cloud access if it can be justified. It's chaos.
There was also a push to build a lot of home-grown tools over the last 3-5 years. All of that is being ripped out in favor of vendor provided solutions.
The primary things that seem to stick around are the snowflakes, as sad as that is. The only teams that don't seem stressed by all that change are the ones who just do their own thing. I fought the good fight, jumping the latest and greatest stuff for over a decade and trying to put systems in place to get others to buy in as well, but I'm tired and this latest crop of tools seemed like they were destine for failure from day one.
So instead of testing your companies CICD pipeline, contributing to it and making it overall better for everyone you now have a snow flake system only you use and know how to maintain?
I try to align when possible. I think your solution is hacky and costs the company much more in the long run.
While my system is admittedly hacky, anyone I would hand development off to would be able to pick it up much easier than the company's pipeline, as it uses the tools they know and is pretty basic... none of them know the CICD pipeline.
I used to try to align when possible, but lately it hasn't made a lot of sense. Keeping up with the tooling ends up being the whole job, as they change it every 6 months. This site is a side project to help out the team, not my primary job. I can't be dropping everything each time they decide to switch vendors, change up their variables names, or whatever else they decide to do.
Sounds like your system is way more efficient than the CICD... in the short term, provided you're the only person that ever needs to use it (and/or nothing ever changes)
I'm the only person who ever uses it. If for some reason I left, what I have would be easier for the people who it would be handed off to, as they know Ansible, but have no idea how the CICD pipelines work. It's primarily an automation team, not an app development team.
> what I have would be easier for the people who it would be handed off to
This seems like a very large assumption.
> as they know Ansible, but have no idea how the CICD pipelines work
I know Ansible very well. I have never set up a CICD system and would much prefer to write a bunch of Ansible playbooks for myself than to do so. My company has a CICD system already set up (I have no idea how it works internally). Setting up a pipeline and understanding how that pipeline works as a user of the CICD system is still a lot easier for me than figuring out someone else's Ansible playbook repo. Adding jobs to that pipeline is still easier than writing new Ansible tasks. CICD is almost always easier for dev teams regardless of their experience once the initial complexity of CICD setup/hosting/maintenance is already handled.
If you don't already have a CICD set up, then setting up one is likely overkill for small teams. If it's already there (and maintained by a well-resourced team), use it, unless the CICD platform being used is absurdly bad for some reason (I've used a few and they have been largely fine / interchangeable).
It's a really basic playbook. There is a server build block that basically says install Apache, PHP, etc. and edit a few config files. And another block that just does a pull from GIT. As far as playbooks go, it's pretty basic and easy to understand. I'm not looking at it right now, but it's probably fewer than 10 tasks. It's not not one of those with thousand of lines referencing dozens of roles with complex when conditions and jinja2 on everything, like I often see.
As mentioned in another comment, the site has been running on VMs for almost 10 years and not containerized. The CICD pipeline is built around Kubernetes and I have no real reason to move to Kubernetes other than to say it's running in containers. The VMs have never had issues, it's a relatively small internal site, so I don't need to worry about scaling issues. K8 is also in the process of being replaced, so moving to it now seems silly. I use the pipeline for another project I'm working on with another team and while it's needed for the stack they're using, it's bad. No one seems to know how it works, I haven't spoke to a single developer who has a positive thing to say about it, it's very poorly documented, and any time there is an issue it requires messaging a bunch of random people until you find someone with the right blend of tribal knowledge and time to help. I've had a half dozen people come to me just to find out where to see the deployment status or to know how to promote something to upper environments... and I wasn't given that information until 8 months in. Seems like it would be day 1 stuff that would be easily documented.
For a hackathon they made something to do a one-click setup of a project that I would likely use, but outside of the hackathon, that feature isn't available and it would require months of work when factoring in meetings, waiting on other teams, etc to get setup without that. It's just not worth it for my use case.
This is surely resume (or consultant) driven development in action. High Scalability™ is like an epic side quest in the game of making software. The final boss at the end of the quest, Yagni, can only be defeated by those who have reached a large enough scale already. Many attempt the side quest too early and as a result get so distracted that they suffer delays or even lose the whole game.
It really depends on what you're going to run and what experience you have.
Personally, I would run k8s from the day I expect to run the code as separate deployment to running it from my IDE. I have the skills to do so, and I found I lose way less time doing so than trying to do "classic" server setup, even if it's just one server.
By "small team" I assume they meant a start-up - i.e the team is the company's entire dev.
As someone who works in a small team within a large corp where we use k8s for everything large and small, I did not read that statement as applying to me.
Good APIs between your code (that means programmatic interfaces, not necessarily over the network) help alot.
If you have a subsystem conform to a good API and is well behaved it can be rewritten. It is allowed to have some tech debt.
If the subsystem is sprawled through other subsystems that tech debt doesn't get paid down because now you need an expert in all those areas to fix it.
I swear I have worked places where there is code no one currently at the company can understand. At best you micro-fix for bugs and back the hell away!
Good APIs and interfaces are everything. I call it "the hotdog principle" when I am mentoring folks who are new to the concept. A good API is like a hot dog: Hot dogs are delicious. That is all you need to know about them. You don't want to know whats inside them or details of how they get made. You will be sad if you do.
It's funny and gross enough to stick most of the time.
Here we go again! Trying to use a financial term to describe "bad" past decisions. I do not see this term used when we build bridges or houses -- we either say they were time or financial constraints or it was a bad job. And certainly intermediate (temporary or not) supporting constructions are not viewed as debt.
Nobody built the pyramids in a day.
Here is a different attempt on definition: An occurrence on debt is when a higher up decides that a product/construction must be ready when it clearly is not, and thus convinces/bullies the engineers and put their signature on it.
Analogies can be a great tool to communicate or for prose but don't take them literally.
I agree with you but your definition was never “take shortcuts that reduce quality”. Ward Cunningham was explicit about this. “TD was about taking the time to incorporate new understanding and learnings into the code, if you don’t you accrue TD”.
Unfortunately your description is what has become the common usage of TD, and now the term is meaningless.
Frankly speaking, today, TD is just a neutral term to externalize blame/responsibility.
Yes, I was trying to give a contemporary definition. But I do find it interesting that it is a notion in our field. I agree with you and your disappointment. Thank you for the comment.
this seems wrong, the examples given aren't examples of technical debt. Starting a monorepo isn't technical debt, changing to something else later isn't technical debt. Technical debt would happen when doing that change you delay a part of the migration, or leave out some testing because its too tricky, or anything you are delaying till some time later that ideally should be done now.
It depends. If you knew that you would certainly need a distributed architecture in the future but you still implemented it as a monorepo, then it's technical debt. Obviously it's not if the requirements have changed.
that's not really technical debt, if mono repo makes sense at the moment, and you will change later, it's not technical debt, that's just a choice, there's no debt. While you are using the monorepo, there's nothing to "pay off". If after you make the change, but you don't do it fully, then you start having things to "pay off".
That's the opposite of what I wrote. My scenario was you knew that it didn't make sense for that project but you still did it because of time/budget/resource reasons. You start paying off the day you can't scale it anymore.
time/budget/resource reasons means it DOES make sense. It's like saying you want a Ferrari, but at the moment all you can afford is a bicycle... its a start, it gets you going, and it maybe not what you want long term, and you can't realistically go long distance, but it gets you started. They are just strategic decisions, not technical debt.
There's a difference between deliberate technical debt and accidental debt. More often what I have seen is accidental debt - not a deliberate choice between tradeoffs but accumulation of poor quality code and architecture due to inexperience, poor time planning, and constant changes in business direction that are common to new companies.
If business requirements change, that isn’t tech debt.
If you don’t implement new high-level (business) insights or postpone this work, you are accruing technical debt. At least, that’s what Ward Cunningham stated.
Technical debt is the gap between what you think is best and what you actually implement. If you think a test should be added, but you don't write one yet, that's technical debt. If you feel like separate repos would be best in the future, but you choose a monorepo for now, that's technical debt.
Please stop using that term. The idea was to communicate "technical stuff that needs to be done" to product management and sometimes have a lever to rework the shortcuts taken due to business pressure - in that case it should be called product debt, and not tech debt. Product took out the money (earlier release), not technology. And needs to pay it back with scheduling development work.
What I see is that the term makes technology look bad, doesn't work as a lever and demotivates developers.
Be a craftman, don't do technical debt. For time to market create a zerocode/crappy MVP that you throw away. Schedule real technical debt like DB migrations and do them as needed.
I do agree that the term feels like a misnomer, but mainly because it gives an air of respectability to simple bad engineering with great cost. The idea is that debt is accrued intentionally and strategically, but everyone is so used to constantly referring to tech debt that just doing a job well has been completely forgotten. If there are real problems, can we just have a list of tasks to improve things? Maybe it's just boredom but the term 'tech debt' doesn't feel that it's doing much good any more.
> For time to market create a zerocode/crappy MVP that you throw away.
Unfortunately what tends to happen is that as the business sees early success and starts to grow, there is never a stopping point where people are happy to start again and build from scratch 'properly'. That crappy MVP will be extended, adjusted and corrected, until it's something no longer throwaway but also not well built/tested/hosted/monitored/etc.
Maybe slow down to go faster is a better approach.
57 comments
[ 2.8 ms ] story [ 124 ms ] threadSomewhere along the way, it became fashionable to build out your devops on day one as though you’ll be Google tomorrow, but to write your code without any architectural foresight as though the only thing that matters is tomorrow’s deployment.
It’s always nice to see people advocating towards a thoughtful balance.
But the calculus changes if instead of EC2 nodes you can have a helm chart that deploys directly to a managed kubernetes services like EKS or GKE?
If you have a lot of experience with any interface, you can do it really efficiently, but if you don't, kubernetes can make your experience worse than anything you'd come up with on a cloud provider.
Even with ECS which is way simpler than K8 for small teams using Fargate has been incredibly beneficial in terms of debt/time.
Then next step would be to move the entire application onto Lambda so we're not paying for downtime, but that's a whole headache and a half.
Then you'll need non trivial amount of knowledge of how it works to avoid many pitfalls (the helm chart will probably only go so far, it will still need small tweaks for your specific use case)
So it's a combination of upfront time investment, ongoing tuning, and recurring maintenance events.
> Coming from AWS to Linode VPS with Caddy + Docker felt great
I guess that depends what you were using with AWS, but Fargate + k8s is a pretty amazing workflow.
> Im wondering at what point (N requests per second ?) it becomes tech dept
I make these calls for our org, and I wouldn't call it tech debt based on the performance or scale, but because of maintenance. As an example, if you needed to hire a new person/replacement person for your team, would it be easier to find someone who can maintain k8s or docker compose?
The other area it shines for is the ecosystem. If you need to add monitoring to your app (e.g. with Prometheus), adding it to a k8s cluster is `helm install prometheus`
My last team was the same, there was ~100 developers on the main app, ~50 on the ancillary services, and a ballpark of ~10 on the "platform". I was just a team member on that team. Current org is 15 devs, 30 people total.
> My concern is that there is a cost to magic of adding deps like that.
Of course there is, there's such thing as a free lunch. It's up to you to decide which is right for your team/org. The cost of doing it manually is developer time, tech debt and the bus factor. The cost of using helm is introducing an abstraction. Profesionally, the bus factor and developer time of home rolling a solution make relying on existing solutions a very appealing proposition. Having to maintain a set of half baked shell scripts is exactly what tools exist to solve, and I've had more issues with deployments where someone has bunged together a bash script that makes varios assumptions about tooling, versions, installation paths, working directories, path formats, etc. than I have with problems related to our actual deployment infrastructure.
> I guess my question is at what org scale the tradeoffs are right
The cost of switching is huge, and likely not worth it unless you have genuine problems that would be solved by moving. For a greenfield project, frankly I'd think you're wrong to use docker-compose on any new project. Either you're managing the underlying infrastructure (ec2/vps style) or you're not (fargate/DO App Platform style), and in either case docker compose requires roughly the same amount of investment for a smaller return on a less actively developed product objectively solves less problems and requires more work to integrate.
[0] https://github.com/marketplace/actions/deploy-to-kubernetes-...
Whatever claims about user friendliness, I didn't figure a stable way to do deploy separate docker-compose.yml's from same user at the same time.
That's a big part of many k8s pitches I've seen but in my experience it hasn't been representative of reality (or at least for every part of infrastructure you no longer need to run there's a new part of k8s architecture/config to learn & manage).
For me the selling points are:
1. For the time-cost of upfront complexity and setup, you get extremely easy and quick subsequent addition of infra/tooling & scaling thereof. 2. There are certain patterns of tooling that k8s enables (relatively easily), that would have been difficult/impossible with more traditional setups.
I don't think it makes infra management overhead less, it just reorganises it so it's more upfront and then less later while scaling.
To build the servers I was running on I wrote an Ansible job to install everything and copy the code down. All I had to do was add a couple flags to that to allow it to just deploy new changes. Instead of getting into the CICD pipeline I just took a few minute to make a little web front end for myself to kick off the job and watch the job status (we have Tower, but it's painfully slow). I can now deploy in 18 seconds with 1 click. Seems good enough. I'm sure I could set up a web hook to eliminate the click, but it's not a big deal.
At my company, it would involve going to weekly meetings to get all the updates and changes happening as they keep developing it, one-off meetings to get onboarded and setup... I'm pretty sure they also designed the system for Kubernetes, so I figured it would also mean containerizing the site, as it has been running on VMs for almost 10 years. Those weekly meetings also mean constant changes so things keep working (which I'm not dealing with on a different site that is going though CICD). All of this in the midst of a lot of bureaucracy. It's also a custom home grown solution, so it's not like AWS knowledge I can take with me or google.
It likely would have taken several months for little to no tangible benefit. No one on my team knows how CICD works. I'm the only one who has actually used it, and even my use has been pretty limited. It took 8 months for someone to bother to tell me where I could see build status/errors, at first they just told me to "wait about 10 minutes". There is no documentation anywhere. It's all tribal knowledge and no one seems to have the time to share it.
The only way to maintain my sanity is to not get too involved in it all. It will likely all change and go away within the next 2 years. Nothing lasts lone enough to get too invested in.
Yeah - this is key. If you're already set up to _not_ be containerised, then it's likely not going to be worthwhile unless you are actively feeling some of the pain points around running on VMs.
> It's also a custom home grown solution, so it's not like AWS knowledge I can take with me or google.
One saving grace is that a huge amount of what k8s does is abstracts the aws-ness from underneath. Running a live application on k8s on aws is three separate skills - running a live app, running apps on k8s and running k8s on aws. Two of three of those still transfer!
> fit. No one on my team knows how CICD works. I'm the only one who has actually used it, and even my use has been pretty limited.
That's fair, and it's up to you to make the call as to whether it's worth it or not. If your team aren't willing then the use cases are very limited, and that's a political fight!
> The only way to maintain my sanity is to not get too involved in it all. It will likely all change and go away within the next 2 years. Nothing lasts lone enough to get too invested in.
I've been a driver for improving developer best practices on every team I've worked on. Not everything has always stuck, but I've found that processes that can be automated tend to have high rates of success, and processes that rely on humans tend to be sidestepped. I spoke to a coworker from a previous job last night who told me that some of the process fixes I made as a cog in the wheel have survived 2 years without me there. All is not lost!
Yeah, it's just a matter of trying to pick those horses that seem likely to stick around, or that actually make like easier. Ansible is one of those; the CICD process and everything around it is not. We've changed out container platform 3 times in 3 years, and just kicked off a project to change it a 4th time, while also opening up some public cloud access if it can be justified. It's chaos.
There was also a push to build a lot of home-grown tools over the last 3-5 years. All of that is being ripped out in favor of vendor provided solutions.
The primary things that seem to stick around are the snowflakes, as sad as that is. The only teams that don't seem stressed by all that change are the ones who just do their own thing. I fought the good fight, jumping the latest and greatest stuff for over a decade and trying to put systems in place to get others to buy in as well, but I'm tired and this latest crop of tools seemed like they were destine for failure from day one.
I try to align when possible. I think your solution is hacky and costs the company much more in the long run.
I used to try to align when possible, but lately it hasn't made a lot of sense. Keeping up with the tooling ends up being the whole job, as they change it every 6 months. This site is a side project to help out the team, not my primary job. I can't be dropping everything each time they decide to switch vendors, change up their variables names, or whatever else they decide to do.
Sounds like your system is way more efficient than the CICD... in the short term, provided you're the only person that ever needs to use it (and/or nothing ever changes)
This seems like a very large assumption.
> as they know Ansible, but have no idea how the CICD pipelines work
I know Ansible very well. I have never set up a CICD system and would much prefer to write a bunch of Ansible playbooks for myself than to do so. My company has a CICD system already set up (I have no idea how it works internally). Setting up a pipeline and understanding how that pipeline works as a user of the CICD system is still a lot easier for me than figuring out someone else's Ansible playbook repo. Adding jobs to that pipeline is still easier than writing new Ansible tasks. CICD is almost always easier for dev teams regardless of their experience once the initial complexity of CICD setup/hosting/maintenance is already handled.
If you don't already have a CICD set up, then setting up one is likely overkill for small teams. If it's already there (and maintained by a well-resourced team), use it, unless the CICD platform being used is absurdly bad for some reason (I've used a few and they have been largely fine / interchangeable).
As mentioned in another comment, the site has been running on VMs for almost 10 years and not containerized. The CICD pipeline is built around Kubernetes and I have no real reason to move to Kubernetes other than to say it's running in containers. The VMs have never had issues, it's a relatively small internal site, so I don't need to worry about scaling issues. K8 is also in the process of being replaced, so moving to it now seems silly. I use the pipeline for another project I'm working on with another team and while it's needed for the stack they're using, it's bad. No one seems to know how it works, I haven't spoke to a single developer who has a positive thing to say about it, it's very poorly documented, and any time there is an issue it requires messaging a bunch of random people until you find someone with the right blend of tribal knowledge and time to help. I've had a half dozen people come to me just to find out where to see the deployment status or to know how to promote something to upper environments... and I wasn't given that information until 8 months in. Seems like it would be day 1 stuff that would be easily documented.
For a hackathon they made something to do a one-click setup of a project that I would likely use, but outside of the hackathon, that feature isn't available and it would require months of work when factoring in meetings, waiting on other teams, etc to get setup without that. It's just not worth it for my use case.
Personally, I would run k8s from the day I expect to run the code as separate deployment to running it from my IDE. I have the skills to do so, and I found I lose way less time doing so than trying to do "classic" server setup, even if it's just one server.
Everything else would only mean less golden path support, less synergies etc.
As someone who works in a small team within a large corp where we use k8s for everything large and small, I did not read that statement as applying to me.
If you have a subsystem conform to a good API and is well behaved it can be rewritten. It is allowed to have some tech debt.
If the subsystem is sprawled through other subsystems that tech debt doesn't get paid down because now you need an expert in all those areas to fix it.
I swear I have worked places where there is code no one currently at the company can understand. At best you micro-fix for bugs and back the hell away!
It's funny and gross enough to stick most of the time.
That said sometimes abstractions leak. I was using Firebase and getting the public Url for a blob given it’s path is an async call! God knows why.
Nobody built the pyramids in a day.
Here is a different attempt on definition: An occurrence on debt is when a higher up decides that a product/construction must be ready when it clearly is not, and thus convinces/bullies the engineers and put their signature on it.
Analogies can be a great tool to communicate or for prose but don't take them literally.
Unfortunately your description is what has become the common usage of TD, and now the term is meaningless.
Frankly speaking, today, TD is just a neutral term to externalize blame/responsibility.
That's the opposite of what I wrote. My scenario was you knew that it didn't make sense for that project but you still did it because of time/budget/resource reasons. You start paying off the day you can't scale it anymore.
If you make a bad choice with no valid reason then it's just a bad choice, not technical debt.
If business requirements change, that isn’t tech debt.
If you don’t implement new high-level (business) insights or postpone this work, you are accruing technical debt. At least, that’s what Ward Cunningham stated.
What I see is that the term makes technology look bad, doesn't work as a lever and demotivates developers.
Be a craftman, don't do technical debt. For time to market create a zerocode/crappy MVP that you throw away. Schedule real technical debt like DB migrations and do them as needed.
> For time to market create a zerocode/crappy MVP that you throw away.
Unfortunately what tends to happen is that as the business sees early success and starts to grow, there is never a stopping point where people are happy to start again and build from scratch 'properly'. That crappy MVP will be extended, adjusted and corrected, until it's something no longer throwaway but also not well built/tested/hosted/monitored/etc.
Maybe slow down to go faster is a better approach.
Yes and from talking to my coachees, this is one of their main problems. I urge them to throw away the MVP when they got funded.
Or clean code because no one will ever touch it again.
Or a feature you only need once and just write it somehow.
And then all this shit stays in your product and starts dragging you down.
This is totally different than when you start with a monolith and know that you might want to rework it later but accept potential future work.
And you should start with a modular monolith anyway.
I don't think it's reasonable to try to redefine the term technical debt.