Ask HN: How to deploy into production large software project?

Automate the low-hanging fruit first (copying files, maybe, or centralized health-checking so you don't have to open logs).

For a lot of improvements you can make a business case other than your quality of life. Health-checking improves reliability, for example. But accept that these will be hard conversations that take a long time. Don't let others convince you this isn't a problem.

Research the tools that solve your problems -- it's easy to argue for adopting an industry-standard tool that solves an industry-standard problem.

Accept that certain pieces of the process will stay manual for a while.

Don't be afraid to go after small improvements -- if you can halve the difficulty, the remaining steps will fall away easily.

You probably have many levels of hierarchy working against you. If you have the opportunity to lead or manage anything, take it.

You can get incredibly fancy and automate everything and deploy tens of times a day etc. but you don't have to. You just have to pick practices that would improve your current situation, and in most cases more automation helps.

In my current job, despite us not having everything down perfectly, we've got to a point where I can tag my branch and run a script that deploys the tagged code as a separate container locally if I'm doing development.

Then once we want to deploy to production, another set of configuration files (also versioned, with the smallest diff possible from the dev config) is fed to the same script, and the service is deployed on the production server.

Then you automate running this process across your servers (are your servers your own? Are they on the cloud? Use the tools relevant to your situation).

Then, there is no need to perform manual monitoring of the logs. Set up a monitoring service to notify you of any worrying logs (warn level and up? Specific patterns? Up to you and depending on what you're logging) and feed that service your logs. Hell, automate pinging these services so you can check whether they're in the state you expect, and make a colour coded minimal dashboard that anyone can check in a sec and find out if the everything is in the state it should be.

If the business is paranoid, then do smoke testing at the end, but you shouldn't need to.

The service listens for manifest change message on a topic, when a team wants to release a new version of an app, they update the url of the jenkins artifacts in a manifest file in git and publish manifest changed message.

Deployment service on all the servers downloads the latest version of the manifest file and updates or installs apps as needed.

I also wrote a custom prometheus exporter to export health and version of services installed on a box. That allows us to track the deployment status.

This setup has worked super reliability for us, very little maintenance and training required. Since deployment occur in parallel, deployment on hundreds of servers takes only a few seconds.

* choose a configuration state manager like Salt, Ansible, Chef, Puppet, etc. Of all these I'd probably say Salt is the simplest for somebody approaching this methodology.

* choose a source control management system -- I'd just go with git, and host something on github or gitlab or bitbucket or whatever.

* choose a secure storage medium for the variables that are going to be loaded into your configuration state manager.

* choose some form of automation management tool, Jenkins, Thoughtworks Go, TravisCI, CircleCI, w/e.

You're going to use these in conjunction to gain the following: when a developer pushes code to a specific branch, this push triggers a build on the automation server, which is able to package the code, and distribute it to the nodes managed by your configuration state manager. Aka, a push to your repo makes it all happen (with testing, etc as gatekeepers to deployment of course.)

I'll describe a basic setup I've encountered in the wild using Salt in a master-agent setup (since it's the most basic imo, aside from headless):

-> developer writes some code

   -> developer pushes that code to github

   -> a POST webhook containing branch info is sent to my Jenkins box

   -> Jenkins initiates a build job that pulls down that branch of code

   -> Jenkins uses make (we use make) to build the code branch.

   -> Jenkins deposits the built code artifacts on a fileshare somewhere.

   -> Jenkins calls to the salt-master to update all nodes.

   -> the salt master (which could just be hosted on the Jenkins i guess) tells all minions to update

   -> all minions pull the new code and HUP themselves.

I use Jenkins Groovy pipelines (which I'm not that crazy about) to ensure that the Jenkins jobs themselves are captured as code and are repeated the same way each time across my make build steps. I work in AWS, so the machines I use, the bootstraps they boot with, the desired state config files, and everything are defined as code in some regard.

You should attempt to push for 'desired state' and move towards 'immutable state'.

I would also strongly consider looking at the other options aside from salt, it's only so-so.