Thanks for posting this. I'm looking to get any application and start deploying through a CI/CD process, to learn how this process works. Any application suggestions for a CI "Hello World"?
I think you mean CD “Hello World”, not CI. You can make a “hello world” web app (or even just an API endpoint) in your favorite language and set it up to deploy on each commit.
CD means each commit that passes tests is deployed
CI means your code is regularly integrated (merged) vs having long QA / code review cycles.
CI and CD work great together but you can have one without the other.
Yes, I want to better understand the process of committing code to the repository of an application, have it run automated tests, report via a communications tool the result, and automatically deploy if successful. Understanding this efficient process is high in demand and thus highly profitable.
I've created and worked on several projects that have involved APIs, reading from Zendesk, and writing to Slack or HTML, among other places. I've supported Jenkins and related tools multiple times in production. Now I want to fill this gap in my knowledge.
Travis is a lot more user-friendly than Jenkins and they run it for you, whereas Jenkins you usually self-host. (Jenkins is a lot more flexible and has a lot going for it, but Travis is much easier to get up and running)
Travis is a third-party service. Jenkins is (usually) software you run yourself. Travis will run the infrastructure for you and will not charge you if your project is public on GitHub. So that makes it an extremely popular choice for open-source projects: sign into Travis with your GitHub credentials, click a checkbox on your repo, and add and commit a .travis.yml file, and now you have CI that is well-integrated with GitHub and you don't have to run any infrastructure or pay for any server hosting.
Of course, you have much less control over Travis than Jenkins. Two cases where Jenkins remains obviously better are when you need custom environments that aren't easily portable to Travis (Windows support is a big one, although there is a company called AppVeyor that is something like a Travis for Windows; licensed software is another big one), or if you have strong security/confidentiality requirements (e.g., you're in a regulated industry) and using random cloud services isn't a good option.
Only comment I would make is that you can use CircleCI instead of Travis to get for free running tests on private repos. The configuration is similar and the benefits as well.
or be willing to move vendor and re-implement the stuff.
For comparison my magic combination(and not all free or realistic for everyone):
- gitlab
- jenkins/looking into gl ci
- linode/cloudafrica(ah-hah hosting is not free)
- sentry is a tough one. We rolling a custom mqttwarn/zabbix solution
- irc(fine I will take your slack and use the irc-bridge pfftt)
I really wanted to use Gitlab for my next side project, but Circle CI doesn't support them, and I really don't want to pay Travis for my shitty proj. Hopefully this changes some day soon.
We use gitlab's built-in runners and pipelines for continuous deployment to kubernetes and it works very well. We self-host our repos but the same tools are available through the hosted version.
GitLab's built-in CI is very good, and simpler than Travis for some things.
You also have the option (unlike Travis) of registering your own runners in your project settings, and you can do this either on gitlab.com or another deployment, without needing to be an admin of the deployment - you can get a key for it in your own project's settings. I'm using this at work, where we have an internal GitLab that I'm not an admin of (and where the GitLab admins probably have no interest in running a CI system), to do a small bit of CI on some things using one of my team's servers as a runner.
> - irc(fine I will take your slack and use the irc-bridge pfftt)
We ditched Slack for Mattermost. It's not entirely up to par (yet), but works for most use-cases. And there's an IRC bridge for it, too (https://github.com/42wim/matterbridge). Oh, and Mattermost comes bundled with GitLab in their omnibus.
Heroku does have a free tier, but it's intended for experiments. It's very limited and wouldn't be acceptable for any serious website, certainly not a 'world class' setup. Unless you're very careful to have one free dyno per account, and avoid using features like staging apps, they will power down your server for several hours a day.
Yeah, the “world-class” hook was more about the quality of these tools than their applicability for use at scale for free - I encourage people to pay real money at various points in the article.
I’m currently running my blog on a single $7/month Heroku Dyno (plus $9/month for Heroku PostgreSQL) behind a free Cloudflare account - with the right caching headers I can easily handle huge amounts of read traffic. I was going to mention Cloudflare in this piece but I decided it didn’t fit the theme of continuous deployment as well as the other tools.
When they say that AI is not going to take over dev jobs, show them this thread and tell them that the time cut down because of automation is going to be spent in something more meaningful, like coding!
So you equate "world-class" with "enterprise"? I disagree. Tools that scale down can also be world-class, and in fact that ability to run cheaply is a big part of what allows startups to get off the ground and bootstrap their way to eventually challenge the big boys. Heck, even if they stay small, they can still have world-class tech. Once you get big and purchasing decisions are made far from the front lines I'd argue it gets harder to really have the best tech.
Well as part of my current world, sure I do. I have to satisfy the needs of not just dev, but operations, compliance, security, business, the people overseas that i've never met, etc. What I'd really like to see are startups that bootstrap this stuff to think that way prior to growing.
My original point was that this was cool for the dev crowd and perfect for the blog use case, but "world"-class seems hyperbolic. But if you are working for other people: people walking around who have strung together various automation stacks are a dime a dozen -- I've literally seen 4 different takes this month. Execution matters, and execution for your audience matters more.
>> Tools that scale down can also be world-class
We are of the same opinion. I love me some elegant free solutions, and I need to because then I can judge what I'm buying when I need to buy something.
>>Once you get big and purchasing decisions are made far from the front lines I'd argue it gets harder to really have the best tech.
I guess I see where you could be coming from, but it doesn't HAVE TO be that way does it? Isn't the whole point of work to empower yourself to affect the company decision-making. Failing at doing so is sort of, without offending you, you not doing your job. At least that's how I look at it.
It's important to understand that you are effectively locking yourself in under a promise of a free tier and should plan to make sure you can afford the paid subscription in a couple of months or years.
Some companies may also sell your data to third-party affiliates, with, or without your consent or knowledge.
Does anyone have a solution for a situation where I have a Django backend and a VueJS front end and want to run the Vue build step before deploying? Currently doing this with Git hooks and committing the resultant build, but that is bad for a number of reasons. This is running on Heroku. Would their release phase allow for this without significantly slowing things down? Would it deal with npm installs?
We have this with GitLab CI, although with React and not Vue (but I don't think this matters). We have two stages, `build-assets` and `build-app`, first one runs Webpack etc, and the second one copies the build artifacts into a Docker image with Django and finalizes the image with collectstatic.
We're using Docker, but I'm sure the same approach can be used with simple tarballs, OS packages (.debs, etc) or deploy-specific git repo (for Heroku)
You do not need to commit your build to git if you are using heroku.
If you add the nodejs buildpack to your heroku project, it should detect and build your package.json automatically on heroku. Works with both yarn and npm.
You just need to make sure to set NODE_ENV=production in your heroku config.
You might be able to solve this using a bin/post_compile script - most standard Heroku buildpacks will run these as part of the build step, and they can be used to run commands that generate files which will then be included in the compiled app that gets deployed to Heroku.
I've done this for several apps. We have a large Jenkins 2 cluster and use declarative Jenkinsfile's. For Jenkins, you can accomplish this by simply setting up a Jenkinsfile that has an agent defined in each stage. Build your VueJS app in the first stage with a NodeJS container, and build the second stage with a Python container. Bonus points for building an actual Docker image as your artifact that has Django inside of it with Apache or nginx serving the VueJS bits too.
Two hours after posting, and this is the only comment about Concourse? jacques_chester must be on holiday.
Is it possible get get Concourse running entirely on free tier stuff? I don't think there's a hosted Concourse service yet - airplane.ci is coming, but still in beta - and i imagine you'd struggle to get a whole setup on a t2.micro.
Codeship (https://codeship.com/features) allows forever free builds for public repos. It has support for both Continuous Integration and Delivery/Deployment. Maybe you can give it a spin as well?
I can second codeship - even though its a bit more clunky than Travis and with less integrations and command-line tools, its has way beefier machines, easily several times faster than what travis offers even to paid customers. And their tools are improving. Keep up the good work.
Codeship founder here. We support .yml file config for our container/Docker based product (called Codeship Pro). We're looking into offering a UI + yml file approach for both products because we see a 50-50 split when it comes to what people prefer. Thanks for the feedback :)
Heroku is awesome for this use case, but there are also great alternatives based on Cloud Foundry. Cloud Foundry is an open source PaaS with support for Heroku buildpacks + docker containers. At a median of ~5€/GB RAM per month they will also be a bit cheaper than Heroku.
Some providers that offer Cloud Foundry including a free tier for small Apps:
Once you need something a bit more heavyweight than Travis (e.g. more CPU) I would highly suggest Buildkite (https://www.buildkite.com). It’s a “bring your own machines” solution, which means you can have beefy machines (e.g. via buildkite on EC2 spot autoscaling), or even machines that run inside production without handing buildkite itself any secrets. We’ve been using it at Ladder and it’s been phenomenal.
Sentry also has self-hosted option available (and it's also opensource).
Instead of GitHub + Travis CI, you can use GitLab (it also has bundled CI capability, based on Docker).
with GitLab, you can also have Continuos Deployment, wired up with the CI solution. The best experience will be deploying to Kubernetes, as it will give you even monitoring, automatically, but you can code custom deployment logic as a step in the CI pipeline.
Not sure bitbucket pipelines are mentioned here. But for free I manage to commit Hugo source files to a private git repository, generate the static files by spinning up a docker container and generating them using Hugo and finally rsyncing the static output to my Webserver. All for free. You only get 50 minutes of time now (for free) in the docker instance but for simple continuous deployment cases like mine it’s very effective.
You might want to look at Gitlab if you want to drop the need for a web server. Their pages support supports Hugo much better than GitHub (which leans toward Jekyll), and is pretty well documented.
Add Netlify and it will continue to be free. I use Hugo as well, Netlify compiles Hugo on it's own, so I don't need workflows, but it is cool to have them.
It’s fully automated. I do a commit and push from any machine that I have cloned the repo to and it gets published automatically.
Now I don’t need to download Hugo, remember the Hugo commands, remember my web server login details, etc. I think it’s a lot less fiddling as all I need to remember now is how to add a file to a git repo to get a post published.
Don't you miss out on Hugo's archetypes doing it that way, or at least need the templates locally? Also you shouldn't need your web server login details for rsync... Hey whatever works, it just seems like substantially more work to me.
Doesn't show any automated verification of code in deployed fashion, no load tests or verifications which would be part of a world class pipeline. Also no canary or red-blue tree type rollout hooked to sentry. At least it has a staging but don't know what they're using it for other than to test migrations before promoting to prod. Nice little write up but far from world class.
Exactly - continuous deployment is about moving some/most testing to production, not eliminating it altogether.
Acceptance testing the system once deployed is the cornerstone of organisational acceptance of the CD doctrine - errors will always happen, it's just that they can now be fixed much faster in production.
Using an acceptance test suite as a deployment safety harness should dramatically increase the quality of tests, and hopefully the application too.
I can't believe the author doesn't mention gitlab. Setting up full CI on gitlab is amazing. They have the easily integratable gitlab runners i.e. travis-ci functionality + different build stages for automatic deployment via ansible/chef/whatever and unlimited free private projects to boot. If you're working on anything that can't be public I highly recommend gitlab. Its an amazing service.
> , but Gitlab pages are also much easier to setup
uuh, what?
i think gitlab pages are wonderful, because you're not restricted to jekyll, but how is it easier to setup? you only need to toggle the checkpoint in the repository settings...
GitLab is great and has come a very long way over the last several years. However, I still gladly pay GitHub $7 a month for my private repos because I feel that GitHub truly does offer a better code review and repository experience. GitLab in certain places is still very rough around the edges, not to mention performance has always been a concern.
To me, GitHub still is the gold standard and will likely be for quite some time.
Not sure how, it's got the same here is your changed files and the changes.
I find GitLab is better on a few point. You can theme it from about 4-5. I prefer the dark theme as it matches my IDE and easier on my eyes. You can also have comments marked as addressed. By default the changes are shown in the page but you can also choose to show side-by-side like in GitHub.
You can host it yourself. It ties in very well to it's own CI/CD service and comes with a docker registry. So your code can be tied easily to your deploys and you see this in the issues.
> can't believe the author doesn't mention gitlab.
I absolutely agree. We use Gitlab in our company for continuous deployment and it works like a charm.
Apart from the other tools I miss some performance monitoring. Sentry is great, but if your application suffers from bad performance after the latest deployment, you won't notice that soon enough.
I recently found Datadog for performance monitoring (it's like statsd++++) and I'm already a big fan. I haven't managed to get it working with my Heroku projects yet though.
91 comments
[ 3.8 ms ] story [ 162 ms ] threadThanks for posting this. I'm looking to get any application and start deploying through a CI/CD process, to learn how this process works. Any application suggestions for a CI "Hello World"?
https://about.gitlab.com/features/gitlab-ci-cd/
CD means each commit that passes tests is deployed
CI means your code is regularly integrated (merged) vs having long QA / code review cycles.
CI and CD work great together but you can have one without the other.
I've created and worked on several projects that have involved APIs, reading from Zendesk, and writing to Slack or HTML, among other places. I've supported Jenkins and related tools multiple times in production. Now I want to fill this gap in my knowledge.
Of course, you have much less control over Travis than Jenkins. Two cases where Jenkins remains obviously better are when you need custom environments that aren't easily portable to Travis (Windows support is a big one, although there is a company called AppVeyor that is something like a Travis for Windows; licensed software is another big one), or if you have strong security/confidentiality requirements (e.g., you're in a regulated industry) and using random cloud services isn't a good option.
or be willing to move vendor and re-implement the stuff.
For comparison my magic combination(and not all free or realistic for everyone):
For hosting, I am looking to host my own server for fun and learning.
[0]: https://docs.gitlab.com/ce/ci/README.html
[1]: https://docs.gitlab.com/ce/ci/runners/README.html
You also have the option (unlike Travis) of registering your own runners in your project settings, and you can do this either on gitlab.com or another deployment, without needing to be an admin of the deployment - you can get a key for it in your own project's settings. I'm using this at work, where we have an internal GitLab that I'm not an admin of (and where the GitLab admins probably have no interest in running a CI system), to do a small bit of CI on some things using one of my team's servers as a runner.
Sentry is open source, you can self-host it...
Yup, definitely look into it. We just killed our last Jenkins instance last week after we found out that everyone eventually moved to the GitLab CI.
> - sentry is a tough one. We rolling a custom mqttwarn/zabbix solution
Why a tough one? I found Sentry easy to self-host. They have pretty good docs: https://docs.sentry.io/server/installation/
> - irc(fine I will take your slack and use the irc-bridge pfftt)
We ditched Slack for Mattermost. It's not entirely up to par (yet), but works for most use-cases. And there's an IRC bridge for it, too (https://github.com/42wim/matterbridge). Oh, and Mattermost comes bundled with GitLab in their omnibus.
your right, sentry! It was tough thinking of alternatives.
It would be nice to see a setup with something like Gitea[0] and/or Jenkins[1] edit and some other slack alternative like mattermost[2]
[0] https://try.gitea.io/
[1] https://jenkins.io/
[2] https://about.mattermost.com/download/
Pretty sure you can self host travis.
I’m currently running my blog on a single $7/month Heroku Dyno (plus $9/month for Heroku PostgreSQL) behind a free Cloudflare account - with the right caching headers I can easily handle huge amounts of read traffic. I was going to mention Cloudflare in this piece but I decided it didn’t fit the theme of continuous deployment as well as the other tools.
Well as part of my current world, sure I do. I have to satisfy the needs of not just dev, but operations, compliance, security, business, the people overseas that i've never met, etc. What I'd really like to see are startups that bootstrap this stuff to think that way prior to growing.
My original point was that this was cool for the dev crowd and perfect for the blog use case, but "world"-class seems hyperbolic. But if you are working for other people: people walking around who have strung together various automation stacks are a dime a dozen -- I've literally seen 4 different takes this month. Execution matters, and execution for your audience matters more.
>> Tools that scale down can also be world-class
We are of the same opinion. I love me some elegant free solutions, and I need to because then I can judge what I'm buying when I need to buy something.
>>Once you get big and purchasing decisions are made far from the front lines I'd argue it gets harder to really have the best tech.
I guess I see where you could be coming from, but it doesn't HAVE TO be that way does it? Isn't the whole point of work to empower yourself to affect the company decision-making. Failing at doing so is sort of, without offending you, you not doing your job. At least that's how I look at it.
[edit: formatting]
It's important to understand that you are effectively locking yourself in under a promise of a free tier and should plan to make sure you can afford the paid subscription in a couple of months or years.
Some companies may also sell your data to third-party affiliates, with, or without your consent or knowledge.
We use this to listen for Github webhooks and execute jobs with Saltstack.
Small correction: It will automatically test the latest commit on every push to every branch
https://github.com/stickfigure/gstrap-skeleton/
Deployments are git checkout production; git merge master; git push. I love being able to use git log to see what changes are live.
We're using Docker, but I'm sure the same approach can be used with simple tarballs, OS packages (.debs, etc) or deploy-specific git repo (for Heroku)
If you add the nodejs buildpack to your heroku project, it should detect and build your package.json automatically on heroku. Works with both yarn and npm.
You just need to make sure to set NODE_ENV=production in your heroku config.
https://devcenter.heroku.com/articles/nodejs-support#build-b...
Here's an example that might be useful: https://github.com/nigma/heroku-django-cookbook/blob/master/...
Is it possible get get Concourse running entirely on free tier stuff? I don't think there's a hosted Concourse service yet - airplane.ci is coming, but still in beta - and i imagine you'd struggle to get a whole setup on a t2.micro.
Codeship (https://codeship.com/features) allows forever free builds for public repos. It has support for both Continuous Integration and Delivery/Deployment. Maybe you can give it a spin as well?
Some providers that offer Cloud Foundry including a free tier for small Apps:
- Meshcloud (hosted in Germany, 2 Locations) https://panel.meshcloud.io/#/register (Disclaimer: I'm a founder there)
- Pivotal Web Services (hosted on AWS US): https://run.pivotal.io/
- IBM Bluemix https://www.ibm.com/cloud-computing/bluemix/ (various locations, a bit more expensive than the other two though)
Also it has a lot of emojis.
Sentry also has self-hosted option available (and it's also opensource).
Instead of GitHub + Travis CI, you can use GitLab (it also has bundled CI capability, based on Docker).
with GitLab, you can also have Continuos Deployment, wired up with the CI solution. The best experience will be deploying to Kubernetes, as it will give you even monitoring, automatically, but you can code custom deployment logic as a step in the CI pipeline.
Now I don’t need to download Hugo, remember the Hugo commands, remember my web server login details, etc. I think it’s a lot less fiddling as all I need to remember now is how to add a file to a git repo to get a post published.
To set it up is simply a matter of starting a Docker image and creating a .drone.yml file (very similar to .travis.yml).
Note: I'm not affiliated with the project in any way. I'm just a happy user who think the project deserves way more buzz.
To make this work, I use Codeship instead of Travis. Codeship builds a Docker image, runs the tests against the image, then pushes the image to Heroku if the tests pass: https://documentation.codeship.com/pro/continuous-deployment...
Acceptance testing the system once deployed is the cornerstone of organisational acceptance of the CD doctrine - errors will always happen, it's just that they can now be fixed much faster in production.
Using an acceptance test suite as a deployment safety harness should dramatically increase the quality of tests, and hopefully the application too.
FYI: I am in no way affiliated with Gitlab.
I've moved all my sites to Gitlab pages generated via Hugo or Jekyll.
Uptime (via Uptime Robot) just doesn't seem as good as GitHub, but it's not bad.
uuh, what?
i think gitlab pages are wonderful, because you're not restricted to jekyll, but how is it easier to setup? you only need to toggle the checkpoint in the repository settings...
Edit: And what I said in original comment. Gh_pages branch was required, but Gitlab could just use the built assets.
To me, GitHub still is the gold standard and will likely be for quite some time.
I find GitLab is better on a few point. You can theme it from about 4-5. I prefer the dark theme as it matches my IDE and easier on my eyes. You can also have comments marked as addressed. By default the changes are shown in the page but you can also choose to show side-by-side like in GitHub.
You can host it yourself. It ties in very well to it's own CI/CD service and comes with a docker registry. So your code can be tied easily to your deploys and you see this in the issues.
I absolutely agree. We use Gitlab in our company for continuous deployment and it works like a charm.
Apart from the other tools I miss some performance monitoring. Sentry is great, but if your application suffers from bad performance after the latest deployment, you won't notice that soon enough.