Ask HN: How do you version control your microservices?
When working on a project consisting of a large number of microservices, what solutions do you employ for controlling the composition of specific versions of each microservice.
I currently use git submodules to track the application as a whole, with commit refs for each "green" version of microservice. This "master" repository is then tested with consumer driven contracts for each of the referenced submodules, with subsequent "green" masters for deployment to staging.
This submodule approach requires a lot of discipline for small teams, and on more than one occasion we have encountered the usual submodule concerns. I'm concerned that this will only become more problematic as the team grows.
What are your thoughts for a replacement process?
52 comments
[ 3.2 ms ] story [ 77.0 ms ] threadMicroservices must be compatible with each other. We cant simply bring up the latest version of "microservice A", because comsuming "microservice B" may not have been updated to account for API changes (which are enforced by testing contracts). Thats what this master repo is for, to track which microservices work with each other.
Obviously, the master application is dependent on the microservices. Microservices are dependent on specific versions of other microservices, etc. That is the problem I am trying to solve.
This should not be the case. Build your services to be backwards compatible. Avoid making breaking changes, but when you do, make a new API version & maintain the previous one until it's no longer used.
i.e. /v1/blah
-- breaking change introduced --
/v1/blah - maintains backwards compatible behavior /v2/asdf - new behavior exposed as v2, update microservice B at some point in the future to use this
As these are internal services (microservices) that only we consume, there is no need to keep old cruft around.
You can also do parametric jobs in jenkins which could allow combining arbitrary microservices versions
Or just version your APIs and declare explicitly what microservice uses what version of the API to communicate with the other service.
The approach is similar to parametric builds in jenkins. The problem is the management of the parameters.
For example:
msA v1.0 & maB v1.0 are individually green, and pass all integration tests. Green build for app.
msB is updated to v1.1. Passes individual tests but fails integration with msA v1.0. Red build for app.
msA is updated to 1.1. Passes individually. Still fails integration with msB v1.1. Red build for app.
However. msA v1.1 is still compatible with msB v1.0, so we could have a green app build with a newer version of msA.
Automating this process is what is becoming cumbersome. We have many more services in dev at any one time.
- What other services each service depends on
- What previously released versions of services worked with what versions of dependecies
- Ability to automatically test new versions of each service against its dependencies
Then you plug all that in to a solver that runs on every build, and voila, you have your Microservice Dependency Constraint Satisfaction Manager Machine™. If you do go this route, it would make for a great open-source project.
Also look into the `repo` tool by AOSP for managing many repositories.
At Clarify.io, we have about 60 repositories and 45 services we deploy.
How do you define what is considered the current version for each microservice in your application?
The reason for being able to push "breaking" changes into the codebase is for future consumption. We control the consumers, which verify that they are able to integrate.
When all consumers have had their contracts updated to be insync with that service, then that service is no longer breaking - even though it hasn't been changed.
Its basically a way of preventing inter-service blockers.
I must admit, I thought these kind of pipelines would have been a lot more commonplace than they appear to be.
Obviously this isn't the exact scenario you're in at the moment, but if you're big enough -- this is exactly what it'll be like.
Implement backwards compatible changes and implement tracking on the users of the old features you want to disable. When your tracking shows nobody is using the old features, delete them.
That is how we manage it.
It's simple and provides traceability as you can go into Jenkins and see the commits associated with that release.
http://www.troyhunt.com/2014/02/your-api-versioning-is-wrong...
You should have a build, test, and deploy pipeline (i.e. continuous deployment) which is triggered on any commit to master of any service. The "test" part should include system/integration tests for all services, deployed into a staging environment. If all tests pass, the service that triggered the commit can rolled out to production. Ideally that rollout should happen automatically, should be phased-in, and should be aborted and rolled back if production monitoring detects any problems.
Im using Martin Fowlers description (whom I admire greatly): "In short, the microservice architectural style is an approach to developing a single application as a suite of small services" http://martinfowler.com/articles/microservices.html
My problem is knowing which microservice dependencies the application has.
> You should have a build, test, and deploy pipeline (i.e. continuous deployment) which is triggered on any commit to master of any service. The "test" part should include system/integration tests for all services, deployed into a staging environment. If all tests pass, the service that triggered the commit can rolled out to production. Ideally that rollout should happen automatically, should be phased-in, and should be aborted and rolled back if production monitoring detects any problems.
We have this, and it is what builds the manifest (as a git repo containing the submodules).
The problems occur when you have multiple commits to different services. If a build is marked "red" because it fails a "future integration" then it just means that it is failing at that point in time. It may, following a commit of a dependency, become valid. However, it would need to have its build manually retriggered in order to be classified as such.
This becomes cumbersome when you have a not insignificant number of services being commit to on a regular basis.
As such, I'm not sure building a single repo of git submodules makes much sense. Maybe for accountability reasons, but not for actual operations. The ·raison d'être· of microservices is to decouple the lifecycle of each service totally. By recombining them into a tagged release repo, you're arguably subverting that intent.
Yep. To solve this problem, your CD system needs to automatically trigger rebuilds (and re-tests) of dependent (downstream) services when a dependency (upstream) changes. The easiest way to do that, in my experience, is to have a manifest in every repo that specifies dependencies as a flat list, which the CD system can parse to build a dependency graph.Tangential point, but -- with this, we're just scratching the surface of the incidental complexity that comes when you move to a microservices architecture. This is the stuff that isn't in all of those exalting blog posts -- beware! :)
Thats pretty much how we manage the contracts.
> To solve this problem, your CD system needs to automatically trigger rebuilds (and re-tests) of dependent (downstream) services when a dependency (upstream) changes
This is GOLD. I think you may have just solved the problem with something so obviously simple that it has made me feel like a fool... Many thanks!!!
You shouldn't have services rebuilding/redeploying other services. We don't insist that Facebook rebuilds their applications if my Facebook app changes. There should be either an (a) API client, (b) stub version of that service's APIs or (c) deployed test service for any services you depend on.
The architecture of microservices is pretty simple. It is a mini version of how the internet works.
But in an organization or company, if Team-A redeploys and breaks Team-B, that's everyone's problem, it's not really fine.
So yeah, microservices are like a mini Internet -- to a point.
Microservices should be designed, developed and deployed as 100% autonomous entities. The whole point of them is that you can reuse individual services for new applications or integrate them with third party (e.g. different team in your company) services without affecting other applications. Otherwise what is the point of doing this ?
Using git submodules absolutely violates this and is a bad idea. I would have separate repos and deploy scripts and then set the version number of each service to the build number which guarantees traceability and makes it easier for change management.
The microservices are indeed "deployed" independently, based on the version (ref) as indicated in the service manifest.
> I would have separate repos and deploy scripts and then set the version number of each service to the build number which guarantees traceability and makes it easier for change management.
They have seperate repos, they have seperate "versions" (we use the git ref, which is constant).
As for why I use submodules, this is (in pseudo-bash) what a service deployment looks like:
Microservices that depend on other microservices obviously are not entirely autonomous. You have to version large API changes and integration-test the whole chain of dependent services on every change.
It's not unlike libraries, just linked via network.
As other people have noted as well in here, you should always keep the interface backwards compatible, if needed make a second version of the API or the messages, but never really have to deploy more than a couple of services who really have changed their behavior. The ones just interacting with those services should experience the same interface, being it a couple of versions older or newer.
I'd recommend watching the ucon videos [1] or Udi Dahan's Advanced Distributed Systems design [2] for more in-depth reference material. If you're transforming a team of engineers I can really advise you to join the latter and afterwards order the videos so you can use them with your team as training material! This is less about microservices as them being micro, but more about setting up a distributed service oriented architecture.
[1] https://skillsmatter.com/conferences/6312-mucon#skillscasts
[2] http://www.udidahan.com/training/
I suppose the synchronisation of consumer and service compatibility is the biggest concern.
So far, everybody is focusing solely on backwards compatibility, but not future compatibility, which is what the contracts are for.
With regards to the backwards compatibility - breaking changes happen! As long as the service remains functional, and remains compatible with its consumer contracts (which can also change) I shouldn't need to worry about deprecating APIs. Anyone keeping deprecated functionality around in an environment where they control the both the services and the consumers is simply asking for problems.
I can't see how it can be possible to not control the composition of microservices. Surely thats exactly what my CI pipeline is doing? Composing a network of compatible services?
Micro services should not be directly talking to each other. This couples them in a way that a small API change can be breaking. Instead use a messaging solution so that each service is passing messages and grabbing messages off a queue to do work. This is the easiest way to prevent coupling. It also allows you to version messages if need be and you can deploy a new service to consume those messages. We use JSON, so we can add to a message with no ill effect, and we are careful about removing any required attributes. So we haven't had a need to version messages, but the ability is there if we find it is needed at some point.
Adding messaging does increase complexity in some ways, but once you pass having a handful of services this is the easiest way to manage it.
As a side note. In our solution we have an API that the website and soon to come mobile app tie to. That API interfaces directly with some data schemas but in many places it simply adds messages to a queue for processing.
All that will happen is that it won't be able to process the message, instead of not being able to serve the request.
This thread seems to be full of mad people determined to make simple concepts incredibly complex.
We use microservices but we do allow services to call services.
All endpoint and data contracts exist separately to services and are used by both the service provider (server) and a consumer so all internal contracts are enforced.
That and a fuck ton of integration testing make it work.
And to be honest I really don't like the architecture. It's much much much better to serve hundreds to thousands of smaller instances of your application than one massive thing divided into lots of services.
This is a ghetto I wish we weren't in to be honest.
I'd also say, microservices are not for every problem, too many people are applying them to the wrong problems (but that is how we all generally learn limits). And a web/app API is generally not where I'd say deploy microservices, but the backend services for that API would make sense. That is exactly what we built now that we are on version 3 of this architecture. API for us is now a totally different beast than the microservices and the whole solution has become a much more manageable with way less churn and confusion. Not to mention deployments are way easier.
If I had to do that then I would definitely be looking at versions message schemas, or contracts like Pact/Pacto but it seems much simpler to just deploy a full set of tested and known to work together services at once than to version each one.
Someone else referred to this pattern as a distributed monolith but I disagree. I still gain the benefits of scaling the size and number of instances of my micro services independently, but remove a lot of the burden of independent versioning and keeping track of the version compatibility matrix.
For anyone interested in a simpler path I think there is a lot of useful information in the book Continuous Delivery by Jez Humble and Paul Hammond.
all microservice architectures look like that sometimes after a day of fighting them.
source: have ~30 microservices in a project at $WORK
The easiest example of a microservice is an email service that all it does it pick up an email off a queue and send it out the door and then send a message that it is complete, with possibly a return from the ESP or SMTP communication.
So keeping it super small, limited functionality makes it easy to debug and simple to scale and manage.
An API might have dozens of endpoints to serve its client(s) and many times cannot be async in nature as a client needs a response now. For example, logging into a website or application, I need to know if the user is authenticated now, not wait for a message to make its way through a series of services. So I don't consider and API and microservices to be the same, although I agree some functionality could be used in either one sometimes so it can be a little confusing.
The only thing that I do agree they share is a contract, in the case of the API it is the endpoint and data, in the case of the service/message it is the message itself.
It's not a great example.
Your distinction might make sense to you, but it's technically wrong. An API means a specific thing. You're almost saying that a car door and a house door aren't both doors. A library has an API and a directly called microservice has an API and a message based microservice has an API. A 'contract', as you call it, is an API.
While it's possible to alleviate some of your pains with versioned APIs to track changes to your data models, you also conflict with data you already have stored in schemaless DBs when those models update.
In a Node or frontend JS stack, I solve that problem with Vers [1]. In any other stack, the idea is fairly simple to replicate: Version your models _inside_ of your code by writing a short diff between it and the previous version every time it changes. Any time you pull data from a DB or accept it via an API, just slip in a call to update it to the latest version. Now your microservice only has to be concerned with the most up-to-date version of this data, and your API endpoints can use the same methods to downgrade any results back to what version that endpoint is using. And frankly that makes versioning your APIs far simpler, as now you move the versioning to the model layer (where all data manipulation really should be) and only need to version the actual API when you change how external services need to interact with it.
And now your other microservices can update to the new schema at their leisure. No more dependency chain driven by your models.
[1] https://github.com/TechnologyAdvice/Vers
How do you ensure that a service can be consumed? Or that an event is constructed with the correct type or parameters? Surely interoperability is the key for any SOA?
Vers looks interesting - I'll have a look at that! Thanks!
If you include the version at the data level, any time it gets passed into a queue or message bus or REST endpoint, your microservice can seamlessly upgrade it to the latest version, which all of its own code has already been updated to use. If a response is required back to the service that originated the request, use your same versioning package (Vers if you go with that) to downgrade it back down to the version the external service is expecting.
If your interface layer is more complex, having responses independent from the data that change, that calls for a versioned API. Either throwing /v1/* or /v2/* into your URLs, or accepting some header that declares the version. But even in this case, you can drastically simplify changes to the model itself by implementing model versioning behind the scenes.
How would you handle deprecation, for example, in the case of a complete service rewrite?
Imagine Twitter or AWS completely rewriting their backend -- if they were to announce to the public that at a specific time, their old API URLs would 404 and the new ones would go live, it would be a wreck. They'd support the old API through deprecated methods and tell users they have X months to migrate away, if they remove that layer at all. Stress-free SOA must employ that same level of discipline in order to stay stress-free.
--And, functionally, the much easier alternative here isn't to re-implement your old API on top of your ground-up rewritten shiny new service, it would be to reduce your old service to an API shell and proxy any requests to the new service in the new format. Far less work that way. Use more traditional API versioning for the much more common updates. Unless you're rewriting your services every other week, in which case you have an entirely different problem ;-)
Thats exactly what the contracts do. The build would fail integration because the client is expecting the older version. This does mean that we don't have multiple incompatible versions of the service in the field, which is a mixed blessing.
> If you completely rewrite a service, it's your responsibility to implement the same interface that you had before on top of it.
Duplicate effort for redundant functionality?
> Then you deprecate it and also publish your new api or data schema. Once you get around to migrating the rest of your application's services away from the deprecated endpoints, the next version of the microservice in question can remove that old code entirely.
Which is exactly what the contracts do, wait until the services have started using the new API before the build is approved. The ONLY difference is that we can remove dead code and apis immediately, rather than doing it in the proverbial "tomorrow".
> Imagine Twitter or AWS completely rewriting their backend -- if they were to announce to the public that at a specific time, their old API URLs would 404 and the new ones would go live, it would be a wreck
None of these services are public facing. The public facing services are exposed via a border-api (which does indeed have a versioned api).
Consider Semver for your interfaces. This is really important.
http://blog.factual.com/docker-mesos-marathon-and-the-end-of...
This approach is almost certainly not a robust, long-term solution, but it has served us well for a couple years, allowing us to evolve our APIs quickly without spending any of our early dev effort on internal versioning.
Whether it's appropriate for you comes down to your reason for using microservices in the first place.
Versioning an API is a decision by the API provider to let the consumer deal with forward and backward compatibility issues. I prefer the approach of focusing on the link relations or media type vice URI or some other technique of versioning because it is consistent in the direction (wrt Hypermedia-based APIs) of the link relations as the point-of-coupling for your API, which makes managing and reasoning about changes to your API less complicated.
Whenever possible, hypermedia-based media type designers should use the technique of extending to make modifications to a media type design. Extending a media type design means supporting compatibility. In other words, changes in the media type can be accomplished without causing crashes or misbehavior in existing client or server implementations. There are two forms of compatibility to consider when extending a media type: forward and backward.
Forward-compatible design changes are ones that are safe to add to the media type without adversely affecting previously existing implementations. Backward-compatible changes are ones that are safe to add to the media type design without adversely affecting future implementations.
In order to support both forward and backward compatibility, there are some general guidelines that should be followed when making changes to media type designs. 1) Existing design elements cannot be removed. 2) The meaning or processing of existing elements cannot be changed. 3) New design elements must be treated as optional.
In short favor extending the media type or link relation and focus on compatibility. Versioning a media type or link relation is essentially creating a new variation on the original, a new media type. Versioning a media type means making changes to the media type that will likely cause existing implementations of the original media type to “break” or misbehave in some significant way. Designers should only resort to versioning when there is no possible way to extend the media type design in order to achieve the required feature or functionality goals. Versioning should be seen as a last resort.
Any change to the design of a media type that does not meet the requirements previously described in are indications that a new version of the media type is needed. Examples of these changes are: 1) A change that alters the meaning or functionality of an existing feature or element. 2) A change that causes an existing element to disappear or become disallowed. 3) A change that converts an optional element into a required element.
While versioning a media type should be seen as a last resort, there are times when it is necessary. The following guidelines can help when creating a new version of a media type. 1) It should be easy to identify new versions of a media type. a) application/vnd.custom+xml application/vnd.custom-v2+xml b) application/custom+JSON;version=1 application/custom+JSON;version=2 c) * REQUEST * PUT /users/1 HTTP/1.1 Host: www.example.org Content-Type: application/vnd.custom+xml Length:xxx Version: 2 … 2) Implementations should reject unsupported versions.
Hope this helps!
The answer, of course, is that you version it. Not put in version control, but manually assign version numbers to it.
You try to make it possible to use several versions at the same time, but that's not always possible. If you have to use only one version, make sure to not make any incompatible changes in a single step, first you deprecate old functionality, some time later you remove it. Some times that's impossible, it's natural but it'll hurt anyway, keep those times at a minimum.
Also, make sure you mark your versions differently for features added and incompatible changes, so that developers can express things like "I'll need an API newer enough for implementing feature X, but old enough so that feature Y is still there".