Anyone running coreOS in production want to comment on their stack? We tried using it around a year+ ago, but etcd/fleet were too unstable and stuff like btrfs and other major features were being changed too often to get the supposed benefits.
It looks significantly more mature now, but I'm still not sure about the buy-in on all their various products.
For the most part we are recommending folks use Kubernetes for new projects over fleet. Kubernetes "completed our sentence" a few months after we started fleet and has rapidly matured. The kubernetes v1.3 release bring a number of new features as well: https://coreos.com/blog/kubernetes-v1.3-preview.html
And on your btrfs question we moved off of btrfs onto ext4 + overlayfs as soon as overlayfs became a viable option in late 2014. You can read more about that here: http://thread.gmane.org/gmane.linux.coreos.devel/1952
I've deployed almost exactly this on a new kubernetes cluster running coreos exclusively with 4 nodes, 3 masters and 3 etcd instances divided over 4 physical machines.
Things I haven't implemented yet is deploying the node-exporters via DaemonSets and the Prometheus config through a ConfigMap. Those are right now done through a cloud-config systemd override and a gluster mount.
Couple of things I ran into. The kubelet's running the kubernetes master components need access to the ssl certificates of the apiserver, otherwise they cannot be scraped by prometheus over https when they communicate with the apiserver over https. And I'm still very confused about a seemingly simple thing of getting a per request response time query. This is what I'm using now:
Two things: I'm confused by what I should set the vector time to `[5m]`. And how I can get the response time of an individual request. We've had a request take 30 seconds but that spike only showed up when viewing the graph over a 3 hour time period. When viewing it over 6 hours or 1 hour it simply will not show up even though it happend in the last hour.
Things I do really love is the ability of configuring services to be scraped by setting annotations to it. Works great when slowly transitioning your services to prometheus style metrics!
If any body has some pointers about querying and visualizing using grafana and prometheus that would be great!
5 comments
[ 3.0 ms ] story [ 24.7 ms ] threadPre-build containers are hosted on Quay.io: https://quay.io/repository/prometheus/prometheus
It looks significantly more mature now, but I'm still not sure about the buy-in on all their various products.
For the most part we are recommending folks use Kubernetes for new projects over fleet. Kubernetes "completed our sentence" a few months after we started fleet and has rapidly matured. The kubernetes v1.3 release bring a number of new features as well: https://coreos.com/blog/kubernetes-v1.3-preview.html
And on your btrfs question we moved off of btrfs onto ext4 + overlayfs as soon as overlayfs became a viable option in late 2014. You can read more about that here: http://thread.gmane.org/gmane.linux.coreos.devel/1952
As for users and maturity some of the videos from Tectonic Summit Day #1 from this blog post might be useful: https://tectonic.com/blog/tectonic-summit-wrap-up.html
Happy to answer any other questions on IRC or email: https://coreos.com/community
Things I haven't implemented yet is deploying the node-exporters via DaemonSets and the Prometheus config through a ConfigMap. Those are right now done through a cloud-config systemd override and a gluster mount.
Couple of things I ran into. The kubelet's running the kubernetes master components need access to the ssl certificates of the apiserver, otherwise they cannot be scraped by prometheus over https when they communicate with the apiserver over https. And I'm still very confused about a seemingly simple thing of getting a per request response time query. This is what I'm using now:
`irate(request_processing_seconds_sum{app="myapp", method=~"$method"}[5m]) / irate(request_processing_seconds_count{app="myapp", method=~"$method"}[5m])`
Two things: I'm confused by what I should set the vector time to `[5m]`. And how I can get the response time of an individual request. We've had a request take 30 seconds but that spike only showed up when viewing the graph over a 3 hour time period. When viewing it over 6 hours or 1 hour it simply will not show up even though it happend in the last hour.
Things I do really love is the ability of configuring services to be scraped by setting annotations to it. Works great when slowly transitioning your services to prometheus style metrics!
If any body has some pointers about querying and visualizing using grafana and prometheus that would be great!