Out of curiosity, who is using Mesos in production?
At GoGuardian, we've got ~10 offline workers, ~10 web services, and ~30 batch/cron jobs and we're starting to see pain in resource utilization. Is anyone running Mesos at a similar scale?
We're using Opsworks/Chef for managing deployments and server configuration. I hate the resulting tight coupling between applications and the servers that they run on. In some cases, we have great utilization (where the application needs to auto-scale based on load and exceeds the capacity of a single box). However, in many other cases, we have significant underutilization (where the application uses a tiny fraction of the box's capacity).
Multi-tenancy (multiple apps on a single box) could alleviate our underutilization. It is possible with Opsworks/Chef, but it isn't a first-class feature.
Finally, we have a rather sad story for our batch/cron jobs. We have a single server and a master crontab for that server. Because of the various types of batch jobs we need to run (shell, python, java, php, node etc.), the cron server needs have runtimes for all of those.
5 comments
[ 3.1 ms ] story [ 26.0 ms ] threadAt GoGuardian, we've got ~10 offline workers, ~10 web services, and ~30 batch/cron jobs and we're starting to see pain in resource utilization. Is anyone running Mesos at a similar scale?
Multi-tenancy (multiple apps on a single box) could alleviate our underutilization. It is possible with Opsworks/Chef, but it isn't a first-class feature.
Finally, we have a rather sad story for our batch/cron jobs. We have a single server and a master crontab for that server. Because of the various types of batch jobs we need to run (shell, python, java, php, node etc.), the cron server needs have runtimes for all of those.
Re underutilization: it's a big problem with clusters in general. Here's a decent talk on the topic:
https://www.youtube.com/watch?v=YpmElyi94AA