The wisdom that comes from running things in production makes open-ended questions about those gotchas useful interview questions.
'looks like you used kubernetes for a major project recently. What were some of the unexpected warts?'
Someone with experience running a service on k8s will have at least a few stories, and how someone describes their own responses to those hurdles can shed light on their personality.
I tend to like these sorts of questions, really shows you how much a person understands about a system. I also like it when its asked of me, though I always wonder how much a rant about the things I've experienced setting up Nomad/Consul/Vault would be productive.
Can you describe what you guys do and your infrastructure? Is there a need to test in production or has it just not been important enough to prevent it?
Your statement doesn't match up at all with what they said nor even make sense. What's the point of a business if you only have a test environment but rarely have a production environment?
You seem to be assuming to know what their development process and infrastructure is like.
The joke is that if you don't have a testing environment, production is your testing environment, which in effect means that you don't have a production environment.
Sorry, cannot describe infrastructure (not a throwaway account).
Prod and Test are out of sync in terms of mission critical software, Test is not powerful enough to handle Prod's tasks, and there is no formal procedure to use Test for testing. So Test is never actually used.
When a bug is found in Prod, it's fixed on Prod, while it's live.
By fix, of course I mean an iteration on and combinations of:
qc: we found bug
dev: it's not a bug
qc: it is a bug: [insert evidence]
dev: it's fixed now
qc: no it's not
dev: how about now?
qc: no
dev: and now?
qc: yes but now this other thing is broken
dev: here's another attempt, please check again
qc: Prod is down
dev: try it now
qc: it's still broken, along with the other broken part, and now these three other parts are broken too
> Envoy’s active healthchecking means that you services get healthchecked by every client. This is mostly okay but (again) services with many clients can get overwhelmed by it.
Man that's brutal. Is that normal in service-mesh frameworks?
11 comments
[ 7.9 ms ] story [ 43.3 ms ] thread'looks like you used kubernetes for a major project recently. What were some of the unexpected warts?'
Someone with experience running a service on k8s will have at least a few stories, and how someone describes their own responses to those hurdles can shed light on their personality.
You seem to be assuming to know what their development process and infrastructure is like.
Prod and Test are out of sync in terms of mission critical software, Test is not powerful enough to handle Prod's tasks, and there is no formal procedure to use Test for testing. So Test is never actually used.
When a bug is found in Prod, it's fixed on Prod, while it's live.
By fix, of course I mean an iteration on and combinations of:
qc: we found bug
dev: it's not a bug
qc: it is a bug: [insert evidence]
dev: it's fixed now
qc: no it's not
dev: how about now?
qc: no
dev: and now?
qc: yes but now this other thing is broken
dev: here's another attempt, please check again
qc: Prod is down
dev: try it now
qc: it's still broken, along with the other broken part, and now these three other parts are broken too
ceo: it's fine, close this ticket
Man that's brutal. Is that normal in service-mesh frameworks?