Ask HN: How to Handle Impossible Situation

2 points by le-mark 6y ago ↗ HN

I'm currently at a Big Dumb Corp. We have a legacy application that actually does pretty well as an add on to the main line of business.

The situation is this; we host the applications on our infrastructure. The company has an infrastructure group that manages the servers, network, databases, etc. We basically throw a .war over a wall and they just take care of it (this is a java shop btw). Due to PII/PHI reasons, development doesn't have access to the prod or UAT environments.

The issue comes when there are problem with a release. We had one release recently that's been very unstable on our largest customers. Everythings fine for smaller customers but when it comes the customers with the really big DB's we have servers going unresponsive, require restarts etc. Needless to say, we don't do load testing, so this was not caught before release. The infrastructure team just rolls back to previous versions if customers complain too much.

It's a real disaster. The problem is the infrastructure team takes no responsibility in trouble shooting or diagnosing the JVM issues, and Dev has no access to do it. If it was me, I'd connect visualvm to a server/jvm and watch it constantly. But I can't, I don't have access and can't get access.

Does anyone have input on this type of situation? What course of action can the development take in this situation? I feel like I'm just whining and pointing fingers, I'd really like to have a solution for this!

2 comments

[ 2.6 ms ] story [ 12.4 ms ] thread

consultutah 6y ago ↗

The best solution in an env like this would be to have a good APM like AppDynamics. If you don’t have that and can’t get it installed, next best would be good logs and a good log aggregator/search tool like splunk. Another option (sorry in advance, this sucks, but “works” to find these issues for some definition of “works”): Branch off the last good version, apply changes one by one (or bisect) and push them out to one of the complaining customers - hopefully with their collaboration. When it breaks again, you know where to start looking. Good luck!

exolymph 6y ago ↗

This isn't a technical problem, it's a business problem. Who are the relevant decision-makers, and what are their constraints when it comes to allocating resources?