Ask HN: How to log full HTTP requests

2 points by eloff 4y ago ↗ HN

I want backups for all data entering my system at the HTTP level. It happens fairly often in my career that the worst data loss bugs happen in ways where you're not covered by backups. The data doesn't get saved, validation goes amok, it gets saved them gets deleted. In all these cases having backups of the data as it enters the system would enable recovery of it.

Are there tools like this? It's not enough to just save it, it would need to be searchable as well. The data may be sensitive so it would need to be encrypted too.

2 comments

[ 3.9 ms ] story [ 16.1 ms ] thread

patoroco 4y ago ↗

If you’re behind a reverse proxy, like NGINX, you could log the traffic and customize the log lines to be more verbose.

After you could index these log files into a search engine, like Elastic Search and perform queries / dashboards through Kibana.

This is the typical stack, so I’m not sure if you’re looking something more sophisticated.

eloff 4y ago ↗

Could you log the headers and body though? I looked into it a little and it looks tricky (requires buffering incoming request body to individual files, which are then your logs of the request body which have to be correlated somehow with the log files.)