Author here! If you're running a Kubernetes cluster, I recommend you check `kubectl version` and see if you're running "Server Version: v1.36.[0,1,2]". If so, you may want to use the one-liner at the end of the article to check your "process_resident_memory_bytes" on each node, and consider restarting kubelet as a temporary workaround to tame the memory leak until v1.36.3 is released.
Can't help but feel this is one of the subtle traps hidden beneath the advice that contexts aren't supposed to be stored. I know it's not always that easy, of course.
Given that this happens on the main loop, one wonders how this escaped release quality checks. Is there another contributing cause that narrows the impact?
Nice finding, golang and kubernetes toolset is always surprisingly enough and friendly for doing such debuggings. Currently me to running this version so kudos to you!
8 comments
[ 10.9 ms ] story [ 262 ms ] threadCan't help but feel this is one of the subtle traps hidden beneath the advice that contexts aren't supposed to be stored. I know it's not always that easy, of course.