In fact, go 1.16 has reverted to using this as the default now. The OS, therefore, might not feel the memory pressure and will not free up the memory even though the container might hit the memory limit and get killed.įortunately, there’s a Golang debug flag to flip this behavior and use MADV_DONTNEED instead, by setting the GODEBUG environment variable to “ madvdontneed=1”. The garbage collector will try to release memory that. However, if you go back to how containers are implemented, these are essentially just processes running under separate Cgroups. If there is no memory left, then the JVM will attempt to free some memory by using the garbage collector. This meant it might not return the memory immediately to the OS, and the OS could choose to reclaim this memory when it felt memory pressure. The potential theory in the Golang issue thread is that Go started using MADV_FREE as the default in go 1.12. Golang / container environment interaction issue We started looking more and came across this thread on similar issues.
It was also interesting to see that the values for HeapReleased were fairly large. In particular, “HeapInuse minus HeapAlloc estimates the amount of memory that has been dedicated to particular size classes, but is not currently being used.” This amount was fairly small, ~3MB in our case. An upper bound on fragmented memory can be easily obtained by subtracting HeapAlloc from HeapInuse.
We added a background thread that periodically logs the MemStats. Luckily it wasn’t too hard to conclude that fragmentation was not the culprit either. The immediate next thought we had was whether this was a memory fragmentation issue because Golang is known to be bad in that aspect. This meant the memory was being held somewhere by the Golang runtime. ~2X less than the memory footprint app engine was seeing). by the in-use objects) was around 1.5G (i.e. One interesting thing the profiles revealed though was that the average heap size (i.e. In particular, the flame graph in the profiler UI would clearly show heavy usage at a specific call site in such cases. In the past, we have seen cases of unclosed Google Remote Procedure Call (gRPC) connections causing such issues, but those were easy to debug using the continuous profiler. Thankfully, a memory leak on the application is not hard to debug in our case since we have access to continuous profiling data using the cloud profiler.
0 kommentar(er)
