I have a scala data processing application that 95% of the time can handle the data thrown at it in memory. The remaining 5% if left unchecked doesn't usually hit OutOfMemoryError, but just gets into a cycle of major GCs that spikes the CPU, prevents background threads from executing and, if it does even finish, takes 10x-50x as long as when it has enough memory.
I've implemented system that can flush data to disk and treat the disk stream as if it was an in-memory iterator. It's usually an order of magnitude slower than memory, but sufficient for these 5% cases. I'm currently triggering by a heuristic of max size of a collection context that tracks the size of various collections involved in the data processing. This works, but really is just an adhoc empirical threshold.
I would much rather react to the JVM getting near the above bad state and flush to disk at that time. I've tried watching memory, but can't find the right combination of eden, old, etc. to reliably predict the death spiral. I've also tried just watching for frequency of major GCs but that also seems to suffer from having a wide range of "too conservative" to "too late".
Any resources for judging JVM health and detecting trouble states would be appreciated.
See Question&Answers more detail:
os 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…