Deadlock, livelock and diagnostics question 146 of 180 senior
A colleague reports a bug that happens once a week in production and never locally. How do you approach it?
- Add retries until it stops
- Correct: Treat rarity as evidence of a race: review for unguarded state, add JFR, reproduce with stress
- Increase the thread pool size
- Ask the team to restart nightly
Once a week means an interleaving that needs an unlikely coincidence, which is the signature of a data race rather than a logic error. The productive order:
- Review the code path for shared mutable state, and for every field ask which lock or happens-before edge covers it. Most races are found this way, not by observation.
- Add cheap invariant checks that fail loudly and capture context when the state is impossible.
- Turn on JFR continuously so the next occurrence comes with evidence instead of a bug report.
- Reproduce deliberately: jcstress for a memory model hypothesis, a load test with more threads than cores, and ARM hardware, where weak ordering exposes what x86 hides.
- Fix the race, not the symptom. A retry or a sleep that makes it rarer converts a weekly bug into a quarterly one that is far harder to find.
Saying "a retry makes it rarer, not absent" is usually what the interviewer is listening for.
Quiz yourself on Deadlock, livelock and diagnostics Suggest a correction