KEMBAR78
Trash Talk! How to Reduce Downtime by Tuning Garbage Collection | PDF
DENISE UNTERWURZACHER • CONFLUENCE ENGINEER • ATLASSIAN
Trash Talk!
How to Reduce Downtime by Tuning Garbage
Collection
GARBAGE COLLECTION BASICS
A CAUTIONARY TALE
TROUBLESHOOTING
Agenda
TUNING BEST PRACTICES
The defaults are
sane
The JVM is built to
react adaptively
The knock-on
effects can be hard
to predict
Don’t mess with it
EAC - extranet.atlassian.com
Node 3Node 2Node 1
5.5gb
VM
5.5gb
VM
5.5gb
VM
EAC - extranet.atlassian.com
1250 daily users
8s
50s
45s
Garbage Collection Basics
Eden Tenured
Young
Generation
Old 

Generation
Survivor+
Survivor TenuredEden
Survivor TenuredEden
Survivor TenuredEden
‘Stop the world’ -
stops the
application running
to collect garbage
Collects garbage
while the
application runs
Serial/Parallel Concurrent
Single or multi-
threaded
Serial/Parallel
Good for small heaps
‘Stop the world’ full
GCs
G1GC
Needs a larger heap
Still some pauses
No full GCs
Young generation
collections are never
concurrent.
Pause is inevitable.
Pause types
To-space allocation
Full GC
Young generation
Mark/remark/cleanup
Serial/parallel, concurrent failover
All collectors
Concurrent
Concurrent
To-space allocation
-XX:G1ReservePercent=20
Benchmarking
Number of pauses
Length of pauses
Time between pauses
<1s
Garbage collection
is hard
Fixes can make
things worse
Benchmarking is
essential
Lessons learned
Benchmarking and
troubleshooting tools
Raw logs
Raw logs
Raw logs
Raw logs
Raw logs
Raw logs
Raw logs
Raw logs
Raw logs
Raw verbose logs
Raw verbose logs
Raw verbose logs
Raw verbose logs
Raw verbose logs
Raw verbose logs
Raw verbose logs
GC Viewer
GC Viewer
1GB heap
GC Viewer
Young generation
GC Viewer
Tenured generation
GC Viewer
Server times
GC Viewer
Pause times
GC Viewer
GC times
GC Viewer
< 0.10 s
GC Viewer
GC Viewer
Full GCs
GC Viewer
GC Viewer
GC Viewer
GC Viewer
GC Viewer
When GC fails
OutOfMemory Error - Java Heap Space
Long pauses
Bad tuning - new generation too small
Heap too large
GC times
Long pauses - allocation failure
To-space

allocation

failure
Full GC
Long pauses - unable to free enough space
Solutions
Long pauses
Heap too large
OOMEs
Increase memory by small
increments.
Decrease memory.
Diagnose, benchmark, test.
Best Practices
Benchmarking
Plan
Change
Measure
Repeat
Tuning Goals
Footprint
Throughput
Latency
Maximum GC pause times
Time spent serving application
requests
The size of the heap
Footprint
Throughput
Pause times
Low pause times + high throughput
Use G1GC and a large heap
Footprint
Throughput
Pause times
Low pause times + low footprint
Use G1GC and a medium sized heap
Footprint
Throughput
Pause times
Low pause times + low footprint
Use G1GC and a medium sized heap
Footprint
Throughput
Pause times
High throughput + low footprint
Use a parallel collector and a small heap
Footprint
Throughput
Pause times
High throughput + low footprint
Use a parallel collector and a small heap
JVM
auto-tuning
3. Footprint
2. Throughput
1. Latency
Maximum GC pause times
Time spent serving application
requests
The size of the heap
Overtuning
Increasing the heap
too much
Parameters that
stop the heap auto-
tuning
-XX:MaxNewSize=2g -XX:NewRatio=3
Hard tuning Soft tuning
Young Generation
-XX:GCTimeLimit=10 -XX:GCTimeRatio=9
Hard tuning Soft tuning
-XX:MaxGCPauseMillis=500
Throughput
-XX:SurvivorRatio=4 -XX:G1ReservePercent=20
Hard tuning Soft tuning
Survivor
Tuning is hard.
Avoid it.
If you have to tune,
tread carefully.
Have a plan.
Benchmark&
Change&
Measure&
Repeat.
What you
can do when
you get
home Be minimal!
Think about goals
Review current state
Where to get
help with
GC
Atlassian Support
Oracle’s docs
Atlassian’s docs
https://confluence.atlassian.com/x/
aACCGw
http://tinyurl.com/caq6y7h
https://support.atlassian.com
Thank you!
DENISE UNTERWURZACHER • CONFLUENCE ENGINEER • ATLASSIAN

Trash Talk! How to Reduce Downtime by Tuning Garbage Collection