KEMBAR78
Mongodb debugging-performance-problems | PDF
MongoDB 
Debugging Performance Problems 
@ConradIrwin
50M crashes/day 
4TB data / 100GB index 
15 nodes / 3* availability zones
I don't care if MongoDB is slow 
I care if my app is slow
How to make app fast again it's slow because 
of the way I'm
What does slow mean?
Performance over time
Performance over time
It's slow :(
Solution 1 
Denormalize
It's fast :)
It's slow :(
db.errors. 
find({project_id: x}). 
sort({ _id: -1}). 
limit(30)
db.errors. 
find({project_id: x}). 
sort({ _id: -1}). 
limit(30). 
explain()
{ 
"cursor" : "BtreeCursor _id_ reverse", 
"isMultiKey" : false, 
"n" : 0, 
"nscannedObjects" : 227756, 
"nscanned" : 227756, 
"nscannedObjectsAllPlans" : 227756, 
"nscannedAllPlans" : 227756, 
"scanAndOrder" : false, 
"indexOnly" : false, 
"nYields" : 1779, 
"nChunkSkips" : 0, 
"millis" : 461, 
"indexBounds" : { 
"_id" : [ 
[ 
{ 
"$maxElement" : 1 
}, 
{ 
"$minElement" : 1 
} 
] 
] 
}, 
"server" : "Jaroussky.local:27017", 
"filterSet" : false 
}
{ "cursor" : 
"BtreeCursor _id_ reverse", 
"nscanned" : 227756, 
"indexBounds" : {"_id" : [ 
[{"$maxElement" : 1}, 
{"$minElement" : 1}] 
] 
} 
}
Solution 2 
Index
db.errors.ensureIndex( 
{project_id: 1, _id: -1} 
)
{"cursor" : "BtreeCursor 
project_id_1__id_1", 
"nscanned" : 6, 
"indexBounds" : { 
"project_id" : [[ 
ObjectId(x),ObjectId(x) 
]], 
"_id" : [[ 
{"$minElement" : 1}, 
{"$maxElement" : 1} 
]] 
}}
It's fast :)
It's slow :(
Only 1 query... 
Indexed properly...
mongostat
insert query update delete getmore command flushes mapped 
5 97 57 *0 95 127|0 0 320g 
5 98 61 *0 113 146|0 0 320g 
8 94 61 *0 95 137|0 0 320g 
vsize res faults locked db idx miss % qr|qw ar|aw 
641g 8.65g 7 bugsnag:47.7% 0 0|0 0|0 
641g 8.66g 2 bugsnag:21.0% 0 0|0 0|0 
641g 8.64g 3 bugsnag:23.4% 0 0|0 0|0 
netIn netOut conn set repl time 
47k 73k 145 bugsnag1 PRI 06:34:05 
63k 99k 146 bugsnag1 PRI 06:34:06 
98k 124k 146 bugsnag1 PRI 06:34:07
command locked db time 
127|0 bugsnag:47.7% 06:34:05 
146|0 bugsnag:21.0% 06:34:06 
137|0 bugsnag:23.4% 06:34:07
Solution 3 
Shard
sh.shardCollection("errors" 
{ 
project_id: 1, 
_id: -1 
})
command locked db time 
83|0 bugsnag:3.1% 06:46:12 
69|0 bugsnag:4.1% 06:46:13 
73|0 bugsnag:2.4% 06:46:14
It's fast :)
It's slow :(
Only 1 query... 
Indexed properly... 
Lock % ok...
iostat
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz 
xvdi 0.00 2.00 15.00 743.00 436.00 7905.50 22.01 
xvdi 0.00 2.00 0.00 663.00 0.00 6706.00 20.23 
xvdi 0.00 1.00 0.00 726.00 0.00 5593.50 15.41 
avgqu-sz await r_await w_await svctm %util 
11.02 14.54 5.33 14.73 0.60 45.60 
8.04 12.12 0.00 12.12 0.51 33.60 
13.01 17.92 0.00 17.92 0.50 36.40
rkB/s wkB/s await %util 
7660 138 14.73 83.60 
2248 2043 12.12 65.20 
7905 436 17.92 45.60
db.stats()['indexSize'] 
= 3852071824 
= 3932614656 
free -b (m3.medium)
Solution 4 
Scale
db.stats()['indexSize'] 
= 3852071824 
= 15775363072 
free -b (r3.large)
It's fast :)
It's slow :(
Main query seems fine... 
"Quick" queries sometimes slow...
mongotop
ns total read write 20 
14-06-22T19:11:35 
bugsnag.events 95ms 0ms 95ms 
bugsnag.errors 80ms 6ms 74ms 
bugsnag.system.namespaces 26ms 26ms 0ms 
bugsnag.projects 12ms 5ms 7ms 
bugsnag.users 15ms 2ms 13ms 
bugsnag.error_aggregates 4ms 0ms 4ms 
bugsnag.deploys 4ms 3ms 1ms 
bugsnag.event_tallies 3ms 0ms 3ms
ns tot r w 
bugsnag.events 95 0 95 
bugsnag.errors 80 6 74 
bugsnag.projects 12 5 7 
bugsnag.users 15 7 8
Solution 5 
Tag shards
It's fast :)
Solution 1 
Denormalize
Solution 2 
Index
Solution 3 
Shard
Solution 4 
Scale
Solution 5 
Tag shards
Solution 6 
...
NewRelic / Skylight 
explain() 
mongostat 
iostat 
mongotop 
...
It will 
be slow
You can 
speed it up
Thanks! 
@ConradIrwin

Mongodb debugging-performance-problems