KEMBAR78
Debugging Lambda timeouts | PDF
Debugging Lambda Timeouts
Yan Cui
theburningmonk.com
@theburningmonk
Developer Advocate @
Independent Consultant
AWS user since 2009
since 2018
yan@lumigo.io
“look for error logs”
Expectation
Reality
“could the function have timed out?”
what caused the
function to time out?
signposts
missing signposts
timeouts are difficult to debug…
mixed in with other errors
no “error” log message
relies on missing signposts to find root cause
How to detect Lambda timeouts
How to mitigate/prevent timeouts
How to find root cause of timeouts
How to detect Lambda timeouts
Easy to implement.
Language agnostic.
Reliable.
Easy to implement.
Language agnostic.
Reliable.
Hard to scale as project grows.
Can’t configure metric dimentions.
Can’t use AWS/Lambda metric namespace.
CloudWatch CloudTrail EventBridge
Lambda
https://amzn.to/2DYGYIn
Time
START TIMED OUT
Time
START TIMED OUT
log timeout error
github.com/middyjs/middy
Error log message for timeouts.
Error log message for timeouts.
Requires language specific implementation.
Can report false positives.
It just works out-of-the-box.
No manual instrumentation required.
Yet another 3rd party solution.
How to debug Lambda timeouts
signposts
Easy to implement.
Hard to scale as project grows.
Only works for your code.
X-Ray
Automated tracing.
Easy to find cause of timeout if after you fnid the transaction.
which are the
errors/timeouts?
Automated tracing.
Easy to find cause of timeout after you fnid the transaction.
Still requires some manual instrumentation.
Still difficult to find timeouts.
It just works out-of-the-box.
No manual instrumentation required.
More than just tracing.
Yet another 3rd party solution.
How to mitigate/prevent timeouts
Step 1. timeout the operation before it times you out
Step 2. execute fallback strategy
DEMO!
API Gateway Lambda
GET /
Lambda
POST
/restaurants/search
API Gateway Lambda
GET /
get-index get-restaurants
search-restaurants
DynamoDB
EventBridge Lambda
notify-restaurant
Lambda
place-order
POST /order
SNS
API Gateway Lambda
GET /
Lambda
POST
/restaurants/search
API Gateway Lambda
GET /
get-index get-restaurants
search-restaurants
DynamoDB
EventBridge Lambda
notify-restaurant
Lambda
place-order
POST /order
SNS
times out 1/4 times
times out most of
the time…
github.com/theburningmonk/debugging-lambda-timeouts-demo
get 25% off for 1st year
@theburningmonk
theburningmonk.com
github.com/theburningmonk
yan@lumigo.io

Debugging Lambda timeouts