KEMBAR78
How to debug slow lambda response times | PDF
Yan Cui, profile picture
Uploaded byYan Cui
321 views

How to debug slow lambda response times

The document discusses strategies for debugging slow AWS Lambda response times, focusing on identifying systemic issues through observability and analyzing function integration points. It emphasizes collecting key performance metrics, the potential impact of cold starts, and the role of tools like AWS X-Ray for tracing. Additionally, it suggests caching strategies and exploring alternative services to mitigate slow dependencies.

Download to read offline
1 / 108
2 / 108
3 / 108
4 / 108
5 / 108
6 / 108
7 / 108
8 / 108
9 / 108
10 / 108
11 / 108
12 / 108
13 / 108
14 / 108
15 / 108
16 / 108
17 / 108
18 / 108
19 / 108
20 / 108
21 / 108
22 / 108
23 / 108
24 / 108
25 / 108
26 / 108
27 / 108
28 / 108
29 / 108
30 / 108
31 / 108
32 / 108
33 / 108
34 / 108
35 / 108
36 / 108
37 / 108
38 / 108
39 / 108
40 / 108
41 / 108
42 / 108
43 / 108
44 / 108
45 / 108
46 / 108
47 / 108
48 / 108
49 / 108
50 / 108
51 / 108
52 / 108
53 / 108
54 / 108
55 / 108
56 / 108
57 / 108
58 / 108
59 / 108
60 / 108
61 / 108
62 / 108
63 / 108
64 / 108
65 / 108
66 / 108
67 / 108
68 / 108
69 / 108
70 / 108
71 / 108
72 / 108
73 / 108
74 / 108
75 / 108
76 / 108
77 / 108
78 / 108
79 / 108
80 / 108
81 / 108
82 / 108
83 / 108
84 / 108
85 / 108
86 / 108
87 / 108
88 / 108
89 / 108
90 / 108
91 / 108
92 / 108
93 / 108
94 / 108
95 / 108
96 / 108
97 / 108
98 / 108
99 / 108
100 / 108
101 / 108
102 / 108
103 / 108
104 / 108
105 / 108
106 / 108
107 / 108
108 / 108
How to debug slow Lambda response times Yan Cui @theburningmonk Developer Advocate, Lumigo AWS Serverless Hero Author of Production-Ready Serverless
Lambda autoscales by traffic
multi-AZ by default
MyApiFunction Worker Worker …
overloaded servers are a thing of the past
observation majority of performance problems originates from a function’s integration points
macro how well is this service performing in general? micro why did this transaction perform poorly?
macro micro identify systemic issues how well is this service performing in general? why did this transaction perform poorly?
how well is this service performing in general? why did this transaction perform poorly? macro micro why did this user get a bad exp?
In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs.
In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. what do we need to collect?
Yan Cui http://theburningmonk.com @theburningmonk Developer Advocate @ AWS user since 2009
Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advisetraining delivery
API Gateway Lambda DynamoDB
API Gateway Lambda DynamoDBhow long did this req take?
what is the state of the world?
In control theory, observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. what are the most important outputs to collect?
macro micro how well is this service performing in general? why did this transaction perform poorly?
API Gateway API GatewayLambda Lambda DynamoDB Service A Service B
API Gateway API GatewayLambda Lambda DynamoDB Service A Service B how long did service B took to respond?
API Gateway API GatewayLambda Lambda DynamoDB Service A Service B how long did service B took to respond? was DynamoDB slow? was it a cold start? could it be API Gateway?
API Gateway
Lambda
Lambda Duration
Lambda time to create and initialize the worker instance
Lambda bit.ly/2QXNVwc
bit.ly/2WL1uj0 Lambda
Lambda time to create and initialize the worker instance
for API functions, use API Gateway’s IntegrationLatency as a proxy for “total response time from Lambda”
DynamoDB
DynamoDB SuccessfulRequestLatency
“I'm facing this problem now with a lambda that usually takes 25 ms but once a week or so takes > 6000 ms and times out.  The lambda's first step is to load a DynamoDB table that only has 8 items.  I'm at a loss to understand how such a simple query could take so long.”
START
START 1st attempt
START 1st attempt exponential backoff (1)
START 1st attempt exponential backoff (1) 2nd attempt exponential backoff (2)
START 1st attempt exponential backoff (1) 2nd attempt exponential backoff (2) 3rd attempt exponential backoff (3)
START 1st attempt exponential backoff (1) 2nd attempt exponential backoff (2) 3rd attempt exponential backoff (3) 4th attempt success!
START 1st attempt exponential backoff (1) 2nd attempt exponential backoff (2) 3rd attempt exponential backoff (3) 4th attempt success! SuccessfulRequestLatency
JavaScript AWS SDK 10 retries Initial exponential backoff of 50ms delay = Math.random() * (Math.pow(2, retryCount) * base) this is Marc Brooker’s fav formula!
10 retries Initial exponential backoff of 50ms delay = Math.random() * (Math.pow(2, retryCount) * base) JavaScript AWS SDK
10 retries Initial exponential backoff of 50ms delay = Math.random() * (Math.pow(2, retryCount) * base) JavaScript AWS SDK danger zone!
Record client-side latency metrics for IO operations
www.youtube.com/watch?v=adtCwnKApWI
Embedded Metric Format (EMF)
Latency [API Gateway]
IntegrationLatency [API Gateway] Latency [API Gateway]
API Gateway’s latency overhead IntegrationLatency [API Gateway] Latency [API Gateway]
Duration [Lambda] API Gateway’s latency overhead IntegrationLatency [API Gateway] Latency [API Gateway]
Duration [Lambda] Lambda’s allocation time API Gateway’s latency overhead IntegrationLatency [API Gateway] Latency [API Gateway]
SuccessfulRequestLatency [DynamoDB] Duration [Lambda] Lambda’s allocation time API Gateway’s latency overhead IntegrationLatency [API Gateway] Latency [API Gateway]
Caller-side DynamoDB latency [custom metric] SuccessfulRequestLatency [DynamoDB] Duration [Lambda] Lambda’s allocation time API Gateway’s latency overhead IntegrationLatency [API Gateway] Latency [API Gateway]
Caller-side retries (mostly) Caller-side DynamoDB latency [custom metric] SuccessfulRequestLatency [DynamoDB] Duration [Lambda] Lambda’s allocation time API Gateway’s latency overhead IntegrationLatency [API Gateway] Latency [API Gateway]
Latency (ms) Time Latency IntegrationLatency Duration Caller-side DynamoDB latency SuccessfulRequestLatency
Latency (ms) Time Latency IntegrationLatency Duration Caller-side DynamoDB latency SuccessfulRequestLatency
how well is this service performing in general? macro
why did this transaction perform poorly? micro
X-Ray
X-Ray
X-Ray can be encapsulated in custom modules
X-Ray doesn’t add latency
X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time)
X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling
X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling X-Ray SDK adds significant overhead
X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling X-Ray SDK adds significant overhead doesn’t trace TCP traffic (RDS/Elasticache)
X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling X-Ray SDK adds significant overhead doesn’t trace TCP traffic (RDS/Elasticache) poor support for saync event sources (only SNS)
X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling X-Ray SDK adds significant overhead doesn’t trace TCP traffic (RDS/Elasticache) poor support for saync event sources (only SNS) doesn’t capture request & response data
X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling X-Ray SDK adds significant overhead doesn’t trace TCP traffic (RDS/Elasticache) poor support for saync event sources (only SNS) doesn’t capture request & response data logs and traces are separate
X-Ray doesn’t add latency can see “system” overhead (e.g. allocation time) built-in sampling X-Ray SDK adds significant overhead doesn’t trace TCP traffic (RDS/Elasticache) poor support for saync event sources (only SNS) doesn’t capture request & response data logs and traces are separate difficult to search
X-Ray good enough for simple workloads when you outgrow X-Ray, look for a 3rd-party tool
answer both macro and micro level questions in just a few clicks!
Support async event sources such as Kinesis, DynamoDB streams and SNS
Support TCP traffic - e.g. RDS and Elasticache
platform.lumigo.io/signup
trace 500K invocations per month for FREE with promo code Yan500 platform.lumigo.io/signup
How to mitigate slow dependencies?
it depends…
can you use another service?
if not, a good caching strategy often helps
bit.ly/3h7Bo41
Client Server 1 Server 2
Client Server 1 Server 2 50ms later
Client Server 1 Server 2
runing required for each service
helps in some cases but can exaspate the problem in other cases
can you use another service?
platform.lumigo.io/signup trace 500K invocations per month for FREE with promo code Yan500
@theburningmonk theburningmonk.com github.com/theburningmonk yan@lumigo.io

More Related Content

PDF
The Need for Async @ ScalaWorld
byKonrad Malawski
 
PPTX
Samuele Resca - REACTIVE PROGRAMMING, DAMN. IT IS NOT ABOUT REACTJS - Codemot...
byCodemotion
 
PPTX
Reactive Programming in Java 8 with Rx-Java
byKasun Indrasiri
 
PDF
Not Only Streams for Akademia JLabs
byKonrad Malawski
 
PDF
[Tokyo Scala User Group] Akka Streams & Reactive Streams (0.7)
byKonrad Malawski
 
PDF
Akka-chan's Survival Guide for the Streaming World
byKonrad Malawski
 
PDF
The things we don't see – stories of Software, Scala and Akka
byKonrad Malawski
 
PDF
Circuit breakers - Using Spring-Boot + Hystrix + Dashboard + Retry
byBruno Henrique Rother
 
The Need for Async @ ScalaWorld
byKonrad Malawski
 
Samuele Resca - REACTIVE PROGRAMMING, DAMN. IT IS NOT ABOUT REACTJS - Codemot...
byCodemotion
 
Reactive Programming in Java 8 with Rx-Java
byKasun Indrasiri
 
Not Only Streams for Akademia JLabs
byKonrad Malawski
 
[Tokyo Scala User Group] Akka Streams & Reactive Streams (0.7)
byKonrad Malawski
 
Akka-chan's Survival Guide for the Streaming World
byKonrad Malawski
 
The things we don't see – stories of Software, Scala and Akka
byKonrad Malawski
 
Circuit breakers - Using Spring-Boot + Hystrix + Dashboard + Retry
byBruno Henrique Rother
 

What's hot

PDF
The Cloud-natives are RESTless @ JavaOne
byKonrad Malawski
 
PDF
Devoxx France: Fault tolerant microservices on the JVM with Cassandra
byChristopher Batey
 
PDF
Reactive java - Reactive Programming + RxJava
byNexThoughts Technologies
 
PDF
Load Testing with RedLine13: Or getting paid to DoS your own systems
byJason Lotito
 
PPTX
utPLSQL: Unit Testing for Oracle PL/SQL
bySteven Feuerstein
 
PPTX
So You Want to Write an Exporter
byBrian Brazil
 
PDF
We Are All Testers Now: The Testing Pyramid and Front-End Development
byAll Things Open
 
PDF
Reactive integrations with Akka Streams
byKonrad Malawski
 
PDF
Async - react, don't wait - PingConf
byJohan Andrén
 
PDF
Computing recommendations at extreme scale with Apache Flink @Buzzwords 2015
byTill Rohrmann
 
PDF
The dark side of Akka and the remedy
bykrivachy
 
PPTX
Back-Pressure in Action: Handling High-Burst Workloads with Akka Streams & Ka...
byReactivesummit
 
PDF
Serverless in production, an experience report (Going Serverless, 28 Feb 2018)
byDomas Lasauskas
 
PDF
Async – react, don't wait
byJohan Andrén
 
PDF
Resilience with Hystrix
byUwe Friedrichsen
 
PDF
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
byBrian Brazil
 
PDF
Voxxed Vienna 2015 Fault tolerant microservices
byChristopher Batey
 
PPTX
Task parallel library presentation
byahmed sayed
 
PDF
Serverless observability - a hero's perspective
byYan Cui
 
PDF
Build reactive systems on lambda
byYan Cui
 
The Cloud-natives are RESTless @ JavaOne
byKonrad Malawski
 
Devoxx France: Fault tolerant microservices on the JVM with Cassandra
byChristopher Batey
 
Reactive java - Reactive Programming + RxJava
byNexThoughts Technologies
 
Load Testing with RedLine13: Or getting paid to DoS your own systems
byJason Lotito
 
utPLSQL: Unit Testing for Oracle PL/SQL
bySteven Feuerstein
 
So You Want to Write an Exporter
byBrian Brazil
 
We Are All Testers Now: The Testing Pyramid and Front-End Development
byAll Things Open
 
Reactive integrations with Akka Streams
byKonrad Malawski
 
Async - react, don't wait - PingConf
byJohan Andrén
 
Computing recommendations at extreme scale with Apache Flink @Buzzwords 2015
byTill Rohrmann
 
The dark side of Akka and the remedy
bykrivachy
 
Back-Pressure in Action: Handling High-Burst Workloads with Akka Streams & Ka...
byReactivesummit
 
Serverless in production, an experience report (Going Serverless, 28 Feb 2018)
byDomas Lasauskas
 
Async – react, don't wait
byJohan Andrén
 
Resilience with Hystrix
byUwe Friedrichsen
 
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
byBrian Brazil
 
Voxxed Vienna 2015 Fault tolerant microservices
byChristopher Batey
 
Task parallel library presentation
byahmed sayed
 
Serverless observability - a hero's perspective
byYan Cui
 
Build reactive systems on lambda
byYan Cui
 

Similar to How to debug slow lambda response times

PDF
Debugging AWS Lambda Performance Issues
byYan Cui
 
PDF
X-Ray distributed tracing proof-of-concept
byAram Alipoor
 
PPTX
Deep Dive: AWS X-Ray London Summit 2017
byRandall Hunt
 
PPTX
Node Summit 2018 - Optimize your Lambda functions
byMatt Lavin
 
PDF
Become a Performance Diagnostics Hero
byTechWell
 
PDF
The present and future of serverless observability (QCon London)
byYan Cui
 
PDF
The present and future of Serverless observability
byYan Cui
 
PDF
The present and future of Serverless observability
byYan Cui
 
PDF
The present and future of serverless observability
byYan Cui
 
PDF
MongoDB Operational Best Practices (mongosf2012)
byScott Hernandez
 
PPTX
AWS Startup Day - Boston 2018 - The Best Practices and Hard Lessons Learned o...
byChris Munns
 
PDF
How to build observability into a serverless application
byYan Cui
 
KEY
bp
byDan Kuebrich
 
PPTX
Performance Forensics - Understanding Application Performance
byAlois Reitbauer
 
PDF
Application Performance Troubleshooting 1x1 - Von Schweinen, Schlangen und Pa...
byrschuppe
 
PDF
#lspe Building a Monitoring Framework using DTrace and MongoDB
bydan-p-kimmel
 
PDF
Observability for Emerging Infra (what got you here won't get you there)
byCharity Majors
 
PDF
CloudWatch hidden features for debugging serverless application
byMarko (ServerlessLife)
 
PDF
User percieved performance
byMike North
 
PPTX
Monitoreo y análisis de aplicaciones "Multi-Tier"
byGeneXus
 
Debugging AWS Lambda Performance Issues
byYan Cui
 
X-Ray distributed tracing proof-of-concept
byAram Alipoor
 
Deep Dive: AWS X-Ray London Summit 2017
byRandall Hunt
 
Node Summit 2018 - Optimize your Lambda functions
byMatt Lavin
 
Become a Performance Diagnostics Hero
byTechWell
 
The present and future of serverless observability (QCon London)
byYan Cui
 
The present and future of Serverless observability
byYan Cui
 
The present and future of Serverless observability
byYan Cui
 
The present and future of serverless observability
byYan Cui
 
MongoDB Operational Best Practices (mongosf2012)
byScott Hernandez
 
AWS Startup Day - Boston 2018 - The Best Practices and Hard Lessons Learned o...
byChris Munns
 
How to build observability into a serverless application
byYan Cui
 
bp
byDan Kuebrich
 
Performance Forensics - Understanding Application Performance
byAlois Reitbauer
 
Application Performance Troubleshooting 1x1 - Von Schweinen, Schlangen und Pa...
byrschuppe
 
#lspe Building a Monitoring Framework using DTrace and MongoDB
bydan-p-kimmel
 
Observability for Emerging Infra (what got you here won't get you there)
byCharity Majors
 
CloudWatch hidden features for debugging serverless application
byMarko (ServerlessLife)
 
User percieved performance
byMike North
 
Monitoreo y análisis de aplicaciones "Multi-Tier"
byGeneXus
 

More from Yan Cui

PDF
How to win the game of trade-offs
byYan Cui
 
PDF
How to choose the right messaging service
byYan Cui
 
PDF
How to choose the right messaging service for your workload
byYan Cui
 
PDF
Patterns and practices for building resilient serverless applications.pdf
byYan Cui
 
PDF
Lambda and DynamoDB best practices
byYan Cui
 
PDF
Lessons from running AppSync in prod
byYan Cui
 
PDF
How to ship customer value faster with step functions
byYan Cui
 
PDF
How serverless changes the cost paradigm
byYan Cui
 
PDF
Why your next serverless project should use AWS AppSync
byYan Cui
 
PDF
Build social network in 4 weeks
byYan Cui
 
PDF
Patterns and practices for building resilient serverless applications
byYan Cui
 
PDF
How to bring chaos engineering to serverless
byYan Cui
 
PDF
Migrating existing monolith to serverless in 8 steps
byYan Cui
 
PDF
Building a social network in under 4 weeks with Serverless and GraphQL
byYan Cui
 
PDF
FinDev as a business advantage in the post covid19 economy
byYan Cui
 
PDF
How to improve lambda cold starts
byYan Cui
 
PDF
What can you do with lambda in 2020
byYan Cui
 
PDF
A chaos experiment a day, keeping the outage away
byYan Cui
 
PDF
What can you do with lambda in 2020
byYan Cui
 
PDF
How to ship customer value faster with step functions
byYan Cui
 
How to win the game of trade-offs
byYan Cui
 
How to choose the right messaging service
byYan Cui
 
How to choose the right messaging service for your workload
byYan Cui
 
Patterns and practices for building resilient serverless applications.pdf
byYan Cui
 
Lambda and DynamoDB best practices
byYan Cui
 
Lessons from running AppSync in prod
byYan Cui
 
How to ship customer value faster with step functions
byYan Cui
 
How serverless changes the cost paradigm
byYan Cui
 
Why your next serverless project should use AWS AppSync
byYan Cui
 
Build social network in 4 weeks
byYan Cui
 
Patterns and practices for building resilient serverless applications
byYan Cui
 
How to bring chaos engineering to serverless
byYan Cui
 
Migrating existing monolith to serverless in 8 steps
byYan Cui
 
Building a social network in under 4 weeks with Serverless and GraphQL
byYan Cui
 
FinDev as a business advantage in the post covid19 economy
byYan Cui
 
How to improve lambda cold starts
byYan Cui
 
What can you do with lambda in 2020
byYan Cui
 
A chaos experiment a day, keeping the outage away
byYan Cui
 
What can you do with lambda in 2020
byYan Cui
 
How to ship customer value faster with step functions
byYan Cui
 

Recently uploaded

PDF
A fool with a tool is still a fool - Plone Conference 2025
byAndreas Jung
 
PDF
NDP Act GAID Simplified: From Confusion to Compliance
byMuiz Adeleke
 
PDF
AGV PROTOCOL BRAND KIT COMPLETE VIEW.pdf
byfagbolade
 
PPTX
CBD Belgium 2025 - The adventures of a Microsoft 365 Platform Owner.pptx
byJasper Oosterveld
 
PDF
From Bots to Brains: Getting Started with Agentic Automation in UiPath
byUiPathCommunity
 
PDF
Session 2 - Agentic Orchestration with UiPath Maestro
byDianaGray10
 
PDF
Atlantix is an AI-first venture studio, building companies with AI at the core
byadmin625551
 
PPTX
"The Art of Web Scraping: Tree Algorithms and JS Magic", Dmytro Tarasenko
byFwdays
 
PDF
Planetek Italia Corporate Profile brochure
byPlanetek Italia Srl
 
PDF
Cassandra vs. ScyllaDB: Evolutionary Differences
byScyllaDB
 
PDF
Getting the Best of TrueDEM – October News & Updates
bypanagenda
 
PDF
Expert Python App Development Company for Modern Enterprises
bySynapseIndia
 
PPTX
Hybrid Active Directory Cyber Resiliency
byFrancesco Faenzi
 
PDF
What's New? ThousandEyes Features and Highlights: October 2025
byThousandEyes
 
PPTX
BioVault.net ADW 2025 CODATA: SyftBox: a General Purpose Solution for Data Vi...
byMadhava Jay
 
PDF
A 4-Terminal Approach to Validating Charge Conservation in MOSFET Compact Models
byEalwan Lee
 
PDF
UiPath DevConnect 2025: Introduction to Agentic Automation
byDianaGray10
 
PDF
What is Google Cloud Platform (GCP)? A 5-Minute Guide for Beginners (2025)
byTeleglobal International
 
PPTX
"Daily workflows with Cursor IDE", Maksym Anisimov
byFwdays
 
PDF
From Data Chaos to Copilot Confidence: A Step-by-Step Microsoft 365 Copilot R...
byNikki Chapple
 
A fool with a tool is still a fool - Plone Conference 2025
byAndreas Jung
 
NDP Act GAID Simplified: From Confusion to Compliance
byMuiz Adeleke
 
AGV PROTOCOL BRAND KIT COMPLETE VIEW.pdf
byfagbolade
 
CBD Belgium 2025 - The adventures of a Microsoft 365 Platform Owner.pptx
byJasper Oosterveld
 
From Bots to Brains: Getting Started with Agentic Automation in UiPath
byUiPathCommunity
 
Session 2 - Agentic Orchestration with UiPath Maestro
byDianaGray10
 
Atlantix is an AI-first venture studio, building companies with AI at the core
byadmin625551
 
"The Art of Web Scraping: Tree Algorithms and JS Magic", Dmytro Tarasenko
byFwdays
 
Planetek Italia Corporate Profile brochure
byPlanetek Italia Srl
 
Cassandra vs. ScyllaDB: Evolutionary Differences
byScyllaDB
 
Getting the Best of TrueDEM – October News & Updates
bypanagenda
 
Expert Python App Development Company for Modern Enterprises
bySynapseIndia
 
Hybrid Active Directory Cyber Resiliency
byFrancesco Faenzi
 
What's New? ThousandEyes Features and Highlights: October 2025
byThousandEyes
 
BioVault.net ADW 2025 CODATA: SyftBox: a General Purpose Solution for Data Vi...
byMadhava Jay
 
A 4-Terminal Approach to Validating Charge Conservation in MOSFET Compact Models
byEalwan Lee
 
UiPath DevConnect 2025: Introduction to Agentic Automation
byDianaGray10
 
What is Google Cloud Platform (GCP)? A 5-Minute Guide for Beginners (2025)
byTeleglobal International
 
"Daily workflows with Cursor IDE", Maksym Anisimov
byFwdays
 
From Data Chaos to Copilot Confidence: A Step-by-Step Microsoft 365 Copilot R...
byNikki Chapple
 

How to debug slow lambda response times