© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SVS407-R
Understanding AWS Lambda
event source mapping
Chris McPeek Jacob Cravinho
(he/him) (he/him)
Principal Solutions Architect Senior Solutions Architect
AWS AWS
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is an event source connector
An event source connector is a class of event processor
services that transform and route events to consumers
•Examples: Amazon EventBridge Pipes and AWS Lambda ESM
•Connects passive event sources with downstream targets
Queue Event source Event
connector target
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is Lambda event source mapping (ESM)
An event source mapping is a Lambda resource that
reads items from stream- and queue-based services and
invokes a function with batches of records
•Processing high-volume data via batch for high efficiency
•Scaling and concurrency management
•Manages polling interactions
Lambda
Queue / Stream
ESM
Lambda
function
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lambda invocation models
Synchronous Asynchronous Poll-based
Amazon Amazon
Amazon
SNS SQS
API Gateway
Amazon Amazon
S3 Kinesis
/order
reqs Changes
event event
AWS Lambda
Synchronous
Lambda Lambda Lambda
function function function
event
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How Lambda ESM works
3. If the Lambda returns successfully
1. Lambda Service polls the queue / stream then the Lambda service advances to
the next set of records and repeats #1
2. The ESM synchronously invokes the
Lambda function with the batch of 4. If the Lambda errors, the
records behavior depends on the event
source configuration
Lambda
Queue / Stream
ESM
Lambda
function
On-failure
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. destination
Error handling – ReportBatchItemFailures
1. Report batch item failure (3,6)
(3)
Lambda
Stream
2. Stream
2. Stream checkpoints
checkpoints to
to 8 7 6 5 4 3 2 1
sequence numbernumber
lowest sequence from (3)
3. Poll records from
updated checkpoint
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Error handling – BisectBatchOnFunctionError
Lambda
8 7 6 5 4 3 2 1 8 7 6 5 4 3 2 1 4 3 2 1
= Record #3 found.
Retry based on MaxRetry
and record age settings
If sent to failure dest. If
4 3 configured or discarded.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service integration
breakdown
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Services
Amazon DocumentDB Amazon Managed Streaming
for Apache Kafka (Amazon
MSK)
Amazon DynamoDB
Self-managed Apache Kafka
Amazon Kinesis
Amazon Simple Queue
Amazon MQ Service (Amazon SQS)
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Whiteboarding
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Resources: serverlessland.com
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Continue your AWS serverless learning
Learn at your Increase your Earn AWS
own pace knowledge Serverless badge
Expand your serverless Use our Ramp-Up Guides Demonstrate your
skills with our learning plans to build your serverless knowledge by achieving
on AWS Skill Builder knowledge digital badges
https://s12d.com/serverless-learning
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Best practices, for everyone
Batch processing Config management
Observability Secrets handling
Powertools for
AWS Lambda REST/GraphQL API Idempotency
Python | TypeScript | Java | .NET BYO middleware
Input/Output validation
Self-documented schemas Feature flags
Caching Streaming Data extraction
*feature set may vary across languages
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Check out these other sessions
SVS216: Serverless data processing with AWS Lambda and Apache Kafka (Builders session)
Wednesday (Dec. 4) @ 8:30a – Mandalay Bay Surf B*
SVS406: Scale streaming workloads with AWS Lambda (Chalk talk)
Thursday (Dec. 5) @ 4:00p – MGM Premier 309*
SVS305: Handle scaling with AWS Lambda (Builders session)
Wednesday (Dec. 4) @11:30a – Mandalay Bay Surf B*
Thursday (Dec. 5) @ 3:30p - MGM Terrace 151*
SVS321: AWS Lambda and Apache Kafka for real-time data processing applications (Breakout
session)
*Times and locations are subject to change;
Look for YouTube after re:Invent! check session catalog for up-to-date information
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you! Please complete the session
survey in the mobile app
Chris McPeek Jacob Cravinho
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
