© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
IOT202
AWS IoT for edge LLM
deployment and execution
Nikit Pednekar Stefano Marzani Joey Lorich
(he/him) (he/him) (he/him)
Principal PMT, AWS IoT AWS WW Tech Leader, Head of Partner Engineering
Automotive Software Defined Vehicles and Enablement
AWS AWS Aptiv
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Speaker introduction
Nikit Pednekar Stefano Marzani Joey Lorich
Principal PMT, AWS IoT AWS WW Tech Leader, Head of Partner Engineering
Automotive Software Defined Vehicles and Enablement
He/Him He/Him He/Him
AWS AWS Aptiv
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
01 Fun fact 07 Vehicle architecture evolution
02 Industry trends 08 Introduction to eLxr
03 LLM benefits and challenges 09 Edge developer journey
04 AWS and LLM value 10 LLM for automotive vertical
proposition
05 Edge LLM vertical approach 11 AWS reference architecture
06 Modernization of distributed 12 Demo
systems
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fun fact about large language models
One command using Voice Assistants (power of LLM)
Imposta la Set the
temperatura a 19 temperature to 19
gradi Celsius degrees Celsius
Réglez la
température à 19 तापमान 19डिग्री
degrés Celsius सेल्सि यस पर सेट करें।
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Edge AI ecosystem market
growth expected close to
US$38 billion by 2026
• Ecosystem • Verticalization
• Opportunities • Technology approach
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LLMs edge processing – Benefits
LLM edge processing empowers applications to be faster, more secure, and more
adaptable to user needs
Reduce latency
Bandwidth efficiency
Enhanced privacy and security
Improved reliability
Customization & personalization
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges
System integration
Performance optimization
Scalability
Model deployment & updates
Data privacy and security
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS value proposition: The Big Loop
AWS Cloud
Partner integration Virtual hardware AWS edge services
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Edge LLM vertical approach
• Advanced driver
• Voice assistants assistance systems
• Quality control
• Home security (ADAS)
• Robotics and
systems • In-car voice assistants
automation
• Smart kitchen • Predictive
• Process optimization
appliances maintenance
• Smart navigation
• Personalized
• Real-time
customer
diagnostics and
experiences
decision support
• Enhanced checkout
• Personalized
processes
treatment plans
• Inventory
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. • Emergency response
management
APTIV: A new approach
for vehicle edge
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Vehicle architecture evolution
SOFTWARE
DEFINED
2025+
IVI
ADAS
FUNCTION DOMAIN ZONE
Historical 2017 2022
50–120+ Supporting incremental Reducing complexity
distributed ECUs
per vehicle
functionality through
domain centralization
through intelligent zone
control and management V2X
LLM
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Introducing
• Leverages Debian’s strong community base EXPANDED PACKAGE ECOSYSTEM
to foster collaboration and innovation
• Optimized and hardened for edge, real-time,
DEBIAN USER SPACE
and high security scenarios
eLxr project
• Provides support for AI/graphics acceleration
DEBIAN KERNELS (STD, RT)
at a reduced time to market
• Offers 24x7 commercial support
• Delivers industry standards and certifications SERVER EDGE
COMMUNITY-DISTRIBUTED
BINARIES
Open source Debian derivative
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Edge developer journey
Test real-world edge deployments with
eLxr on AWS Graviton (ARM) EC2 instances
Manage connected devices and data
with AWS IoT Core
Manage workloads and stream data
with AWS IoT Greengrass
Secure and rapid edge development with
eLxr Pro
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LLMs for automotive
vertical
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
From software-defined to AI-defined vehicles
Cloud-to-edge LLMs orchestration enables faster, more efficient, secure, and scalable solutions for the
automotive industry, driving innovation while enhancing safety, privacy, and user experience
• Real-time decision-making
• Data privacy & security
• Reduced latency &
dependence on connectivity
• Optimized power & resource usage
• Customization & personalization
• Scalability & flexibility
• Enhanced safety features
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
3 steps for SDVs
AWS strategy for SDV has been based on three pillars:
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
3 steps for SDVs
AWS strategy for SDV has been based on three pillars:
• Virtualization
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
3 steps for SDVs
AWS strategy for SDV has been based on three pillars:
• Virtualization
• Platform
engineering
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
3 steps for SDVs
AWS strategy for SDV has been based on three pillars:
• Virtualization
• Platform
engineering
• AI accelerators for
function
engineering and
development
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
3 steps for SDVs +1
AWS strategy for AI-defined vehicles:
- Virtualization
- Platform
engineering
- AI accelerators for
function
engineering and development
- Cloud-to-vehicle edge
orchestration of AI agents
… architecture…
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cloud-to-vehicle edge LLM orchestration
• Cloud-to-edge
orchestration,
facilitated by the use of
eLxr Pro
• Staged SDV
development and
testing
• Automatic model
updates
• IoT integration &
analytics
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The order is important: Cloud-first!
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The order is important: Virtual hardware
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The order is important: Real device
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The order is important: Big loop!
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo time
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo time
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo time
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo time
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo time
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo time
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo time
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo time
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo time
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo time
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo time
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key takeaways
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• LLM market growing rapidly, expected to reach $38B by 2026.
• Edge processing of LLMs offers benefits but faces challenges.
▪ Human supervision and guardrails
▪ Performance assessment and management, simplify if possible setting
performance bar
▪ Heterogeneity in the training/inference execution across different vendors an
issue to work on as an industry
• AWS provides solutions for LLM integration, optimization, and deployment.
• Automotive industry shifting to AI-defined vehicles; AWS strategy focuses on
virtualization, AI acceleration, and cloud-to-vehicle orchestration.
• eLxr, an open-source Debian derivative, developed for edge, real-time, and high-
security scenarios, optimized for automotive AI applications
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related IoT sessions @ re:Invent 2024
IOT307 | Unleashing LLMs at the edge with AWS IoT
IOT316 | Unleash edge computing with AWS IoT Greengrass on NVIDIA Jetson
IOT313 | Build a generative AI industrial diagnosis & troubleshooting assistant
IOT315 | Transforming healthcare with IoT, Amazon Location, and generative AI
AUT310 | End-to-end acceleration of vehicle software development
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you! Please complete the session
survey in the mobile app
© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved.
