KEMBAR78
How Optimizely Scaled its REST API with asyncio | PDF
How Optimizely scaled 

its REST API with asyncio
Nick DiRienzo
@nickdirienzo
Software Engineer
Optimizely
@Optimizely
Vinay Tota
@vinaytota
Senior Software Engineer
Optimizely
We’ll talk about:
• Optimizely’s API story
• What makes an API “good”
• Optimizely’s API gateway
• Improving it with asyncio
• Lessons learned
Where we started
Where we started
Where we started
Optimizely Classic was an
A/B Testing Solution
Optimizely X is an experimentation platform
Optimizely X Web
Visual Editor Based Experimentation
Optimizely X Full Stack
Code Based Experimentation
Optimizely X Web
Visual Editor Based Experimentation
Optimizely X Full Stack
Code Based Experimentation
Optimizely X Web
Visual Editor Based Experimentation
Experimentation Platform
What is happening?
How can we turn a good web UI into
a good API?
What makes an API “good”?
Easy to use APIs
What makes an API “good”?
More than mapping a data model to an
HTTP endpoint
API is the UI for developers
TODO: Add results slide.
Getting Optimizely Results
1. Get Experiment data
Getting Optimizely Results
1. Get Experiment data
2. Get Layer data
Getting Optimizely Results
1. Get Experiment data
2. Get Layer data (whatever that is)
Getting Optimizely Results
1. Get Experiment data
2. Get Layer data (whatever that is)
3. Get Project Settings
Getting Optimizely Results
1. Get Experiment data
2. Get Layer data (whatever that is)
3. Get Project Settings
4. Get results data
1. Get Experiment data
2. Get Layer data (whatever that is)
3. Get Project Settings
4. Get results data
GET /v2/experiments/:id/results
Getting Optimizely Results
Easy to use APIs
What makes an API “good”?
Ensured Trust
Build a business on our APIs
How do we ensure trust?
No surprises
How do we ensure trust?
No surprises
Reliable platform
How do we ensure trust?
No surprises
Reliable platform
Performant platform
Trust can be made with:
Interface Stability
Interface stability means...
Beta & Stable APIs
Interface stability means...
Beta & Stable APIs
No breaking
changes
Interface stability means...
Beta & Stable APIs
No breaking
changes
Versioning
Trust can be made with:
Interface Stability
Reliability
99.999%
Trust can be made with:
Interface Stability
Reliability
Performance
Performance is mission critical
Performance can be a

dealbreaker
Trust can be made with:
Interface Stability
Reliability
Performance
Easy to use APIs
What makes an API “good”?
Ensured Trust Documentation
Write documentation
Write documentation
Generate documentation
Generate documentation with
swagger-ui to ensure docs
are...
Generate documentation with
swagger-ui to ensure docs
are...
Up to Date
Generate documentation with
swagger-ui to ensure docs
are...
Up to Date Comprehensive
Generate documentation with
swagger-ui to ensure docs
are...
Up to Date Comprehensive Accurate
Easy to use APIs
What makes an API “good”?
Ensured Trust Documentation
How can we provide

“good” APIs?
How can we provide
“good” APIs?
Getting Optimizely Results
1. Get Experiment data
2. Get Layer data (whatever that is)
3. Get Project Settings
4. Get results data
1. Get Experiment data
2. Get Layer data (whatever that is)
3. Get Project Settings
4. Get results data
/v2/experiments/:id/results
Getting Optimizely Results
We’ve got data all over
We’ve got data all over
Experiments
We’ve got data all over
Experiments Projects
We’ve got data all over
Experiments Projects Results
We’ve got data all over
Experiments Projects Results
API
Gateway
Benefits of an API gateway
Operations through the API look a lot like
their web UI counterparts

Benefits of an API gateway
Operations through the API look a lot like
their web UI counterparts
Allows us to orchestrate APIs instead of
data models

Benefits of an API gateway
Operations through the API look a lot like
their web UI counterparts
Allows us to orchestrate APIs instead of
data models



Pulls problems away from business logic
into a higher-level



Benefits of an API gateway
Operations through the API look a lot like
their web UI counterparts
Allows us to orchestrate APIs instead of
data models



Pulls problems away from business logic
into a higher-level



Internal and external consumers go
through the same application



api.optimizely.com What we’ve got
Python 3 + Pyramid + pyramid_swagger
We need to do a bunch of things in parallel...
Pyramid and Multiprocessing
with multiprocessing.Pool() as pool, multiprocessing.Manager() as manager:

response_queue = manager.Queue()

try:

# send our requests get back responses from the response_queue

finally:

pool.join()

pool.terminate()
Pyramid and Multiprocessing
with multiprocessing.Pool() as pool, multiprocessing.Manager() as manager:

response_queue = manager.Queue()

try:

# send our requests get back responses from the response_queue

finally:

pool.join()

pool.terminate()
Pyramid and Multiprocessing
with multiprocessing.Pool() as pool, multiprocessing.Manager() as manager:

response_queue = manager.Queue()

try:

# send our requests get back responses from the response_queue

finally:

pool.join()

pool.terminate()
Pyramid and Multiprocessing
with multiprocessing.Pool() as pool, multiprocessing.Manager() as manager:

response_queue = manager.Queue()

try:

# send our requests get back responses from the response_queue

finally:

pool.join()

pool.terminate()
An Unhealthy Situation
AWS Elastic Load Balancer 5XX
An Unhealthy Situation
Textbook case for non blocking IO
AWS Elastic Load Balancer 5XX
Not the Experience We Want
Blocking IO: “Normal” PythonApplicationKernel
Start
Blocking IO: “Normal” PythonApplicationKernel
Start read()
Blocking IO: “Normal” PythonApplicationKernel
Start read()
Initiate read
Blocking IO: “Normal” PythonApplicationKernel
Start read()
Initiate read
Receive
response
Blocking IO: “Normal” PythonApplicationKernel
Start read()
Initiate read
read()
returns
Receive
response
Blocking IO: “Normal” PythonApplicationKernel
Start read()
Initiate read
read()
returns
Receive
response
Nonblocking IOApplicationKernel
Start
Nonblocking IOApplicationKernel
Start read()
Nonblocking IOApplicationKernel
Start read()
Initiate read,
returns
immediately
Nonblocking IOApplicationKernel
Start read()
Initiate read,
returns
immediately
poll()until
data has arrived
Nonblocking IOApplicationKernel
Start read()
Initiate read,
returns
immediately
Receive
response
poll()until
data has arrived
Nonblocking IOApplicationKernel
Start read()
Initiate read,
returns
immediately
read()
Receive
response
poll()until
data has arrived
Nonblocking IOApplicationKernel
Start read()
Initiate read,
returns
immediately
read()
Receive
response
poll()until
data has arrived
Nonblocking IOApplicationKernel
poll(5)
Nonblocking IOApplicationKernel
poll(5) poll(5)
Nonblocking IOApplicationKernel
poll(5) poll(5)poll(5)
Nonblocking IOApplicationKernel
Receive
response
poll(5) poll(5)poll(5)
Nonblocking IOApplicationKernel
read()
Receive
response
poll(5) poll(5)poll(5)
Nonblocking IOApplicationKernel
read()
Receive
response
poll(5) poll(5)poll(5)
Non Blocking IO
Tw
isted
N
odeJS
Tornado
2002
N
etty2004 20092006
EventM
achine
2008
Eventlet
2015
Asyncio
2012
G
o/G
orutines
What’s it good for?
Proxy
What’s it good for?
Proxy
What’s it good for?
Push Updates/Chat
What’s it good for?
Push Updates/Chat
How do you schedule this work?
What framework?
Enter Asyncio
Python 3.4
async def i_take_a_while():
print('slow func start')
await asyncio.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_take_a_while()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_take_a_whilei_run_fast
async def i_take_a_while():
print('slow func start')
await asyncio.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_take_a_while()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_take_a_whilei_run_fast
async def i_take_a_while():
print('slow func start')
await asyncio.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_take_a_while()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_take_a_whilei_run_fast
fast func
start
async def i_take_a_while():
print('slow func start')
await asyncio.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_take_a_while()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_take_a_whilei_run_fast
fast func
start
async def i_take_a_while():
print('slow func start')
await asyncio.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_take_a_while()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_take_a_whilei_run_fast
slow func
start
fast func
start
async def i_take_a_while():
print('slow func start')
await asyncio.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_take_a_while()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_take_a_whilei_run_fast
slow func
start
fast func
start
async def i_take_a_while():
print('slow func start')
await asyncio.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_take_a_while()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_take_a_whilei_run_fast
slow func
start
fast func
start
async def i_take_a_while():
print('slow func start')
await asyncio.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_take_a_while()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_take_a_whilei_run_fast
slow func
start
fast func
start
1 sec after start
fast func
done
async def i_take_a_while():
print('slow func start')
await asyncio.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_take_a_while()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_take_a_whilei_run_fast
slow func
start
fast func
start
1 sec after start
fast func
done
async def i_take_a_while():
print('slow func start')
await asyncio.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_take_a_while()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_take_a_whilei_run_fast
slow func
start
fast func
start
1 sec after start
fast func
done
async def i_take_a_while():
print('slow func start')
await asyncio.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_take_a_while()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_take_a_whilei_run_fast
slow func
start
fast func
start
1 sec after start
fast func
done
3 sec after start
slow func
done
Blocking -> Asyncio
Gunicorn Pyramid Pyramid View
Blocking -> Asyncio
Gunicorn Pyramid Pyramid View
Switch to asyncio worker class http://
docs.gunicorn.org/en/stable/design.html#asyncio-
workers
Blocking -> Asyncio
Gunicorn Pyramid Pyramid View
Switch to aiopyramid https://github.com/housleyjk/aiopyramid
Blocking -> Asyncio
Gunicorn Pyramid Pyramid View
We can rewrite the views incrementally? Right?
async def i_block():
print('slow func start')
time.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_block()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_blocki_run_fast
async def i_block():
print('slow func start')
time.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_block()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_blocki_run_fast
fast func
start
async def i_block():
print('slow func start')
time.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_block()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_blocki_run_fast
fast func
start
async def i_block():
print('slow func start')
time.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_block()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_blocki_run_fast
block func
start
fast func
start
async def i_block():
print('slow func start')
time.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_block()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_blocki_run_fast
block func
start
fast func
start
async def i_block():
print('slow func start')
time.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_block()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_blocki_run_fast
block func
start
fast func
start
3 sec later
block func
done
async def i_block():
print('slow func start')
time.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_block()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_blocki_run_fast
block func
start
fast func
start
3 sec later
block func
done
async def i_block():
print('slow func start')
time.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_block()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_blocki_run_fast
block func
start
fast func
start
3 sec later
block func
done
async def i_block():
print('slow func start')
time.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_block()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_blocki_run_fast
block func
start
fast func
start
3 sec later
block func
done
async def i_block():
print('slow func start')
time.sleep(3)
print('slow func done')
async def i_run_fast():
print('fast func start')
await asyncio.sleep(1)
print('fast func done')
async def run_me():
await asyncio.gather(
i_run_fast(),
i_block()
)
loop = asyncio.get_event_loop()

loop.run_until_complete(run_me())

loop.close()
Event loop i_blocki_run_fast
block func
start
fast func
start
3 sec later
block func
done
fast func
done
Blocking and Nonblocking IO Don’t Mix
Blocking and Nonblocking IO Don’t Mix
asyncio all the things
asyncio all the things
How Do You Make Big Changes?
How Do You Make Big Changes?
How Do You Make Big Changes?
Code is Social
Code is Social
• Better make sure it actually works!
Code is Social
• Better make sure it actually works!
• Pull Request Open 3+ weeks
Code is Social
• Better make sure it actually works!
• Pull Request Open 3+ weeks
• Everyone on team added as a reviewer
Code is Social
• Better make sure it actually works!
• Pull Request Open 3+ weeks
• Everyone on team added as a reviewer
• 63 comments on Pull Request
Code is Social
• Better make sure it actually works!
• Pull Request Open 3+ weeks
• Everyone on team added as a reviewer
• 63 comments on Pull Request
• Meetings for questions/concerns
Happily ever after
AWS Elastic Load Balancer 5XX
Happily ever after
AWS Elastic Load Balancer 5XX
Happily ever after
99th percentile latency
Happily ever after
99th percentile latency
asyncio
• asyncio from the start is ideal
asyncio
• asyncio from the start is ideal
• But can add it later if you have to
asyncio
• asyncio from the start is ideal
• But can add it later if you have to
• Still early, but support will only improve
asyncio
Good APIs are Hard

Good APIs are Hard

• Lots of details to get right
Good APIs are Hard

• Lots of details to get right
• Using the right technologies helps
Good APIs are Hard

• Lots of details to get right
• Using the right technologies helps
• Always keep iterating and improving
Come visit us at our booth!
@Optimizely
Try out Optimizely:
optimizely.com/get-started
Q&A

How Optimizely Scaled its REST API with asyncio