Developing a user-friendly OpenResty
application
OpenResty Con 2017 - Beijing
OpenResty Con 2017 - Beijing 1
$ whoami
Thibault Charbonnier
● Lead Engineer @ Kong (https://konghq.com)
● Main contributor @ Kong API Gateway (https://github.com/Kong/kong)
● OpenResty Contributor on my spare time
● https://github.com/thibaultcha
● https://chasum.net
OpenResty Con 2017 - Beijing 2
A user-friendly OpenResty application?
Initial assumption: Most OpenResty applications that we know of seem
to be private, deployed on internal infrastructures.
If we were to ship an on-premise OpenResty application, it should be
easy to install and deploy:
➔ NGINX processes only (no other daemons in the system)
➔ Minimal libraries dependencies (pure LuaJIT/OpenResty)
➔ Horizontally scalable (clustering)
➔ Platform agnostic
OpenResty Con 2017 - Beijing 3
A user-friendly OpenResty application?
Example: recurring background jobs
➔ Cronjob
*/5 * * * * curl -XGET 'http://localhost:8000/job?hello=world'
vs
➔ ngx.timer API
ngx.timer.every(60 * 5, do_job(), "world")
OpenResty Con 2017 - Beijing 4
What is Kong?
Short introduction to API Gateways
OpenResty Con 2017 - Beijing 5
What is an API Gateway?
It’s a reverse proxy, sitting between your clients and your upstream services
ABSTRACTION LAYER
Items
Client API Gateway
(The client can be another service too) Authentication
Customers
Security
Logging
Transformations Orders
Load-Balancing
...and more.
Invoices
OpenResty Con 2017 - Beijing 6
What is an API Gateway?
Reduce Code Duplication, Orchestrate Common Functionalities
OpenResty Con 2017 - Beijing 7
Kong
Open Source API Gateway
Built with OpenResty.
■ 12,000+ Stars on GitHub
■ 70+ Contributors
■ 107 Meetups
● Open Source ■ 10K Community Members
● Extensible via Plugins (60+ available)
● Sub-millisecond latency on most
use-cases
● Platform Agnostic
● Horizontally Scalable
https://github.com/Mashape/kong https://getkong.org
OpenResty Con 2017 - Beijing 8
Kong
init_by_lua_block {
kong = require 'kong'
kong.init()
}
…
location / {
set $upstream_scheme '';
set $upstream_uri '';
rewrite_by_lua_block { kong.rewrite() }
access_by_lua_block { kong.access() }
proxy_http_version 1.1;
proxy_pass $upstream_scheme://kong_upstream$upstream_uri;
header_filter_by_lua_block { kong.header_filter() }
body_filter_by_lua_block { kong.body_filter() }
log_by_lua_block { kong.log() }
}
OpenResty Con 2017 - Beijing 9
Kong v0.1 dependencies
● Proxy + Lua middleware → OpenResty
● A command line interface (CLI) → PUC-Rio Lua 5.1 ☹
● DNS resolution → dnsmasq ☹
● Clustering of nodes → Serf ☹
● Generate UUIDs → libuuid ☹
● Database → Cassandra
Lots of dependencies for users to install...
OpenResty Con 2017 - Beijing 10
Kong v0.1 dependencies
Processes
● Kong’s CLI
● NGINX
● dnsmasq
● serf
Ports
● 80 + 8001 (NGINX)
● 8053 (dnsmasq)
● 7946 + 7373 TCP/UDP (serf)
Less than ideal for dockerized environments or
firewall rules...
OpenResty Con 2017 - Beijing 11
CLI
Choosing an interpreter
OpenResty Con 2017 - Beijing 12
Build an OpenResty CLI
~ $ kong
Usage: kong COMMAND
Database I/O
[OPTIONS]
The available commands are:
migrations
prepare Start NGINX
reload
restart
start
stop
version
Stop NGINX
Options:
--v verbose
--vv debug
OpenResty Con 2017 - Beijing 13
Build an OpenResty CLI
#!/usr/bin/env lua
require("kong.cmd.init")(arg)
● No FFI (LuaJIT only)
● No support for cosockets (LuaSocket + LuaSec fallback)
● Missing ngx.* API
Lots of fragmentation between our OpenResty and CLI code ☹
OpenResty Con 2017 - Beijing 14
Build an OpenResty CLI
#!/usr/bin/env luajit
require("kong.cmd.init")(arg)
● LuaJIT FFI available
● No support for cosockets (LuaSocket + LuaSec fallback)
● Missing ngx.* API
An improvement but fragmentation is still very much of an issue ☹
OpenResty Con 2017 - Beijing 15
Build an OpenResty CLI
#!/usr/bin/env resty
require("kong.cmd.init")(arg)
● Runs in timer context thanks to https://github.com/openresty/resty-cli
● Cosockets available
● ngx.* API available
● LuaJIT FFI available
We can reuse our OpenResty and CLI code
No PUC-Rio Lua dependency
OpenResty Con 2017 - Beijing 16
Build an OpenResty CLI
● Proxy + Lua middleware → OpenResty
● A command line interface (CLI) → PUC-Lua 5.1 OpenResty
● DNS resolution & Load balancing → dnsmasq
● Clustering of nodes → Serf
● Generate UUIDs → libuuid
● Database → Cassandra
OpenResty Con 2017 - Beijing 17
Using resty-cli in busted
OpenResty Con 2017 - Beijing 18
Using resty-cli in busted
describe('Busted unit testing framework', function()
it('should be easy to use', function()
assert.truthy('Yup.')
end)
it('should have lots of features', function()
Kong’s test framework is busted since 2014 -- deep check comparisons!
assert.same({ table = 'great'}, { table = 'great' })
https://github.com/Olivine-Labs/busted
-- or check by reference!
assert.is_not.equals({ table = 'great'},
{ table = 'great'})
assert.falsy(nil)
assert.error(function() error('Wat') end)
end)
end)
OpenResty Con 2017 - Beijing 19
Using resty-cli in busted
We changed the interpreter from PUC-Rio
Lua to resty-cli
-- ./rbusted
#!/usr/bin/env resty
-- Busted command-line runner
require 'busted.runner'({ standalone = false })
https://github.com/thibaultcha/lua-resty-busted
OpenResty Con 2017 - Beijing 20
Using resty-cli in busted
-- t/sanity_spec.lua
describe("openresty script", function()
it("should run in ngx_lua context", function()
assert.equal(0, ngx.OK)
assert.equal(200, ngx.HTTP_OK)
end)
it("should yield", function()
ngx.sleep(3)
assert.is_true(1 == 1)
end)
end)
OpenResty Con 2017 - Beijing 21
Using resty-cli in busted
Improving busted for OpenResty development
~ $ rbusted --o=tap t/sanity_spec.lua
ok 1 - openresty script should run in ngx_lua context
ok 2 - openresty script should yield
1..2
Now we can test our OpenResty code with busted!
OpenResty Con 2017 - Beijing 22
UUID generation
And PRNG seeding
OpenResty Con 2017 - Beijing 23
Removing the libuuid dependency
● PUC-Rio Lua - https://github.com/Tieske/uuid
○ Slowest implementation
○ Generates invalid v4 UUIDs
● libuuid binding (Lua C API) - https://github.com/Kong/lua-uuid
○ Safe underlying implementation
○ External dependency
● libuuid binding (LuaJIT FFI) - https://github.com/bungle/lua-resty-uuid
○ Safe underlying implementation
○ External dependency
● LuaJIT - https://github.com/thibaultcha/lua-resty-jit-uuid
○ Seems to be the fastest implementation
○ Uses LuaJIT’s PRNG
OpenResty Con 2017 - Beijing 24
Removing the libuuid dependency
LuaJIT 2.1.0-beta1 with 1e+06 UUIDs
UUID v4 (random) generation
1. resty-jit-uuid took: 0.064228s 0%
2. FFI binding took: 0.093374s +45%
3. C binding took: 0.220542s +243%
4. Pure Lua took: 2.051905s +3094%
UUID v3 (name-based and MD5) generation if supported
1. resty-jit-uuid took: 1.306127s
UUID v5 (name-based and SHA-1) generation if supported
1. resty-jit-uuid took: 4.834929s
UUID validation if supported (set of 70% valid, 30% invalid)
1. resty-jit-uuid (JIT PCRE enabled) took: 0.223060s
2. FFI binding took: 0.256580s
3. resty-jit-uuid (Lua patterns) took: 0.444174s
OpenResty Con 2017 - Beijing 25
Caution: PRNG seeding in NGINX workers
init_by_lua_block {
--math.randomseed(ngx.time()) <-- AVOID
}
init_worker_by_lua_block {
math.randomseed(ngx.time() + ngx.worker.pid())
math.randomseed = function()end -- ensure we prevent re-seeding
}
● Be wary of calling math.randomseed() in init_by_lua
● Seeding in init_worker_by_lua is safer
● Still, some external dependencies may call math.randomseed() again
A possible solution: https://github.com/openresty/lua-resty-core/pull/92
OpenResty Con 2017 - Beijing 26
Build an OpenResty CLI
● Proxy + Lua middleware → OpenResty
● A command line interface (CLI) → PUC-Lua 5.1 OpenResty
● DNS resolution & Load balancing → dnsmasq
● Clustering of nodes → Serf
● Generate UUIDs → libuuid OpenResty
● Database → Cassandra
OpenResty Con 2017 - Beijing 27
DNS Resolution
OpenResty Con 2017 - Beijing 28
DNS Resolution
NGINX does not use the system resolver, and needs a
user-specified name server.
More often than not, this deceives users:
➔ Ignores /etc/resolv.conf ☹
➔ Ignores /etc/hosts ☹
➔ No support for SRV records ☹
➔ Unusable with balancer_by_lua ☹
OpenResty Con 2017 - Beijing 29
DNS Resolution
Temporary solution: dnsmasq
~ $ kong start --vv
...
2017/03/01 14:45:35 [debug] found 'dnsmasq' executable at /usr/sbin/dnsmasq
2017/03/01 14:45:35 [debug] starting dnsmasq: /usr/sbin/dnsmasq -p 8053 --pid-file=/usr/local/kong/pids/dnsmasq.pid -N -o
--listen-address=127.0.0.1
http { ● Parses /etc/resolv.conf
resolver 127.0.0.1:8053 ipv6=off; ● Parses /etc/hosts
● Support for SRV records
...
} ● Still unusable with balancer_by_lua ☹
● New dependency ☹
dnsmasq daemon
OpenResty Con 2017 - Beijing 30
DNS Resolution
To remove our dnsmasq dependency, and use balancer_by_lua, we
must resolve DNS records in the Lua land.
Part of the solution: https://github.com/openresty/lua-resty-dns
● Pure Lua, bundled with OpenResty
● Resolves, A, AAAA, CNAME, SRV records (and more)
● No /etc/hosts parsing ☹
● No /etc/resolv.conf parsing ☹
● No results caching ☹
● No DNS load-balancing ☹
OpenResty Con 2017 - Beijing 31
DNS Resolution
lua-resty-dns-client - https://github.com/Kong/lua-resty-dns-client
Author: Thijs Schreijer (@tieske)
● Built on top of lua-resty-dns
● Parses /etc/hosts
● Parses /etc/resolv.conf
● Built-in cache & asynchronous querying
● Built-in DNS load-balancing
●
OpenResty Con 2017 - Beijing 32
DNS Resolution
● Proxy + Lua middleware → OpenResty
● A command line interface (CLI) → PUC-Lua 5.1 OpenResty
● DNS resolution & Load balancing → dnsmasq OpenResty
● Clustering of nodes → Serf
● Generate UUIDs → libuuid OpenResty
● Database → Cassandra
OpenResty Con 2017 - Beijing 33
Clustering
OpenResty Con 2017 - Beijing 34
Clustering
● Kong nodes connected to the same database (PostgreSQL or Cassandra)
share the same configuration.
● To limit database traffic, Kong nodes maintain their own cache.
● lua-shared-dict + lua-resty-lock allow Kong to avoid the “dogpile effect”
(cache stampede).
http {
lua_shared_dict kong_cache ${{MEM_CACHE_SIZE}};
…
}
OpenResty Con 2017 - Beijing 35
Clustering
Temporary solution: Serf (https://www.serf.io/)
~ $ kong start --vv
...
2017/05/22 14:30:13 [debug] found 'serf' executable in $PATH
2017/05/22 14:30:13 [debug] starting serf agent: nohup serf agent -profile 'wan' -bind '0.0.0.0:7946' -log-level 'err' -rpc-addr
'127.0.0.1:7373' -event-handler
'member-join,member-leave,member-failed,member-update,member-reap,user:kong=/usr/local/kong/serf/serf_event.sh' -node
'dev_0.0.0.0:7946_470b634076b94e2aa6a0bb7bce7673f7' > /usr/local/kong/logs/serf.log 2>&1 & echo $! > /usr/local/kong/pids/serf.pid
2017/05/22 14:30:14 [verbose] serf agent started
2017/05/22 14:30:14 [verbose] auto-joining serf cluster
2017/05/22 14:30:14 [verbose] registering serf node in datastore
2017/05/22 14:30:14 [verbose] cluster joined and node registered in datastore
● Provides inter-nodes gossiping
● New dependency ☹
● Additional ports and firewall rules ☹
● Additional cross-datacenter communication ☹
OpenResty Con 2017 - Beijing 36
Clustering
The overhead of the OpenResty + Serf pattern
LB LB
us-west-1 us-east-1
K K K K K K
Serf Serf Serf Serf Serf Serf
Cassandra Cassandra
37
OpenResty Con 2017 - Beijing 37
Clustering
Our desired high-level view of a Kong cluster
LB LB
us-west-1 us-east-1
K K K K K K
Cassandra Cassandra
38
OpenResty Con 2017 - Beijing 38
Clustering
We removed our Serf dependency by introducing a pub/sub mechanism between
OpenResty and PostgreSQL/Cassandra.
● Workers write in a “channel” with broadcast(channel, data, nbf)
● Other workers subscribe to it with subscribe(channel, callback)
● A combination of ngx.timer and lua-resty-lock allows for a safe polling mechanism
● Introducing new configuration properties:
db_update_frequency/db_update_propagation/db_cache_ttl
● Upon invalidation event received: ngx.shared.cache:delete(key)
https://github.com/Kong/kong/blob/master/kong/cluster_events.lua
OpenResty Con 2017 - Beijing 39
Clustering
lua-resty-mlcache
NGINX
● Multi-level caching (lua-resty-cache + worker worker worker
L1
lua_shared_dict) with LRU eviction
Lua cache Lua cache Lua cache
● TTL and negative (miss) TTL
● Built-in mutex mechanism with
lua-resty-lock to prevent dogpile L2 lua_shared_dict
effects
● Multiple instances supported
callback
I/O fetch
Database, API, I/O...
https://github.com/thibaultcha/lua-resty-mlcache
OpenResty Con 2017 - Beijing 40
DNS Resolution
● Proxy + Lua middleware → OpenResty
● A command line interface (CLI) → PUC-Lua 5.1 OpenResty
● DNS resolution & Load balancing → dnsmasq OpenResty
● Clustering of nodes → Serf OpenResty
● Generate UUIDs → libuuid OpenResty
● Database → Cassandra
OpenResty Con 2017 - Beijing 41
Inter-workers communication
OpenResty Con 2017 - Beijing 42
Inter-workers communication
Invalidating Lua-land cache (lua-resty-lru) requires inter-workers
communication, a long-requested OpenResty feature.
lua-resty-worker-events - https://github.com/Kong/lua-resty-worker-events
Author: Thijs Schreijer (@tieske)
● Pub/sub mechanism via lua_shared_dict
● Multiple channels
● Automatic polling via ngx.timer
Ideally, a binding API for cosockets will one day replace lua_shared_dict based
solutions!
OpenResty Con 2017 - Beijing 43
Conclusion
OpenResty Con 2017 - Beijing 44
Conclusion
One by one, we’ve eliminated all external dependencies. Kong now is a
pure OpenResty application.
● Proxy + Lua middleware → OpenResty
● A command line interface (CLI) → PUC-Lua 5.1 OpenResty
● DNS resolution & Load balancing → dnsmasq OpenResty
● Clustering of nodes → Serf OpenResty
● Generate UUIDs → libuuid OpenResty
● Database → Cassandra
OpenResty Con 2017 - Beijing 45
Conclusion
We’ve open sourced several libraries to the OpenResty community!
● https://github.com/Kong/lua-resty-dns-client
● https://github.com/Kong/lua-resty-worker-events
● https://github.com/thibaultcha/lua-resty-mlcache
● https://github.com/thibaultcha/lua-resty-jit-uuid
● https://github.com/thibaultcha/lua-resty-busted
● And more!
● https://github.com/thibaultcha/lua-cassandra (see my LuaConf
2017 talk in Rio de Janeiro: https://youtu.be/o8mbOT3Veeo)
● https://github.com/thibaultcha/lua-resty-socket
OpenResty Con 2017 - Beijing 46
Conclusion
Wishlist:
● Support for cosockets in init_by_lua
● Native inter-workers communication
● Support for SSL client certificates for cosockets
(https://github.com/openresty/lua-nginx-module/pull/997)
● Support for ngx.rawlog() API
(https://github.com/openresty/lua-resty-core/pull/128)
● Support for /etc/hosts parsing
(https://github.com/openresty/openresty/pull/247)
OpenResty Con 2017 - Beijing 47
Thank you!
Questions?
OpenResty Con 2017 - Beijing 48
Bonus
OpenResty Con 2017 - Beijing 49
lua-cjson empty array encoding
local cjson = require "cjson"
local rows = {} -- fetch from db
-- before
cjson.encode({ data = rows })
--[[
{
"data":{}
}
--]]
-- now
setmetatable(rows, cjson.empty_array_mt)
cjson.encode({ data = rows })
--[[
{
"data":[]
https://github.com/openresty/lua-cjson/pull/6
}
--]]
OpenResty Con 2017 - Beijing 50
lua-resty-socket
https://github.com/thibaultcha/lua-resty-socket
Compatibility module for cosocket/LuaSocket.
● Automatic fallback to LuaSocket in non-OpenResty, or non-supported
OpenResty contexts (e.g. init_by_lua)
● Support for SSL via LuaSec fallback
● Full interoperability
local socket = require "resty.socket"
local sock = socket.tcp()
sock:settimeout(1000) ---> 1000ms converted to 1s if LuaSocket
sock:getreusedtimes(...) ---> 0 if LuaSocket
sock:setkeepalive(...) ---> calls close() if LuaSocket
sock:sslhandshake(...) ---> LuaSec dependency if LuaSocket
OpenResty Con 2017 - Beijing 51
Friendly error logs with ngx.log
local errlog = require "ngx.errlog"
errlog.rawlog(ngx.NOTICE, "hello world")
2017/07/09 19:36:25 [notice] 25932#0: *1 [lua] content_by_lua(nginx.conf:51):5:
hello world, client: 127.0.0.1, server: localhost, request: "GET /log
HTTP/1.1", host: "localhost"
● Raw output to error_log
● Customizable stacktrace level report
https://github.com/openresty/lua-resty-core/pull/128
OpenResty Con 2017 - Beijing 52
Thank you!
Questions?
OpenResty Con 2017 - Beijing 53
