Reactive server with netty

Highload reactive server
with Netty

Dmitriy Dumanskiy
Blynk, CTO
Java blog : https://habrahabr.ru/users/doom369/topics
DOU : https://dou.ua/users/DOOM/articles/

Makers problem
● Http/s
● Mqtt
● WebSockets
● Own binary protocol

Blynk
10000 req/sec
3 VM * 2 cores, 60$
25% load
10k of local installations

Why netty?
Cassandra
Apache Spark
Elasticsearch
Graylog
Neo4j
Vert.x
HornetQInfinispan
Finagle
Async-http-client
Firebase
Akka
CouchbasePlay framework
Redisson

Why netty?
● Less GC
● Optimized for Linux based OS

Why netty?
● Less GC
● High performance buffers

Why netty?
● Less GC
● Well defined threading model

Why netty?
● Less GC
● Well defined threading model
● HTTP, HTTP/2, SPDY, SCTP, TCP,
UDP, UDT, MQTT, etc

When to use?
● Performance is critical

When to use?
● Own protocol

When to use?
● Own protocol
● Full control over network
(so_reuseport, tcp_cork,
tcp_fastopen, tcp_nodelay, etc)

When to use?
● Own protocol
● Game engines (agario, slither,
minecraft)

When to use?
● Own protocol
● Game engines
● <3 reactive

Non-Blocking
● Few threads
● No context switching
● No memory consumption

Non-Blocking
new Channel
read / write
Selector
Thread
new Channel
read / write
new Channel
read / write

java.nio.channels.Selector
Selector selector = Selector.open();
channel.configureBlocking(false);
SelectionKey key = channel.register(selector, SelectionKey.OP_READ);
while(true) {
selector.select();
Set<SelectionKey> selectedKeys = selector.selectedKeys();
Iterator<SelectionKey> keyIterator = selectedKeys.iterator();
while(keyIterator.hasNext()) {
key = keyIterator.next();
if (key.isReadable()) { ... }
}
}

Selector selector = Selector.open(); // creating selector
while(true) {
selector.select();
}
}

//registering channel with selector, listening for READ events only
while(true) {
selector.select();
}
}

while(true) {
selector.select(); //blocking until we get some READ events
}
}

while(true) {
selector.select();
//now we have channels with some data
}
}

while(true) {
selector.select();
//do something with data
if (key.isReadable()) { key.channel() }
}
}

Flow
Selector
SelectionKey
Channel
ChannelPipeline

Flow
ChannelPipeline
fireEvent()
invokeChannelRead() executor.execute()
invokeChannelRead()

Minimal setup
ServerBootstrap b = new ServerBootstrap();
b.group(
new NioEventLoopGroup(1),
new NioEventLoopGroup()
) .channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer() {...});
ChannelFuture f = b.bind(8080).sync();
f.channel().closeFuture().sync();

Minimal setup
b.group(
new NioEventLoopGroup(1), //IO thread
new NioEventLoopGroup()

Minimal setup
b.group(
new NioEventLoopGroup() //worker threads

Minimal setup
b.group(
new NioEventLoopGroup() //worker threads
.childHandler(new ChannelInitializer() {...}); //pipeline init

Minimal setup
new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) {
final ChannelPipeline pipeline = ch.pipeline();
pipeline.addLast(new MyLogicHere());
}
};

ChannelPipeline
● Inbound event ->
ChannelInboundHandler (CIHA)
● Outbound event ->
ChannelOutboundHandler (COHA)

ChannelInboundHandler
public interface ChannelInboundHandler extends ChannelHandler {
...
void channelRegistered(ChannelHandlerContext ctx);
void channelActive(ChannelHandlerContext ctx);
void channelRead(ChannelHandlerContext ctx, Object msg);
void userEventTriggered(ChannelHandlerContext ctx, Object evt);
void channelWritabilityChanged(ChannelHandlerContext ctx);
...
}

void initChannel(SocketChannel ch) {
ch.pipeline()
.addLast(new MyProtocolDecoder())
.addLast(new MyProtocolEncoder())
.addLast(new MyLogicHandler());
}
Own tcp/ip server

Channel
MyProtocolDecoder
MyLogicHandler
Own tcp/ip server
Channel
MyProtocolEncoder
MyLogicHandler

Handlers
HttpServerCodec
ChannelTrafficShapingHandler
IdleStateHandler
ReadTimeoutHandler
ChunkedWriteHandler
SslHandler
LoggingHandler
RuleBasedIpFilter
StringDecoder
JsonObjectDecoder
Base64DecoderJZlibDecoder
JZlibDecoder
Lz4FrameDecoder
ProtobufDecoder
ObjectDecoder
XmlFrameDecoder

ch.pipeline()
.addLast(new HttpRequestDecoder())
.addLast(new HttpResponseEncoder())
.addLast(new MyHttpHandler());
}
Http Server

ch.pipeline()
.addLast(new HttpServerCodec())
}
OR

ch.pipeline()
.addLast(sslCtx.newHandler(ch.alloc()))
}
Https Server

ch.pipeline()
.addLast(sslCtx.newHandler(ch.alloc()))
.addLast(new HttpContentCompressor())
}
Https Server + content gzip

@Override
public void channelRead(Context ctx, Object msg) {
//pass flow processing to next handler
super.channelRead(ctx, msg);
}
Pipeline flow

@Override
//stop request processing
return;
}
Pipeline flow

If (msg instanceOf LoginMessage) {
LoginMessage login = (LoginMessage) msg;
if (isSuperAdmin(login)) {
ctx.pipeline().remove(this);
ctx.pipeline().addLast(new SuperAdminHandler());
}
}
Pipeline flow on the fly

ChannelFuture cf = ctx.writeAndFlush(response);
cf.addListener(new ChannelFutureListener() {
@Override
public void complete(ChannelFuture future) {
future.channel().close();
}
});
}
Pipeline futures

@Override
//close connection after message was delivered
cf.addListener(ChannelFutureListener.CLOSE);
}
Pipeline futures

@Override
...
cf.addListener(future -> {
...
});
}
Pipeline futures

ChannelFuture cf = session.sendMsgToFriend(msg);
cf.addListener(new ChannelFutureListener() {
@Override
public void complete(ChannelFuture future) {
future.channel().writeAndFlush(“Delivered!”);
}
});
}
Pipeline futures

Pipeline blocking IO
Non blocking pools Blocking pools
IO Event Loops
DB
Worker Event Loops
Mailing
File system

if (msg instanceof HttpRequest) {
HttpRequest req = (HttpRequest) msg;
if (req.method() == GET && req.uri().eq(“/users”)) {
Users users = dbManager.userDao.getAllUsers();
ctx.writeAndFlush(new Response(users));
}
}

if (req.method() == POST && req.uri().eq(“/email”)) {
mailManager.sendEmail();
}
}

if (req.method() == GET && req.uri().eq(“/property”)) {
String property = fileManager.readProperty();
ctx.writeAndFlush(new Response(property));
}
}
}

...
blockingThreadPool.execute(() -> {
Users users = dbManager.userDao.getAllUsers();
ctx.writeAndFlush(new Response(users));
});
}

● Thread.sleep()

● Thread.sleep()
● java.util.concurrent.*

● Thread.sleep()
● Intensive operations

● Thread.sleep()
● Intensive operations
● Any blocking IO (files, db, smtp, etc)

@Override
public void channelInactive(Context ctx) {
HardwareState state = getState(ctx.channel());
if (state != null) {
ctx.executor().schedule(
new DelayedPush(state), state.period, SECONDS
);
}
}
EventLoop is Executor!

if (msg instanceof FullHttpRequest) {
FullHttpRequest request = (FullHttpRequest) msg;
User user = sessionDao.checkCookie(request);
...
}
}
Request state

private static AttributeKey<User>
USER_KEY = AttributeKey.valueOf("user");
ctx.channel().attr(USER_KEY).set(user);
Request state

if (msg instanceof FullHttpRequest) {
FullHttpRequest request = (FullHttpRequest) msg;
User user = sessionDao.checkCookie(request);
ctx.channel().attr(USER_KEY).set(user);
}
}
Request state

if (isSsl(in)) {
enableSsl(ctx);
} else {
if (isGzip()) {
enableGzip(ctx);
} else if (isHttp(in)) {
switchToHttp(ctx);
}
Port unification

Back pressure
if (channel.isWritable()) {
channel.writeAndFlush(msg);
}

Back pressure
BackPressureHandler
coming soon...

Performance
https://www.techempower.com/benchmarks/#section=data-r13&hw=ph&test=plaintext

<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-epoll</artifactId>
<version>${netty.version}</version>
<classifier>${os}</classifier>
</dependency>
Native transport

Bootstrap b = new Bootstrap();
b.group(new EpollEventLoopGroup());
b.channel(EpollSocketChannel.class);
Native transport

SslContextBuilder.forServer()
.sslProvider(SslProvider.OpenSsl);
JNI OpenSslEngine

<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>${netty.boring.ssl.version}</version>
<classifier>${os}</classifier>
</dependency>
JNI OpenSslEngine

● Netty-tcnative
● netty-tcnative-libressl
● netty-tcnative-boringssl-static
JNI OpenSslEngine

Own ByteBuf
● Reference counted
● Pooling by default
● Direct memory by default
● LeakDetector by default
● Reduced branches, range-checks

Own ByteBuf
● ByteBufAllocator.buffer(size);
● ctx.alloc().buffer(size);
● channel.alloc().buffer(size);

Less system calls
for (Message msg : messages) {
ctx.writeAndFlush(msg);
}

Less system calls
for (Message msg : messages) {
ctx.write(msg);
}
ctx.flush();

Thread Model
ChannelFuture inCf = ctx.deregister();
inCf.addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture cf) {
targetLoop.register(cf.channel())
.addListener(completeHandler);
}
});

Reusing Event Loop
new ServerBootstrap().group(
new EpollEventLoopGroup(1),
new EpollEventLoopGroup()
).bind(80);

Reusing Event Loop
EventLoopGroup boss = new EpollEventLoopGroup(1);
EventLoopGroup workers = new EpollEventLoopGroup();
boss,
workers
).bind(80);
boss,
workers
).bind(443);

Use direct buffers
ctx.writeAndFlush(
new ResponseMessage(messageId, OK)
);

Use direct buffers
ByteBuf buf = ctx.alloc().buffer(3);//pool
buf.writeByte(messageId);
buf.writeShort(OK);
ctx.writeAndFlush(buf);

Less allocations
ByteBuf msg = makeResponse(...);
msg.retain(targets.size() - 1);
for (Channel ch : targets) {
ch.writeAndFlush(msg);
}

Void promise
ctx.writeAndFlush(
response
);

Void promise
ctx.writeAndFlush(
response, ctx.voidPromise()
);

Reuse handlers
@Sharable
public class StringDecoder extends
MessageToMessageDecoder<ByteBuf> {
...
}

Prefer context
ctx.channel().writeAndFlush();

Prefer context
ctx.channel().writeAndFlush();
ctx.writeAndFlush();

Simpler - faster
ChannelInboundHandlerAdapter
does nothing, but fast

Simpler - faster
ByteToMessageDecoder
does some work, but slower

Simpler - faster
ReplayingDecoder
does job for you, but slowest

Turn off leak detection
ResourceLeakDetector.setLevel(
ResourceLeakDetector.Level.DISABLED);

What else?
● ASCIIString
● FastThreadLocal
● Unsafe
● Optimized Encoders

● Really fast
● Low GC load
● Flexible
● Rapidly evolve
● Cool support
Summary

● Hard
● Memory leaks
● Still have issues
Summary

https://github.com/blynkkk/blynk-server

Reactive server with netty

More Related Content

What's hot

Similar to Reactive server with netty

Recently uploaded

Reactive server with netty