Reactive Programming at Cloud-Scale and Beyond

InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/reactive-cloud-scale

Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide


 Tesla

 mid 2000s





 LINQ to Events
Nikola Tesla
www.knowledgeoftoday.org


 Tier-splitting
 metadata







 asynchronous
www.Wikipedia.org




 Events are not first-class objects

[RunOnClient]
public event EventHandler<MouseEventArgs> MouseMoved;
// Runs in cloud
public void CloudCanvas()
{
MouseMoved += (o, e) => { /* do stuff */ };
}
An electric eel…


 delegates

Action a = new Action(Foo); // explicit creation of delegate instance
Action b = Foo; // method group conversion
Action c = () => { … }; // creates anonymous method
void Foo() { … }
event Action Bar // metadata that refers to …
{
add { … } // add accessor
remove { … } // remove accessor
}
http://en.wikipedia.org/wiki/First-class_citizen


 duality enumerator

interface IObservable<out T>
{
IDisposable Subscribe(IObserver<T> observer);
}
interface IObserver<in T>
{
void OnNext(T value);
void OnError(Exception error);
void OnCompleted();
}
Notification grammar
OnNext* (OnError | OnCompleted)?
“Gang of four” book
Addison-Wesley


 Composition
 Composition
interface IScheduler
{
IDisposable Schedule(Action work);
…
}
static class Observable
{
static IObservable<T> Where<T>(this IObservable<T> source, Func<T, bool> f);
static IObservable<R> Select<T, R>(this IObservable<T> source, Func<T, R> p);
…
}
Function composition
www.Wikipedia.org

 Function composition
www.Wikipedia.org
public static class Observable
{
public static IObservable<T> Merge<T>(this IObservable<T> xs, IObservable<T> ys)
{
return Create<T>(observer =>
{
var gate = new object();
return new CompositeDisposable
{
xs.Subscribe(x => { lock (gate) { observer.OnNext(x); } }, …),
ys.Subscribe(y => { lock (gate) { observer.OnNext(y); } }, …),
};
});
}
}
First-class: can build
extension methods
Composition of
resource management

 layering


 asynchrony and time


 virtual time historical time
public static IObservable<T> Return<T>(T value, IScheduler scheduler)
{
return Create<T>(obs => scheduler.Schedule(() => { obs.OnNext(value);
obs.OnCompleted(); }));
}
Space-time
www.Wikipedia.org


 push pull

Category theory
www.Wikipedia.org
interface IObservable<out T>
{
IDisposable Subscribe(IObserver<T> observer);
}
interface IObserver<in T>
{
void OnNext(T value);
void OnError(Exception error);
void OnCompleted();
}
interface IEnumerable<out T>
{
IEnumerator<T> GetEnumerator();
}
interface IEnumerator<out T> : IDisposable
{
bool MoveNext() throws Exception;
T Current { get; }
void Reset();
}



 Await Stephen Kleene
Kleene star (closure)
www.Wikipedia.org
Func<T> f
var x = wait f();
IEnumerable<T> xs
foreach (var x in xs) {
f(x);
}
Task<T> t
var x = await t;
IObservable<T> xs
xs.Subscribe(x => {
f(x);
});
OneMany
Synchronous Asynchronous


 Code-as-data


Alan Kay
Homoiconicity, 1969
www.Wikipedia.org
IEnumerable<T> xs
from x in xs where f(x) …
IQueryable<T> xs
IObservable<T> xs
IQbservable<T> xs
CodeData
Pull Push

 Events




 scalable abstraction



Satya Nadella
“Cloud-first, mobile-first”
www.businessinsider.com


 Bing


 Cortana


 Cloud and device



 Optimization



 Reliability






Richard Feynman
Westview Press

from w in weather
where w.City == “Seattle”
select w
flights
.Where(f => f.Code == “BA49”)
.DelaySubscription(departure – 24 * H)
.TakeUntil(arrival + 24 * H)
.Select(f => f.Status)
userLocation
.Sample(1 * M)
.DistinctUntilChanged()
.Select(here =>
traffic(here, meetingLocation)
.TakeUntil(meeting – minimumTime)
.Select(t => Timer(meeting – t.EstimatedTime))
.StartWith(Timer(meeting – estimatedTime))
.Switch())
.Switch()
.Select(_ => “Leave now for ” + subject)
.DelaySubscription(meeting – estimatedTime)
Temporal query
operators
Device-side
event stream
// Insert cloud-side observable sequence
// of time-to-leave timer here
Cloud-side
event stream
Higher-order
query operators
Remember
tier-splitting?

 IReactiveProcessing



Bing cloud
IRP
Partner1 cloud
IRP
Partner2 cloud
IRP
Tablet
IRP
Phone
IRP
Node
IRP
Node
IRP Node
IRP

 Rx



 distributed








 Proxies
 Definitions
 Metadata


 Hyper-cube





Sync Async
IntrinsicExtrinsic
Higher dimensions
Calabi-Yau manifold
www.Wikipedia.org


 Proxies
 Definitions
 Metadata


 Hyper-cube
 Synchronous versus asynchronous
 Extrinsic versus intrinsic identities
 Code-as-data versus code
 Reliable versus non-reliable
 Etc.
Higher dimensions
Calabi-Yau manifold
www.Wikipedia.org
Async
Extrinsic
E.g. this is the
client-side API
Need string theory


 context




 proxies

var conn = new ReactiveServiceConnection(endpoint);
var ctx = new ClientContext(conn);
var traffic = ctx.GetObservable<TrafficInfo>(trafficUri);
var http = ctx.GetObserver<HttpData>(httpUri);
Explicit identifiers
provided for artifacts



 extrinsic identifiers
 Async


var subscription = await traffic.Where(t => t.Road == “I-90”)
.Select(t => new HttpData { Uri = myService,
Body = t.ToString() })
.SubscribeAsync(http, mySubUri);
Explicit identifiers
provided for artifacts

 SubscribeAsync
 expression tree



 serialized



 IRP

var subscription = await traffic.Where(t => t.Road == “I-90”)
.Select(t => new HttpData { Uri = …, Body = … })
.SubscribeAsync(http, mySubUri);
Invoke
rx://subscribe Invoke
rx://select Invoke
rx://where trafficUri
httpUri
λ
t => new { bing://http/uri = …,
bing://http/body = … }
λ t => t.Road == “I-90”
Unbound parameters
processed by binder
Structural typing to
reduce coupling

 definition of artifacts



// Define
await ctx.DefineObservableAsync<string, TrafficInfo>(
trafficByRoadUri,
road => traffic.Where(t => t.Road == road));
// Proxies
var trafficByRoad = ctx.GetObservable<string, TrafficInfo>(trafficByRoadUri);
trafficByRoad(“I-90”).…


 parameterized observables



var xs = ctx.GetObservable<int>(xsUri);
var iv = ctx.GetObserver<int>(ivUri);
await xs.Where(x => x > 0).Select(x => x * x).SubscribeAsync(iv, …);
var xs = ctx.GetObservable<int>(xsUri);
var iv = ctx.GetObserver<int>(ivUri);
var where = ctx.GetObservable<IARQ<int>, Expr<Func<int, bool>>, int>(whereUri);
var select = ctx.GetObservable<IARQ<int>, Expr<Func<int, int>>, int>(selectUri);
await select(where(xs, x => x > 0), x => x * x).SubscribeAsync(iv, …);

 [KnownResource] attributes can be used everywhere
 Provide shorthand syntax for Get* operations
 Enable code generation using the Metadata facilities
 What’s IAsyncReactiveQbservable<T>?
 Async because it’s client-side (cf. SubscribeAsync)
 Reactive because we had to disambiguate with Rx concepts
 Qbservable because it’s expression tree based
static class AsyncReactiveQbservable
{
[KnownResource(whereUri)]
static IAsyncReactiveQbservable<T> Where<T>(this IAsyncReactiveQbservable<T> xs,
Expression<Func<T, bool>> filter) {…}
}

 properties


 tooling


 delegation





 recovery
 density
 reliability
 Scalable

 checkpointing
 Acknowledge / replay



 Coordinator engines

Event Processing
Execution Engine Node1
Rx
IRP
Event Processing
Execution Engine Node2
Rx
IRP
Checkpoint
storage
Save
Load
Checkpoint
storage
Save
Load
Reliable
messaging
OnNext
Replay
Ack

 traverse the operator tree






 ISubscribable<T>

interface ISubscribable<out T>
{
ISubscription Subscribe(ISubscriber<T> subscriber);
}
interface ISubscription : IDisposable
{
void Accept(ISubscriptionVisitor v);
}





 Physical schedulers


 logical schedulers

 Unit of pausing



 onion




 GitHub




 bartde@microsoft.com
Service
Host
Data model
Query engine
Operators

Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/reactive-
cloud-scale

Reactive Programming at Cloud-Scale and Beyond

More Related Content

What's hot

Similar to Reactive Programming at Cloud-Scale and Beyond

More from C4Media

Recently uploaded

Reactive Programming at Cloud-Scale and Beyond