Download as PDF, PPTX
![Word Count
var spark = require(‘eclairjs');
var sc = new spark.SparkContext("local[*]", "foo");
var file = __dirname + '/dream.txt';
var rdd = sc.textFile(file);
var rdd2 = rdd.flatMap(function(sentence) {
return sentence.split(" ");
});
var rdd3 = rdd2.filter(function(word) {
return word.trim().length > 0;
});
var rdd4 = rdd3.mapToPair(function(word, Tuple) {
return new Tuple(word.toLowerCase(), 1);
}, [spark.Tuple]);
var rdd5 = rdd4.reduceByKey(function(value1, value2) {
return value1 + value2;
});
var rdd6 = rdd5.mapToPair(function(tuple, Tuple) {
return new Tuple(tuple[1], tuple[0]);
}, [spark.Tuple]);
var rdd7 = rdd6.sortByKey(false);
rdd7.take(10).then(function(val) {
console.log("Success:", val);
});
Spark Operator](https://image.slidesharecdn.com/9bdavidfallside-160614185453/85/EclairJS-Node-Js-Apache-Spark-5-320.jpg)
![Word Count
var spark = require(‘eclairjs');
var sc = new spark.SparkContext("local[*]", "foo");
var file = __dirname + '/dream.txt';
var rdd = sc.textFile(file);
var rdd2 = rdd.flatMap(function(sentence) {
return sentence.split(" ");
});
var rdd3 = rdd2.filter(function(word) {
return word.trim().length > 0;
});
var rdd4 = rdd3.mapToPair(function(word, Tuple) {
return new Tuple(word.toLowerCase(), 1);
}, [spark.Tuple]);
var rdd5 = rdd4.reduceByKey(function(value1, value2) {
return value1 + value2;
});
var rdd6 = rdd5.mapToPair(function(tuple, Tuple) {
return new Tuple(tuple[1], tuple[0]);
}, [spark.Tuple]);
var rdd7 = rdd6.sortByKey(false);
rdd7.take(10).then(function(val) {
console.log("Success:", val);
});
Spark Operator](https://image.slidesharecdn.com/9bdavidfallside-160614185453/75/EclairJS-Node-Js-Apache-Spark-5-2048.jpg)
More Related Content
Similar to EclairJS = Node.Js + Apache Spark
![RTP_AR_Basic_Learners' Workbook_KS2 [FOR REPRODUCTION] (1).pdf](https://cdn.slidesharecdn.com/ss_thumbnails/rtparbasiclearnersworkbookks2forreproduction1-251016024943-e51a16ac-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
![Matrix and determinant URT [Autosaved].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/matrixanddeterminanturtautosaved-251018190340-9e6a6deb-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
EclairJS = Node.Js + Apache Spark
- 1. ECLAIRJS = NODE.JS+ APACHE SPARK David Fallside IBM
- 2. Why EclairJS? • Digitalbusiness looking to improve customer interactions, capture “perishable” insights (Forrester), use new data sources • Today’s interactive & user-facing applications often developed in JavaScript using Node.js • npm provides largest (Node.js) package repo (www.modulecounts.com) • Handles very large numbers of simultaneous requests • Compute-intensive workloads handed off to back-end engines • Apache Spark as a back-end engine • Scalable, static & streaming data, Spark SQL, ML analytics, graph engine • But no Spark API for Node.js/JavaScript, hence EclairJS • So let’s look at an EclairJS application …
- 3.
- 4. Program Flow Kafka Spark SQLTempTable Spark Node.js Radial Graph UI Airport Selection Flight Data
- 5. Word Count var spark= require(‘eclairjs'); var sc = new spark.SparkContext("local[*]", "foo"); var file = __dirname + '/dream.txt'; var rdd = sc.textFile(file); var rdd2 = rdd.flatMap(function(sentence) { return sentence.split(" "); }); var rdd3 = rdd2.filter(function(word) { return word.trim().length > 0; }); var rdd4 = rdd3.mapToPair(function(word, Tuple) { return new Tuple(word.toLowerCase(), 1); }, [spark.Tuple]); var rdd5 = rdd4.reduceByKey(function(value1, value2) { return value1 + value2; }); var rdd6 = rdd5.mapToPair(function(tuple, Tuple) { return new Tuple(tuple[1], tuple[0]); }, [spark.Tuple]); var rdd7 = rdd6.sortByKey(false); rdd7.take(10).then(function(val) { console.log("Success:", val); }); Spark Operator
- 6. EclairJS Stack Node.js Application EclairJS-Node Desktop, etcWeb Browser Cluster/Driver Toree* EclairJS-Nashorn Java, Nashorn Spark Context EclairJS-Nashorn Java, Nashorn Spark Executor Jupyter Gateway Jupyter NB Server Cloud/IT Cluster/Worker *Toree in Apache Incubator
- 7. Notebooks • Notebooks designedfor (data) scientists, widely used for data cleaning and transformation, numerical simulation, statistical modeling, etc • Appear in browser as cells, may contain live code, visualizations, formatted text, widgets, etc • Jupyter notebooks have pluggable kernel architecture to enable different languages (jupyter.org) • EclairJS provides JavaScript kernel so data engineers and web developers can try-out code and work with data in notebooks
- 8.
- 9. In Closing • EclairJSfor web application development in Node.js and JavaScript • For Data Engineers with JavaScript in Notebooks • Project under active development in Github, eclairjs.org • Examples, documentation, getting-started, etc • EclairJS Node and EclairJS Nashorn • Open source, Apache v2 license • Looking for collaborators!
- 10. THANK YOU. eclairjs.org fallside atus.ibm.com