Download to read offline
Uploaded byBryan Reinero
Mongo db &_spark
The document describes Spark and how it relates to other big data technologies like HDFS, YARN, and Mesos. It shows how Spark can be used both with these technologies or on its own. Key Spark concepts discussed include Resilient Distributed Datasets (RDDs), transformations, actions, and lineage. It also provides examples of using Spark with MongoDB to read data from a MongoDB collection into an RDD.
More Related Content
Similar to Mongo db &_spark
Mongo db &_spark
- 1.
- 2.
- 3.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28. execut or Worker Node execut or Worker Node Driver ResilientDistributed Datasets
- 29. Resilient Distributed Datasets f(x’’)= yParellelize = xt(x) = x’t(x’) = x’’
- 30. t(x) = x’t(x’)= x’’f(x’’) = x’’’Parellelize = x Parallelization
- 31. t(x) = x’t(x’)= x’’f(x’’) = x’’’Parellelize = x Transformations
- 32. Tranformations filter( func ) union(func ) intersection( set ) distinct( n ) map( function )
- 33. t(x) = x’t(x’)= x’’f(x’’) = x’’’Parellelize = x Action
- 34.
- 35. f(x) = x’f(x’)= x’’t(x’’) = x’’’Parellelize = x Lineage
- 36.
- 37.
- 38.
- 39.
- 40.
- 41. { "_id" : ObjectId("4f16fc97d1e2d32371003e27"), "body": "the scrimmage is still up in the air. "subFolder" : "notes_inbox", "mailbox" : "bass-e", "filename" : "450.", "headers" : { "X-cc" : "", "From" : "michael.simmons@enron.com", "Subject" : "Re: Plays and other information", "X-Folder" : "Eric_Bass_Dec2000Notes FoldersNotes inbox", "Content-Transfer-Encoding" : "7bit", "X-bcc" : "", "To" : "eric.bass@enron.com", "X-Origin" : "Bass-E", "X-FileName" : "ebass.nsf", "X-From" : "Michael Simmons", "Date" : "Tue, 14 Nov 2000 08:22:00 -0800 (PST)", "X-To" : "Eric Bass", "Message-ID" : "<6884142.1075854677416.JavaMail.evans@thyme>", "Content-Type" : "text/plain; charset=us-ascii", "Mime-Version" : "1.0" } }
- 42. { "_id" : "gretchen.hardeway@enron.com|shirley.crenshaw@enron.com", "value": 2 } { "_id" : "kmccomb@austin-mccomb.com|brian@enron.com", "value" : 2 } { "_id" : "sally.beck@enron.com|sandy.stone@enron.com", "value" : 2 }
- 43.
- 44. Spark Configuration Configuration conf= new Configuration(); conf.set( "mongo.job.input.format", "com.mongodb.hadoop.MongoInputFormat” ); conf.set( "mongo.input.uri", "mongodb://localhost:27017/db.collection” );
- 45. Spark Context JavaPairRDD<Object, BSONObject>documents = context.newAPIHadoopRDD( conf, MongoInputFormat.class, Object.class, BSONObject.class );
- 46. Spark Context JavaPairRDD<Object, BSONObject>documents = context.newAPIHadoopRDD( conf, MongoInputFormat.class, Object.class, BSONObject.class );
- 47. Spark Context JavaPairRDD<Object, BSONObject>documents = context.newAPIHadoopRDD( conf, MongoInputFormat.class, Object.class, BSONObject.class );
- 48. Spark Context JavaPairRDD<Object, BSONObject>documents = context.newAPIHadoopRDD( conf, MongoInputFormat.class, Object.class, BSONObject.class );
- 49. Spark Context JavaPairRDD<Object, BSONObject>documents = context.newAPIHadoopRDD( conf, MongoInputFormat.class, Object.class, BSONObject.class );
- 50.
- 51. Deployment Artifacts Hadoop Connector Jar FatJar Java Driver Jar
- 52. Spark Submit /usr/local/spark-1.5.1/bin/spark-submit --classcom.mongodb.spark.examples.DataframeExample --master local Examples-1.0-SNAPSHOT.jar
- 53.
- 54. JavaRDD<Message> messages =documents.map ( new Function<Tuple2<Object, BSONObject>, Message>() { public Message call(Tuple2<Object, BSONObject> tuple) { BSONObject header = (BSONObject)tuple._2.get("headers"); Message m = new Message(); m.setTo( (String) header.get("To") ); m.setX_From( (String) header.get("From") ); m.setMessage_ID( (String) header.get( "Message-ID" ) ); m.setBody( (String) tuple._2.get( "body" ) ); return m; } } );
- 55.
- 57.
- 60.
- 62. THANK S!{ Name: ‘BryanReinero’, Title: ‘Developer Advocate’, Twitter: ‘@blimpyacht’, Email: ‘bryan@mongdb.com’ }
Editor's Notes
- #29 A fault-tolerant collection of elements operated on in parallel best suited for batch applications
- #45 MongoInputFormat allows us to read from a live MongoDB instance. We could also use BSONFileInputFormat to read BSON snapshots.
- #46 JavaPamongodbConfig
- #47 JavaPamongodbConfig
- #48 JavaPamongodbConfig
- #49 JavaPamongodbConfig
- #50 JavaPamongodbConfig