Introduction to MapReduce & hadoop

Introduction to MapReduce & hadoop

• 1.
  Introduction to Hadoopand MapReduce Colin Su, Tagtoo
• 2.
  Advertisement System Architecture(now)
• 3.
  Advertisement System Architecture(future) • Grid • Ad Server • Data Highway • Steaming Computing
• 4.
  Grid • Core: • Datamining • Machine Learning • Collecting data from users, logs and calculate out the strategy • Sort our data in a proper form, them we could use it anytime Data -> Information
• 5.
  Ad Server • Ranking •According the “information” in Grid, decide which AD should be advertised • show proper ads to website visitors
• 6.
  Data Highway • Transferyour data to the proper place
• 7.
  Stream Computing • Core: •logging • feedback • anti-cheating • pricing • post-process everything thrown out from Ad Server, and feedback useful information to Grid • be the entrance of advertisement system
• 8.
  Hadoop • an open-sourcesoftware framework for data scientists • derives from Google’s MapReduce and Google File System (GFS) papers • written in Java • could be divided in to 2 components: • MapReduce • HDFS (Hadoop distributed file system) • a yellow elephant
• 9.
  Why Hadoop? • movingcomputation is much cheaper and easier than moving data • “Big Data”, the amount of data becomes too large, need a effective way to manage it • so does computation • high fault-tolerance • developed by Yahoo!
• 10.
  MapReduce • a programmingmodel for processing “large data sets” with a “parallel, distributed” algorithm on a cluster • different from map/reduce, the conception of functional programming, but actually they have the same idea, “divide and conquer” • proposed by Google
• 11.
  Functional “map/reduce” • map()/reduce()in Python • map(function(elem), list) -> list • reduce(function(elem1, elem2), list) -> single result • e.g. • map(lambda x: x*2, [1,2,3,4]) => [2,4,6,8] • reduce(lambda x,y: x+y, [1,2,3,4]) => 10
• 12.
  Parallel “MapReduce” 5Steps • prepare the map() input for mappers • mappers run the map() code -> generated intermediate pairs • dispatch intermediate pairs to reducers • reducers run the reduce() code, aggregate the results • prepare output from the result of reduce()
• 13.
  Example of “MapReduce”Word Count map() reduce()
• 14.
  Example of “MapReduce”Word Count • Original Input Apple Orange Mongo Orange Grapes Plum ...
• 15.
  Example of “MapReduce”Word Count • Prepare data for mappers Apple Orange Mongo Orange Grapes Plum ...
• 16.
  Example of “MapReduce”Word Count • map() to useful record (Apple, 1) Apple Orange Mongo (Orange, 1) (Mongo, 1) Intermediate key/value pair
• 17.
  Example of “MapReduce”Word Count • sort and shuffle (Apple, 1) (Mongo, 1) (Apple, 1) (Orange, 1) Reducer (Apple, 1) (Mongo, 1) (Apple, 1) (Orange, 1) (Orange, 1) Shuffle to Reducers (Orange, 1) (Orange, 1) (Apple, 1) (Mongo, 1) (Apple, 1) (Mongo, 1) unsorted Sorted (Orange, 1) Reducer (Mongo, 1) (Mongo, 1) Reducer
• 18.
  Example of “MapReduce”Word Count • Reduce() (Apple, 1) (Apple, 1) (Apple, 2) Reducer (Orange, 1) (Orange, 1) (Orange, 1) Reducer (Orange, 3)
• 19.
  Example of “MapReduce”Word Count • Generate Output (Apple, 2) (Orange, 3) (Grapes, 1) (Plum, 5) Apple 2 Orange 3 Grapes 1 Plum 5 WordCount.txt
• 20.
  Hadoop Infrastructure • Pig:Programming Language for MapReduce • Thrift: cross-language communication, just like Google’s ProtoBuffer • Zookeeper: cluster management Hadoop Hadoop Other Services Thrift MapReduce Pig Hadoop HDFS Hadoop Hadoop ZooKeeper