Download as PDF, PPTX
Uploaded byQAware GmbH
Clickstream Analysis with Apache Spark
The document discusses clickstream analysis using Apache Spark, highlighting the challenges of handling large data volumes in real-time web and ad tracking. It introduces a proposed architecture for efficient data processing, incorporating aspects like event ingestion, user journey mapping, and real-time analytics. The document also emphasizes lessons learned regarding optimal architecture design, user counting techniques, and system monitoring for effective data management.
More Related Content
Similar to Clickstream Analysis with Apache Spark
Clickstream Analysis with Apache Spark
- 1. CLICKSTREAM ANALYSIS WITH APACHESPARK Andreas Zitzelsberger
- 2.
- 3. ONE POT TORULE THEM ALL Web Tracking Ad Tracking ERP CRM ▪ Products ▪ Inventory ▪ Margins ▪ Customer ▪ Orders ▪ Creditworthiness ▪ Ad Impressions ▪ Ad Costs ▪ Clicks & Views ▪ Conversions
- 4. ONE POT TORULE THEM ALL Retention Reach Monetarization steer … ▪ Campaigns ▪ Offers ▪ Contents
- 5. REACT ON WEBSITE TRAFFIC IN REAL TIME Image: https://www.flickr.com/photos/nick-m/3663923048
- 6. SAMPLE RESULTS Geolocated andgender- specific conversions. Frequency of visits Performance of an ad campaign
- 7. THE CONCEPTS Image: RandyPaulino
- 8. THE FIRST SKETCH (=real-time) SQL
- 10. CALCULATING USER JOURNEYS CV VT VT VT C X C V V V V V V V V C V V C V V V VT VT V V V VT C V X Event stream: User journeys: Web / Ad tracking KPIs: ▪ Unique users ▪ Conversions ▪ Ad costs / conversion value ▪ … V X VT C Click View View Time Conversion
- 11.
- 12. „LARRY & FRIENDS“ARCHITECTURE Runs not well for more than 1 TB data in terms of ingestion speed, query time and optimization efforts
- 13. Image: adweek.com Nope. Sorry, noBig Data.
- 14. „HADOOP & FRIENDS“ARCHITECTURE Aggregation takes too long Cumbersome programming model (can be solved with pig, cascading et al.) Not interactive enough
- 15.
- 16. Κ-ARCHITECTURE Cumbersome programming model Over-engineered: Weonly need 15min real-time ;-) Stateful aggregations (unique x, conversions) require a separate DB with high throughput and fast aggregations & lookups.
- 17.
- 18.
- 19. The Final Architecture* *)Maybe called µ-architecture one day ;-)
- 20. FUNCTIONAL ARCHITECTURE Strange Events IngestionRawEvent Stream Collection Events Processing Analytics Warehouse Fact Entries Atomic Event Frames Data Lake Master Data Integration ▪ Buffers load peeks ▪ Ensures message delivery (fire & forget for client) ▪ Create user journeys and unique user sets ▪ Enrich dimensions ▪ Aggregate events to KPIs ▪ Ability to replay for schema evolution ▪ The representation of truth ▪ Multidimensional data model ▪ Interactive queries for actions in realtime and data exploration ▪ Eternal memory for all events (even strange ones) ▪ One schema per event type. Time partitioned. ▪ Fault tolerant message handling ▪ Event handling: Apply schema, time-partitioning, De-dup, sanity checks, pre-aggregation, filtering, fraud detection ▪ Tolerates delayed events ▪ High throughput, moderate latency (~ 1min)
- 21. SERIAL CONNECTION OFSTREAMING AND BATCHING IngestionRaw Event Stream Collection Event Data Lake Processing Analytics Warehouse Fact Entries SQL Interface Atomic Event Frames ▪ Cool programming model ▪ Uniform dev&ops ▪ Simple solution ▪ High compression ratio due to column-oriented storage ▪ High scan speed ▪ Cool programming model ▪ Uniform dev&ops ▪ High performance ▪ Interface to R out-of-the-box ▪ Useful libs: MLlib, GraphX, NLP, … ▪ Good connectivity (JDBC, ODBC, …) ▪ Interactive queries ▪ Uniform ops ▪ Can easily be replaced due to Hive Metastore ▪ Obvious choice for cloud-scale messaging ▪ Way the best throughput and scalability of all evaluated alternatives
- 22. public Map<Long, UserJourney> sessionize(JavaRDD<AtomicEvent>events) { return events // Convert to a pair RDD with the userId as key .mapToPair(e -> new Tuple2<>(e.getUserId(), e)) // Build user journeys .<UserJourneyAcc>combineByKey( UserJourneyAcc::create, UserJourneyAcc::add, UserJourneyAcc::combine) // Convert to a Java map .collectAsMap(); }
- 23.
- 24. APACHE FLINK ■ Also has a nice, Spark-like API ■Promises similar or better performance than spark ■ Looks like the best solution for a κ- Architecture ■ But it’s also the newest kid on the block
- 25. EVENT VERSUS PROCESSINGTIME ■ There’s a difference between even time (te) and processing time (tp). ■ Events arrive out-of order even during normal operation. ■ Events may arrive arbitrary late. Apply a grace period before processing events. Allow arbitrary update windows of metrics.
- 26. EXAMPLE Minute Hour Day Week Month Quarter Year I U U U U U U I U U U U U U U Resolution in Time Time dtp tp tp: Processing Time ti: Ingestion time te: Event Time dtp: Aggregation time frame dtw: Grace period : Insert fact : Update fact dtw te ti
- 27.
- 28. BEST-OF-BREED INSTEAD OFCOMMODITY SOLUTIONS ETL Analytics Realtime Analytics Slice & Dice Data Exploration Polyglot Processing http://datadventures.ghost.io/2014/07/06/polyglot-processing
- 29. POLYGLOT ANALYTICS Data Lake Analytics Warehouse SQL lane R lane Timeseries lane Reporting Data Exploration Data Science
- 30. NO RETENTION PARANOIA DataLake Analytics Warehouse ▪ Eternal memory ▪ Close to raw events ▪ Allows replays and refills into warehouse Aggressive forgetting with clearly defined retention policy per aggregation level like: ▪ 15min:30d ▪ 1h:4m ▪ … Events Strange Events
- 31. BEWARE OF THEHIPSTERS Image: h&m
- 32. ENSURE YOUR SOFTWARERUNS LOCALLY The entire architecture must be able to run locally. Keep the round trips low for development and testing. Throughput and reaction times need to be monitored continuously. Tune your software and the underlying frameworks as needed.
- 33. TUNE CONTINUOUSLY IngestionRaw Event Stream CollectionEvent Data Lake Processing Analytics Warehouse Fact Entries SQL Interface Atomic Event Frames Load generator Throughput & latency probes System, container and process monitoring
- 34. IN NUMBERS Overall deveffort until the first release: 250 person days Dimensions: 10 KPIs: 26 Integrated 3rd party systems: 7 Inbound data volume per day: 80GB New data in DWH per day: 2GB Total price of cheapest cluster which is able to handle production load:
- 36.
- 37.
- 38. CALCULATING UNIQUE USERS ■We need an exact unique user count. ■ If you can, you should use an approximation such as HyperLogLog. U1 U2 U3 U1 U4 Time Users 3 UU 2 UU 4 UU Flajolet, P.; Fusy, E.; Gandouet, O.; Meunier, F. (2007). "HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm". AOFA ’07: Proceedings of the 2007 International Conference on the Analysis of Algorithms.
- 39.
- 40. CHOOSING WHERE TOAGGREGATE Ingestion Event Data Lake Processing Analytics Warehouse Fact Entries Analytics Atomic Event Frames 1 2 3 - Enrichment - Preprocessing - Validation The hard lifting. - Processing steps that can be done at query time. - Interactive queries.