Downloaded 262 times
Uploaded byDmitry Anoshin
Building Modern Data Platform with Microsoft Azure
This document provides an overview of building a modern cloud analytics solution using Microsoft Azure. It discusses the role of analytics, a history of cloud computing, and a data warehouse modernization project. Key challenges covered include lack of notifications, logging, self-service BI, and integrating streaming data. The document proposes solutions to these challenges using Azure services like Data Factory, Kafka, Databricks, and SQL Data Warehouse. It also discusses alternative implementations using tools like Matillion ETL and Snowflake.
More Related Content
Similar to Building Modern Data Platform with Microsoft Azure
More from Dmitry Anoshin
![Agentic Systems and Compliance - A brief intro [1.2]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticsystemsandcompliace-1-251018025303-958a42ec-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
![Matrix and determinant URT [Autosaved].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/matrixanddeterminanturtautosaved-251018190340-9e6a6deb-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Building Modern Data Platform with Microsoft Azure
- 1. Building Modern Cloud AnalyticsSolution Dmitry Anoshin
- 2. Outline • About Me •Role of Analytics • History of Cloud • Analytics powered by Microsoft Azure • DW modernization Project • Use cases and Challenges • Alternative Solution with Azure
- 3.
- 4. About Myself • Workwith Business Intelligence since 2007
- 5.
- 6. Technical Skills Matrix 2015 2010 2007 Data Warehouse ETL/ELT Business Intelligence BigData Cloud Analytics (AWS, Azure, GCP) Machine Learning 2019
- 7. Other Activities Jumpstart Sno wflake:A Step- by-Step Guide to Modern Cloud Analytics. • Victoria Power BI andVictoria SQL Server meetup • Victoria andVancouverTableau User Group • Conferences (EDW 2018, 2019, Data Architecture Summit) • Amazon internal conferences
- 8.
- 9. BusinessValue Stakeholders Employees Customers Value ”Thegoal of any organization is to generateValue” The Future of Competition. https://www.amazon.com/Future-Competition-Co-Creating-Unique-Customers/dp/1578519535
- 10. BIValue Chain Stakeholders EmployeesCustomers Value Decisions Data Value creation based on effective decisions Effective decisions based on accurate information
- 11. For Data tobe a differentiator, customers need to be able to… • Capture and store new non-relational data at PB-EB scale in real time • Discover value in a new type of analytics that go beyond batch reporting to incorporate real-time, predictive, voice, and image recognition • Democratize access to data in a secure and governed way New types of analytics Dashboards Predictive Image Recognition VoiceReal-time New types of data
- 12.
- 13. Cloud Early History 1970 TimeSharing Concept by GE 1977 Cloud symbol used in ARPANET 1990 VPN by telecom 1993 Cloud refer to Distributed Computing 1994 Cloud metaphor for virtualized services
- 14. Cloud Recent History 2002 AWS 2006 AWSElastic Compute Cloud 2006 Google Docs 2008 Google App Engine 2008 Microsoft Announced Azure 2010 Microsoft Azure
- 15. Why moving tothe Cloud? • Elasticity • Pay for what you need • Fail fast • Fast time to market • Secure • Reliable • Business SLA
- 16. Downsides of on-premisesolution Scale Constrained Up-front cost Maintenance Resources Tuning and Deployment
- 17. Cloud Restrictions ->Hybrid Clouds Sensitive Data Data Moving Cost Public/Private Cloud
- 18.
- 19. Cloud Service Models– friendly version
- 20.
- 21. Microsoft Azure forAnalytics
- 22. Data Analytics withAzure • Data Factory • Integration Service • Kafka • Event Hub • Data Lake Gen 1 • Data Lake Gen 2 • Blob Storage • HD Insight • Data Lake Analytics • Streaming Analytics • PolyBase • CosmosDB • SQL DW • Analysis Service • SQL Database • SQL Server in VM • Cosmos DB Data Integration and Transformation Data Warehouse and Data bases Big Data • Analysis Service • ML Analytics • Business Intelligence Analytics
- 23.
- 24. BI/DW (before) Storage LayerSourceLayer Ad-hoc SQL SFTP Data Warehouse ETL (PL/SQL)Files Inventory Sales Access Layer
- 25. Cloud Migration Strategy Lift& Shift • Typical Approach • Move all-at-once • Target platform then evolve • Approach gets you to the cloud quickly • Relatively small barrier to learning new technology since it tends to be a close fit Split & Flip • Split application into logical functional data layers • Match the data functionality with the right technology • Leverage the wide selection of tools onAWS to best fit the need • Move data in phases — prototype, learn and perfect
- 26. Migration Approach Useful tools: •Total Cost Ownership (TCO) Calculator • Azure Database Migration Service • Azure Migration Assistant
- 28.
- 29. What is AzureDW? • Decouple Storage and Compute • MPP • Distribution Styles: Hash/Robin/Replicat e
- 30.
- 31. SQL Database vsSQL Data Warehouse
- 32. What is AzureData Factory? Azure Data Factory (ADF) is Microsoft’s fully managed ELT service in the cloud that’s delivered as a Platform as a Service (PaaS)
- 33. Lack of Notification Problem:Users are missing emails or they jump to spam. Solution: Leverage Messenger with Webhooks. (Slack, Chime or so on).
- 34. Lack of Logging Problem:We didn’t have any detail logs about our ETL performance and we didn’t have any insights. Solution: Collecting logs and events. In addition, we are able to collect logs on any level of jobs and transformation.
- 35. Self-Service BI Problem: BusinessUsers wants Interactive and Self-Service tool. Fast time to Market and less dependency on IT. Solution: Implement modern Visual Analytics Platform
- 36. Marketing Automation Problem: Marketingteam wants “Move Fast and Break Things”. Solution: Using ADF the gave Marketing template jobs and they doing their jobs themselves. Affiliates Insights
- 37. Integration with BI Problem:Having best BI tool doesn’t guaranty good SLA. Solution: Build Integration between Matillion ETL and Tableau based on Trigger. Add data quality checks.
- 38. Evolving to Cloud DataAnalytics Platform
- 39. Streaming Data Problem: Organizationis using NoSQL database and mobile application. It is critical to deliver near real time analytics Solution: Using Apache Kaffka, we are able to stream data into the Data lake and query this data in near real time Data Lake Dashboard Kafka CosmoDB Mobile App
- 40. Clickstream Analytics Problem: Businesswants to analyze Bots traffics and discover broken URLs. Access logs are ~50GB per day, 5600 log files per day. Solution: Leveraging Databricks in order to produce Parquet file and store in Azure Data Lake Gen2. User are able query it with T-SQL and BI Tools. Databricks ParquetBlob Storage Access Logs Load Balancer Data Lake Data Factory SQL DW Query with SQL or Databricks
- 41. DevOps onboarding Problem: Solutionisn’t reliable and could easy break. As a result end users will experience bad experience and it will affect business decisions. Solution: Onboarding Continuous Integration methodology for Cloud Data Platform • Agile and Kanban board • Code branching (Git) • Gated check-ins • Automated Tests • Build • Release
- 42. Evolving to CloudData Analytics Platform
- 43.
- 44.
- 45.
Editor's Notes
- #14 The cloud symbol was used to represent networks of computing equipment in the original ARPANET by as early as 1977 The term cloud was used to refer to platforms for distributed computing as early as 1993, when Apple spin-off General Magic and AT&T used it in describing their (paired) Telescript and PersonaLink technologies.
- #15 The cloud symbol was used to represent networks of computing equipment in the original ARPANET by as early as 1977 The term cloud was used to refer to platforms for distributed computing as early as 1993, when Apple spin-off General Magic and AT&T used it in describing their (paired) Telescript and PersonaLink technologies.