Azure Data Factory Data Wrangling with Power Query

Azure Data Factory Data Wrangling with Power Query

• 1.
  Azure Data Factory:Data Wrangling Power Query in ADF Updated Public Preview Q1 CY21
• 2.
  What is DataWrangling?  Code-free data exploration and data prep  Operationalize Power Query as an activity by translating M script in ADF data flow script  Execute Power Query as a pipeline activity using the ADF data flow serverless, scaled-out, ADF-managed Apache Spark engine  Essentially acts as a data-first entry point to building ADF data flows
• 3.
  ADF Data WranglingUse Cases  Data Engineer is building an ETL process in ADF uses PQ to explore data using data profiling  Business Analyst is a PQ desktop user and wishes to operationalize their M query in a data pipeline that sinks data in the Lake  Data Engineer needs to prep data for modeling and ETL by using a data-first approach. Creates a PQ wrangling activity and adds it to pipeline.  Trimming strings  Data type conversions  Rename columns  Remove columns  Value prop  “Data Wrangling in ADF”, not “Power Query lift-and-shift”
• 4.
  ADF Data WranglingRoadmap – PQ Activity  Continue to add more M functions to fold into Spark  Add more native connectors that work in both ADF & Power Query  Enable V-Net in Power Query Online data wrangling experience in ADF  Launch PQ activity in Synapse Pipelines  Enable interactive monitoring similar to Copy and Data Flow
• 5.
  Additional resources Documentation List of tutorialvideos Expression language reference Performance guide ADF twitter ADF tech community blog