Data Engineering on Microsoft Azure (DP-203)

Course Overview

1 - Introduction to data engineering on Azure

• What is data engineering
• Important data engineering concepts
• Data engineering in Microsoft Azure

2 - Introduction to Azure Data Lake Storage Gen2

• Understand Azure Data Lake Storage Gen2
• Enable Azure Data Lake Storage Gen2 in Azure Storage
• Compare Azure Data Lake Store to Azure Blob storage
• Understand the stages for processing big data
• Use Azure Data Lake Storage Gen2 in data analytics workloads

3 - Introduction to Azure Synapse Analytics

• What is Azure Synapse Analytics
• How Azure Synapse Analytics works
• When to use Azure Synapse Analytics

4 - Use Azure Synapse serverless SQL pool to query files in a data lake

• Understand Azure Synapse serverless SQL pool capabilities and use cases
• Query files using a serverless SQL pool
• Create external database objects

5 - Use Azure Synapse serverless SQL pools to transform data in a data lake

• Transform data files with the CREATE EXTERNAL TABLE AS SELECT statement
• Encapsulate data transformations in a stored procedure
• Include a data transformation stored procedure in a pipeline

6 - Create a lake database in Azure Synapse Analytics

• Understand lake database concepts
• Explore database templates
• Create a lake database
• Use a lake database

7 - Secure data and manage users in Azure Synapse serverless SQL pools

• Choose an authentication method in Azure Synapse serverless SQL pools
• Manage users in Azure Synapse serverless SQL pools
• Manage user permissions in Azure Synapse serverless SQL pools

8 - Secure data and manage users in Azure Synapse serverless SQL pools

• Choose an authentication method in Azure Synapse serverless SQL pools
• Manage users in Azure Synapse serverless SQL pools
• Manage user permissions in Azure Synapse serverless SQL pools

9 - Analyze data with Apache Spark in Azure Synapse Analytics

• Get to know Apache Spark
• Use Spark in Azure Synapse Analytics
• Analyze data with Spark
• Visualize data with Spark

10 - Transform data with Spark in Azure Synapse Analytics

• Modify and save dataframes
• Partition data files
• Transform data with SQL

11 - Use Delta Lake in Azure Synapse Analytics

• Understand Delta Lake
• Create Delta Lake tables
• Create catalog tables
• Use Delta Lake with streaming data
• Use Delta Lake in a SQL pool

12 - Build a data pipeline in Azure Synapse Analytics

• Understand pipelines in Azure Synapse Analytics
• Create a pipeline in Azure Synapse Studio
• Define data flows
• Run a pipeline

13 - Use Spark Notebooks in an Azure Synapse Pipeline

• Understand Synapse Notebooks and Pipelines
• Use a Synapse notebook activity in a pipeline
• Use parameters in a notebook

14 - Introduction to Azure Synapse Analytics

• What is Azure Synapse Analytics
• How Azure Synapse Analytics works
• When to use Azure Synapse Analytics

15 - Use Azure Synapse serverless SQL pool to query files in a data lake

• Understand Azure Synapse serverless SQL pool capabilities and use cases
• Query files using a serverless SQL pool
• Create external database objects

16 - Analyze data with Apache Spark in Azure Synapse Analytics

• Get to know Apache Spark
• Use Spark in Azure Synapse Analytics
• Analyze data with Spark
• Visualize data with Spark

17 - Use Delta Lake in Azure Synapse Analytics

• Understand Delta Lake
• Create Delta Lake tables
• Create catalog tables
• Use Delta Lake with streaming data
• Use Delta Lake in a SQL pool

18 - Analyze data in a relational data warehouse

• Design a data warehouse schema
• Create data warehouse tables
• Load data warehouse tables
• Query a data warehouse

19 - Build a data pipeline in Azure Synapse Analytics

• Understand pipelines in Azure Synapse Analytics
• Create a pipeline in Azure Synapse Studio
• Define data flows
• Run a pipeline

20 - Analyze data in a relational data warehouse

• Design a data warehouse schema
• Create data warehouse tables
• Load data warehouse tables
• Query a data warehouse

21 - Load data into a relational data warehouse

• Load staging tables
• Load dimension tables
• Load time dimension tables
• Load slowly changing dimensions
• Load fact tables
• Perform post load optimization

22 - Manage and monitor data warehouse activities in Azure Synapse Analytics

• Scale compute resources in Azure Synapse Analytics
• Pause compute in Azure Synapse Analytics
• Manage workloads in Azure Synapse Analytics
• Use Azure Advisor to review recommendations
• Use dynamic management views to identify and troubleshoot query performance

23 - Secure a data warehouse in Azure Synapse Analytics

• Understand network security options for Azure Synapse Analytics
• Configure Conditional Access
• Configure authentication
• Manage authorization through column and row level security
• Manage sensitive data with Dynamic Data Masking
• Implement encryption in Azure Synapse Analytics

24 - Plan hybrid transactional and analytical processing using Azure Synapse Analytics

• Understand hybrid transactional and analytical processing patterns
• Describe Azure Synapse Link

25 - Implement Azure Synapse Link with Azure Cosmos DB

• Enable Cosmos DB account to use Azure Synapse Link
• Create an analytical store enabled container
• Create a linked service for Cosmos DB
• Query Cosmos DB data with Spark
• Query Cosmos DB with Synapse SQL

26 - Implement Azure Synapse Link for SQL

• What is Azure Synapse Link for SQL?
• Configure Azure Synapse Link for Azure SQL Database
• Configure Azure Synapse Link for SQL Server 2022

27 - Get started with Azure Stream Analytics

• Understand data streams
• Understand event processing
• Understand window functions

28 - Ingest streaming data using Azure Stream Analytics and Azure Synapse Analytics

• Stream ingestion scenarios
• Configure inputs and outputs
• Define a query to select, filter, and aggregate data
• Run a job to ingest data

29 - Visualize real-time data with Azure Stream Analytics and Power BI

• Use a Power BI output in Azure Stream Analytics
• Create a query for real-time visualization
• Create real-time data visualizations in Power BI

30 - Explore Azure Databricks

• Get started with Azure Databricks
• Identify Azure Databricks workloads
• Understand key concepts
• Data governance using Unity Catalog and Microsoft Purview

31 - Perform data analysis with Azure Databricks

• Ingest data with Azure Databricks
• Data exploration tools in Azure Databricks
• Data analysis using DataFrame APIs

32 - Use Apache Spark in Azure Databricks

• Get to know Spark
• Create a Spark cluster
• Use Spark in notebooks
• Use Spark to work with data files
• Visualize data

33 - Manage data with Delta Lake

• Get started with Delta Lake
• Manage ACID transactions
• Implement schema enforcement
• Data versioning and time travel in Delta Lake
• Data integrity with Delta Lake

34 - Build data pipelines with Delta Live Tables

• Explore Delta Live Tables
• Data ingestion and integration
• Real-time processing

35 - Deploy workloads with Azure Databricks Workflows

• What are Azure Databricks Workflows?
• Understand key components of Azure Databricks Workflows
• Explore the benefits of Azure Databricks Workflows
• Deploy workloads using Azure Databricks Workflows

36 - Introduction to Microsoft Purview

• What is Microsoft Purview?
• How Microsoft Purview works
• When to use Microsoft Purview

37 - Discover trusted data using Microsoft Purview

• Search for assets
• Browse assets
• Use assets with Power BI
• Integrate with Azure Synapse Analytics

38 - Catalog data artifacts by using Microsoft Purview

• Register and scan data
• Classify and label data
• Search the data catalog

39 - Manage Power BI assets by using Microsoft Purview

• Register and scan a Power BI tenant
• Search and browse Power BI assets
• View Power BI metadata and lineage

40 - Integrate Microsoft Purview and Azure Synapse Analytics

• Catalog Azure Synapse Analytics data assets in Microsoft Purview
• Connect Microsoft Purview to an Azure Synapse Analytics workspace
• Search a Purview catalog in Synapse Studio
• Track data lineage in pipelines

Prerequisites

Successful students start this course with knowledge of cloud computing and core data concepts and professional experience with data solutions.

• KT-AZ-900t00a- Azure Fundamentals
• KT-DP-900 - Microsoft Azure Data Fundamentals