Master Data Engineering using Azure Data Analytics

Master Data Engineering using Azure Data Analytics, available at $64.99, has an average rating of 4.41, with 195 lectures, based on 212 reviews, and has 2276 subscribers.

You will learn about Data Engineering leveraging Services under Azure Data Analytics such as Azure Storage, Data Factory, Azure SQL, Synapse, Databricks, etc. Setup Development Environment using Visual Studio Code on Windows Building Data Lake using Azure Storage (Blob and ADLS) Build Data Warehouse using Azure Synapse Implement ETL Logic using ADF Data Flow with Azure Storage as Source and Target In Depth Coverage of Orchestration using ADF Pipeline Overview of Azure SQL and Azure Synapse Serverless and Dedicated Pool Features Implement ETL Logic using ADF Data Flow with Azure SQL as Source and Azure Synapse as Target Using Data Copy to copy data between different sources and targets Performance Tuning Scenarios of ADF Data Flow and Pipelines Build Big Data Solutions using Azure Databricks Overview of Spark SQL and Pyspark Data Frame APIs Build ELT Pipelines using Databricks Jobs and Workflows Orchestrate Databricks Notebooks using ADF Pipelines This course is ideal for individuals who are Beginner or Intermediate Data Engineers who want to learn Key Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc or Intermediate Application Engineers who want to explore Data Engineering using Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc or Data and Analytics Engineers who want to learn Data Engineering Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc or Testers who want to learn key skills to test Data Engineering applications built using Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc It is particularly useful for Beginner or Intermediate Data Engineers who want to learn Key Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc or Intermediate Application Engineers who want to explore Data Engineering using Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc or Data and Analytics Engineers who want to learn Data Engineering Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc or Testers who want to learn key skills to test Data Engineering applications built using Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc.

Enroll now: Master Data Engineering using Azure Data Analytics

Summary

Title: Master Data Engineering using Azure Data Analytics

Price: $64.99

Average Rating: 4.41

Number of Lectures: 195

Number of Published Lectures: 195

Number of Curriculum Items: 195

Number of Published Curriculum Objects: 195

Original Price: $199.99

Quality Status: approved

Status: Live

What You Will Learn

Data Engineering leveraging Services under Azure Data Analytics such as Azure Storage, Data Factory, Azure SQL, Synapse, Databricks, etc.

Setup Development Environment using Visual Studio Code on Windows

Building Data Lake using Azure Storage (Blob and ADLS)

Build Data Warehouse using Azure Synapse

Implement ETL Logic using ADF Data Flow with Azure Storage as Source and Target

In Depth Coverage of Orchestration using ADF Pipeline

Overview of Azure SQL and Azure Synapse Serverless and Dedicated Pool Features

Implement ETL Logic using ADF Data Flow with Azure SQL as Source and Azure Synapse as Target

Using Data Copy to copy data between different sources and targets

Performance Tuning Scenarios of ADF Data Flow and Pipelines

Build Big Data Solutions using Azure Databricks

Overview of Spark SQL and Pyspark Data Frame APIs

Build ELT Pipelines using Databricks Jobs and Workflows

Orchestrate Databricks Notebooks using ADF Pipelines

Who Should Attend

Beginner or Intermediate Data Engineers who want to learn Key Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc

Intermediate Application Engineers who want to explore Data Engineering using Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc

Data and Analytics Engineers who want to learn Data Engineering Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc

Testers who want to learn key skills to test Data Engineering applications built using Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc

Target Audiences

Beginner or Intermediate Data Engineers who want to learn Key Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc

Intermediate Application Engineers who want to explore Data Engineering using Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc

Data and Analytics Engineers who want to learn Data Engineering Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc

Testers who want to learn key skills to test Data Engineering applications built using Azure Analytics Services for Data Engineering such as Azure Storage, ADF, Synapse, Databricks, etc

Data Engineering is all about building Data Pipelines to get data from multiple sources into Data Lakes or Data Warehouses and then from Data Lakes or Data Warehouses to downstream systems. As part of this course, I will walk you through how to build Data Engineering Pipelines using Azure Data Analytics Stack. It includes services such as Azure Storage (both Blob and ADLS), ADF Data Flow, ADF Pipeline, Azure SQL, Azure Synapse, Azure Databricks, and many more.

As part of this course, first, you will go ahead and set up the environment to learn using VS Code on Windows and Mac.

Once the environment is ready, you need to sign up for Azure Portal. We will provide all the instructions to sign up for Azure Portal Account including reviewing billing as well as getting USD 200 Credit valid for up to a month.

We typically use Azure Storage as Data Lake. As part of this course, you will learn how to use Azure Storage as Data Lake along with how to manage the files in Azure Storage using tools such as Azure Storage Explorer.

ADF (Azure Data Factory)is used for both ETL as well as Orchestration. First, you will understand how to perform ETL using ADF Data Flow. The source and target will be Files in Azure Storage Account. As part of this process, you will also learn how to set up Linked Services and Data Sets in ADF (Azure Data Factory).

Once ADF Data Flow is ready, you will go ahead and build Pipeline for Orchestration using ADF Pipeline. You will also learn how to parameterize and also how to take care of baseline load.

You will also understand key performance tuning techniques using ADF Pipeline such as controlling the number of partitions, custom integration runtimes (IR), etc.

Azure provides RDBMS as different services for Postgres, SQL Server, etc. You will learn how to set up Azure SQL Once the Azure SQL is set up, you will also understand how to create required tables and run queries against them.

ADF provides ADF Data Copy to copy data from different sources and different targets. Once the Database tables are ready you will use ADF Data Copy to copy data into the tables.

Azure provides Synapse Analytics for Data Warehouse. You will get an overview of both serverless as well as dedicated pools. You will end up setting up a Dedicated Pool for ETL using ADF.

Once Azure SQL and Azure Synapseare ready, you will build ETL Pipeline using ADF Data Flow and Orchestrate using ADF Pipeline.

Azure Databricks is the service for Big Data Processing using Spark Engine. You will learn how to set up Azure Databricks, integrate with ADLS, and also managing secrets.

You will also get an overview of Spark SQL and Pyspark Data Frame APIs using Azure Databricks.

You will also build ELT Pipeline using Databricks Jobs and Workflows where tasks are defined based on Pyspark as well as Spark SQL.

You will also understand how to build ADF Pipelines to orchestrate Databricks Notebooks.

Course Curriculum

Chapter 1: Udemy Introduction for Data Engineering using Azure

Lecture 1: Introduction to Data Engineering using Azure Data Analytics

Lecture 2: Overview of Additional Costs associated with Azure

Lecture 3: Overview of Udemy Interface for new students

Chapter 2: Setup Environment for Data Engineering using Azure

Lecture 1: Setup VS Code on Windows

Lecture 2: Setup Python 3.9 on Windows

Lecture 3: Configure Environment Variable PATH for Python on Windows

Lecture 4: Integrate VSCode with Python on Windows

Chapter 3: Getting Started with Azure for Data Engineering

Lecture 1: Sign up for Azure Portal

Lecture 2: Sign up for Azure Subscription

Lecture 3: Overview of Azure CLI and Azure Cloud Shell

Lecture 4: Setup Azure CLI on Windows or Mac or Linux

Lecture 5: Configure Azure CLI against Azure Portal Account

Lecture 6: Overview of Cost Management and Billing in Azure Portal

Lecture 7: Review Resources used by Azure Cloud Shell

Chapter 4: Getting Started with Azure Resource Groups

Lecture 1: Create Azure Resource Group using Azure Portal

Lecture 2: Add Storage Account as Resource to Azure Resource Group

Lecture 3: Overview of Azure Resource Groups and Resources

Chapter 5: Setup Data Sets for Data Engineering

Lecture 1: Download Data Sets for Data Engineering from Git Repository

Lecture 2: Create Container with in Azure Storage Account

Lecture 3: Review Upload Feature of Azure Storage Account using Azure Portal

Lecture 4: Setup Azure Storage Explorer on Windows or Mac

Lecture 5: Upload Local Folder into Azure Storage Container using Storage Explorer

Lecture 6: Validate Data Sets using Azure Portal

Lecture 7: Create ADLS Storage Account in Azure

Lecture 8: Upgrade Azure Blob Storage to ADLS Gen 2

Chapter 6: Getting Started with Azure Data Factory

Lecture 1: Introduction to Getting Started with Azure Data Factory

Lecture 2: Setup Azure Data Factory and Launch ADF Studio

Lecture 3: Overview of Azure Data Factory Studio

Lecture 4: Create ADF Linked Service to Storage Account

Lecture 5: Create ADF Dataset using ADF Studio

Lecture 6: Review ADF Dataset CSV Properties

Lecture 7: Create Azure Dataset for Sink using Parquet

Lecture 8: Understand the Schema of Data Set

Lecture 9: Create Data Flow Source using Azure Dataset

Lecture 10: Define Cache Sink to ADF Data Flow

Lecture 11: Create ADF Pipeline for File Format Converter

Lecture 12: Run and Review ADF Data Pipelines

Lecture 13: Update ADF Data Flow with ADLS Dataset as Sink

Lecture 14: Conclusion to Getting Started with Azure Data Factory

Lecture 15: Exercise – Simple ADF Data Flow and Pipeline for Order Items

Chapter 7: ADF Data Flow for ETL Logic to Compute Daily Product Revenue

Lecture 1: Introduction to ADF Data Flow for ETL Logic to Compute Daily Product Revenue

Lecture 2: Create Data Flow to Compute Daily Product Revenue

Lecture 3: Filter Transformation in ADF Data Flow

Lecture 4: Create ADF Pipeline to Validate Data Flow

Lecture 5: Create ADF Integration Runtime to run ADF Pipelines

Lecture 6: Validate Custom ADF Integration Runtime using ADF Pipeline

Lecture 7: ADF Data Flow Filter Transformation using in

Lecture 8: ADF Data Flow Join Trasformation between 2 Data Sets

Lecture 9: Validate ADF Data Flow Join Transformation using ADF Pipeline

Lecture 10: ADF Data Flow Aggregate Transformation to Compute Daily Product Revenue

Lecture 11: ADF Data Flow Sink to Save Results to Azure Storage using Parquet

Lecture 12: Run and Review ADF Pipeline with ETL Data Flow

Lecture 13: Access JSON Code of ADF Data Flow and Pipeline

Chapter 8: Run ADF Pipelines Dynamically using Parameters

Lecture 1: Introduction to Running ADF Pipelines Dynamically using Parameters

Lecture 2: Create ADF Data Set using Parameter for Dynamic Path

Lecture 3: Define Parameter and Use in Filter Transformation of ADF Data Flow

Lecture 4: Create ADF Pipeline with Parameter

Lecture 5: Run ADF Pipeline with Parameters

Chapter 9: Run Baseline ETL Loads using ADF Pipeline

Lecture 1: Overview of Common ADF Pipeline Activities

Lecture 2: Overview of ADF Pipeline ForEach

Lecture 3: Create ADF Pipeline for Baseline load using ForEach and Execute Pipeline

Lecture 4: Run ADF Pipeline for Baseline Load

Chapter 10: Performance Tuning of ADF Data Flows and Pipelines

Lecture 1: Introduction to Prerformance Tuning of ADF Data Flows and Pipelines

Lecture 2: Create Integration Runtime with right Compute Size

Lecture 3: Troubleshoot Performance Bottleneck of Baseline ADF Pipeline

Lecture 4: Reduce Cluster Startup Time using Custom Integration Runtime

Lecture 5: Using Paralllel in ADF Pipeline ForEach Activity

Lecture 6: Troubleshoot Shuffling and Too Many Small Files Issue

Lecture 7: Reduce Shuffle Partitions in ADF Data Flow Aggregate Transformation

Lecture 8: Conclusion of Performance Tuning of ADF Pipelines and Data Flows

Chapter 11: Getting Started with Azure SQL Database

Lecture 1: Setup Azure SQL Database Server

Lecture 2: Setup Database in Azure SQL Database Server

Lecture 3: Overview of SQL Server Databases in Azure

Lecture 4: Setup Azure Data Studio on Windows or Mac or Linux

Lecture 5: Connect to Azure SQL Database using Azure Data Studio

Chapter 12: ADF Data Copy to Copy Data From Files to SQL Server Tables

Lecture 1: Create table in Azure SQL Database

Lecture 2: Create Linked Service and Dataset for Azure SQL Database Table

Lecture 3: Copy ADF Dataset into a folder

Lecture 4: Create ADF Pipeline with Data Copy to Copy CSV Data to SQL Table

Lecture 5: Define Mapping in ADF Data Copy

Lecture 6: Merge from CSV to SQL Table using ADF Pipeline Data Copy

Lecture 7: Exercise to Copy Data to SQL Table using ADF Data Copy

Chapter 13: Getting Started with Azure Synapse Analytics

Lecture 1: Create Azure Synapse Analytics Workspace

Lecture 2: Getting Started with Azure Synapse Studio

Lecture 3: Overview of Azure Synapse Serverless SQL Pool

Lecture 4: Link Azure Storage Account with Azure Synapse Workspace

Lecture 5: Generate Azure Synapse Query using ADLS Files

Instructors

Durga Viswanatha Raju Gadiraju
CEO at ITVersity and CTO at Analytiqs, Inc

Pratik Kumar

Sathvika Dandu

Madhuri Gadiraju

Sai Varma

Phani Bhushan Bozzam

Rating Distribution

1 stars: 5 votes

2 stars: 5 votes

3 stars: 15 votes

4 stars: 76 votes

5 stars: 111 votes

Frequently Asked Questions

How long do I have access to the course materials?

You can view and review the lecture materials indefinitely, like an on-demand channel.

Can I take my courses with me wherever I go?

Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don’t have an internet connection, some instructors also let their students download course lectures. That’s up to the instructor though, so make sure you get on their good side!