Practical Apache Airflow

Practical Apache Airflow, available at $39.99, has an average rating of 4.05, with 59 lectures, based on 22 reviews, and has 163 subscribers.

You will learn about Airflow, Developing Data Pipeline, Python, Pandas Hands on data pipeline development and comparison with other technology Installation, Configuration of Airflow DAGs, Creating workflow, Operators, Tasks, dependency management, Hooks, Connections Different Executors like Local, Celery and Sequential and differences Airflow Architecture in detail Advanced concepts like Xcoms, Branching, Variables and DAG Chaining Authentication and Log storage to S3 Airflow on Docker This course is ideal for individuals who are Students who want to learn developing data pipeline or Students and Professional wanting to learn Airflow or Professional looking to implement Airflow for their team and organization It is particularly useful for Students who want to learn developing data pipeline or Students and Professional wanting to learn Airflow or Professional looking to implement Airflow for their team and organization.

Enroll now: Practical Apache Airflow

Summary

Title: Practical Apache Airflow

Price: $39.99

Average Rating: 4.05

Number of Lectures: 59

Number of Published Lectures: 59

Number of Curriculum Items: 59

Number of Published Curriculum Objects: 59

Original Price: $19.99

Quality Status: approved

Status: Live

What You Will Learn

Airflow, Developing Data Pipeline, Python, Pandas

Hands on data pipeline development and comparison with other technology

Installation, Configuration of Airflow

DAGs, Creating workflow, Operators, Tasks, dependency management, Hooks, Connections

Different Executors like Local, Celery and Sequential and differences

Airflow Architecture in detail

Advanced concepts like Xcoms, Branching, Variables and DAG Chaining

Authentication and Log storage to S3

Airflow on Docker

Who Should Attend

Students who want to learn developing data pipeline

Students and Professional wanting to learn Airflow

Professional looking to implement Airflow for their team and organization

Target Audiences

Students who want to learn developing data pipeline

Students and Professional wanting to learn Airflow

Professional looking to implement Airflow for their team and organization

Data engineering is a field that can be thought as a superset of business intelligence and data warehousing which brings more elements from software engineering. The reason data engineering exists today is because companies have massive treasure troves of data, but to provide value the data must be extracted. Data engineering provides the toolbox and is how we make sense of that data quickly and effectively.

When it comes to managing data collection, munging and consumption, data pipeline frameworks play a significant role and with the help of Apache Airflow, task of creating data pipeline is not only easy but its actually fun. Originated from AirBnb, Airflow soon became part of the very core of their tech stack.

The data infrastructure ecosystem has yet to show any sign of converging into something more manageable. It seems like we’re still in a huge phase of expansion where every new day bring new distributed database, new frameworks, new libraries and new teammates. As these systems get more complicated and evolve rapidly, it becomes even more important to have something like Apache Airflow that brings everything together in a sane place where every little piece of the puzzle can be orchestrated properly with sane APIs.

So in this course we will be learning as how to reach feature completeness with this amazing orchestration tool called Apache Airflow. You will not only learn to setup the environment but also learn how to create workflow pipeline with real world example so don’t wait and sign-up today and get started.

Looking forward to seeing you in this course!

Course Curriculum

Chapter 1: Introduction

Lecture 1: Introduction

Lecture 2: Who this course is for

Lecture 3: Dags and Pipeline

Lecture 4: What is Airflow?

Lecture 5: Why Use Airflow?

Lecture 6: Comparison between Luigi, Azkaban, Oozi

Lecture 7: Quiz Time

Chapter 2: Architecture

Lecture 1: Airflow Architecture

Lecture 2: Quiz Time

Chapter 3: Installation

Lecture 1: Installing Airflow

Lecture 2: Environment variable setup

Lecture 3: Hands on: setting environment variable and starting web server

Lecture 4: Hands on : setting encryption to secure connection secrets

Lecture 5: Quiz Time

Chapter 4: Airflow Configuration

Lecture 1: Airflow Configuration Overview

Lecture 2: Configuration options: ORM Configuration

Lecture 3: Configuration Option: Maximum Active Runs Explained

Lecture 4: Configuration Option: Maximum Active Runs Explained Continued

Lecture 5: Configuration Options: Additional Configuration Settings

Lecture 6: Quiz Time

Chapter 5: Developing Your First Data Pipeline

Lecture 1: Problem Statement

Lecture 2: Hands on: Project Setup

Lecture 3: Hands on: Data Retrieval from File System

Lecture 4: Hands on: Merging DataFrames

Lecture 5: Hands on: Aggregation Using Pandas

Lecture 6: Hands on: Database Connectivity

Lecture 7: Hands on: Creating Dags

Lecture 8: Summary

Chapter 6: DAG Chaining

Lecture 1: Dynamic Flow: Pattern -1 and Pattern – 2 Explained

Lecture 2: Dynamic Flow: Pattern – 3 Overview

Lecture 3: Dynamic Flow: Pattern – 4 Discussed

Lecture 4: Dynamic Flow: Exploring Pattern – 5

Lecture 5: Hands on: What are Airflow Hooks?

Lecture 6: Hands on: Exploring Task Branching

Lecture 7: Hands on: Passing and Modifying Variables

Lecture 8: Hands on: Xcom in Depth

Chapter 7: Authentication

Lecture 1: Airflow behavior with Python 2 vs Python 3

Lecture 2: Setting Authentication

Lecture 3: Setting Authentication Demo

Lecture 4: Hands On : Creating user with Python 3

Lecture 5: Creating user with Python2

Chapter 8: Airflow: Log storage to Cloud

Lecture 1: Writing Logs to S3

Lecture 2: Code: How to store Logs to S3

Lecture 3: Demo: Logs publishing to S3

Chapter 9: Airflow on Docker

Lecture 1: Intro to setting airflow in docker

Lecture 2: Installing Docker

Lecture 3: Building docker container for Airflow

Lecture 4: Building containers with Local and Celery Executors

Lecture 5: Dockerfile Explained

Lecture 6: Assignment

Lecture 7: Further on Airflow Containers

Chapter 10: REST APIs

Lecture 1: Test REST API Server

Lecture 2: REST API Continued

Lecture 3: REST API

Chapter 11: SLAs

Lecture 1: What is an SLA?

Lecture 2: SLA Implementation

Lecture 3: SLA Demo

Lecture 4: (Optional) SLA – Implementation with Airflow explained

Chapter 12: Airflow Command Line

Lecture 1: Useful Command Lines for Airflow

Instructors

CGI GEN
Learn by Doing. Practical Approach to Solve Common Problems

Rating Distribution

1 stars: 2 votes

2 stars: 1 votes

3 stars: 7 votes

4 stars: 5 votes

5 stars: 7 votes

Frequently Asked Questions

How long do I have access to the course materials?

You can view and review the lecture materials indefinitely, like an on-demand channel.

Can I take my courses with me wherever I go?

Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don’t have an internet connection, some instructors also let their students download course lectures. That’s up to the instructor though, so make sure you get on their good side!