The Complete Hands-On Introduction to Apache Airflow

The Complete Hands-On Introduction to Apache Airflow, available at $84.99, has an average rating of 4.52, with 133 lectures, 13 quizzes, based on 10632 reviews, and has 63038 subscribers.

You will learn about Create plugins to add functionalities to Apache Airflow. Using Docker with Airflow and different executors Master core functionalities such as DAGs, Operators, Tasks, Workflows, etc Understand and apply advanced concepts of Apache Airflow such as XCOMs, Branching and SubDAGs. The difference between Sequential, Local and Celery Executors, how do they work and how can you use them. Use Apache Airflow in a Big Data ecosystem with Hive, PostgreSQL, Elasticsearch etc. Install and configure Apache Airflow Think, answer and implement solutions using Airflow to real data processing problems This course is ideal for individuals who are People being curious about data engineering. or People who want to learn basic and advanced concepts about Apache Airflow. or People who like hands-on approach. It is particularly useful for People being curious about data engineering. or People who want to learn basic and advanced concepts about Apache Airflow. or People who like hands-on approach.

Enroll now: The Complete Hands-On Introduction to Apache Airflow

Summary

Title: The Complete Hands-On Introduction to Apache Airflow

Price: $84.99

Average Rating: 4.52

Number of Lectures: 133

Number of Quizzes: 13

Number of Published Lectures: 103

Number of Published Quizzes: 11

Number of Curriculum Items: 152

Number of Published Curriculum Objects: 118

Original Price: $39.99

Quality Status: approved

Status: Live

What You Will Learn

Create plugins to add functionalities to Apache Airflow.

Using Docker with Airflow and different executors

Master core functionalities such as DAGs, Operators, Tasks, Workflows, etc

Understand and apply advanced concepts of Apache Airflow such as XCOMs, Branching and SubDAGs.

The difference between Sequential, Local and Celery Executors, how do they work and how can you use them.

Use Apache Airflow in a Big Data ecosystem with Hive, PostgreSQL, Elasticsearch etc.

Install and configure Apache Airflow

Think, answer and implement solutions using Airflow to real data processing problems

Who Should Attend

People being curious about data engineering.

People who want to learn basic and advanced concepts about Apache Airflow.

People who like hands-on approach.

Target Audiences

People being curious about data engineering.

People who want to learn basic and advanced concepts about Apache Airflow.

People who like hands-on approach.

Apache Airflow is an open-source  platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have.

In this course you are going to learn everything you need to start using Apache Airflow through theory and pratical videos. Starting from very basic notions such as, what is Airflow and how it works, we will dive into advanced concepts such as, how to create plugins and make real dynamic pipelines.

Course Curriculum

Chapter 1: Course Introduction

Lecture 1: Prerequisites

Lecture 2: Course Objectives

Lecture 3: Who I am?

Lecture 4: Development Environment

Chapter 2: Getting Started with Airflow

Lecture 1: Why Airflow?

Lecture 2: What is Airflow?

Lecture 3: Core Components

Lecture 4: Core Concepts

Lecture 5: Airflow is not

Lecture 6: The Different Architectures

Lecture 7: How does it work?

Lecture 8: [Practice] Installing Apache Airflow

Lecture 9: What is Docker?

Lecture 10: The docker-compose file

Lecture 11: Key Takeaways

Chapter 3: The important views of the Airflow UI

Lecture 1: The DAGs View

Lecture 2: The Grid View

Lecture 3: The Graph View

Lecture 4: The Landing Times View

Lecture 5: The Calendar View

Lecture 6: The Gantt View

Lecture 7: The Code View

Lecture 8: Wrap up!

Chapter 4: Coding Your First Data Pipeline with Airflow

Lecture 1: The Project

Lecture 2: Advices

Lecture 3: What is a DAG?

Lecture 4: DAG Skeleton

Lecture 5: What is an Operator?

Lecture 6: Providers

Lecture 7: Create a Table

Lecture 8: Create a connection

Lecture 9: The secret weapon!

Lecture 10: What is a Sensor?

Lecture 11: Is the API available?

Lecture 12: Create the connection user_api

Lecture 13: Extract users

Lecture 14: Process users

Lecture 15: Before running process_user

Lecture 16: What is a Hook?

Lecture 17: Store users

Lecture 18: Order matters!

Lecture 19: Your DAG in action!

Lecture 20: DAG Scheduling

Lecture 21: Backfilling: How does it work?

Lecture 22: Wrap up!

Chapter 5: The New Way of Scheduling DAGs

Lecture 1: Why do you need that feature?

Lecture 2: What is a Dataset?

Lecture 3: Adios schedule_interval!

Lecture 4: Create the Producer DAG

Lecture 5: Create the Consumer DAG

Lecture 6: Track your Datasets with the new view!

Lecture 7: Wait for many datasets

Lecture 8: Dataset limitations

Chapter 6: Databases and Executors

Lecture 1: Whats an executor?

Lecture 2: The default config

Lecture 3: The Sequential Executor

Lecture 4: The Local Executor

Lecture 5: The Celery Executor

Lecture 6: The current config

Lecture 7: Add the DAG parallel_dag.py into the dags folder

Lecture 8: Monitor your tasks with Flower

Lecture 9: Remove DAG examples

Lecture 10: Running tasks on Celery Workers

Lecture 11: What is a queue?

Lecture 12: Add a new Celery Worker

Lecture 13: Create a queue to better distribute tasks

Lecture 14: Send a task to a specific queue

Lecture 15: Concurrency, the parameters you must know!

Chapter 7: Implementing Advanced Concepts in Airflow

Lecture 1: Adios repetitive patterns

Lecture 2: Add the DAG group_dag.py

Lecture 3: How to use SubDAGs?

Lecture 4: Adios SubDAGs, welcome TaskGroups!

Lecture 5: Add the DAG xcom_dag.py

Lecture 6: Sharing data between tasks with XComs

Lecture 7: [Practice] XComs in action!

Lecture 8: Choosing a specific path in your DAG

Lecture 9: [Practice] Executing a task according to a condition

Lecture 10: Trigger rules or how tasks get triggered

Instructors

Marc Lamberti
Apache Airflow Expert, Data Engineer

Rating Distribution

1 stars: 97 votes

2 stars: 147 votes

3 stars: 915 votes

4 stars: 3762 votes

5 stars: 5716 votes

Frequently Asked Questions

How long do I have access to the course materials?

You can view and review the lecture materials indefinitely, like an on-demand channel.

Can I take my courses with me wherever I go?

Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don’t have an internet connection, some instructors also let their students download course lectures. That’s up to the instructor though, so make sure you get on their good side!