Apache Airflow 2.0 - Complete Distributed Configuration

Apache Airflow 2.0 : Complete Distributed Configuration, available at $49.99, has an average rating of 3.65, with 38 lectures, based on 29 reviews, and has 204 subscribers.

You will learn about Gain complete understanding of the apache airflow Learn different types of Executors and its working principles Own a scalable distributed airflow setup which could be shared by multi teams in your organisation This course is ideal for individuals who are Curious on Big Data Technologies It is particularly useful for Curious on Big Data Technologies.

Enroll now: Apache Airflow 2.0 : Complete Distributed Configuration

Summary

Title: Apache Airflow 2.0 : Complete Distributed Configuration

Price: $49.99

Average Rating: 3.65

Number of Lectures: 38

Number of Published Lectures: 37

Number of Curriculum Items: 38

Number of Published Curriculum Objects: 37

Original Price: ?1,499

Quality Status: approved

Status: Live

What You Will Learn

Gain complete understanding of the apache airflow

Learn different types of Executors and its working principles

Own a scalable distributed airflow setup which could be shared by multi teams in your organisation

Who Should Attend

Curious on Big Data Technologies

Target Audiences

Curious on Big Data Technologies

Airflow is a platform created by the community to programmatically author, schedule and monitor workflows. Airflow is one of the best open source orchestrators and it is used widely because it is simplicity, scalability and extensibility.

Main goal of this course is to achieve an Airflow distributed setup using Celery Executor and be able to run more than 100 jobs or DAGs in parallel at any instance in time. I cover Sequential, Local and Celery Executor in this course. We acquire few EC2 instances from AWS and configure these executors.

Airflow community recently released Airflow 2.0. It contains many amazing features like HA Scheduler, Massive performance improvement on scheduler as well as celery workers etc. I myself fascinated by airflow 2.0 performance and migrated all airflow 1.x to 2.x in my organisation.

I am adding a new module on apache airflow 2.0. Here, we begin the module by learning new enhancements and HA architecture of airflow 2.0. Next, we install Webserver, Scheduler, Celery workers and Flower components. At the end, we configure multiple schedulers and observe its performance.

In addition to this, we explore salient features like Login, Email alerting and Logs management.

By the end of this course, you own a great distributed airflow setup which could be shared by multi teams in your organisation.

Course Curriculum

Chapter 1: Introduction

Lecture 1: Introduction

Lecture 2: Motivation

Chapter 2: Install Airflow and Web Server walkthrough

Lecture 1: Installation

Lecture 2: Fundamental Concepts

Lecture 3: Airflow Config Walkthrough Part – 1

Lecture 4: Airflow Config Walkthrough Part – 2

Lecture 5: Airflow config hands-on section

Chapter 3: Sequential Executor with SQLite

Lecture 1: Components and Architecture

Chapter 4: Local Executor with Mysql

Lecture 1: Introduction

Lecture 2: Deep dive into Local Executor

Lecture 3: System Configuration

Lecture 4: Acquire 2 VMs from AWS

Lecture 5: (Doc) Install and configure Mysql

Lecture 6: Setup Local Executor

Lecture 7: Login

Lecture 8: Email setup, Templates and Priority

Lecture 9: Logs management

Lecture 10: Mysql airflow database walkthrough

Lecture 11: Benchmarking

Chapter 5: Celery Executor with Mysql and RabbitMQ

Lecture 1: Introduction

Lecture 2: Components and Architecture

Lecture 3: System Configuration

Lecture 4: (Doc) Install and Configure RabbitMQ

Lecture 5: Configure the Master Node

Lecture 6: Configure Worker nodes

Lecture 7: Sync Logs to S3 part – 1

Lecture 8: Sync Logs to S3 part – 2

Lecture 9: Flower

Lecture 10: Benchmarking

Chapter 6: Apache Airflow 2.0

Lecture 1: Enhancements

Lecture 2: Proposed HA Architecture

Lecture 3: Install Airflow 2.0.1

Lecture 4: Install Webserver

Lecture 5: Install Scheduler

Lecture 6: Install Celery Worker

Lecture 7: Install Flower

Lecture 8: Install Multiple Schedulers

Instructors

Ganesh Dhareshwar
Senior Data Engineer at Delivery Hero

Rating Distribution

1 stars: 0 votes

2 stars: 1 votes

3 stars: 7 votes

4 stars: 6 votes

5 stars: 15 votes

Frequently Asked Questions

How long do I have access to the course materials?

You can view and review the lecture materials indefinitely, like an on-demand channel.

Can I take my courses with me wherever I go?

Definitely! If you have an internet connection, courses on Udemy are available on any device at any time. If you don’t have an internet connection, some instructors also let their students download course lectures. That’s up to the instructor though, so make sure you get on their good side!