Tutorialspoint

Celebrating 11 Years of Learning Excellence! Use: TP11

Apache Airflow 2.0 using Docker, Docker Swarm

person icon Ganesh Dhareshwar

4.6

Apache Airflow 2.0 using Docker, Docker Swarm

Setup Resilient Airflow 2.0.1 using Docker and Docker Swarm

updated on icon Updated on Jun, 2025

language icon Language - English

person icon Ganesh Dhareshwar

English [CC]

category icon Development ,Database and Design Development,

Lectures -20

Resources -6

Duration -1 hours

Lifetime Access

4.6

price-loader

Lifetime Access

30-days Money-Back Guarantee

Training 5 or more people ?

Get your team access to 10000+ top Tutorials Point courses anytime, anywhere.

Course Description

Why Am I conducting this course?
When I started configuring airflow in my organisation, I spent many weeks on writing Docker Compose files for each airflow component.

The Airflow community provides a single Docker compose file, which installs all the components on a single machine. It works fine for practicing purposes. But in production, we setup each component on different machines. Also, there is no Docker image available on the Docker registry to start airflow through Docker Swarm.

Overall, I contributed many sleepless nights to achieve fault-tolerant, resilient, distributed, Highly Available airflow using Docker Swarm. I consolidated all my learning and knowledge into this course. So that others don't need to struggle like mine.

What do you Learn From this Course?
The primary objective of this course is to achieve resilient airflow using the Docker and Docker Swarm. I am using the latest stable airflow (2.0.1) throughout this course.

At first, we study all the required Docker concepts. Don't worry if you have no prior experience on Docker. I cover all Docker-necessary concepts that are used in this course. In the second module, we investigate Airflow 2.0 and understand the additional advantage over Airflow 1.x. We discover the airflow HA architecture and discuss each system requirement.

After this, we acquire these machines from AWS and start containerising one-by-one applications using Docker Compose.
At the end, we run multiple airflow schedulers and benchmark them.

The last module is on Docker Swarm, and we witness how easy it is to setup the entire airflow just by running a few Swarm commands.
In the beginning, we learn all the swarm concepts, architecture, commands, and networking. We translate the docker compose file into the docker service command. Subsequently, we create a swarm cluster and trigger one-by-one airflow services on it.

Goals

  • Apache Airflow 2.0 Architectures and Enhancements
  • Docker and Docker Swarm commands
  • Setting up airflow 2.0.1 using Docker Compose and Docker Swarm

Prerequisites

  • Familiar with Docker concepts and commands
  • Good to have knowledge on Apache Airflow 1.x
Apache Airflow 2.0 using Docker, Docker Swarm

Curriculum

Check out the detailed breakdown of what’s inside the course

Introduction

1 Lectures
  • play icon Course Introduction 03:06 03:06

Docker

4 Lectures
Tutorialspoint

Airflow 2.0 using Docker Compose

7 Lectures
Tutorialspoint

Docker Swarm

8 Lectures
Tutorialspoint

Instructor Details

Ganesh Dhareshwar

Ganesh Dhareshwar

Course Certificate

Use your certificate to make a career change or to advance in your current career.

sample Tutorialspoint certificate

Our students work
with the Best

Related Video Courses

View More

Annual Membership

Become a valued member of Tutorials Point and enjoy unlimited access to our vast library of top-rated Video Courses

Subscribe now
Annual Membership

Online Certifications

Master prominent technologies at full length and become a valued certified professional.

Explore Now
Online Certifications

Talk to us

1800-202-0515