#AIRFLOW 2.0 TUTORIAL UPDATE#
#AIRFLOW 2.0 TUTORIAL FOR WINDOWS 10#
#AIRFLOW 2.0 TUTORIAL WINDOWS 10#

To solve this issue you must do additional steps (unfortanutely there is no more quick workaround, check: and ):ĭocker volume create -name volume-postgresql -d localĢ.

#AIRFLOW 2.0 TUTORIAL WINDOWS 10#

If you try to work on Windows 10 & run docker-compose on it you will got an issue for postgres service:įATAL: data directory '/var/lib/postgresql/data/pgdata' has wrong ownership Apache Airflow Image was updated to version 1.10.6.added samples for article (docker-compose-volume-packages.yml, packages.pth, added commented lines to Dockerfile).Added docker-compose-with-celery.yml based on official image.Docker Compose files with puckel images moved to docker_with_puckel_image.

#AIRFLOW 2.0 TUTORIAL FOR WINDOWS 10#

At the bottom of readme added note for Windows 10 users.

Creted docker-compose config based on official image (required docker-compose version 3.7 and higher):.

All compose files with puckel_image moved to docker_with_puckel_image.

Added restart_policy to services in docker-compose.

#AIRFLOW 2.0 TUTORIAL UPDATE#

Update Apache Airflow version to 1.10.12.Change init db command to 'airflow db init'.Update Apache Airflow version to 1.10.14.Added separate docker-compose file for Apache Airflow 2.0 version.

Added FAQ section with issues that might help.

Added example for Apached Airflow 2.0 with 2 workers.

In docker-compose files for Airflow 2.0 scheduler service restart police changed to 'any' because for some reason it exist with 0 if error in DB and init is not finished yet, so restart policy 'on-failure' does not works.

all files with version 1.* & puckel images moved to 'archive' folder.

Source files for article with description on Medium.Īpache Airflow with LocalExecutor: Airflow with CeleryExecutor: Python dependencies to docker-compose cluster without re-build images Updated version to 2.5.0

Main Docker Compose Cluster based on apache/airflow Imageĭocker-compose config based on official image (required docker-compose version 3.7 and higher):Īnd env file with config setting for Airflow (used in docker-compose-with-celery-executor.yml): So this docker-compose files became 'legacy'Īnd all sources moved to 'docker_with_puckel_image'. Now, already exist official image apache/airflow. Those articles was created before release of official Apache Airflow Docker image and they use puckel/docker-airflow. Thank you! Apache Airflow with Docker Compose examples If you had any troubles & you successfully solve it - please open an issue with solution, I will add it to this readme.md file. And you can see the errors like this:ĭocker-compose -f up -build postgres If you share low count of resources for Docker or you have a machine with low perfomance, up&run PostgreSQL for the first time can take a significant time. Problem with connection to PostgreSQL (at first time run):

No DAGs in UI in Airflow 2.0 & failed airflow init on second runs - Not resolved yetĪlso at the end of this README.md file exists section with some information wor WSL users.

In issues you can find some cases when something goes wrong and maybe it will help you to solve own issue. This docker-compose file tested by me in MacOS mostly, some time I can up & run it on wsl (but not each update).

Wait until all services will succesfull up & open FAQ & HelpĮxists different behaviour of Docker Compose on different OS relative to file system specifications, work with access rights & etc.

Airflow 2.0 tutorial