What is the correct permission for directories that will be used in airflow docker?

Question

Hi I want to run an airflow setup using the following docker-compose file

version: "3.8"

services:
  postgres:
    image: postgres:12
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=airflow
      - POSTGRES_DB=airflow
      - PGDATA=/var/lib/postgresql/data/pgdata
    ports:
      - "5433:5432"
    volumes:
      - ~/airflow-data/pg:/var/lib/postgresql/data/pgdata

  scheduler:
    image:apache/airflow
    restart: always
    depends_on:
      - postgres
      - webserver
    env_file:
      - .env
    ports:
      - "8793:8793"
    volumes:
      - ./airflow-dags:/opt/airflow/dags
      - ~/airflow-data/logs:/opt/airflow/logs
    command: scheduler
    healthcheck:
      test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 3
  webserver:
    image:apache/airflow
    hostname: webserver
    restart: always
    depends_on:
      - postgres
    env_file:
      - .env
    volumes:
      - ./airflow-dags:/opt/airflow/dags
      - ~/airflow-data/logs:/opt/airflow/logs
      - ./scripts:/opt/airflow/scripts
    ports:
      - "8080:8080"
    entrypoint: ./scripts/webserver-entrypoint.sh
    healthcheck:
      test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
      interval: 30s
      timeout: 30s
      retries: 32

As you can see that I am going to use ~/airflow-data/ to store logs and ./airflow-dags to store dags file, when I start these containers using docker-compose up I can see in the logs that the airflow user inside the docker image simply do not have permission to read/write to these locations which are created by hand using mkdir -p outside the container. I am just wondering, what is the correct permission I should give to these directories? Of course 777 is an option but I just feel that is too blunt, is there any other option that can make sure only my user (outside the docker container) and the airflow user can use these files and no one else?

Thanks!

question from:https://stackoverflow.com/questions/65853736/what-is-the-correct-permission-for-directories-that-will-be-used-in-airflow-dock

Answer 1

Waitting for answers