Airflow does not respect depends_on_past when catchup = True?

Question

I have an Airflow 2.0 DAG defined with a start_date in the past, catchup = True and max_active_runs = 5. The goal is to process data from the start_date with up to five DAG Runs executing in parallel.

Certain tasks are defined with depends_on_past = True, however I observe that they do sometimes execute in parallel and it seems depends_on_past is not respected.

Why is this ? Is it possible to process historical data with parallel DAG Runs, with either catchup or backfill, and prevent certain tasks in the DAG from executing in parallel ?

question from:https://stackoverflow.com/questions/65844294/airflow-does-not-respect-depends-on-past-when-catchup-true

Answer 1

Why is this?

According to documentation this should not happen. Refer to this answer.

Is it possible to process historical data with parallel DAG Runs, with either catchup or backfill, and prevent certain tasks in the DAG from executing in parallel?

You would benefit from the pools. Pools allow you to control parallelism on tasks of your choice. It is referred in this answer as well.

P.S. Would have suggested in comments but I can't.