Broken DAG: No module named 'psycopg2' when using AWS Airflow Managed Service

Question

I'm trying to use the PostgresHook inside a DAG in AWS Airflow Managed Service as the following way:

from airflow.hooks.postgres_hook import PostgresHook

The Airflow version that uses this service is 1.10.12, but when I upload this DAG the Airflow UI shows me the "Broken DAG: No module named 'psycopg2' " error.

I have the requirements.txt file defined with these modules but none seems to be working:

psycopg2-binary
psycopg2
tableauserverclient
google-auth
botocore
apache-airflow[postgres]

Does anyone know if there is a workaroundabout this issue? There is not so much info about that in AWS forum page.

question from:https://stackoverflow.com/questions/65890992/broken-dag-no-module-named-psycopg2-when-using-aws-airflow-managed-service

Answer 1

I have no problem using psycopg2 with MWAA.

in my requirements i just have psycopg2-binary not psycopg2.

Here is a dag i like to use to list all the pip packages installed on my MWAA airflow environement:

import os
from datetime import timedelta

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.utils.dates import days_ago

DAG_ID = os.path.basename(__file__).replace('.py', '')

DEFAULT_ARGS = {
    'owner': 'Louis',
    'depends_on_past': False,
    'email_on_failure': False,
    'email_on_retry': False
}

with DAG(
        dag_id=DAG_ID,
        default_args=DEFAULT_ARGS,
        description='Print all installed Python packages',
        dagrun_timeout=timedelta(hours=2),
        start_date=days_ago(1),
        schedule_interval=None,
        tags=['bash']
) as dag:
    list_python_packages_operator = BashOperator(
        task_id='list_python_packages',
        bash_command='python3 -m pip list'
    )

list_python_packages_operator

Hope it helps debugging your issue.