kubernetes - How to run a python function or script somescript.py on the KubernetesPodOperator in airflow?

Question

I am running a Celery Executor and I'm trying to run some python script in the KubernetesPodOperator. Below are examples of what I have tried that didn't work. What am I doing wrong?

Running sctipt

org_node = KubernetesPodOperator(
    namespace='default',
    image="python",
    cmds=["python", "somescript.py" "-c"],
    arguments=["print('HELLO')"],
    labels={"foo": "bar"},
    image_pull_policy="Always",
    name=task,
    task_id=task,
    is_delete_operator_pod=False,
    get_logs=True,
    dag=dag
)

Running function load_users_into_table()

def load_users_into_table(postgres_hook, schema, path):
  gdf = read_csv(path)
  gdf.to_sql('users', con=postgres_hook.get_sqlalchemy_engine(), schema=schema)

org_node = KubernetesPodOperator(
    namespace='default',
    image="python",
    cmds=["python", "somescript.py" "-c"],
    arguments=[load_users_into_table],
    labels={"foo": "bar"},
    image_pull_policy="Always",
    name=task,
    task_id=task,
    is_delete_operator_pod=False,
    get_logs=True,
    dag=dag
)

question from:https://stackoverflow.com/questions/65858358/how-to-run-a-python-function-or-script-somescript-py-on-the-kubernetespodoperato

Answer 1

KubernetesPodOperator launches a Kubernetes pod that runs a container as specified in the operator's arguments.

First Example

In the first example, the following happens:

• KubernetesPodOperator instructs K8s to lunch a pod and prepare to run a container in it using the python image (the image parameter) from hub.docker.com (the default image registry)
• ENTRYPOINT of the python image is replaced by ["python", "somescript.py" "-c"] (the cmd parameter)
• CMD of the python image is replaced by ["print('HELLO')"] (the arguments parameter)
• ...
• The container is run

So, the complete command that is run in the container is

python somescript.py -c print('HELLO')

Obviously, the official Python image from Docker Hub does not have somescript.py in its working directory. Even if did, it probably would have been not the one that you wrote. That is why the command fails with something like:

python: can't open file 'somescrit.py': [Errno 2] No such file or directory

Second Example

In the second example, pretty much the same happens as in the first example, but the command that is run in the container (again based on the cmd and arguments parameters) is

python somescript.py -c None

(None is the string representation of the load_users_into_table()'s return value)

This command fails, because of the same reasons as in the first example.

How It Could be Done (a Sketch)

You could build a Docker image with somescript.py and all its dependencies. Push the image to an image registry. Specify the image, ENTRYPOINT, and CMD in the corresponding parameters of KubernetesPodOperator.