Categories
Mastering Development System & Network

Apache Airflow BashOperator() missing environment variables because running with /bin/sh instead of /bin/bash

I’m having trouble with Apache Airflow DAG BashOperator() executing using /bin/sh instead of what I would prefer /bin/bash and getting an in complete set of bash environment variables on Ubuntu 18.04 .

I have one server that it works correctly using /bin/bash and another that it does not and uses /bin/sh instead and I haven’t been able to figure out why yet or how to fix it. Has anyone encountered this or resolved this same issue?

I verified that the linux user running the BashOperator has /bin/bash as their shell in /etc/passwd

The Airflow DAG important parts:

dag = DAG(
    'zz_environment_test',
    default_args=default_args,
    description='Ariflow Environment Test',
    catchup=False,
    schedule_interval=None)

#commands 
#step 1 : Cut release branch from develop for Stage
environment_test_command = """
cd %s;bash zz_environment_test.sh %s;
""" % (release_target_directory, release_target_directory)

step_1_environment_test = BashOperator(
    task_id='step_1_environment_test',
    depends_on_past=False,
    bash_command=environment_test_command,
    dag=dag)


step_1_environment_test

In zz_environment_test.sh I have

whoami

printenv > mytest/bash_from_airflow_printenv.txt

Which on the environment that has the correct environment variables has value

SHELL=/bin/bash

And the one that isn’t working has

SHELL=/bin/sh

Leave a Reply

Your email address will not be published. Required fields are marked *