Categories
Mastering Development

Airflow:How to select bigquery table data in to dataframe

I am new in airflow. I created my first dag below, selecting data from google big query table & saving it to a pd dataframe. Need suggestion in below Where should i provide connection id of my big query As pd.read_gbq requires authintication, how to handle same in airflow dags import os import pandas as […]

Categories
Mastering Development

AirflowTaskTimeout after setting execution_timeout

My Airflow DAG keeps failing on the only task that I have. I declared the execution_timeout as 300 seconds, but it keeps crashing after around 37 seconds. The task consists in scraping a website, without Chromedriver. I’m on Linux, Raspberry PI. Here is the code: from datetime import timedelta import importlib import sys from airflow.operators.bash_operator […]

Categories
Mastering Development System & Network

Apache Airflow BashOperator() missing environment variables because running with /bin/sh instead of /bin/bash

I’m having trouble with Apache Airflow DAG BashOperator() executing using /bin/sh instead of what I would prefer /bin/bash and getting an in complete set of bash environment variables on Ubuntu 18.04 . I have one server that it works correctly using /bin/bash and another that it does not and uses /bin/sh instead and I haven’t […]

Categories
Mastering Development

Airflow ExternalTaskSensor does not trigger task

I’m trying to add a cross dag dependency using ExternalTaskSensor but haven’t been able to get it to work. Dag A has schedule_interval=None as it doesn’t have a fixed schedule and is triggered externally by a file creation event. Dag B should execute once Dag A has completed. Here is code for dag_a and dag_b. […]

Categories
Mastering Development

Sqs sensor in airflow not triggering when there is a new message

I am new to Airflow. I am trying to run my dag whenever there is a message in SQS. I am using SQSSensor to do the same. It triggered for the first run but after that it doesn’t invoke when there is a new message. Please let me know if i am missing something. default_args […]

Categories
Mastering Development

Airflow: Master Dag with ExternalTaskSensor gets stuck forever

The requirement is to have DAG run one after the other and on success of each DAG I have a Master DAG in which I am calling all the DAG to get executed one after the other in sequence Also in each of the dag_A, dag_B, dag_C I have to given schedule_interval = None and […]

Categories
Mastering Development

Airflow BranchPython Operator – Continue After Branch

I have the following operators as shown below. I am able to visually see the the graph representation looks correct. However, I am not able to get the functionality to work properly as in the paths will not keep going past either branch. Regardless of the date, neither path will keep going to task_05. The […]

Categories
Mastering Development

How to trigger a Airflow task only when new partition/data in avialable in the AWS athena table using DAG in python?

I have a scenerio like a below : Trigger a Task 1 and Task 2 only when new data is avialable for them in source table ( Athena). Trigger for Task1 and Task2 should happen when a new data parition in a day. Trigger Task 3 only on the completion of Task 1 and Task […]

Categories
Mastering Development

Understanding Airflow’s execution_date and schedule

New to airflow coming from cron, trying to understand how the execution_date macro gets applied to the scheduling system and when manually triggered. I’ve read the faq, and setup a schedule to what I expected would execute with the correct execution_date macro filled in. I would like to run my dag weekly, on Thursday at […]

Categories
Mastering Development

Airflow subdag codeview shows code of main-dag

I am new to Airflow and I followed the tutorial on the official page (https://airflow.readthedocs.io/en/stable/tutorial.html) and added a subdag to the tutorial dag. When I zoom into the subdag on the web-UI and click on code, the code of the main-dag is shown. Also when I click on details of the subdag the filename of […]