Mastering Development

Airflow:How to select bigquery table data in to dataframe

I am new in airflow. I created my first dag below, selecting data from google big query table & saving it to a pd dataframe. Need suggestion in below Where should i provide connection id of my big query As pd.read_gbq requires authintication, how to handle same in airflow dags import os import pandas as […]


ERROR – SSH operator error: timed out in Airflow while connecting with ec2 instance

I am getting a timeout error while trying to connect with ec2 instance. I can connect to the machine through ssh command: ssh -i keypair.pem myuser@ec2IPaddress My connection file has the following things: conn Id: ssh_custom Conn Type: SSH Host: ec2IPaddress Username: myuser Port: 8888 Extra: {“key_file”:”/home/ubuntu/keypair.pem”, “no_host_key_check”: “false”, “allow_host_key_change”: “true”, “timeout”: “45”} My dag […]