[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Create Connections in Airflow operator at runtime

Hello maintainers

I have to run tasks on a remote EMR cluster, which is created (and later
destroyed) using EmrCreateJobFlowOperator
So the 'conn_id' in my SSHHook will be known only at runtime. Is it
possible to create (and also destroy) either of these from within an
Airflow operator?

   1. *Connection*
   in Airflow's db)
   2. *Environment Variable* (should be accessible to all downstream tasks
   and not just current task as told here


Here's the link
to my (elaborated) question on SO
Thank You

*Shubham Gupta*
Software Engineer