|
- How to create a conditional task in Airflow - Stack Overflow
Airflow 1 x Airflow has a BranchPythonOperator that can be used to express the branching dependency more directly The docs describe its use: The BranchPythonOperator is much like the PythonOperator except that it expects a python_callable that returns a task_id The task_id returned is followed, and all of the other paths are skipped
- airflow users create command not working with 3. 0 version
Run pip install apache-airflow-providers-fab to install fab auth manager and set the below variable in airflow cfg file to enable fab auth manager auth_manager = airflow providers fab auth_manager fab_auth_manager FabAuthManager After you set this, you should be able to create users using 'airflow users create' command
- How to define a timeout for Apache Airflow DAGs?
I'm using Airflow 1 10 2 but Airflow seems to ignore the timeout I've set for the DAG I'm setting a timeout period for the DAG using the dagrun_timeout parameter (e g 20 seconds) and I've got a task which takes 2 mins to run, but Airflow marks the DAG as successful!
- How to Trigger a DAG on the success of a another DAG in Airflow using . . .
from typing import List from airflow models baseoperator import BaseOperator from airflow models dag import DAG from airflow operators dagrun_operator import TriggerDagRunOperator from airflow utils trigger_rule import TriggerRule # DAG object my_dag: DAG = DAG(dag_id='my_dag', start_date= ) # a list of 'tail' tasks: tasks that have no
- First time login to Apache Airflow asks for username and password, what . . .
user: airflow password: airflow If you want to create another account with docker use this (you have to be in the same folder of the docker-compose yaml file): docker-compose run airflow-worker airflow users create --role Admin --username admin --email admin --firstname admin --lastname admin --password admin
- airflow - Run DAG at specific time each day - Stack Overflow
In Airflow the scheduling is calculated by start_date + schedule interval Airflow execute the job at the END of the interval Airflow execute the job at the END of the interval This is consistent with how data pipelines usually works
- In Airflow UI, how to add SQL Server connection type
FROM apache airflow RUN pip install apache-airflow-providers-microsoft-mssql \ pip install apache-airflow-providers-microsoft-azure \ pip install apache-airflow-providers-odbc \ pip install pyodbc Now launch a terminal in the project directory, and build a docker image from the Dockerfile, using the command
- python - Running airflow tasks dags in parallel - Stack Overflow
By default, Airflow uses SequentialExecutor which would execute task sequentially no matter what So to allow Airflow to run tasks in Parallel you will need to create a database in Postges or MySQL and configure it in airflow cfg (sql_alchemy_conn param) and then change your executor to LocalExecutor in airflow cfg and then run airflow initdb
|
|
|