Airflow benchmarks

Benchmark conclusions

Conclusions for each benchmark run for all engines in all languages and settings

Summary

Airflow was the slowest in all categories, with high orchestration overhead and poor responsiveness to both lightweight and long-running tasks. It suffers from long assignment delays and inefficient scaling, making it unsuitable for performance-critical workflows.

Airflow setup

We set up Airflow version 2.7.3 using the docker-compose.yaml referenced in Airflows official documentation.

The DAG was the following:

ITER = 10     # respectively 40
FIBO_N = 33   # respectively 10

with DAG(
    dag_id="bench_{}".format(ITER),
    schedule=None,
    start_date=datetime(2023, 1, 1),
    catchup=False,
    tags=["benchmark"],
) as dag:
    for i in range(ITER):
        @task(task_id=f"task_{i}")
        def task_module():
            return fibo(FIBO_N)
        fibo_task = task_module()

        if i > 0:
            previous_task >> fibo_task
        previous_task = fibo_task

Fibonacci 40 iterations, n=10

This benchmark measures the performance of Airflow when handling a large number of lightweight tasks. The tasks are Fibonacci calculations with 40 iterations and a base value of 10.

Python

Visualization

Statistics

	Airflow	Windmill	Windmill Dedicated
Total duration (in seconds)	116.221	4.383	2.092
Assignment	75.112 (64.63%)	1.836 (41.89%)	1.795 (85.80%)
Execution	12.516 (10.77%)	2.215 (50.54%)	0.122 (5.83%)
Transition	28.593 (24.60%)	0.332 (7.57%)	0.175 (8.37%)

Timing details

View task timing details

Task	Created at	Started at	Completed at
task_00	0.000	4.335	4.752
task_01	6.236	8.710	8.923
task_02	9.792	11.117	11.320
task_03	12.157	13.513	13.733
task_04	13.804	15.413	15.622
task_05	16.201	17.587	17.849
task_06	18.902	20.227	20.432
task_07	21.262	22.691	22.958
task_08	24.015	25.349	25.558
task_09	26.368	28.158	28.635
task_10	29.361	31.035	31.357
task_11	31.861	36.245	37.062
task_12	38.868	42.180	42.388
task_13	42.641	44.027	44.280
task_14	45.321	46.676	46.877
task_15	47.676	49.073	49.298
task_16	50.432	51.786	51.999
task_17	52.415	53.852	54.051
task_18	54.155	55.564	55.771
task_19	56.575	58.346	58.781
task_20	59.254	60.999	61.355
task_21	62.071	63.671	64.079
task_22	64.366	66.011	66.442
task_23	67.061	68.619	68.866
task_24	69.601	71.842	72.303
task_25	73.373	77.495	78.212
task_26	78.428	79.896	80.134
task_27	81.199	82.495	82.741
task_28	83.665	84.958	85.153
task_29	85.205	86.561	86.766
task_30	87.690	89.357	89.778
task_31	90.419	91.970	92.282
task_32	93.024	94.610	95.031
task_33	95.636	97.495	97.745
task_34	98.857	100.626	100.877
task_35	101.926	103.271	103.477
task_36	103.915	105.523	105.875
task_37	105.996	107.412	107.622
task_38	108.409	112.610	113.214
task_39	114.054	115.998	116.221

Windmill Comparison

Task	Created at	Started at	Completed at
task_00	0.000	0.003	0.059
task_01	0.067	0.113	0.171
task_02	0.180	0.226	0.280
task_03	0.290	0.335	0.389
task_04	0.398	0.446	0.501
task_05	0.510	0.558	0.614
task_06	0.622	0.669	0.725
task_07	0.732	0.780	0.834
task_08	0.842	0.889	0.942
task_09	0.950	0.997	1.052
task_10	1.061	1.108	1.166
task_11	1.175	1.220	1.274
task_12	1.283	1.330	1.385
task_13	1.394	1.440	1.494
task_14	1.503	1.550	1.605
task_15	1.612	1.661	1.716
task_16	1.723	1.770	1.823
task_17	1.831	1.878	1.930
task_18	1.939	1.986	2.041
task_19	2.049	2.096	2.152
task_20	2.161	2.209	2.266
task_21	2.274	2.320	2.376
task_22	2.384	2.431	2.486
task_23	2.495	2.542	2.596
task_24	2.604	2.652	2.706
task_25	2.715	2.761	2.816
task_26	2.825	2.872	2.925
task_27	2.933	2.979	3.033
task_28	3.042	3.090	3.145
task_29	3.154	3.201	3.269
task_30	3.278	3.325	3.382
task_31	3.391	3.437	3.493
task_32	3.501	3.548	3.602
task_33	3.611	3.660	3.715
task_34	3.723	3.770	3.823
task_35	3.833	3.879	3.934
task_36	3.942	3.990	4.045
task_37	4.053	4.101	4.157
task_38	4.165	4.212	4.268
task_39	4.277	4.324	4.383

Windmill Dedicated Comparison

Task	Created at	Started at	Completed at
task_00	0.000	0.019	0.022
task_01	0.029	0.073	0.077
task_02	0.081	0.125	0.128
task_03	0.134	0.179	0.182
task_04	0.187	0.231	0.234
task_05	0.239	0.284	0.287
task_06	0.292	0.338	0.341
task_07	0.345	0.391	0.394
task_08	0.398	0.444	0.447
task_09	0.451	0.497	0.500
task_10	0.505	0.549	0.552
task_11	0.557	0.603	0.606
task_12	0.610	0.655	0.659
task_13	0.663	0.709	0.712
task_14	0.716	0.761	0.764
task_15	0.768	0.814	0.817
task_16	0.821	0.867	0.870
task_17	0.876	0.921	0.924
task_18	0.929	0.973	0.976
task_19	0.981	1.027	1.030
task_20	1.035	1.080	1.083
task_21	1.087	1.132	1.135
task_22	1.139	1.186	1.189
task_23	1.193	1.238	1.241
task_24	1.246	1.292	1.295
task_25	1.299	1.345	1.348
task_26	1.352	1.398	1.401
task_27	1.405	1.451	1.454
task_28	1.458	1.504	1.507
task_29	1.512	1.557	1.560
task_30	1.564	1.611	1.614
task_31	1.618	1.664	1.667
task_32	1.671	1.717	1.720
task_33	1.724	1.770	1.773
task_34	1.777	1.823	1.826
task_35	1.830	1.876	1.879
task_36	1.884	1.930	1.933
task_37	1.937	1.983	1.986
task_38	1.991	2.036	2.039
task_39	2.043	2.089	2.092

Fibonacci 10 iterations, n=33

This benchmark measures the performance of Airflow when handling a small number of heavyweight tasks. The tasks are Fibonacci calculations with 10 iterations and a base value of 33.

Python

Visualization

Statistics

	Airflow	Windmill	Windmill Dedicated
Total duration (in seconds)	54.668	8.347	7.701
Assignment	22.058 (40.35%)	0.428 (5.13%)	0.370 (4.80%)
Execution	28.272 (51.72%)	7.832 (93.83%)	7.205 (93.56%)
Transition	4.338 (7.94%)	0.087 (1.04%)	0.126 (1.64%)

Timing details

View task timing details

Task	Created at	Started at	Completed at
task_00	0.000	4.347	6.910
task_01	7.315	9.690	16.387
task_02	16.545	18.361	20.077
task_03	20.130	21.785	23.487
task_04	23.869	25.319	27.463
task_05	28.061	29.665	32.354
task_06	33.210	34.996	37.498
task_07	38.378	39.938	41.754
task_08	42.366	43.933	45.887
task_09	46.281	50.179	54.668

Windmill Comparison

Task	Created at	Started at	Completed at
task_00	0.000	0.002	0.846
task_01	0.858	0.906	1.705
task_02	1.715	1.761	2.539
task_03	2.548	2.595	3.365
task_04	3.375	3.421	4.206
task_05	4.215	4.263	5.033
task_06	5.042	5.089	5.857
task_07	5.866	5.913	6.684
task_08	6.693	6.740	7.519
task_09	7.529	7.579	8.347

Windmill Dedicated Comparison

Task	Created at	Started at	Completed at
task_00	0.000	0.023	0.745
task_01	0.776	0.797	1.518
task_02	1.546	1.571	2.292
task_03	2.298	2.340	3.057
task_04	3.063	3.114	3.845
task_05	3.874	3.889	4.608
task_06	4.614	4.661	5.380
task_07	5.385	5.433	6.151
task_08	6.158	6.208	6.925
task_09	6.933	6.981	7.701