You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/modules/airflow/pages/troubleshooting/index.adoc
+48Lines changed: 48 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,53 @@
1
1
= Troubleshooting
2
2
3
+
== Azure Blob Storage Logging
4
+
5
+
Azure's `ADLS` can be used to store Airflow task logs.
6
+
7
+
Assume a regular storage container in Azures ADLS backend: this can be accessed with either the `adls[s]` or `wasb` connector using the https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/connections/adls_v2.html[Azure Data Lake Storage Gen2 Connection] or the https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/connections/wasb.html[Microsoft Azure Blob Storage Connection] respectively.
8
+
9
+
If `ADLS` is used as a task log backend it must be accessed via `wasb` and thus the configuration in the environment should look like:
<1> This env var is only used for wasb connections.
26
+
<2> Note that the <container-name> is *not* referenced.
27
+
<3> This connection can be defined in the AirflowUI or declared as an environment variable.
28
+
29
+
Due to this open https://github.com/apache/airflow/issues/58946[issue] with Airflow, it's recommended to use `wasb-<folder-name>` rather then `wasb://<folder-name>` as using the latter option would assume the target location looks like this:
30
+
[source,text]
31
+
----
32
+
<container-name>
33
+
└── wasb:/
34
+
└── tasklogs/
35
+
└── dag_id=...
36
+
----
37
+
However the workaround will result in
38
+
[source,text]
39
+
----
40
+
<container-name>
41
+
└── wasb-tasklogs/
42
+
└── dag_id=...
43
+
----
44
+
45
+
The `Azure Blob Storage Connection` will offer the optional field `Host` which should have a value looking like this:
0 commit comments