> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/documentation-fr-en/data-and-ai-project/dataflow/analyser-les-executions-et-logs.md).

# Analyze runs and logs

Every Dataflow run — whether manual, scheduled, or event-driven — generates detailed logs. These logs let you track the execution step by step, spot any errors, and diagnose problems.

***

## Accessing run details

From the Dataflow list, the History tab, or from the Executions tab in the action bar of the branch of the Dataflow you are interested in:

* Choose the run to analyze
* Each run contains:
  * the **overall status** (success, failure, cancelled),
  * the **date and duration** of execution,
  * the **cluster used**,
  * the **trigger** (manual, recurrence, event)
  * the **list of executed steps and associated logs**
* You can click the transformation to analyze to view the logs. You can also display the executed code, the parent datasets, or preview the data

{% hint style="info" %}
You can find your branch's run by typing its name in the search bar
{% endhint %}

<figure><img src="/files/74a34474646e021b17fdd59b235d88fb26ac5212" alt="" width="563"><figcaption></figcaption></figure>

From the Dataflow, you can also click each transformation.

#### Queue in the cluster <a href="#position-dans-la-file-dattente" id="position-dans-la-file-dattente"></a>

When a Dataflow is launched, its status changes to `Pending` : it joins the **cluster queue.**

The **position** displayed under the status indicates its execution order. If the position is number 1, the Dataflow will be executed first.

The queue is **shared by the entire cluster** :

* If several sessions are available, Dataflows can run in parallel (one session per Dataflow).
* If only one session is active, the following Dataflows wait for the current execution to finish.

To **reduce the waiting time**, the Platform Manager can **add a session**. This is taken into account almost immediately, making it possible to unblock pending executions.

## View results

After each run (manual or scheduled), you can **view the results:** click on an **output dataset** to display its **preview** (up to the first 100 rows) as well as its full schema (columns, types, nullable, etc.).

{% hint style="info" %}
If the **preview does not appear** when it should, it is often a schema issue
{% endhint %}

## Analyze logs

### Logging best practices

* The `print()` of Python transformations are clearly visible in the logs. However, avoid overusing them (especially on entire DataFrames) to keep the output readable.
* You can configure the level of detail of the logs in your transformation:

```python
import logging 
root_logger = logging.getLogger()
root_logger.setLevel(logging.INFO)
```

### Spark cluster landmarks

**Quick diagnosis** : any error between “Iceberg catalog: cleyrop” (start) and “Table … initialized” (end of transformation) comes from the transformation code.

{% code title="log start" %}

```log
INFO - Cleyrop client initialized.
… INFO - Iceberg catalog: cleyrop
```

{% endcode %}

{% code title="log end" %}

```log
… INFO - Table "<techname>.branch_<branch>" initialized
… INFO - Writing dataframe to table "<techname>"
… INFO - Writing mode: "WriteMode.OVERWRITE"
… INFO - Converting PySpark pandas dataframe to Spark dataframe
```

{% endcode %}

### Copy logs

You can **copy logs** from the interface (right-click > Copy).

This makes it possible to share the diagnosis with another user or with technical support.

## Known errors & fixes

#### **Error: `executor lost` or OOM**

* **Cause** : comes from the **memory resource** (Out Of Memory)
* **Possible fixes** :
  * Increase the memory (RAM) or the number of executors allocated to the cluster.
  * If certain steps require heavy use of Pandas (local in-memory processing), it is better to isolate them in a separate Dataflow to limit the load on the main cluster

#### **Error: p**no output Dataset preview or schema compatibility

* **Cause** **likely** : schema compatibility, encoding issue
* **Possible fixes** :
  * Check the [**compatibility**](/docs/documentation-fr-en/data-and-ai-project/dataflow/creer-un-dataflow/configurer-un-dataset-de-sortie.md#table-de-compatibilite-schema) between engines / library used

#### Other errors

```log
java.lang.UnsupportedOperationException: Not a supported type: void
```

* **Cause** : an entirely empty column.
* **Possible fixes** :
  * Fill the column using fillna() (e.g. df\['col'].fillna(0)), or
  * Explicitly type the column using astype() (e.g. df.astype({'col': "string"})).

<pre class="language-log"><code class="lang-log"><strong>TypeError: Dataframe must be a PySpark pandas or Spark dataframe
</strong></code></pre>

* **Cause** : the Spark transformation does not return a pandas-on-Spark or Spark DataFrame.
* **Fix** : return a pyspark.pandas.DataFrame (or a pyspark.sql.DataFrame) and use pyspark.pandas instead of pandas.

```log
TypeError: 'Dataframe' object has no attribute 'collect'
```

* **Cause** : the Python transformation does not return a Polars LazyFrame.
* **Fix** : return a polars.lazyframe.frame.LazyFrame (use .lazy() if you start from a Polars DataFrame).

***

## Diagnostic best practices

* Read the logs **from bottom to top** : the most recent error is often the root cause.
* If the error concerns a table, check the **schema consistency** (missing types or columns).
* Add custom logs in your transformations with `print()` or `logger.info()` to provide context for the run.