> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/documentation-fr-en/data-and-ai-project/dataflow/creer-un-dataflow/configurer-un-dataset-de-sortie.md).

# Configure an output Dataset

## Configure metadata

For transformations, you can configure the output Dataset information:

* **Name**
* **Description**
* **Owner**
* **Sensitivity** :
  * If the dataset comes from at least one input Dataset, the output Dataset will have the maximum sensitivity of its parents
  * Otherwise you can choose its sensitivity level. In that case, it cannot be changed after production deployment
* **Classification** : bronze, silver, gold
* **Labels** : labels are free keywords that you can associate with a dataset to make it easier to search and organize in the library and catalog. You can choose a label from the list of existing ones or add one by typing it. You can manage labels by clicking the ... next to a label name
* **Refresh mode**
  * *replace* → old data are replaced
  * *append* → new data are added

{% hint style="warning" %}
The **refresh mode**t cannot be changed after the first execution of the parent transformation.

**During a refresh, the applied schema policy is *****schema merging*****. Concretely:**

* **Matching existing columns:**\
  Columns with the *same name* as those in the Dataset schema must also have the *same type* ➝ *If the type does not match, the file is rejected.*
* **Additional or missing columns:**\
  Extra or missing columns are allowed ➝ *Missing columns will simply be filled with empty values.*
  {% endhint %}

### Add descriptions to columns

Each column of a dataset can be documented with a text description to explain its business meaning, transformation logic, or specific constraints.

For datasets from Dataflow, descriptions can be edited in Draft mode. Once the Dataflow is in production, they can only be changed from the dataset page after the production merge, and not in development branches.

## Reference the dataset in the library

By default, an output dataset is not visible in the project library. You can explicitly choose to reference it to make it available to other users.

The **Reference** button is accessible:

* directly from the Dataflow graph, above the output dataset,
* or from the output dataset detail page.

You can also reference several output datasets at once when **deploying** the Dataflow: a list of all output datasets is displayed, with an option **Select all**.

The referenced dataset appears in the library once the Dataflow is deployed to production.

{% hint style="warning" %}
This action is irreversible.
{% endhint %}

## Reset data

It is possible to reset a dataset's data without deleting its schema, only when it is handled via a draft Dataflow or a branch. This function is especially useful when you are iterating on a transformation with an output Dataset in mode *Append*

**How to perform this action**

* Go to the Dataflow branch → click the Dataset to reset
* A button <i class="fa-arrows-rotate-reverse">:arrows-rotate-reverse:</i> Reset data appears above the Dataset

On validation

* A deletion script is executed (PySpark or Python depending on the cluster).
* The data is deleted without touching the schema.
* A status update is displayed in the Dataflow execution logs.

#### Restrictions

* Cannot reset a **dataset deployed to production**.
* Not possible from the dataset page alone: the action is only available in the Dataflow.
* The branch owner is the only one authorized to perform the action.

## Manage the output dataset schema

A dataset schema (columns and types) is **shared across all branches of the same Dataflow**.

* Once a column is created, **its type can no longer be changed** without deleting the column or the dataset.
* The schema acts as a **shared structural contract:** all branches must keep the same definition for the same column.
* If a **column needs to change** (e.g. int → string), you must either delete the column or create a new dataset with the correct type.

{% hint style="info" %}
There may be **local columns** within a branch, but they all inherit the same global dataset schema.
{% endhint %}

Some libraries (Pandas, Spark, Polars) do not use the same typing conventions.

### Supported types and schema compatibility

<table data-search="false"><thead><tr><th width="112.55078125">Cleyrop type</th><th width="107.2734375">Pandas</th><th>Pandas-on-Spark (PySpark)</th><th width="122.5390625">Spark SQL</th><th width="95.44921875">Polars</th><th>Notes</th></tr></thead><tbody><tr><td>Integer</td><td>int32 / Int32</td><td>IntegerType()</td><td>int</td><td>Int32</td><td>32-bit integer.</td></tr><tr><td>Long</td><td>int64 / Int64</td><td>LongType()</td><td>bigint</td><td>Int64</td><td>64-bit integer. Compatible as long as there are no null values (prefer nullable types).</td></tr><tr><td>Float</td><td>float32</td><td>FloatType()</td><td>float</td><td>Float32</td><td>Single-precision floating point.</td></tr><tr><td>Double</td><td>float64</td><td>DoubleType()</td><td>double</td><td>Float64</td><td>Double-precision floating point, beware of implicit conversions (rounding).</td></tr><tr><td>Decimal</td><td>object (Decimal)</td><td>DecimalType(19,2)</td><td>decimal(19,2)</td><td>Decimal(38,s)</td><td>Exact precision (19 digits total, 2 maximum after the decimal point) — no floating-point rounding.</td></tr><tr><td>Boolean</td><td>bool</td><td>BooleanType()</td><td>boolean</td><td>Boolean</td><td>⚠️ None in Pandas can become an unsupported null in Spark without casting.</td></tr><tr><td>String<br>Character string</td><td>object / string</td><td>StringType()</td><td>string</td><td>Utf8</td><td>Prefer string (pandas >= 1.5) for compatibility.</td></tr><tr><td>Date</td><td>datetime64[ns]</td><td>DateType()</td><td>date</td><td>Date</td><td>Date only, without time.</td></tr><tr><td>Timestamp</td><td>datetime64[ns]</td><td>TimestampNTZType()</td><td>timestamp_ntz</td><td>Datetime("us", None)</td><td>Date and time, without time zone.</td></tr><tr><td>Timestamp with time zone</td><td>datetime64[ns, tz]</td><td>TimestampType()</td><td>timestamp</td><td>Datetime("us", "UTC")</td><td>Date and time with explicit time zone.</td></tr><tr><td>Binary</td><td>bytes / object</td><td>BinaryType()</td><td>binary</td><td>Binary</td><td>Binary data (hashes, blobs, inline files). Manual type selection required (no automatic detection).</td></tr><tr><td>Array</td><td>object (list)</td><td>ArrayType(StringType())</td><td>array&#x3C;...></td><td>List(inner)</td><td>Array of typed values.</td></tr><tr><td>Struct</td><td>object (dict)</td><td>StructType([])</td><td>struct&#x3C;...></td><td>Struct({...})</td><td>Nested structure. Detected automatically.</td></tr><tr><td>Map</td><td>object (dict)</td><td>MapType(k,v)</td><td>map&#x3C;k,v></td><td>List(Struct)</td><td>Nested keys/values. ⚠️ Polars has no native Map type: it uses a list of structures.</td></tr></tbody></table>

### Delete a column <a href="#creer-un-dataset-depuis-un-dataframe-pandas-version-polars" id="creer-un-dataset-depuis-un-dataframe-pandas-version-polars"></a>

When preparing an output dataset, it is common to want to remove an unnecessary column (e.g. handling error, empty column, test, obsolete attribute).

This action irreversibly deletes the column from the schema and the associated data.

**How to perform this action**

* Go to the Dataflow → output dataset preview panel.
* A button <i class="fa-circle-trash">:circle-trash:</i> Delete appears for each column.
* Select the desired column and confirm

{% hint style="warning" %}
It is possible to delete a column in the schema if:

* The Dataflow is in **draft**
* After production deployment, possible in a **branch** of the Dataflow only **if the column is not** **in the schema of the `main`**
  {% endhint %}

#### Restrictions

These rules are important to ensure data consistency and security.

| Constraint                                                  | Detail                                               |
| ----------------------------------------------------------- | ---------------------------------------------------- |
| Cannot delete a column from an output dataset in production | Production datasets are immutable.                   |
| Cannot delete a column present in the deployed main dataset | Even if you are in a branch.                         |
| Cannot delete the **last column** of a dataset              | Iceberg format limitation.                           |
| Deletion possible in any branch                             | But only by the dataset owner (creator or DF owner). |

In case of an error (protected column or last column), an explanatory popup informs the user.

### Column type mapping <a href="#creer-un-dataset-depuis-un-dataframe-pandas-version-polars" id="creer-un-dataset-depuis-un-dataframe-pandas-version-polars"></a>

#### Pandas <> PySpark <a href="#creer-un-dataset-depuis-un-dataframe-pandas-version-pyspark" id="creer-un-dataset-depuis-un-dataframe-pandas-version-pyspark"></a>

You can create a PySpark Pandas dataframe from a Pandas dataframe with the function: `ps.from_pandas(pdf)`.

Warning, using this function can lead to column type issues that will not be the same between the Pandas dataframe and PySpark Pandas.

To be sure of getting the same dataframe after conversion, we recommend using the code:

{% code title="column type mapping between pandas and pyspark pandas" %}

```python
df = <your-pandas-dataframe>

def map_pandas_dtype_to_spark(pandas_dtype):
    if pd.api.types.is_string_dtype(pandas_dtype):
        return 'string'
    elif pd.api.types.is_numeric_dtype(pandas_dtype):
        if pd.api.types.is_integer_dtype(pandas_dtype):
            return 'int'
        elif pd.api.types.is_float_dtype(pandas_dtype):
            return 'float'
    elif pd.api.types.is_datetime64_any_dtype(pandas_dtype):
        return 'datetime64[ns]'
    else:
        return 'string'

# Get column types for the Spark schema
column_types = {col: map_pandas_dtype_to_spark(df[col].dtype) for col in df.columns}


# Convert a Pandas DataFrame into a Spark DataFrame with the specified column types
psdf = ps.DataFrame(data=df.to_dict(), columns=df.columns, dtype=str).astype(column_types)
```

{% endcode %}

#### Pandas <> Polars

You can create a Polars LazyFrame (which will be automatically converted into a DataFrame when the transformation is executed) from a Pandas dataframe with the function: `polars.from_pandas(dataframe).lazy()`. ([Polars documentation](https://docs.pola.rs/api/python/stable/reference/api/polars.from_pandas.html))

Warning, using this function can lead to column type issues that will not be the same between the Pandas dataframe and Polars.

To be sure of getting the same dataframe after conversion, we recommend using the code:

{% code title="column type mapping between pandas and polars" %}

```python
df = <your-pandas-dataframe>

def map_pandas_dtype_to_polars(pandas_dtype):
    if pd.api.types.is_string_dtype(pandas_dtype):
        return polars.String
    elif pd.api.types.is_numeric_dtype(pandas_dtype):
        if pd.api.types.is_integer_dtype(pandas_dtype):
            return polars.Int64
        elif pd.api.types.is_float_dtype(pandas_dtype):
            return polars.Float64
    elif pd.api.types.is_datetime64_any_dtype(pandas_dtype):
        return polars.Datetime("ns")
    else:
        return polars.String

# Get column types for the Polars schema
schema_overrides = {col: map_pandas_dtype_to_polars(df[col].dtype) for col in df.columns}


# Convert a Pandas DataFrame into a Polars DataFrame with the specified column types
lf = polars.from_pandas(df, schema_overrides=schema_overrides, nan_to_null=True, rechunk=True,).lazy()
```

{% endcode %}

## Best practices

* Prefer **explicit types** (string, int64, float64) to avoid automatic conversions.
* Make sure the output schema is correct and consistent with the other branches.