> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/v4.7-en/data-and-ai-project/dataflow/creer-un-dataflow/configurer-un-dataset-de-sortie.md).

# Configure an output Dataset

Master governance, schema, and updates of produced data

## Configure metadata

For transformations, you can configure the output Dataset information:

* **Name**
* **Description**
* **Owner**
* **Sensitivity** :
  * If the dataset comes from at least one input Dataset, the output Dataset will have the highest sensitivity of its parents
  * Otherwise, you can choose its sensitivity level. In that case, it will no longer be editable after deployment to production
* **Classification** : bronze, silver, gold
* **Labels** : labels are free keywords that you can associate with a dataset to make it easier to find and organize in the library and catalog. You can choose a label from the existing list or add one by typing it in. You can manage labels by clicking the ... next to a label name
* **Refresh mode**
  * *replace* → old data is replaced
  * *append* → new data is added
* **Schema policy** : **Schema merge** (default), **Schema replacement**, **Permissive fixed schema** or **Strict fixed schema**. It defines how the dataset reacts when a run produces a different schema (extra or missing columns, type changes). The details of each policy are described in [Schema policy](/docs/v4.7-en/data-and-ai-project/datasets/politique-de-schema.md).

{% hint style="warning" %}
The **refresh mode** cannot be changed after the first run of the parent transformation.

The **schema policy** is defined on the `main` branch as long as the Dataflow is not deployed to production; it can no longer be changed afterwards.
{% endhint %}

### Add descriptions to columns

Each column in a dataset can be documented with a text description to explain its business meaning, transformation logic, or specific constraints.

For datasets coming from Dataflow, descriptions can be edited in Draft mode. Once the Dataflow is in production, they can only be changed from the dataset details page after merging to production, and not in development branches.

## Reference the dataset in the library

By default, an output dataset is not visible in the project library. You can explicitly choose to reference it to make it available to other users.

The **Reference** button is available:

* directly from the Dataflow graph, above the output dataset,
* or from the output dataset details page.

You can also reference several output datasets at once when **deploying** the Dataflow: a list of all output datasets is displayed, with an option **Select all**.

The referenced dataset appears in the library once the Dataflow is deployed to production.

{% hint style="warning" %}
This action is irreversible.
{% endhint %}

## Reset data

It is possible to reset the data of a dataset without deleting its schema, only when it is handled through a draft Dataflow or a branch. This function is particularly useful when you iterate on a transformation with an output Dataset in *Append*

**How to perform this action**

* Go to the Dataflow branch → click on the Dataset to reset
* A button <i class="fa-arrows-rotate-reverse">:arrows-rotate-reverse:</i> Reset data appears above the Dataset

On validation

* A deletion script is executed (PySpark or Python depending on the cluster).
* The data is deleted without affecting the schema.
* A status update is displayed in the Dataflow execution logs.

#### Restrictions

* Unable to reset a **dataset deployed to production**.
* Not possible from the dataset page alone: the action is only available in the Dataflow.
* The branch owner is the only person authorized to perform the action.

## Manage the output dataset schema

The schema of a dataset (columns and types) is **shared across all branches of the same Dataflow**.

The way the dataset reacts when a run produces a different schema (extra or missing columns, type changes) depends on its **schema policy**, chosen in the output dataset metadata on the `main` before production deployment: see [Schema policy](/docs/v4.7-en/data-and-ai-project/datasets/politique-de-schema.md).

* Once a column is created, **its type can no longer be changed** without deleting the column or the dataset.
* The schema acts as a **shared structural contract:** all branches must keep the same definition for the same column.
* If a **column needs to evolve** (e.g. int → string), you must either delete the column or create a new dataset with the correct type.

{% hint style="info" %}
There can be **local columns** within a branch, but they all inherit the same global dataset schema.
{% endhint %}

Some libraries (Pandas, Spark, Polars) do not have the same typing conventions.

### Supported types and schema compatibility

<table data-search="false"><thead><tr><th width="112.55078125">Cleyrop type</th><th width="107.2734375">Pandas</th><th>Pandas-on-Spark (PySpark)</th><th width="122.5390625">Spark SQL</th><th width="95.44921875">Polars</th><th>Notes</th></tr></thead><tbody><tr><td>Integer</td><td>int32 / Int32</td><td>IntegerType()</td><td>int</td><td>Int32</td><td>32-bit integer.</td></tr><tr><td>Long</td><td>int64 / Int64</td><td>LongType()</td><td>bigint</td><td>Int64</td><td>64-bit integer. Compatible as long as there are no null values (prefer nullable types).</td></tr><tr><td>Float</td><td>float32</td><td>FloatType()</td><td>float</td><td>Float32</td><td>Single-precision floating point.</td></tr><tr><td>Double</td><td>float64</td><td>DoubleType()</td><td>double</td><td>Float64</td><td>Double-precision floating point, watch out for implicit conversions (rounding).</td></tr><tr><td>Decimal</td><td>object (Decimal)</td><td>DecimalType(19,2)</td><td>decimal(19,2)</td><td>Decimal(38,s)</td><td>Exact precision (19 digits total, 2 max after the decimal point) — no floating-point rounding.</td></tr><tr><td>Boolean</td><td>bool</td><td>BooleanType()</td><td>boolean</td><td>Boolean</td><td>⚠️ None in Pandas can become a non-supported null in Spark without casting.</td></tr><tr><td>String<br>Character string</td><td>object / string</td><td>StringType()</td><td>string</td><td>Utf8</td><td>Prefer string (pandas >= 1.5) for compatibility.</td></tr><tr><td>Date</td><td>datetime64[ns]</td><td>DateType()</td><td>date</td><td>Date</td><td>Date only, no time.</td></tr><tr><td>Timestamp</td><td>datetime64[ns]</td><td>TimestampNTZType()</td><td>timestamp_ntz</td><td>Datetime("us", None)</td><td>Date and time, no time zone.</td></tr><tr><td>Timestamp with time zone</td><td>datetime64[ns, tz]</td><td>TimestampType()</td><td>timestamp</td><td>Datetime("us", "UTC")</td><td>Date and time with explicit time zone.</td></tr><tr><td>Binary</td><td>bytes / object</td><td>BinaryType()</td><td>binary</td><td>Binary</td><td>Binary data (hashes, blobs, inline files). Manual type selection required (no automatic detection).</td></tr><tr><td>Array</td><td>object (list)</td><td>ArrayType(StringType())</td><td>array&#x3C;...></td><td>List(inner)</td><td>Array of typed values.</td></tr><tr><td>Struct</td><td>object (dict)</td><td>StructType([])</td><td>struct&#x3C;...></td><td>Struct({...})</td><td>Nested structure. Detected automatically.</td></tr><tr><td>Map</td><td>object (dict)</td><td>MapType(k,v)</td><td>map&#x3C;k,v></td><td>List(Struct)</td><td>Nested keys/values. ⚠️ Polars does not have a native Map type: it uses a list of structs.</td></tr></tbody></table>

### Delete a column <a href="#creer-un-dataset-depuis-un-dataframe-pandas-version-polars" id="creer-un-dataset-depuis-un-dataframe-pandas-version-polars"></a>

When preparing an output dataset, it is common to want to remove an unnecessary column (e.g.: manipulation error, empty column, test, obsolete attribute).

This action irreversibly deletes the column from the schema and its associated data.

**How to perform this action**

* Go to the Dataflow → output dataset preview panel.
* A button <i class="fa-circle-trash">:circle-trash:</i> Delete appears for each column.
* Select the desired column and confirm

{% hint style="warning" %}
It is possible to delete a column in the schema if:

* The Dataflow is in **draft**
* After production deployment, possible in a **branch** of the Dataflow only **if the column is not** **in the schema of the `main`**
  {% endhint %}

#### Restrictions

These rules are important to ensure data consistency and security.

| Constraint                                                  | Detail                                               |
| ----------------------------------------------------------- | ---------------------------------------------------- |
| Cannot delete a column from a production output dataset     | Production datasets are immutable.                   |
| Cannot delete a column present in the deployed main dataset | Even if you are in a branch.                         |
| Cannot delete the **last column** of a dataset              | Iceberg format limitation.                           |
| Deletion possible in any branch                             | But only by the dataset owner (creator or DF owner). |

In case of an error (protected column or last column), an explanatory popup informs the user.

### Column type mapping <a href="#creer-un-dataset-depuis-un-dataframe-pandas-version-polars" id="creer-un-dataset-depuis-un-dataframe-pandas-version-polars"></a>

#### Pandas <> Pyspark <a href="#creer-un-dataset-depuis-un-dataframe-pandas-version-pyspark" id="creer-un-dataset-depuis-un-dataframe-pandas-version-pyspark"></a>

You can create a PySpark Pandas dataframe from a Pandas dataframe with the function: `ps.from_pandas(pdf)`.

Be careful, using this function can lead to column type issues that will not be the same between the Pandas and PySpark Pandas dataframe.

To make sure you get the same dataframe after conversion, we recommend using the code:

{% code title="column type mapping between pandas and pyspark pandas" %}

```python
df = <your-pandas-dataframe>

def map_pandas_dtype_to_spark(pandas_dtype):
    if pd.api.types.is_string_dtype(pandas_dtype):
        return 'string'
    elif pd.api.types.is_numeric_dtype(pandas_dtype):
        if pd.api.types.is_integer_dtype(pandas_dtype):
            return 'int'
        elif pd.api.types.is_float_dtype(pandas_dtype):
            return 'float'
    elif pd.api.types.is_datetime64_any_dtype(pandas_dtype):
        return 'datetime64[ns]'
    else:
        return 'string'

# Get the column types for the Spark schema
column_types = {col: map_pandas_dtype_to_spark(df[col].dtype) for col in df.columns}


# Convert a Pandas DataFrame to a Spark DataFrame with the specified column types
psdf = ps.DataFrame(data=df.to_dict(), columns=df.columns, dtype=str).astype(column_types)
```

{% endcode %}

#### Pandas <> Polars

You can create a Polars LazyFrame (which will automatically be transformed into a DataFrame when the transformation runs) from a Pandas dataframe with the function: `polars.from_pandas(dataframe).lazy()`. ([Polars documentation](https://docs.pola.rs/api/python/stable/reference/api/polars.from_pandas.html))

Be careful, using this function can lead to column type issues that will not be the same between the Pandas and Polars dataframe.

To make sure you get the same dataframe after conversion, we recommend using the code:

{% code title="column type mapping between pandas and polars" %}

```python
df = <your-pandas-dataframe>

def map_pandas_dtype_to_polars(pandas_dtype):
    if pd.api.types.is_string_dtype(pandas_dtype):
        return polars.String
    elif pd.api.types.is_numeric_dtype(pandas_dtype):
        if pd.api.types.is_integer_dtype(pandas_dtype):
            return polars.Int64
        elif pd.api.types.is_float_dtype(pandas_dtype):
            return polars.Float64
    elif pd.api.types.is_datetime64_any_dtype(pandas_dtype):
        return polars.Datetime("ns")
    else:
        return polars.String

# Get the column types for the Polars schema
schema_overrides = {col: map_pandas_dtype_to_polars(df[col].dtype) for col in df.columns}


# Convert a Pandas DataFrame to a Polars DataFrame with the specified column types
lf = polars.from_pandas(df, schema_overrides=schema_overrides, nan_to_null=True, rechunk=True,).lazy()
```

{% endcode %}

## Best practices

* Prefer **explicit types** (string, int64, float64) to avoid automatic conversions.
* Make sure the output schema is correct and consistent with the other branches.