> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/v4.7-en/data-and-ai-project/datasets/politique-de-schema.md).

# Schema policy

Choose how a dataset behaves when new data does not have exactly the same schema as it does.

The **schema policy** defines what happens when new data (new file, refresh of a data source, execution of a Dataflow) does not have exactly the same schema as the dataset: extra or missing columns, type changes.

It is prompted at the step **Refresh** when creating any dataset (local import, data source, multiple import) and in the metadata of a Dataflow output dataset.

***

## The four policies

<table><thead><tr><th width="230">Policy</th><th>Behavior</th></tr></thead><tbody><tr><td><strong>Schema merge</strong> (default)</td><td>New columns are added, missing columns are kept with empty values. The type of existing columns cannot change.</td></tr><tr><td><strong>Schema replacement</strong> <mark style="background-color:blue;">beta</mark></td><td>The schema adapts to each import: new columns are added and columns absent from the new data are removed. For existing columns, a type change is accepted only if the conversion is done without data loss.</td></tr><tr><td><strong>Permissive fixed schema</strong> <mark style="background-color:blue;">beta</mark></td><td>The schema remains fixed: extra columns are ignored and missing columns remain empty. The type of existing columns cannot change.</td></tr><tr><td><strong>Strict fixed schema</strong> <mark style="background-color:blue;">beta</mark></td><td>The schema remains fixed: column names and types must match exactly, otherwise the import is rejected.</td></tr></tbody></table>

In all cases, an impossible type change (for example, a text column to a number) rejects the file: the refresh record indicates the column in question.

<figure><img src="https://4159280578-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMHNFH6aA4j5ucH4XFBXy%2Fuploads%2Fgit-blob-d7636cbf44361736cfec4a2d8808899b35d7519c%2F4.7-bulk-schema-policy.png?alt=media" alt="Sélecteur de politique de schéma à l&#x27;étape Rafraîchissement"><figcaption></figcaption></figure>

{% hint style="warning" %}
The schema policy is chosen **at creation** of the dataset and can no longer be modified afterward, including for a local import. It is visible in the **Information** tab of the dataset details page. To change it, create a new dataset.

For a Dataflow output dataset, it is defined on the branch `main` as long as the Dataflow is not deployed to production.
{% endhint %}

## Multiple files for the same dataset

When several files feed **a single** dataset (local import of multiple files, multi-file refresh, SFTP or SharePoint deposit), processing depends on the **refresh mode** :

* **Replacement** (*replace*): the files are first aggregated together according to the logic of **schema merge** (columns with the same name and same type, missing columns filled with empty values) to form a single intermediate table. It is this table that is then compared to the dataset: the refresh succeeds or fails depending on the dataset's schema policy and the consistency between the intermediate table and the target schema.
* **Append** (*append*): the files are processed **one by one** ; each is compared to the dataset according to its schema policy. A rejected file does not prevent the others from being imported.

{% hint style="info" %}
To get one dataset per file rather than a single dataset, use[multiple import](/docs/v4.7-en/data-and-ai-project/datasets/gerer-les-datasets.md#creer-plusieurs-datasets-en-une-fois-import-multiple).
{% endhint %}

## Schema policy and manual schema modification

After [renamed or deleted a column](/docs/v4.7-en/data-and-ai-project/datasets/modifier-le-schema-dun-dataset.md), a new import of the original file behaves according to the policy:

* with **Schema merge**, the renamed or deleted column is recreated, because the file header no longer matches any column in the dataset;
* with **Strict fixed schema**, the import is rejected.

Adjust your source files after a schema change.

***

## Best practices

* **Schema merge** is suitable for most uses: it absorbs columns that appear without breaking refreshes.
* **Strict fixed schema** guarantees compliance for regulatory or contractual data: any deviation is detected immediately.
* **Schema replacement** is suited to sources whose structure changes often, bearing in mind that missing columns are removed from the dataset.