> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/v4.7-en/data-governance/datasets-et-meta-donnees.md).

# Datasets and metadata

Datasets are the structured datasets in Cleyrop with their governance, metadata, data, and access

The **Datasets** are at the heart of data governance in Cleyrop.

They represent the storage and processing units for structured data: each dataset contains a set of rows and columns, from internal or external sources, and forms the basis of any transformation, analysis, or visualization flow.

***

A dataset is a set of structured data stored in tabular form (rows and columns).

It can come from:

* an external source (Excel file, CSV, database, API, etc.), or
* an internal source (another dataset, dataflow, collection from a repository).

Each dataset has:

* a **schema** (columns, types, format),
* of **metadata** (owner, origin, sensitivity, labels, last update),
* and a **refresh history** tracking its updates.

## Dataset states

Each dataset can be in a functional state, reflecting its availability:

| State       | Description                                                              |
| ----------- | ------------------------------------------------------------------------ |
| Initialized | The dataset has been created, but has not yet been populated.            |
| Refreshing  | A data update is currently running.                                      |
| Valid       | The last refresh succeeded. The dataset is usable.                       |
| Warning     | The last refresh encountered partial errors. The dataset remains usable. |
| Error       | All refreshes have failed. The dataset is not usable.                    |

## Refreshes

A refresh updates the dataset’s data from its source.

It can be triggered:

* automatically, according to a defined schedule,
* manually, by the dataset owner.

**History** : all refreshes are tracked from the tab *History*, allowing failures, run durations, and processed volumes to be analyzed.

### Refresh statuses

| Status      | Meaning                                                         |
| ----------- | --------------------------------------------------------------- |
| Created     | The refresh has been recorded but not yet started.              |
| Pending     | The refresh is queued.                                          |
| In progress | Data import is in progress.                                     |
| Success     | The dataset was successfully updated.                           |
| Warning     | Anomalies were detected (partial files, minor inconsistencies). |
| Failure     | The refresh failed (connection, format, access right…).         |
| Skipped     | The refresh was canceled or skipped.                            |

## Sensitivity and classification

### Sensitivity levels

Sensitivity determines the confidentiality and access conditions of the dataset.

It is defined at creation and automatically inherited from its source (datasource or another dataset).

| Level      | Description                                  |
| ---------- | -------------------------------------------- |
| Internal   | Accessible to all users                      |
| Sensitive  | Accessible to members of authorized projects |
| Restricted | Accessible to members of authorized projects |

**Inheritance from a source**

* If the datasource or source dataset is **internal**, the new dataset is **internal**.
* If the origin is sensitive, the dataset inherits the sensitive level.
* If the origin is restricted, it remains restricted.
* If several source datasets are used in a dataflow, the sensitivity level of the result is the most **restrictive of the sources**.

### Classification levels

Classification describes the maturity and transformation of the data:

<table><thead><tr><th width="370.046875">Level</th><th>Meaning</th></tr></thead><tbody><tr><td>🥉 Bronze</td><td>Raw, untransformed data.</td></tr><tr><td>🥈 Silver</td><td>Cleaned data ready for analysis.</td></tr><tr><td>🥇 Gold</td><td>Final data ready to be published (API, Agents, business applications).</td></tr></tbody></table>

## Labels and metadata

Users can add labels to a dataset when creating it to make it easier to search for and understand the data in the library.

* Labels are configured in the step *Metadata*.
* They can be chosen from an **existing list or created freely.**

**Label examples** :

\[finance], \[HR], \[open-data], \[production], \[geo-reference]

## Storage and export

Datasets are stored in optimized tabular format (parquet) and can be viewed on the platform via the *Dataset*.

Each dataset contains:

* a schema (columns, types, constraints),
* metadata (origin, owner, update frequency),
* a link to its source or descendant dataflows.

{% hint style="info" %}
Exports are made in CSV format.
{% endhint %}

***

## Governance best practices

* **Name** your datasets clearly (business prefixes, version, domain).
* **Automate** refreshes to ensure data is up to date.
* Use **labels** to ensure good understanding.
* Track the history of **refreshes** to detect anomalies.
* **Classify** your datasets (bronze, silver, gold) to organize transformation flows.