> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/documentation-fr-en/data-governance/datasets-et-meta-donnees.md).

# Datasets and metadata

The **Datasets** are at the heart of data governance in Cleyrop.

They represent the storage and processing units for structured data: each dataset contains a set of rows and columns, from internal or external sources, and forms the basis of any transformation, analysis, or visualization flow.

***

A dataset is a set of structured data stored in tabular form (rows and columns).

It may come from:

* an external source (Excel file, CSV, database, API, etc.), or
* an internal source (another dataset, dataflow, collection from a repository).

Each dataset has:

* a **schema** (columns, types, format),
* to **metadata** (owner, source, sensitivity, labels, last update),
* and a **refresh history** tracking its updates.

## Dataset states

Each dataset can be in a functional state, reflecting its availability:

| Status              | Description                                                                |
| ------------------- | -------------------------------------------------------------------------- |
| Initialized         | The dataset has been created, but has not yet been populated.              |
| Refresh in progress | A data update is currently running.                                        |
| Valid               | The latest refresh succeeded. The dataset is usable.                       |
| Warning             | The latest refresh encountered partial errors. The dataset remains usable. |
| Error               | All refreshes have failed. The dataset is not usable.                      |

## Refreshes

A refresh updates the dataset's data from its source.

It can be triggered:

* automatically, according to a defined schedule,
* manually, by the dataset owner.

**History** : all refreshes are tracked from the *History*, making it possible to analyze failures, execution times, and processed volumes.

### Refresh statuses

| Status      | Meaning                                                         |
| ----------- | --------------------------------------------------------------- |
| Created     | The refresh has been recorded but not yet started.              |
| Pending     | The refresh is queued.                                          |
| In progress | Data import is in progress.                                     |
| Success     | The dataset was updated successfully.                           |
| Warning     | Anomalies were detected (partial files, minor inconsistencies). |
| Failure     | The refresh failed (connection, format, access rights...).      |
| Ignored     | The refresh was canceled or ignored.                            |

## Sensitivity and classification

### Sensitivity levels

Sensitivity determines the confidentiality and access conditions for the dataset.

It is defined at creation and automatically inherits from its source (datasource or another dataset).

| Level      | Description                                  |
| ---------- | -------------------------------------------- |
| Internal   | Accessible to all users                      |
| Sensitive  | Accessible to members of authorized projects |
| Restricted | Accessible to members of authorized projects |

**Inheritance from a source**

* If the datasource or source dataset is **internal**, the new dataset is **internal**.
* If the source is sensitive, the dataset inherits the sensitive level.
* If the source is restricted, it remains restricted.
* If several source datasets are used in a dataflow, the sensitivity level of the result is the most **restrictive of the sources**.

### Classification levels

Classification describes the maturity and transformation of the data:

<table><thead><tr><th width="370.046875">Level</th><th>Meaning</th></tr></thead><tbody><tr><td>🥉 Bronze</td><td>Raw, untransformed data.</td></tr><tr><td>🥈 Silver</td><td>Cleaned data ready for analysis.</td></tr><tr><td>🥇 Gold</td><td>Final data ready to be distributed (dashboards, API, AI assistants).</td></tr></tbody></table>

## Labels and metadata

Users can add labels to a dataset when it is created to make it easier to search and understand the data in the library.

* Labels are configured in the step *Metadata*.
* They can be chosen from an **existing list or freely created.**

**Example labels** :

\[finance], \[HR], \[open-data], \[production], \[geo-reference]

## Storage and export

Datasets are stored in an optimized tabular format (parquet) and can be consulted on the platform via the view *Dataset*.

Each dataset contains:

* a schema (columns, types, constraints),
* metadata (source, owner, update frequency),
* a link to its source or downstream dataflows.

{% hint style="info" %}
Exports are done in CSV format.
{% endhint %}

***

## Governance best practices

* **Name** your datasets clearly (business prefixes, version, domain).
* **Automate** refreshes to ensure data is up to date.
* Use **labels** to ensure good understanding.
* Track the **refresh history** to detect anomalies.
* **Classify** your datasets (bronze, silver, gold) to organize transformation flows.