> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/documentation-fr-en/data-governance/sources-de-donnees.md).

# Data sources

Data sources make it possible to ingest, synchronize, and expose data securely, while ensuring traceability and access governance.

There are two main families:

1. The **external datasources,** which make it possible to directly connect data systems (SQL, S3, API, Snowflake, etc.).
2. The **Repositories** : secure SFTP servers used for file exchanges with external partners.

***

## External datasources

A Datasource is an object representing an external data source.

It contains all the information needed to connect to this source and retrieve the data in a Cleyrop Dataset.

Each datasource uses a specific connector, adapted to the source type.

### Creating a datasource

Adding a datasource follows the same principle regardless of the connector. From the Data sources > Datasourcee menu:

1. Click **Create**
2. Select the connector type
3. Enter the requested **connection information** (URL, credentials, keys, etc.)
4. **Test** the connection
5. Enter the metadata: name, description, **sensitivity**

A summary sheet is then available for each datasource:

* connection settings,
* list of imported datasets,
* connectivity status.

{% hint style="success" %}
For each data source, we recommend creating a dedicated service account with the minimum necessary privileges.
{% endhint %}

### Connection settings by connector type

#### 🗄️ Relational databases

Find below the parameters to fill in to configure a type Oracle / PostgreSQL / Microsoft SQL / MySQL datasource.

<figure><img src="/files/58627646b8335f9ffd652ee59738fa5f542b13a3" alt=""><figcaption></figcaption></figure>

**Required fields**

<table><thead><tr><th width="157.328125">Field</th><th width="264.0703125">Description</th><th>Example</th></tr></thead><tbody><tr><td><strong>Host</strong></td><td>Domain name or IP address of the database server.<br></td><td><code>db.mydomain.com</code> or <code>192.168.1.10</code><br>⚠️ <em>Do not include a protocol (e.g.: <code>jdbc:</code>) or a port.</em></td></tr><tr><td><strong>Port</strong></td><td>Network port used by the database. Pre-filled according to the connector.</td><td>Oracle: 1521<br>PostgreSQL: 5432<br>SQL Server: 1433<br>MySQL: 3306</td></tr><tr><td><strong>Database / SID / Service name</strong></td><td>Name of the target database (or SID for Oracle).</td><td><code>production_db</code></td></tr><tr><td><strong>Username</strong></td><td>Read-only account (recommended: dedicated account).</td><td><code>cleyrop_reader</code></td></tr><tr><td><strong>Password</strong></td><td>Associated password.</td><td>—</td></tr></tbody></table>

**Best practices and points of attention**

* Check that the server allows connections from the Cleyrop IP (firewall / security group).
* For Oracle, the field **Host** must include the SID or Service Name if your instance requires it (e.g.: `monserveur.com/ORCL`).
* SQL Server may require the instance name (`host\\instance`).
* If the connection fails: check whether SSL/TLS is required or not.

#### ❄️ Snowflake

<figure><img src="/files/8bce5056e9dd866e83340eea3c64f0918c228ea6" alt=""><figcaption></figcaption></figure>

<table><thead><tr><th width="166.28125">Field</th><th width="315.19140625">Description</th><th>Example</th></tr></thead><tbody><tr><td><strong>Host</strong></td><td>Snowflake instance address in format (without https://): <code>[account].snowflakecomputing.com</code></td><td><code>xy12345.snowflakecomputing.com</code></td></tr><tr><td><strong>Warehouse</strong></td><td>Compute warehouse used to run queries.</td><td><code>COMPUTE_WH</code></td></tr><tr><td><strong>Database</strong></td><td>Target database.</td><td><code>ANALYTICS_DB</code></td></tr><tr><td><strong>Username</strong></td><td>Service account.</td><td><code>CLEYROP_USER</code></td></tr><tr><td><strong>Password</strong></td><td>Password or private key.</td><td>—</td></tr><tr><td><strong>Role (optional)</strong></td><td>Snowflake role to use.</td><td><code>ANALYST_ROLE</code></td></tr></tbody></table>

**Best practices and points of attention**

* Use a dedicated role with:
  * `USAGE` on warehouse
  * `USAGE` on database & schema
  * `SELECT` on tables
* If no role is provided → default role used (often a source of access errors).
* Find your host in your account details: Account/Server URL

#### 🔌 **API**

<figure><img src="/files/bd80ec340cede6b13c4b40a8ebd3012a135429b8" alt=""><figcaption></figcaption></figure>

**Fields**

| Field                  | Description                                                                  | Example                     |
| ---------------------- | ---------------------------------------------------------------------------- | --------------------------- |
| **Base URL**           | Root URL of the API (without a specific endpoint).                           | `https://api.mydomain.com`  |
| **Endpoint (URL end)** | Path used to test the connection.                                            | `/v1/users`                 |
| **Query Params**       | Parameters added to the URL to filter or specify the request.                | `?limit=100&status=active`  |
| **Headers**            | Technical information sent with each request (authentication, format, etc.). | `Authorization: Bearer xxx` |

{% hint style="info" %}
The parameters and headers are optional but may be required if your API needs authentication or specific filters.
{% endhint %}

**HTTP methods**

The API connector supports several HTTP methods:

<table><thead><tr><th width="224.92578125">Method</th><th>Use</th></tr></thead><tbody><tr><td><strong>GET</strong></td><td>Retrieve data (default behavior)</td></tr><tr><td><strong>POST</strong></td><td>Send data in the request body</td></tr><tr><td><strong>PUT</strong></td><td>Update a complete resource</td></tr><tr><td><strong>PATCH</strong></td><td>Partially update a resource</td></tr></tbody></table>

For the POST, PUT, and PATCH methods, a **request body** can be configured as key/value pairs (JSON format).

{% hint style="info" %}
The parameters defined in the Datasource can be overridden at the Dataset level. The Dataset value then takes precedence over that of the Datasource.
{% endhint %}

**Token authentication**

If your API requires token authentication, enable the option **Enable token authentication** in the dedicated section.

<figure><img src="/files/ce481f8284587ca736dfdd74a92b44766417b777" alt=""><figcaption></figcaption></figure>

**Token request:**

| Field                              | Description                                                 | Example                               |
| ---------------------------------- | ----------------------------------------------------------- | ------------------------------------- |
| **Token URL**                      | Authentication endpoint URL                                 | `https://api.example.com/oauth/token` |
| **HTTP Method**                    | Method used to retrieve the token                           | `POST`                                |
| **Token property in the response** | Name of the field containing the token in the JSON response | `access_token`                        |

You can also configure:

* **Parameters** : parameters to send in the token request (e.g.: `grant_type: client_credentials`)
* **Headers** : headers to send in the token request (e.g.: `Content-Type: application/x-www-form-urlencoded`)
* **Body** : JSON key/value pairs to send in the token request body

**Token injection:**

Once retrieved, the token is automatically injected into API calls. Configure:

| Field                | Description                              | Example                 |
| -------------------- | ---------------------------------------- | ----------------------- |
| **Injection target** | Where to inject the token in the request | `Header`                |
| **Property name**    | Target header or parameter name          | `Authorization`         |
| **Property pattern** | Token injection format                   | `Bearer {ACCESS_TOKEN}` |

{% hint style="info" %}
The token is automatically retrieved before each collection and injected into calls according to the defined configuration.
{% endhint %}

#### 🔌 **Business Central**

**Prerequisite:**

* Have an application registered in **Microsoft Entra ID** (Azure AD), with:
  * the permissions **Application** *Dynamics 365 Business Central* (`API.ReadWrite.All`)
  * admin consent granted
  * a generated client secret

**Fields**

<table><thead><tr><th width="202.33984375">Field</th><th>Description</th></tr></thead><tbody><tr><td><strong>Environment</strong></td><td>Name of your Business Central environment as configured in the Business Central administration center.</td></tr><tr><td><strong>Tenant ID</strong></td><td>Azure AD directory (tenant) ID, available in Azure Portal > Azure Active Directory > Overview.</td></tr><tr><td><strong>Client ID</strong></td><td>Application (client) ID from your Azure AD app registration. The application must have permissions for the Dynamics 365 Business Central API.</td></tr><tr><td><strong>Password</strong></td><td>Client secret generated in Azure</td></tr></tbody></table>

{% hint style="info" %}
Test the connection so you can move to the next step.
{% endhint %}

#### **🔌 Salesforce**

**Prerequisite:**

* Have a Salesforce domain name
* Have login credentials (username and password)

**Fields**

<table><thead><tr><th width="227.2109375">Field</th><th>Description</th></tr></thead><tbody><tr><td><strong>API version</strong></td><td>Salesforce API version to use, from the list of versions supported by Cleyrop.</td></tr><tr><td><strong>Domain</strong></td><td>Salesforce organization domain name.</td></tr><tr><td><strong>Username</strong> </td><td>Consumer Key (client ID) of your Salesforce Connected App</td></tr><tr><td><strong>Password</strong> </td><td>Consumer Secret associated with your Salesforce Connected App.</td></tr></tbody></table>

{% hint style="info" %}
Test the connection so you can move to the next step.
{% endhint %}

{% hint style="warning" %}
The filters applied to the data must comply with Salesforce SOQL syntax (operators `=`, `!=`, etc.).
{% endhint %}

#### ☁️ **S3**

<figure><img src="/files/10d27f9ff4f0ca13d0cb70b811d7f40b8ba629e2" alt=""><figcaption></figcaption></figure>

| Field             | Description               | Example                              |
| ----------------- | ------------------------- | ------------------------------------ |
| **URL**           | S3 service endpoint.      | `https://s3.eu-west-1.amazonaws.com` |
| **Region**        | AWS region of the bucket. | `eu-west-1`                          |
| **Access Key**    | IAM access key.           | `AKIA...`                            |
| **Secret Key**    | Associated secret key.    | —                                    |
| **Bucket**        | Target bucket name.       | `data-production`                    |
| **Path (prefix)** | Specific subfolder.       | `exports/2025/`                      |

#### Sensitivity inheritance

{% hint style="danger" %}
The **sensitivity choice** is crucial because the children of the Datasource will inherit it.
{% endhint %}

When a Dataset is created from a datasource, it automatically inherits the sensitivity level of its source. When a Dataset is created from multiple datasources, the highest sensitivity is taken.

| Datasource sensitivity            | Sensitivity of the created dataset |
| --------------------------------- | ---------------------------------- |
| Internal & Internal               | Internal                           |
| Sensitive & Internal              | Sensitive                          |
| Restricted & Sensitive & Internal | Restricted                         |

### Monitoring and connectivity statuses

Each datasource has a connectivity status visible in the administration panel:

<table><thead><tr><th width="172.56640625">Status</th><th>Meaning</th></tr></thead><tbody><tr><td>✅ Valid</td><td>The last 5 refreshes completed without error.</td></tr><tr><td>⚠️ Watch</td><td>One of the last 5 refreshes encountered an error.</td></tr><tr><td>❌ Error</td><td>The last 5 refreshes failed.</td></tr></tbody></table>

***

## Secure SFTP repositories

Repositories are based on an SFTP server whose spaces are **isolated** and made available to **exchange files securely** between the platform and **external partners** (providers, institutions, clients…).

Each repository corresponds to an independent storage space, accessible via:

* authentication by **SSH key** (recommended) or
* authentication by **password**.

They serve as secure buffer zones (“pot zones”) for depositing or retrieving external files before integration into a dataset.

### Create a repository

Creating a repository is reserved for **Platform Managers** from the Data sources > Repositories panel.

When creating:

* choose a **clear** name (e.g. sftp-partner-finance),
* **SSH Key** : add one or more public **SSH** keys allowed (Ed25519, ECDSA, RSA or FIDO/hardware key) to secure your repository.

{% hint style="info" %}
Following the **version 4.6 upgrade and increased security measures**, it is no longer possible to create a new repository with password access. Existing repositories configured with a password remain accessible for the moment but will no longer be by 31/12/2026. We recommend migrating these repositories quickly to repositories with SSH access.
{% endhint %}

{% hint style="warning" %}
The repository name is **visible to platform users,** avoid confidential information.
{% endhint %}

Then, you will be able to:

* **browse files** : view the folder and file list from Repository > Explorer
* **edit** the repository: add an SSH key
* **delete** the repository: delete files and access

### Retention period and automatic cleanup

Files deposited in a repository are subject to **automatic cleanup** :

* by default, the retention period is **7 days** after the last file modification,
* this duration is configurable between 1 and 30 days,
* upon expiration, files are deleted, but the folder tree is preserved.

This policy ensures a **controlled storage space** while preserving the **logical structure** of the repository.

### Connection and use (e.g. FileZilla)

Authorized users can access a repository via any SFTP client (FileZilla, Cyberduck, WinSCP, etc.).

**FileZilla example:**

1. Open File > Site Manager
2. Click New site
3. Enter:
   * Protocol: SFTP – SSH File Transfer Protocol
   * Host: (address provided in the repository sheet)
   * Port: 22 (default)
   * User: repository identifier
   * Authentication method: password or private key
4. Click Quickconnect

{% hint style="info" %}
The exact connection information (host, port, user, SSH key or password) is available in Configuration > Repositories > Management > Repository Sheet.
{% endhint %}

***

## Governance best practices

* Clearly name your **datasources** (SFTP\_Partners\_2025, PostgreSQL\_Prod, etc.).
* Prefer SSH key or token authentication over password authentication.
* **Revoke** immediately the **unused accesses**.
* Monitor the **connectivity states** to avoid recurring ingestion failures.