> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/v4.7-en/data-and-ai-project/corpus-de-documents.md).
# Document corpus
Build document knowledge bases for your Agents and Le Druide.
A corpus is a set of unstructured text documents serving as a knowledge base for Agents and Le Druide.
The documents are **vectorized** and stored in a database optimized for semantic search thanks to the **RAG** (Retrieval-Augmented Generation).
A corpus thus makes it possible to extract insights and generate content in natural language from internal documents.
Corpora are managed from the **Library** of the project, item **Corpus**.
***
## Create a corpus
1. From the project Library, open **Corpus** then click on **Create**.
2. **Source** : choose the origin of the documents:
* **Work Data** : the files placed in the project's Work Data;
* a **SharePoint datasource** added to the project, to use directly the files from a SharePoint site (see [Data Sources](/docs/v4.7-en/data-governance/sources-de-donnees.md#ms-sharepoint)). As with any datasource, it must first be **authorized in the project** : from **Library > Datasources**, click on **Add a Datasource** then **Request access** ; the datasource appears as a source once the request is accepted.
3. **Files** : select the files or folders to include. The entire tree of a selected folder is added. Files whose format is not supported are grayed out.
4. For a SharePoint source, also choose the **storage mode** and the **update mode** (see below).
5. **Information** : give the corpus a name, an owner, and a description.
6. Click on **Create** : vectorization starts. Once the status changes to *Valid*, the corpus becomes usable in the project.
{% hint style="warning" %}
For a source **Work Data**, **placed in Work Data** before creating the corpus.
Supported formats: **PDF, DOCX, TXT, PPTX, CSV, TSV, TAB**
For a Word file (`.docx`), only paragraph text is indexed: the content of the **tables**, the **headers** and the **footers** is not taken into account. Copy the information found there into the body of the document.
The embedding model used is the platform's default model (BGE-M3 by default). It is indicated in the corpus sheet.
{% endhint %}
{% hint style="danger" %}
A corpus is linked to the embedding model with which it was created. If the Platform Manager changes the platform's default embedding model, existing corpora can no longer be used in Le Druide as of today: they must be recreated.
{% endhint %}
#### Storage mode (SharePoint source)
* **Duplicate files** (default): a copy of the file is kept on the platform. The documents cited as sources remain accessible at all times, even if the original is moved or deleted. Access rights are managed in the platform, independently of the source.
* **Point to the source** : no copy is kept, only indexed excerpts are stored on the platform. The full document is read directly from SharePoint: if it has been moved or deleted, or if the user does not have access to it, they will not be able to consult it in full.
The storage mode cannot be changed after creation.
#### Update mode (SharePoint source)
* **Manual** : you manage the documents added to or removed from the corpus yourself.
* **Synchronized** : files added, modified, or deleted on SharePoint are reflected in the corpus at each synchronization. See [Synchronize a corpus with SharePoint](#synchroniser-un-corpus-avec-sharepoint).
{% hint style="info" %}
A valid corpus is accessible to all project members. Its **Unique Identifier**, displayed at the top of its sheet, allows it to be unambiguously identified, for example in a question asked to Le Druide.
{% endhint %}
### States and statuses of a Corpus
#### States
A corpus can be in one of the following states:
* **Created** : the Corpus has been created but the files have not yet been vectorized
* **Valid:** the Corpus is valid and can be used by platform users to create Agents and query them. At least one document has been successfully vectorized
* **Error** : the Corpus is in error and cannot be used by platform users. No file has been successfully vectorized.
#### File statuses
Corpus documents can have several statuses:
* **Created** the file has been selected but has not yet been vectorized
* **In progress** : vectorization is in progress
* **Success** : vectorization was completed successfully
* **Failure** : vectorization failed. The reason is indicated (file unreadable, rejected by antivirus, unsupported format, etc.).
## Query a corpus
Once the corpus is valid:
* **From an Agent** : associate the corpus with the knowledge scope of an [Agent](/docs/v4.7-en/data-and-ai-project/studio-ai/agents.md#creer-un-agent) via the tool **Cleyrop Corpus**. The Agent searches for relevant passages in the corpus and responds by citing its sources.
* **From Le Druide** : activate the tool **Cleyrop Corpus** and ask your questions in natural language. Le Druide searches across all corpora you have access to; mention the name or unique identifier of a corpus to target the search. See [Query a corpus of documents](/docs/v4.7-en/le-druide/interroger-un-corpus-de-documents.md).
## Modify a corpus
You can enrich or clean up a corpus at any time.
#### Add documents
1. Select the corpus to enrich.
2. Click on **Add documents.**
3. Select the files to integrate (documents already present are grayed out).
4. Start the integration: the files are vectorized and the metadata updated.
Each addition is logged with:
* the date and time,
* the status (*success* or *failure*).
#### Delete documents
1. Select the corpus to modify.
2. Click Delete documents.
3. Choose the files to remove.
4. Confirm the deletion.
Deleted documents are removed from Agent results and their metadata deleted.
The deletion history remains viewable (date, time, status).
{% hint style="info" %}
As long as a deletion or addition is in progress, no new action can be initiated on the corpus.
On a **synchronized**, manual addition and deletion of documents are not available: the content is driven by synchronization. Modify the scope via **Set recurrence**.
{% endhint %}
## Synchronize a corpus with SharePoint
A corpus created from a SharePoint datasource in **Synchronized** mode is updated automatically: at each synchronization, files added or modified on SharePoint are integrated, and files deleted or removed from the scope are removed from the corpus.
**Set recurrence**
1. From the corpus sheet, click on **Set recurrence**.
2. Adjust the synchronized folders and files if needed.
3. Set the frequency: every N hours, days, weeks (with weekdays) or months (on a given date or day), at the chosen time. The time is expressed in **UTC** (Paris time − 1 h in winter, − 2 h in summer). The minimum frequency is one hour; by default, synchronization occurs daily at 00:00 UTC.
4. Click on **Apply**.
Without configured recurrence, synchronization remains **manual** : start it from the corpus sheet with the **Synchronize**. It starts within the following minute.
{% hint style="info" %}
* Only one synchronization at a time: if a synchronization is already in progress, a manual request is rejected and a scheduled occurrence is queued.
* If the configured scope contains no files, synchronization is ignored: the corpus is never emptied by mistake.
* If the platform's default embedding model has changed since the corpus was created, synchronization fails with an explicit message: recreate the corpus.
* The actions **Synchronize** and **Set recurrence** are reserved for the corpus owner and the project owner.
{% endhint %}
**Track synchronizations**
The **History** tab of the corpus sheet lists each refresh with its identifier, status (Success, Failure, Pending, In progress, Ignored, Warning), execution date, and trigger (manual or recurrence). The details of a refresh show the duration, the applied scope, and the list of **modified files** (additions, modifications, deletions) with their processing date and status.
In case of failure, the message associated with the status indicates the cause: SharePoint authentication expired (reconnect the datasource), datasource deleted, source unreachable, folder or file in the scope not found.
## Delete a corpus
To delete a corpus, click the 🗑️ icon on the corresponding card in the Corpus list.
Deletion is permanent and immediately removes the knowledge base from the project.
***
## Best practices
* Group documents by theme (e.g. “Client contracts”, “Internal procedures”).
* Check the language and quality of the files to optimize vectorization.
* Avoid excessively large or noisy files (imperfect OCR, scans).
* Regularly update your corpora to keep AI responses relevant. For a SharePoint corpus, prefer the **Synchronized** mode with a recurrence adapted to the update rate of your documents.
