> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/documentation-fr-en/data-and-ai-project/dataflow/collaborer-et-versionner.md).

# Collaborate and version

Cleyrop’s branching system makes it possible for multiple people to collaborate efficiently on the same Dataflow while preserving the stability of the production version.

Each user can create their own branches, develop or test changes there, then merge them once validated.

***

## Versioning principles

**Purpose** : to allow data, business, or technical teams to work in parallel, test new logic, or fix errors without any risk to production

{% hint style="success" %}
Each **transformation** and **dataset** is versioned by branch, which makes it possible to work on **evolutions**, test, iterate **without impacting production**
{% endhint %}

* Each Dataflow has a main branch called `main`.
* You can create as many work branches as needed (e.g. feature/client-cleanup, fix/date-typing).
* Each branch is independent: changes made in one branch do not affect the others until they are merged.
* The datasets produced by each branch are isolated and versioned: a dataset created in feature/optimization does not replace the one in main.
* Once validated, a branch can be merged into main.

## Create a new branch

{% hint style="info" %}
The Dataflow must be in **Production** to create branches
{% endhint %}

Once the Dataflow owner has [Deployed the Dataflow](/docs/documentation-fr-en/data-and-ai-project/dataflow/executer-deployer-et-automatiser.md#publier-un-dataflow) :

1. Open the Dataflow
2. Click on `Create a Branch`
3. Give it a clear name (e.g. feature/new\_rule\_pricing) without special characters
4. The new branch is automatically created from the contents of `main` : all transformations, global variables, and the dataset list are retrieved. The global environment variables from main can be modified locally.

{% hint style="warning" %}
For the output datasets present in the branch `main` : the **schema is retrieved** from the branch `main` but **without the data**.

A **first run** of the parent transformation will be needed to populate the branch data and **preview** the branch dataset.
{% endhint %}

You can find the list of branches in the Branch tab of the Dataflow action bar:

<figure><img src="/files/d84ba8008e79bd2434cb7487c712229c336c4b16" alt="" width="375"><figcaption></figcaption></figure>

From each branch, you can navigate from the **action bar** click on:

* **Variables** : to view the global variables created and create local variables or override a global variable locally
* **Cluster** : to identify or modify a specific execution cluster
* **Versions** : to find the list of versions for the branch
* **Executions** : to find the list of executions for the branch

Once the branch is created:

* You can freely modify the code, variables, schedule, or output datasets.
* Runs made from this branch write to the branch dataset and do not affect the main branch

### Using variables within a branch

* The global variables created in `main` are copied into each new branch.
* They can be redefined locally: a local variable with the same name hides the global variable.
* **During a merge**, global variables are not automatically overwritten: only the values explicitly modified in `main` are retained.

{% hint style="info" %}
To test different environments (sandbox vs prod), create branch-specific local variables.
{% endhint %}

### Define a cluster per branch

Each branch can be attached to a different cluster (Spark or Python).

This makes it possible:

* to test different configurations (versions, resources),
* to avoid saturating the main cluster,
* or to run tests on a lighter environment.

To change a branch’s cluster: open the Cluster tab from the Dataflow branch, then click Edit.

### Schema management in branches

Users can modify the schema of an output dataset (e.g. deleting a column) in Dataflow branches, to work in parallel without impacting production.

#### Modify the schema in a branch

{% hint style="warning" %}
To perform this action, you must be **branch owner** and the column must not be in a dataset present in the branch `main` production
{% endhint %}

When working in a Dataflow branch, you can adjust the schema of your output datasets before merging your changes into `main`.

This includes:

* removing an obsolete column,
* cleaning up an intermediate dataset,
* harmonizing the schema before deployment.

The action is available in the Dataflow branch → Click on the Dataset → dataset preview panel → delete button on the column

#### Terms of use

Deleting a column is possible only in the following cases:

| Context                                        | Deletion allowed? |
| ---------------------------------------------- | ----------------- |
| Output dataset still in draft (never deployed) | Yes               |
| Output dataset in a branch                     | Yes (owner only)  |
| Dataset already deployed in main               | No                |
| Existing column in the deployed schema of main | No                |
| Column = last column                           | No                |

If a condition is not met, a clear error message will guide you.

### Delete a transformation from `main` from a branch

In a Dataflow, it is possible to delete a transformation (node) originating from `main` from a branch, without **automatically deleting** the dataset it had produced.

This action makes it possible to safely modify the Dataflow while keeping access to the historical dataset already present in the project library.

**To perform this action**

* Create a branch
* Delete the transformation
* Submit a merge request

Upon validation

* the transformation disappears from the graph of `main`,
* the associated dataset remains available in the project Library marked as orphaned (with no active producer). It can be deleted manually.

## Merge a branch

{% hint style="success" %}
To guarantee a **consistency between transformations and data**, when the merge request is approved, the **target branch will be executed**
{% endhint %}

Once the branch has been validated and tested:

1. Click on `Submit`.
2. Choose the **target branch**. You can choose to **delete the source branch** after the merge
3. Fill in the information (title, description)

<figure><img src="/files/0c6ffa0ba4f63eae10fe3d4e0abcdd0cdb1b4a55" alt="" width="375"><figcaption></figcaption></figure>

4. Choose **the approver** who will receive a notification

The approver can **analyze the entire set of changes simply and visually** : transformation modification, dataset addition, metadata modification

Upon validation,

* The changes are merged into the target branch
* The request appears as Approved in the list of requests
* The **target branch will be executed** in order to take the changes into account and ensure a **consistency** between transformations and data

<figure><img src="/files/901d81d456e10eb7225164ae563ce4983a643044" alt=""><figcaption></figcaption></figure>

You can list all **change requests** (past and still open) from the `Requests` tab in the Dataflow action bar.

### Resolve a merge conflict

If incompatible changes have been made to the same transformation in two different branches, the merge request appears with the status **Conflict**.

{% hint style="warning" %}
The branch owner, who has edit rights, must perform the conflict resolution operations
{% endhint %}

To resolve the conflict, the branch owner must perform the following steps:&#x20;

* Open the request to access the resolution screen. Each conflicting transformation is listed individually in the panel **List of changes**.
* For each conflicting transformation, a comparison screen is displayed:
  * On the left, the **differences between the branches**
  * On the right, the content of the branch you are merging, **editable**
* Choose between:
  * keeping this content as is by clicking on **Keep my version:** it will then replace the one from the target branch
  * edit the code directly in the right panel to build a merged version: the button then becomes **Save the new version**.

{% hint style="info" %}
This conflict resolution applies to **Python or SQL**. Editing is not available for Low Code transformations.
{% endhint %}

Once all conflicts are resolved, the merge can be validated normally.

<figure><img src="/files/7617388ac944ee5e8b511a3add45c710b5667f6f" alt=""><figcaption></figcaption></figure>

<figure><img src="/files/b9703398089b4593eaee4d5ef299e82d8db83f35" alt=""><figcaption></figcaption></figure>

***

## Best practices

* Create one branch per topic (feature, fix, test).
* Give **clear names** (feature/…, fix/…, test/…).
* Only merge into main after **full validation** of the code and produced datasets.
* **Clean up** obsolete branches regularly.
* Enable **local variables** to isolate your test environments.