> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/documentation-fr-en/data-and-ai-project/dataflow/creer-un-dataflow/configurer-le-cluster-dexecution.md).

# Configure the execution cluster

Each Dataflow runs on a cluster. The cluster determines the language used (Spark or Python), the version of the available libraries, as well as the resources (CPU, RAM, storage) allocated to the run.

This configuration ensures that each process runs in an isolated, reproducible, and secure environment.

Cleyrop provides, in each environment, **a Spark cluster and a Python cluster permanently available** :

* **No startup delay** : the clusters are always active.
* **Automatic fallback**

## Set a cluster per branch

Each branch of a Dataflow can be associated with a different cluster.

This allows you to:

* test a specific configuration or version without impacting production,
* run test branches on lighter clusters,
* reserve a high-performance cluster for critical branches (e.g. main or production).

{% hint style="success" %}
When a custom cluster is available, you can assign it to your Dataflow to benefit from a more powerful environment or one specific to your use case: either when creating it or in the'**Cluster tab of the Dataflow action bar**
{% endhint %}

When launching a run, Cleyrop chooses the cluster defined for the current branch. If no cluster is explicitly defined, the cluster compatible with the default Dataflow type is automatically used.

* Spark Dataflow → run on a Spark cluster (driver + executors).
* Python Dataflow → run on a dedicated Python cluster.

**Change the cluster** of the branch from the Dataflow branch Cluster tab button `Edit`

<figure><img src="/files/a79baed58b2a763398c658cd886c165a7086a2a4" alt="" width="375"><figcaption></figcaption></figure>

## Create or use a custom cluster

{% hint style="warning" %}
Only Platform Managers can manage clusters (creation, scaling down, configuration)
{% endhint %}

Platform Managers [can create and manage new clusters](/docs/documentation-fr-en/administration/gerer-les-clusters.md) tailored to specific needs. By defining:

* the amount of resources (CPU, RAM),
* the engine version (Python),
* the number of sessions

These clusters make it possible to provide:

| Parameter              | Description                                                                                |
| ---------------------- | ------------------------------------------------------------------------------------------ |
| CPU                    | Number of cores allocated per run. More CPU = faster but more expensive execution.         |
| RAM                    | Amount of available memory. Required for processing large volumes or in-memory operations. |
| Python / Spark version | Defines the compatibility of the installed libraries.                                      |
| Available libraries    | List of Python or Spark packages installed in the cluster.                                 |

***

## Best practices

* For development branches, prefer a separate cluster to avoid overloading the main cluster.
* Do not manually create Spark sessions: they are automatically managed by Cleyrop.