> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/documentation-fr-en/data-and-ai-project/dataflow/comprendre-les-dataflows.md).

# Understand Dataflows

Dataflows are at the heart of the Cleyrop platform.

They make it possible to design **transformation pipelines** of data, by linking *datasets* existing ones to transformation steps (code, SQL or low code) to produce new datasets ready to be used in analyses, dashboards, or AI applications.

Designed with a **visual and collaborative approach**, Dataflows make it easier to understand and evolve processing over time.

They enable mixed teams, **technical and business,** to work together, visually document transformations, and encourage rapid onboarding of new joiners.

### What is a Dataflow

A Dataflow is a visual data processing pipeline.

Each step takes one or more *Datasets*, applies a transformation, then writes the result to a new dataset.

Dataflows allow:

* to**orchestrate multiple processes** in a coherent flow,
* to **track and version** each step,
* to**automate execution** and publication of results.

<figure><img src="/files/074008746f487f245234028623f675f232a16d25" alt=""><figcaption></figcaption></figure>

### Types of Dataflows

There are two main types of Dataflows in Cleyrop:

<table data-card-size="large" data-view="cards"><thead><tr><th>Type</th><th>Main use</th><th>Languages</th><th>Input / Output required</th><th>Performance</th></tr></thead><tbody><tr><td>SPARK</td><td>ETL / distributed processing</td><td>PySpark, SQL, Low Code</td><td>Python (Polars / Pandas)</td><td>High (distributed cluster)</td></tr><tr><td>PYTHON</td><td>Automation / collection / lightweight scripts</td><td>Python (Polars)</td><td>No</td><td>Medium (local or multi-process execution)</td></tr></tbody></table>

### Components of a Dataflow

A Dataflow consists of:

* **Input datasets** : data sources used in transformations.
* **Transformations** : processing steps (Python, SQL, Low Code).
* **Output datasets** : produced datasets.
* **Execution cluster** : computing environment (Spark or Python).
* **Environment variables** : optional parameters (keys, URLs, etc.).

### How Dataflows run

When a Dataflow is launched:

* Each transformation reads its input datasets if they exist.
* The code runs on the chosen cluster (Spark or Python).
* The output dataset is written to the Cleyrop catalog
* Execution logs are recorded and can be viewed.

All **executions and versions are versioned by branch**, guaranteeing **traceability** of processes.

### Best practices

* Avoid pandas calls in Spark transformations.
* Break long flows into several Dataflows to simplify debugging and make it easier to get started.
* Use environment variables to manage your sensitive parameters.
* Document the produced datasets to encourage reuse.