> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/v4.7-en/data-and-ai-project/dataflow/comprendre-les-dataflows.md).

# Understand Dataflows

Discover how Dataflows work.

Dataflows are at the heart of the Cleyrop platform.

They make it possible to design **transformation chains** of data by linking *datasets* existing datasets to transformation steps (code, SQL or low code) to produce new datasets ready to be used in analyses or AI applications.

Designed with a **visual and collaborative approach**, Dataflows make it easier to understand and evolve processing over time.

They allow mixed teams, **technical and business,** to work together, visually document transformations, and promote the rapid onboarding of newcomers.

### What is a Dataflow

A Dataflow is a graphical data processing pipeline.

Each step takes one or more *Datasets*, applies a transformation, then writes the result to a new dataset.

Dataflows allow you to:

* to**orchestrate multiple processes** in a coherent flow,
* to **track and version** each step,
* to**automate execution** and the publication of results.

<figure><img src="https://4159280578-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FMHNFH6aA4j5ucH4XFBXy%2Fuploads%2Fgit-blob-14c5826ce114f6a2d337c55b49f95256a843fe4d%2Fimage.png?alt=media" alt=""><figcaption></figcaption></figure>

### Types of Dataflows

There are two main types of Dataflows in Cleyrop:

<table data-card-size="large" data-view="cards"><thead><tr><th>Type</th><th>Primary use</th><th>Languages</th><th>Required Input/Output</th><th>Performance</th></tr></thead><tbody><tr><td>SPARK</td><td>ETL / distributed processing</td><td>PySpark, SQL, Low Code</td><td>Python (Polars / Pandas)</td><td>High (distributed cluster)</td></tr><tr><td>PYTHON</td><td>Automation / collection / lightweight scripts</td><td>Python (Polars)</td><td>No</td><td>Medium (local or multiprocess execution)</td></tr></tbody></table>

### Components of a Dataflow

A Dataflow consists of:

* **Input datasets** : data sources used in transformations.
* **Transformations** : processing steps (Python, SQL, Low Code).
* **Output datasets** : produced datasets.
* **Execution cluster** : computing environment (Spark or Python).
* **Environment variables** : optional parameters (keys, URLs, etc.).

### How Dataflows run

When a Dataflow is launched:

* Each transformation reads its input datasets if present.
* The code runs on the chosen cluster (Spark or Python).
* The output dataset is written to the Cleyrop catalog
* Execution logs are recorded and can be viewed.

All **executions and versions are tracked by branch**, guaranteeing the **traceability** of the processing in full.

### Best practices

* Avoid pandas calls in Spark transformations.
* Break long flows into multiple Dataflows to simplify debugging and make them easier to get to grips with.
* Use environment variables to manage your sensitive parameters.
* Document the produced datasets to encourage their reuse.