> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/documentation-fr-en/data-and-ai-project/dataflow.md).

# Dataflow

Dataflows are the heart of the data transformation engine in Cleyrop.

They make it possible to create **visual, reproducible, and collaborative pipelines** to transform, enrich, and publish your data within the platform.

A Dataflow can be seen as a **sequence of logical steps** (transformations) connected to each other to produce one or more **Output datasets**.

## Why use Dataflows?

Dataflows were designed to:

* 🧩 **Unify processing** Spark, Python, SQL, and Low Code in the same environment.
* 👥 **Facilitate collaboration** between technical and business profiles thanks to a visual interface.
* 🔁 **Ensure traceability** and the **reproducibility** of processing (versioning, logs, executions).
* ⚙️ **Optimize performance** through managed, ready-to-use clusters.
* 📦 Ensure **quality and governance** thanks to versioned and typed datasets.

***

## The lifecycle of a Dataflow

Each Dataflow follows a cycle in several stages

{% stepper %}
{% step %}
**Creation**

You define the type of Dataflow (Spark or Python), its variables, and its output dataset.
{% endstep %}

{% step %}
**Transformation**

You add processing steps: code, SQL, or Low Code.
{% endstep %}

{% step %}
**Execution**

You run the flow manually or schedule it for recurring execution.
{% endstep %}

{% step %}
**Analysis**

You review the logs, results, and quality of the datasets produced.
{% endstep %}

{% step %}
**Publication**

Once validated, the Dataflow can be published to feed other projects, dashboards, or AI assistants.
{% endstep %}

{% step %}
**Collaboration and evolution**

After production deployment, other project members can create branches to evolve the Dataflow
{% endstep %}
{% endstepper %}

{% hint style="success" %}
Dataflows ensure a **clear separation** between test and production environments thanks to their **branches and roles**.
{% endhint %}

## Dataflow structure

A Dataflow consists of several elements:

| Element                 | Description                                                                             |
| ----------------------- | --------------------------------------------------------------------------------------- |
| Transformations         | Processing steps executed sequentially (Python, PySpark, SQL, or Low Code).             |
| Environment variables   | Reusable parameters injected into the code (e.g., API keys, S3 paths…).                 |
| Input / output datasets | Data consumed or produced by the Dataflow.                                              |
| Execution cluster       | Technical environment in which the processing is executed.                              |
| Logs and executions     | Detailed history of the processing carried out.                                         |
| Branches                | Isolated copies of the Dataflow allowing work in parallel without impacting production. |

## Navigate the documentation

| Step                                                                                                                          | Objective                                                                               |
| ----------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| [Understand Dataflows](/docs/documentation-fr-en/data-and-ai-project/dataflow/comprendre-les-dataflows.md)                    | Discover the fundamental concepts, best practices, code examples, and roles.            |
| [Create a Dataflow](/docs/documentation-fr-en/data-and-ai-project/dataflow/creer-un-dataflow.md)                              | Learn how to define, configure, and run your flows: transformation, dataset, schema.... |
| [Analyze executions and logs](/docs/documentation-fr-en/data-and-ai-project/dataflow/analyser-les-executions-et-logs.md)      | Understand the results, visualize the outputs, and fix errors.                          |
| [Collaborate and version](/docs/documentation-fr-en/data-and-ai-project/dataflow/collaborer-et-versionner.md)                 | Work together on the same Dataflow without risk.                                        |
| [Configure alerts](/docs/documentation-fr-en/data-and-ai-project/dataflow/configurer-des-alertes.md)                          | Be automatically notified of executions.                                                |
| [Transform without code (Low Code)](/docs/documentation-fr-en/data-and-ai-project/dataflow/transformer-sans-code-low-code.md) | Use the visual interface to manipulate your data.                                       |
| [Use Modules](/docs/documentation-fr-en/data-and-ai-project/dataflow/utiliser-des-modules.md)                                 | Share reusable functions across your Dataflows.                                         |

***

## Best practices

* Create a **branch for each change** to secure your tests.
* Always check the **output dataset schema** before publication.
* Use modules to **share** functions across projects.
* Enable **failure alerts** for your production Dataflows.
* **Document** your transformations in their description to improve the graph's readability.