> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/v4.7-en/data-and-ai-project/dataflow.md).

# Dataflow

Design, run, and monitor your data processing in Cleyrop.

Dataflows are the heart of the data transformation engine in Cleyrop.

They make it possible to create **visual, reproducible, and collaborative pipelines** to transform, enrich, and publish your data within the platform.

A Dataflow can be seen as a **sequence of logical steps** (transformations) connected together to produce one or more **output datasets**.

## Why use Dataflows?

Dataflows were designed to:

* 🧩 **Unify processing** Spark, Python, SQL, and Low Code in the same environment.
* 👥 **Facilitate collaboration** between technical and business profiles thanks to a visual interface.
* 🔁 **Ensure traceability** and **reproducibility** of processing (versioning, logs, runs).
* ⚙️ **Optimize performance** through managed, ready-to-use clusters.
* 📦 Ensure **quality and governance** thanks to versioned and typed datasets.

***

## The lifecycle of a Dataflow

Each Dataflow follows a cycle in several stages

{% stepper %}
{% step %}
**Creation**

You define the type of Dataflow (Spark or Python), its variables, and its output dataset.
{% endstep %}

{% step %}
**Transformation**

You add processing steps: code, SQL, or Low Code.
{% endstep %}

{% step %}
**Execution**

You run the flow manually or schedule it for recurring execution.
{% endstep %}

{% step %}
**Analysis**

You review the logs, results, and quality of the produced datasets.
{% endstep %}

{% step %}
**Publication**

Once validated, the Dataflow can be published to feed other projects, Dataflows, or Agents.
{% endstep %}

{% step %}
**Collaboration and evolution**

After production deployment, other project members can create branches to evolve the Dataflow
{% endstep %}
{% endstepper %}

{% hint style="success" %}
Dataflows ensure a **clear separation** between test and production environments thanks to their **branch and role**.
{% endhint %}

## Dataflow structure

A Dataflow consists of several elements:

| Element                 | Description                                                                            |
| ----------------------- | -------------------------------------------------------------------------------------- |
| Transformations         | Processing steps executed sequentially (Python, PySpark, SQL, or Low Code).            |
| Environment variables   | Reusable parameters injected into the code (e.g., API keys, S3 paths...).              |
| Input / output datasets | Data consumed or produced by the Dataflow.                                             |
| Execution cluster       | Technical environment in which the processing is executed.                             |
| Logs and runs           | Detailed history of the processing performed.                                          |
| Branches                | Isolated copies of the Dataflow that allow parallel work without impacting production. |

## Navigating the documentation

| Step                                                                                                              | Objective                                                                                |
| ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| [Understanding Dataflows](/docs/v4.7-en/data-and-ai-project/dataflow/comprendre-les-dataflows.md)                 | Discover the fundamental concepts, best practices, code examples, and roles.             |
| [Create a Dataflow](/docs/v4.7-en/data-and-ai-project/dataflow/creer-un-dataflow.md)                              | Learn how to define, configure, and run your flows: transformation, dataset, schema .... |
| [Analyze runs and logs](/docs/v4.7-en/data-and-ai-project/dataflow/analyser-les-executions-et-logs.md)            | Understand the results, visualize the outputs, correct the errors.                       |
| [Collaborate and version](/docs/v4.7-en/data-and-ai-project/dataflow/collaborer-et-versionner.md)                 | Work with several people on the same Dataflow without risk.                              |
| [Configure alerts](/docs/v4.7-en/data-and-ai-project/dataflow/configurer-des-alertes.md)                          | Be automatically notified of runs.                                                       |
| [Transform without code (Low Code)](/docs/v4.7-en/data-and-ai-project/dataflow/transformer-sans-code-low-code.md) | Use the visual interface to manipulate your data.                                        |
| [Use Modules](/docs/v4.7-en/data-and-ai-project/dataflow/utiliser-des-modules.md)                                 | Share reusable functions across your Dataflows.                                          |

***

## Best practices

* Create a **branch per change** to secure your tests.
* Always check the **output dataset schema** before publishing.
* Use modules to **share** functions across projects.
* Enable **failure alerts** for your production Dataflows.
* **Document** your transformations in their description to improve the readability of the graph.