> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/documentation-fr-en/data-and-ai-project/dataflow/comprendre-les-dataflows/les-bonnes-pratiques.md).

# Best practices

This page brings together the key recommendations for creating, maintaining, and scaling your Dataflows safely.

They cover both technical aspects (execution, typing, performance) and collaborative ones (branches, roles, documentation).

***

## General

* Create a **branch for each change** to isolate your tests without impacting production.
* Always check the **output dataset schema**.
* **Document** your transformations and variables in their description to make the graph easier to read.
* Keep consistent naming conventions (naming transformations, datasets, modules…).

## Typing and schema

* **A dataset schema is common to all branches.**

  Once a column is created, its type can no longer be changed without deleting the column or the dataset.
* Check the **compatibility** of types between Pandas, Pandas-on-Spark and Polars before publishing a dataset.
* **Explicit cast recommended** (e.g. astype("string"), to\_pandas\_on\_spark()…) to avoid implicit errors.
* Before each merge, compare the output schema with that of main to avoid future structural conflicts.

## Performance and execution

* Favor **distributed libraries** : Spark or Polars rather than pure Pandas.
* Avoid to\_pandas() calls that bring data back to the driver (risk of OOM).
* On Spark, never recreate **the SparkSession manually** : it is managed by Cleyrop.
* Use **suitable clusters** for the data size and test on a lightweight cluster before production.

## Collaboration and governance

* Define **roles and responsibilities clearly** : branch owner, merge approver, project manager.
* Merge only after **functional and technical validation**.
* Use **local variables** to avoid accidental use of production variables.
* Enable **failure alerts** on all critical Dataflows.
* Regularly delete **obsolete branches** and **orphan datasets**.

## Maintenance and reuse

* Use **modules** to share functions or dictionaries common to several Dataflows.
* Systematically test a module in a **test Dataflow** before deployment.
* During a **module deployment**, keep in mind that all the Dataflows that use it will be impacted.
* Clean up **old Dataflows** to keep a clear view of the project space.

## Visualization and monitoring

* You can **preview the produced datasets** directly from the graph or the latest run report.
* Each preview shows the **full schema** and the first 100 rows of the dataset.
* If the preview does not display, it is often a schema or encoding issue.
* Regularly check the **execution logs** to identify recurring errors or slowdowns.