> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/documentation-fr-en/data-and-ai-project/dataflow/creer-un-dataflow/ecrire-une-transformation.md).

# Write a transformation

A transformation corresponds to a processing step in a Dataflow.

It takes one or more datasets as input, applies code or a rule, then produces a new dataset (or a temporary result).

Cleyrop offers several transformation modes depending on the type of Dataflow chosen: Spark, Python, SQL or Low Code.

***

## Write a Spark transformation

### PySpark transformation

Spark transformations are designed for distributed processing on large volumes. They run on a Spark cluster (driver + executors) **managed by Cleyrop**.

They run on a Spark cluster, with automatic parallelization of computations.

{% hint style="warning" %}

* `cleyrop_datasets["projet.nom_dataset"]` **returns a DataFrame *****pandas-on-Spark*** (`pyspark.pandas.DataFrame`).
* **No SparkSession to create** : initialization and configuration are fully managed by Cleyrop.
* You must **return a DataFrame** (Spark or pandas-on-Spark) at the end of the transformation
  {% endhint %}

{% hint style="success" %}
You can [**Access Work Data**](/docs/documentation-fr-en/data-and-ai-project/donnees-de-travail-fichiers/utiliser-les-donnees-dans-un-script.md#utiliser-dans-le-dataflow) in a PySpark transformation
{% endhint %}

The**autocomplete** helps insert the correct syntax: type at least three letters of a Dataset name in the editor.

**Example:**

{% code title="PySpark transformation" %}

```python
voitures = cleyrop_datasets["projet.voitures"]
voitures_modifiees = voitures.with_column("prix", voitures["prix"] * 1.2)
return voitures_modifiees
```

{% endcode %}

The code must return a Spark DataFrame (pyspark.pandas.DataFrame or DataFrame).

**Conversion to Spark DataFrame**

In some cases (Spark APIs not available in pandas-on-Spark, MLlib uses, advanced SQL functions), convert to a Spark DataFrame:

{% code title="Spark DataFrame conversion" %}

```python
voitures_pos = cleyrop_datasets["projet.voitures"]          # pandas-on-Spark
voitures_spark = voitures_pos.to_spark()                    # Spark DataFrame
from pyspark.sql import functions as F
res = voitures_spark.withColumn("prix", F.col("prix") * 1.2)
return res                           # return in pandas-on-Spark
```

{% endcode %}

💡 If you convert to a Spark DataFrame, Cleyrop automatically handles passing the result to the output dataset.

#### Points to watch

* **Avoid `to_pandas()`**, which brings the data back to the driver → **risk of OOM**.
* If you use `to_spark()`, make sure to return the Spark DataFrame, not a locally transformed object.
* Never return a list, dict or any other raw Python type when the transformation feeds an output dataset.
* Keep a “distributed” logic: no Python loops over millions of rows.

### SQL transformation

SQL transformations allow Spark SQL code to be executed directly on input datasets (Spark Dataflows only).

{% code title="SQL transformation" %}

```sql
SELECT modele, prix * 1.2 AS prix
FROM projet.voitures
WHERE pays = 'France'
```

{% endcode %}

The input dataset is recognized by its Unique Identifier (projet.nom\_du\_dataset).

### Low Code transformation

The [Low Code mode](/docs/documentation-fr-en/data-and-ai-project/dataflow/transformer-sans-code-low-code.md) allows you to create visual transformations without writing code.

Each block corresponds to an operation (filter, join, aggregate, rename, etc.).

Low Code transformations can be combined with Python or SQL.

***

## Write a Python transformation (Polars)

Python transformations run on a dedicated Python cluster, ideal for:

* light processing,
* automation,
* API calls or business operations

{% hint style="warning" %}

* `cleyrop_datasets["projet.nom_dataset"]` **returns a polars LazyFrame**
* If the transformation has an output dataset, the code must absolutely return an object **Polars LazyFrame** (recommended) or a Pandas DataFrame.
* Use **Pandas** **only for libraries that are not compatible with Polars**.
  {% endhint %}

{% hint style="success" %}
You can use the [**PyAi library**](/docs/documentation-fr-en/the-factory/deployer-des-apps-et-des-tools.md#utiliser-la-librairie-de-composants-cleyrop) **in Python 3.11 and 3.12 clusters.**

You can [**Access Work Data**](/docs/documentation-fr-en/data-and-ai-project/donnees-de-travail-fichiers/utiliser-les-donnees-dans-un-script.md#utiliser-dans-le-dataflow) in a Python transformation
{% endhint %}

Example:

{% code title="Python transformation " %}

```python
import polars as pl
voitures = cleyrop_datasets["projet.voitures"]  # Polars LazyFrame
voitures_modifiees = voitures.with_columns((pl.col("prix") * 1.2).alias("prix"))
return voitures_modifiees
```

{% endcode %}

#### Why use Polars?

💡 By default, Cleyrop runs Python Dataflows in Polars (LazyFrame).

Polars is the modern, high-performance library recommended by Cleyrop. It combines speed, safety, and lazy computation.

* High-performance: native parallelization (multicore).
* Memory-efficient: deferred computation, no full loading.
* Lazy execution: Cleyrop can automatically optimize chained processing.
* Interoperable: simple conversions to Pandas if needed.

***

{% hint style="warning" %}
**Deletion of a transformation node** : after deleting a node, save the Dataflow before creating a new one. If you create a transformation with the same name as a recently deleted node without saving beforehand, a conflict may prevent it from working properly.
{% endhint %}

## Use environment variables

You can declare [**environment variables** ](/docs/documentation-fr-en/data-and-ai-project/dataflow.md#ajouter-une-variable-denvironnement)(e.g. API\_KEY, S3\_PATH) directly accessible in your transformations:

```python
import os

api_key = os.getenv("API_KEY")

print(f"API key: {api_key}")
```

#### Global and local variables

* **Global** : created in the main branch (main), accessible in all branches.
* **Local** : specific to a branch, they override global variables with the same name.
* When a local variable and a global variable have the same name, the local one takes precedence when executed in the corresponding branch.

***

## Best practices

* Define local variables in the main branch if you want to prevent accidental use of production variables.
* Avoid hardcoding sensitive values in your transformations.
* To test a Dataflow in different environments (e.g. sandbox vs prod), simply change the value of the variable without modifying the code.