> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/documentation-fr-en/data-and-ai-project/dataflow/creer-un-dataflow/choisir-le-bon-type-de-dataflow.md).

# Choose the right type of Dataflow

When creating a Dataflow, you must choose its execution type: Spark or Python.

This choice is essential because it determines the available languages, the data processing mode, and how datasets are read and produced.

<table data-card-size="large" data-view="cards"><thead><tr><th>Type</th><th>Main use</th><th>Languages</th><th>Input / Output required</th><th>Performance</th></tr></thead><tbody><tr><td>SPARK</td><td>ETL / distributed processing</td><td>PySpark, SQL, Low Code</td><td>Python (Polars / Pandas)</td><td>High (distributed cluster)</td></tr><tr><td>PYTHON</td><td>Automation / collection / lightweight scripts</td><td>Python (Polars)</td><td>No</td><td>Medium (local or multi-process execution)</td></tr></tbody></table>

***

## Spark Dataflow

Spark Dataflows are designed for distributed processing and large-scale structured data processing.

They use a Spark cluster made up of a driver and several executors to parallelize computations.

#### When to use it

* You are working with large datasets (several GB to TB).
* You need joins, aggregations, or complex transformations.
* You want to chain several processing steps with structured outputs.
* You want to benefit from Low Code mode or Spark SQL.

#### Available languages

* **PySpark** : distributed transformations in Python.
* **SQL** : Spark SQL queries executed on the cluster.
* **Low Code** : visual transformation interface (join, filter, group, etc.).
* The list of libraries used is available at the Cluster level

#### Datasets

* Required input dataset (you start from one or more existing sources).
* Required output dataset (each Spark transformation produces one).

#### Example

```python
voitures = cleyrop_datasets["projet.voitures"]
voitures_modifiees = voitures.with_column("prix", voitures["prix"] * 1.2)
return voitures_modifiees
```

## Python Dataflow

Python Dataflows are more flexible and lightweight.

They allow combining free-form Python code, API calls, automations, or non-distributed processing.

#### When to use it

* You need to retrieve or transform external data (API, S3, files).
* You perform business processing or automations.
* You work with light or unstructured volumes.
* You do not necessarily want to produce a structured dataset.

#### Languages and libraries

* Standard Python compatible with Polars (recommended)
* The list of libraries used is available at the Cluster level

#### Datasets

* Optional input dataset (you can start without an existing dataset).
* Optional output dataset (some transformations do not produce structured data).

#### Example

```python
import requests, polars as pl

response = requests.get("https://api.exemple.com/data").json()
df = pl.DataFrame(response).lazy()
return df
```

***

## Best practices

* Choose Spark if your processing handles structured and large data.
* Choose Python if your processing is focused on automation, integration, or orchestration.
* Do not mix the two types within the same business logic: prefer several dedicated, interconnected Dataflows.
* Document the purpose of the Dataflow from the start to make maintenance and handoff easier.