> For the complete documentation index, see [llms.txt](https://cleyrop.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cleyrop.gitbook.io/docs/documentation-fr-en/data-and-ai-project/donnees-de-travail-fichiers/utiliser-les-donnees-dans-un-script.md).

# Use data in a script

The Work Data of a Cleyrop project can be handled in two main ways depending on your use case:

* From a Dataflow — to consume, transform, or enrich files in an automated processing pipeline.
* From a Notebook (Codelab) — to explore, test, or manually analyze the data stored in the project's bucket.

To use the Work Data, you can use the library `cleyrop.connectors` and the object `CleyropS3` available in Python transformations.

The library is included by default in the **Python Clusters** and **PySpark**.

{% hint style="warning" %}

* To use it in a transformation in a **Dataflow** you must first initialize the client
* In a Codelab, you do not need to run this command; you can directly use `cleyrop_s3` as the S3 client
  {% endhint %}

### S3 client initialization

#### In a Dataflow

You must explicitly create a CleyropS3 client:

```python
from cleyrop.connectors import CleyropS3

client = CleyropS3()
```

#### In a Notebook (Codelab)

A client is already available under the name **`cleyrop_s3`**.

In the examples below, you can therefore replace:

```python
client = CleyropS3()
```

with

```python
client = cleyrop_s3
```

### Summary of the public CleyropS3 methods

<table data-full-width="true"><thead><tr><th width="126.60546875">Method</th><th width="202.8125">Description</th><th width="421.3125">Expected parameters</th><th>Return type</th></tr></thead><tbody><tr><td><code>discover</code></td><td>List the files in a folder</td><td><code>{"key": "path_dossier"}</code></td><td>Pandas DataFrame</td></tr><tr><td><code>read_bytes</code></td><td>Read a binary file</td><td><code>{"path": "path_fichier"}</code></td><td><code>bytes</code></td></tr><tr><td><code>get_df</code></td><td>Load a tabular file</td><td><code>{"key": "path_fichier"}</code> (+ <code>sheet_name</code>)</td><td>Pandas DataFrame</td></tr><tr><td><code>write_bytes</code></td><td>Write a file</td><td><code>{"path": "...", "body": bytes}</code></td><td><code>None</code></td></tr><tr><td><code>move</code></td><td>Move a file</td><td><code>{"source_key": "...", "destination_key": "..."}</code></td><td><code>None</code></td></tr><tr><td><code>delete</code></td><td>Delete a file</td><td><code>{"key": "path_fichier"}</code></td><td><code>None</code></td></tr></tbody></table>

## Read a file

The method `read_bytes` allows you to read the raw binary content of a file stored in the Work Data (PDF, image, Excel, etc.): it returns the file bytes directly.

#### Parameters

* Expected parameter: `path` (file path after the project root)
* Return type: `bytes`

{% hint style="info" %}
**Example** CA Analysis Project, commande.csv file in the 2019 subfolder in the Consolidation folder. The full path is projet-ca/consolidation/2019

The path\_file is consolidation/2019/commande.csv
{% endhint %}

**Example: reading a PDF file**

{% code title="Dataflow example : read pdf file" %}

```python
from cleyrop.connectors import CleyropS3

client = CleyropS3()

pdf_bytes = client.read_bytes({
    "path": "path_file"
})
```

{% endcode %}

## List the files in a folder

In a Dataflow or notebook in the project, you can list the files in a Work Data folder.

Enter the **path\_dossier** : path of the folder to explore after the root. The output dataset will be a Pandas DataFrame.

{% hint style="info" %}
**Example** CA Analysis Project, 2019 subfolder in the Consolidation folder. The full path is projet-ca/consolidation/2019

The path\_dossier is consolidation/2019
{% endhint %}

```python
# Reading files from Cleyrop "Workspace"
from cleyrop.connectors import CleyropS3

# Create a CleyropS3 client
client = CleyropS3()

# List all files in a bucket
discover_params = {
    'key': 'path_dossier'
}
client.discover(discover_params)
```

## Create a file

Create a CSV file from a Polars, PySpark, or Pandas DataFrame in a workspace folder.

Enter the **path\_new\_file** = path to the file to create after the root

{% hint style="info" %}
**Example** CA Analysis Project, file to create in the Consolidation folder.

The path\_new\_file is consolidation/new\_file.csv
{% endhint %}

```python
from cleyrop.connectors import CleyropS3
import io
import polars as pl

df = cleyrop_datasets["project.datasetname"].collect()

# Create a CleyropS3 client
client = CleyropS3()

# Serialize to CSV (in-memory string → bytes)from cleyrop.connectors import CleyropS3
csv_bytes = df.write_csv().encode("utf-8")

client.write_bytes({
    "path": "path_new_file",  
    "body": csv_bytes
})
```

## Move a file

In a Dataflow or notebook in the project, you can move a file from the Work Data.

Enter the **origin\_path\_file** and the **target\_path\_file** = origin and target paths (after the root) of the file to move

{% hint style="info" %}
**Example** CA Analysis Project, commande.csv file in the 2019 subfolder in the Consolidation folder.

The origin\_path\_file is consolidation/2019/commande.csv

The target\_path\_file is consolidation/2018/commande.csv
{% endhint %}

```python
# Reading files from Cleyrop "Workspace"
from cleyrop.connectors import CleyropS3

# Create a CleyropS3 client
client = CleyropS3()

# Move a file from one location to another
move_params = {
    'source_key': 'origin_path_file',
    'destination_key': 'target_path_file'
}
client.move(move_params)
```

## Delete a file

In a Dataflow or notebook in the project, you can move a file from the Work Data.

Enter the **path\_file** = path of the file to delete after the root

{% hint style="danger" %}
Deletion is permanent.
{% endhint %}

{% hint style="info" %}
**Example** CA Analysis Project, commande.csv file in the 2019 subfolder in the Consolidation folder. The full path is projet-ca/consolidation/2019

The path\_file is consolidation/2019/commande.csv
{% endhint %}

```python
# Reading files from Cleyrop "Workspace"
from cleyrop.connectors import CleyropS3

# Create a CleyropS3 client
client = CleyropS3()

# Delete a file 
delete_params = {'key': 'path_file'}
client.delete(delete_params)
```

## Create a dataset from a file

In a **Dataflow**, you can create a **Pandas DataFrame** from a CSV or Excel file stored in the Work Data.

Enter the **path\_file** = path to the file to process after the root

{% hint style="info" %}
**Example** CA Analysis Project, commande.csv file in the 2019 subfolder in the Consolidation folder. The full path is projet-ca/consolidation/2019

The path\_file is consolidation/2019/commande.csv
{% endhint %}

```python
# Reading files from Cleyrop "Workspace"
from cleyrop.connectors import CleyropS3
# Create a CleyropS3 client
client = CleyropS3()

# Get Dataframe from a file
read_params = {'key': 'path_file'}
df = client.get_df(read_params)

return df
```

**For an Excel file**

```python
# Reading files from Cleyrop "Workspace"
from cleyrop.connectors import CleyropS3

# Create a CleyropS3 client
client = CleyropS3()

# Get Dataframe from a file
file_params = {'key': 'path_file'}
excel_file = client.file(file_params)

# Get sheetnames from an excel file
sheets = pd.ExcelFile(excel_file).sheet_names

df = client.get_df(file_params, sheet_name='sheet_name')
#df = client.get_df(file_params) first sheet by default

return df
```

{% hint style="info" %}

### For a Spark Dataflow

Transform the Pandas DataFrame into a PySpark DataFrame

\
import pyspark.pandas as ps

df = ps.from\_pandas(df)
{% endhint %}