# Run the project · Big data analysis with Chromia and PySpark · Learn Chromia

> The script writes the dummy data to the database of whichever node is currently active.

Source: https://chromia.com/learn/big-data/project-run/

Big data analysis with Chromia and PySparkLesson 4 of 6

1. [01Analyzing Chromia blockchain data with PySpark](https://chromia.com/learn/big-data/introduction/)
2. [02Set up your project](https://chromia.com/learn/big-data/setup/)
3. [03Prepare the project](https://chromia.com/learn/big-data/project-launch/)
4. [04Run the project](https://chromia.com/learn/big-data/project-run/)
5. [05Blockchain components](https://chromia.com/learn/big-data/blockchain-side-description/)
6. [06Python components](https://chromia.com/learn/big-data/python-side-description/)

[Learn](https://chromia.com/learn/)[Big data analysis with Chromia and PySpark](https://chromia.com/learn/big-data/)

04 / 06Intermediate

# Run the project

## Run PySpark

### Navigate to the PySpark directory

shell

```nolines
cd pyspark
```

### Generate dummy data

The script writes the dummy data to the database of whichever node is currently active.

shell

```nolines
python seed_products.py
```

### Execute PySpark functionality

The next step pulls every record from the node, loads the records into a PySpark DataFrame, and runs a set of analyses on it.

> **Note:** Large datasets can trigger a Java OutOfMemoryError, which happens when PySpark exhausts its heap memory. You can avoid this by adding the environment variables below to your `.env` file before you start the script:

**On Linux, macOS, or WSL:**

env

```
# Set these environment variables before running your script
export SPARK_DRIVER_MEMORY=2g
export SPARK_EXECUTOR_MEMORY=2g
export SPARK_DRIVER_MAXRESULTSIZE=1g
```

**On Windows:**

env

```
# Set these environment variables before running your script
set SPARK_DRIVER_MEMORY=2g
set SPARK_EXECUTOR_MEMORY=2g
set SPARK_DRIVER_MAXRESULTSIZE=1g
```

Then start the script:

shell

```nolines
python get_products_paginated.py
```

Written for this site from Chromia’s public course. Code samples are the course’s own.

[← Previous**Prepare the project**](https://chromia.com/learn/big-data/project-launch/)[Next →**Blockchain components**](https://chromia.com/learn/big-data/blockchain-side-description/)

On this page

* [Run PySpark](#run-pyspark)