Run the project

Run PySpark

Navigate to the PySpark directory

cd pyspark

Generate dummy data

The script writes the dummy data to the database of whichever node is currently active.

python seed_products.py

Execute PySpark functionality

The next step pulls every record from the node, loads the records into a PySpark DataFrame, and runs a set of analyses on it.

Note: Large datasets can trigger a Java OutOfMemoryError, which happens when PySpark exhausts its heap memory. You can avoid this by adding the environment variables below to your .env file before you start the script:

On Linux, macOS, or WSL:

# Set these environment variables before running your scriptexport SPARK_DRIVER_MEMORY=2gexport SPARK_EXECUTOR_MEMORY=2gexport SPARK_DRIVER_MAXRESULTSIZE=1g

On Windows:

# Set these environment variables before running your scriptset SPARK_DRIVER_MEMORY=2gset SPARK_EXECUTOR_MEMORY=2gset SPARK_DRIVER_MAXRESULTSIZE=1g

Then start the script:

python get_products_paginated.py