# Set up your project · Big data analysis with Chromia and PySpark · Learn Chromia

> Complete the following setup steps before you start:

Source: https://chromia.com/learn/big-data/setup/

Big data analysis with Chromia and PySparkLesson 2 of 6

1. [01Analyzing Chromia blockchain data with PySpark](https://chromia.com/learn/big-data/introduction/)
2. [02Set up your project](https://chromia.com/learn/big-data/setup/)
3. [03Prepare the project](https://chromia.com/learn/big-data/project-launch/)
4. [04Run the project](https://chromia.com/learn/big-data/project-run/)
5. [05Blockchain components](https://chromia.com/learn/big-data/blockchain-side-description/)
6. [06Python components](https://chromia.com/learn/big-data/python-side-description/)

[Learn](https://chromia.com/learn/)[Big data analysis with Chromia and PySpark](https://chromia.com/learn/big-data/)

02 / 06Intermediate

# Set up your project

## Prerequisites

* You can write basic Python.
* You know the core ideas behind blockchains.
* You have a basic grasp of PySpark and Big Data processing.
* You have Python 3.8 or later and the related libraries.
* You work inside a Python virtual environment.

## Set up the Chromia development environment

Complete the following setup steps before you start:

Set up PostgreSQL database

## Set up PostgreSQL database

Rell needs PostgreSQL 16.3. You can use the IDE without a database, but you can't run a node. A console or a remote postchain app runs without a database.

By default, Rell expects these database settings:

* database: `postchain`
* user: `postchain`
* password: `postchain`

## Install

1. Install Homebrew by following the [Homebrew installation guide](https://brew.sh/).
2. Install PostgreSQL:

   shell

   ```
   brew install postgresql@16
   brew services start postgresql@16
   createuser -s postgres
   ```
3. Create and configure the database:

   plsql

   ```nolines
   psql -U postgres -c "CREATE DATABASE postchain WITH TEMPLATE = template0 LC_COLLATE = 'C.UTF-8' LC_CTYPE = 'C.UTF-8' ENCODING 'UTF-8';" -c "CREATE ROLE postchain LOGIN ENCRYPTED PASSWORD 'postchain'; GRANT ALL ON DATABASE postchain TO postchain;"
   ```

   note

   A "peer authentication failed" error means you need to switch the authentication method from `peer` to `md5`. Make this change in the `pg_hba.conf` file of your `psql` database.

1. Install PostgreSQL:

   bash

   ```nolines
   sudo apt install postgresql-16
   ```
2. Create and configure the database:

   bash

   ```nolines
   sudo -u postgres psql -c "CREATE DATABASE postchain WITH TEMPLATE = template0 LC_COLLATE = 'C.UTF-8' LC_CTYPE = 'C.UTF-8' ENCODING 'UTF-8';" -c "CREATE ROLE postchain LOGIN ENCRYPTED PASSWORD 'postchain'; GRANT ALL ON DATABASE postchain TO postchain;"
   ```

1. Install Docker by following the [Docker installation guide](https://docs.docker.com/engine/install/).
2. Create and configure the database:

   dockerfile

   ```nolines
   docker run --name postgres -e POSTGRES_USER=postchain -e POSTGRES_PASSWORD=postchain -p 5432:5432 -d postgres:16.3-alpine3.20
   ```

   note

   The Alpine image of PostgreSQL is used here because its default collation settings are correct. You can also set the collation explicitly with this environment variable:

   bash

   ```nolines
   POSTGRES_INITDB_ARGS="--lc-collate=C.UTF-8 --lc-ctype=C.UTF-8 --encoding=UTF-8"
   ```

1. Get the PostgreSQL installer from the [PostgreSQL download page](https://www.postgresql.org/download/windows/).
2. Run the installer, then add the PostgreSQL folder that holds the binaries to your environment variables. In the Command Prompt (CMD), run the command below. Check that the path to your binaries is correct and substitute the real version number for the `<version>` placeholder:

   shell

   ```nolines
   setx POSTGRESQL "C:\Program Files\PostgreSQL\<version>\bin"
   ```
3. Close and reopen the Command Prompt (CMD) so the new environment variables take effect. Then run these two commands, one after the other, to prepare the database:

   plsql

   ```nolines
   psql -U postgres -c "CREATE DATABASE postchain WITH TEMPLATE = template0 LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8' ENCODING 'UTF-8';"
   psql -U postgres -c "CREATE ROLE postchain LOGIN ENCRYPTED PASSWORD 'postchain'; GRANT ALL ON DATABASE postchain TO postchain;"
   ```

   note

   Some Windows versions produce permission errors or faulty PostgreSQL installations. If you hit these problems, install Docker and run PostgreSQL in a container instead.

Install Chromia CLI

## Install Chromia CLI

Follow these steps to install and update the [Chromia CLI](https://gitlab.com/chromaway/core-tools/chromia-cli).

## Prerequisite

Check that you have the following in place first:

* **PostgreSQL database**: Follow [the PostgreSQL setup guide](https://docs.chromia.com/get-started/installation/#set-up-postgresql-database).
* **RELL\_JAVA environment**: Chromia CLI runs on a Java runtime, version 21 or later. The package managers handle this dependency, so you normally don't configure it yourself. To choose a specific Java runtime for Chromia CLI, set the RELL\_JAVA environment variable to the location of a valid Java installation.

## Installation

Install Chromia CLI through a package manager, or download it from [the Chromia CLI package registry](https://gitlab.com/chromaway/core-tools/chromia-cli/-/packages).

On macOS, install Chromia CLI (`chr`) as follows:

1. Install Homebrew if you don't have it yet:

   bash

   ```nolines
   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
   ```
2. Register the Chromia repository with Homebrew:

   bash

   ```nolines
   brew tap chromia/core https://gitlab.com/chromaway/core-tools/homebrew-chromia.git
   ```
3. Install Chromia CLI:

   bash

   ```nolines
   brew install chromia/core/chr
   ```

   info

   To pin a particular Chromia CLI version, run:

   shell

   ```
   brew install chromia/core/chr@<version>
   brew unlink chr
   brew link chr@<version>
   ```

   To see which versions are available, run `brew search chr`.
4. Confirm the installation by printing the version:

   bash

   ```nolines
   chr --version
   ```

On Linux or WSL (Windows Subsystem for Linux), install Chromia CLI (`chr`) as follows:

1. Fetch Chromia's `apt-repo` public key and add it to your system's trusted keyrings:

   bash

   ```nolines
   curl -fsSL https://apt.chromia.com/chromia.gpg | sudo tee /usr/share/keyrings/chromia.gpg
   ```
2. Add the Chromia repository to your package sources:

   bash

   ```nolines
   echo "deb [arch=amd64 signed-by=/usr/share/keyrings/chromia.gpg] https://apt.chromia.com stable main" | sudo tee /etc/apt/sources.list.d/chromia.list
   ```
3. Refresh your package sources:

   bash

   ```nolines
   sudo apt-get update
   ```

   info

   If you added `apt.chromia.com` before Chromia CLI version `0.16.0`, run this command so the repository can update:

   bash

   ```nolines
   sudo apt-get --allow-releaseinfo-change update
   ```
4. After the update finishes, install Chromia CLI:

   bash

   ```nolines
   sudo apt-get install chr
   ```
5. Confirm the installation by printing the version:

   bash

   ```nolines
   chr --version
   ```

On Windows, install Chromia CLI (`chr`) with [Scoop](https://scoop.sh/):

1. Install Scoop if you don't have it yet. Run this in PowerShell as Administrator:

   powershell

   ```nolines
   iwr -useb get.scoop.sh | iex
   ```
2. Add the Chromia bucket to Scoop:

   powershell

   ```nolines
   scoop bucket add chromia https://gitlab.com/chromaway/core-tools/scoop-chromia/
   ```
3. Add the Java bucket to Scoop:

   powershell

   ```nolines
   scoop bucket add java
   ```

   With this bucket, Scoop can fetch openjdk21, which chromia-cli needs during installation.
4. Install Chromia CLI:

   powershell

   ```nolines
   scoop install chr
   ```
5. Confirm the installation by printing the version:

   powershell

   ```nolines
   chr --version
   ```

## Updating Chromia CLI

Get the newest Chromia CLI from [the package registry](https://gitlab.com/chromaway/core-tools/chromia-cli/-/packages). If a package manager installed it, update it with the command for your platform:

shell

```nolines
brew update
brew upgrade chr
```

shell

```nolines
sudo apt-get update
sudo apt-get install chr
```

shell

```nolines
scoop update
scoop update chr
```

## Docker

You can run Chromia CLI inside a standalone Linux container where it is already installed. Set up the [PostgreSQL database](https://docs.chromia.com/get-started/installation/#set-up-postgresql-database) first.

The published images require Docker to be installed and configured on your host. The Docker documentation explains how to [install Docker](https://docs.docker.com/get-docker/) on Windows, Mac, and Linux.

### Start the Docker container with Chromia CLI pre-installed

To run the latest Chromia CLI, pass the CLI image name and `chr` to the `docker run` command.

shell

```nolines
docker run --rm -v $(pwd):/usr/app registry.gitlab.com/chromaway/core-tools/chromia-cli/chr:<latest version> chr
```

Windows Command Prompt Users

The `$(pwd)` syntax doesn't work in Windows Command Prompt (cmd). Use one of these instead:

* **PowerShell**: `$(Get-Location)` or `$(pwd)`
* **Command Prompt**: `%CD%`

**Command Prompt example:**

cmd

```nolines
docker run --rm -v %CD%:/usr/app registry.gitlab.com/chromaway/core-tools/chromia-cli/chr:<latest version> chr
```

note

Configure your `chromia.yml` file for your platform:

* **Mac**: set `database:host` to `host.docker.internal`.
* **Windows**: set `database:host` to `host.docker.internal` (Docker Desktop). If the node still cannot reach your database, try `172.17.0.1`, the default Docker bridge address on Linux.
* **Linux**: add the `--network=host` argument to your Docker commands.

Without these settings, Chromia CLI can't reach the PostgreSQL instance.

To update or remove the Docker image, consult the [Docker command line reference](https://docs.docker.com/engine/reference/commandline/docker/).

bash

```
#!/bin/bash

# Allocate a pseudo-TTY one when run in interactive mode
if [ -t 0 ] && [ -t 1 ] ; then TTY="--tty"; else TTY=""; fi

docker run \
  # Sets the network to host to not need to change the database hostname (linux only)
  --network=host \
  # Set timezone based on system settings (linux only)
  -e TZ=$(cat /etc/timezone) \
  # Sets process ownership to current user
  --user $(id -u):$(id -g) \
  --mount type=bind,source="/etc/passwd",target=/etc/passwd,readonly \
  --mount type=bind,source="/etc/group",target=/etc/group,readonly \
  # Configures ssh-agent (only needed if chr install is called on non-public repositores)
  -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK \
  --volume "$SSH_AUTH_SOCK:$SSH_AUTH_SOCK" \
  --mount type=bind,source="${HOME}/.ssh",target=${HOME}/.ssh,readonly \
  --mount type=bind,source="${HOME}/.config/jgit",target=${HOME}/.config/jgit \
  # Mounts current folder into the container (Use `Get-Location` on PowerShell)
  --mount type=bind,source="$(pwd)",target=/usr/app \
  --interactive ${TTY} \
  --rm \
  registry.gitlab.com/chromaway/core-tools/chromia-cli/chr:${CHR_VERSION:-latest} chr "$@"
```

Windows Command Prompt Users

This bash script relies on `$(pwd)`, which Windows Command Prompt doesn't support. On Windows:

* **PowerShell**: run the command unchanged; PowerShell also understands `$(pwd)`
* **Command Prompt**: swap `$(pwd)` for `%CD%` in the mount command

**Mount line adjusted for Command Prompt:**

cmd

```nolines
--mount type=bind,source="%CD%",target=/usr/app \
```

## Clone the project repository

shell

```nolines
git clone https://bitbucket.org/chromawallet/big-data-spark/src/main/
```

## Navigate to blockchain components

shell

```nolines
cd main/rell/
```

## Install Rell dependencies

shell

```nolines
chr install
```

## Navigate to PySpark components

shell

```nolines
cd pyspark
```

## Install PySpark dependencies

### Set up a virtual environment

**Linux/WSL:**

shell

```
# Create virtual environment
python3 -m venv venv

# Activate virtual environment
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt
```

**Windows:**

shell

```
# Create virtual environment
python -m venv venv

# Activate virtual environment
venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt
```

**Note:** Activate your virtual environment before you run any of the project's Python scripts.

Written for this site from Chromia’s public course. Code samples are the course’s own.

[← Previous**Analyzing Chromia blockchain data with PySpark**](https://chromia.com/learn/big-data/introduction/)[Next →**Prepare the project**](https://chromia.com/learn/big-data/project-launch/)

On this page

* [Prerequisites](#prerequisites)
* [Set up the Chromia development environment](#set-up-the-chromia-development-environment)
* [Install](#install)
* [Prerequisite](#prerequisite)
* [Installation](#installation)
* [Updating Chromia CLI](#updating-chromia-cli)
* [Docker](#docker)
* [Clone the project repository](#clone-the-project-repository)
* [Navigate to blockchain components](#navigate-to-blockchain-components)
* [Install Rell dependencies](#install-rell-dependencies)
* [Navigate to PySpark components](#navigate-to-pyspark-components)
* [Install PySpark dependencies](#install-pyspark-dependencies)