Preprocess movie data

You will download a real-world movie dataset and turn it into a clean, structured CSV file. The next step embeds the plot summaries from this file as vectors. The remaining metadata is used to display search results.

1. Download the dataset

From the python/ folder, run:

python download_data.py

The script downloads the CMU Movie Summary Corpus and unpacks it into a MovieSummaries/ folder at the root of your project.

2. Preprocess the data

Once the download finishes, run:

python preprocess.py

The script loads the raw plot summaries and the movie metadata, joins them, and writes the cleaned result to:

• data/movie_data.csv

What’s in the dataset?

Every row of the cleaned CSV has these fields:

• wiki_id: unique movie identifier that ties the data together
• title: name of the movie
• plot: complete plot summary, which is the input for vector embedding
• release_date
• box_office
• runtime
• languages
• countries
• genres

The plot field is converted to a vector and stored on-chain together with the movie metadata. This gives you a decentralized system that supports semantic search.

What’s next?

Next, you will create sentence embeddings from the movie plots with the model you chose earlier.