A Big Data Analytics Laboratory project that analyzes BoardGameGeek game data using Apache Hadoop MapReduce and Apache Pig.
The project contains:
- 3 Hadoop MapReduce analyses
- 5 Apache Pig analyses
- Dataset preprocessing using Python
- HDFS-based data processing
- Result files and execution screenshots
- Pig result visualizations
- A cleaned dataset sample
- Final laboratory report
The objective of this project is to process and analyze a BoardGameGeek dataset using Hadoop and Apache Pig.
The analysis focuses on game ownership, user ratings, publishers, and primary categories.
As required for the project:
- 3 MapReduce programs were implemented
- 5 Apache Pig analyses were implemented
- Hive was not used
The project uses the BoardGameGeek Reviews dataset from Kaggle.
Dataset source: BoardGameGeek Reviews on Kaggle
The file used for this project was:
games_detailed_info.csv
- Records: 21,631
- Columns: 56
- Duplicate game IDs: 0
- Malformed CSV rows: 0
After preprocessing:
- Records: 21,347
- Fields: 13
- Primary categories: 81
- Primary publishers: 3,824
The category and publisher columns in the original dataset may contain multiple values. To preserve one record per game, the first listed category and first listed publisher were selected as primaryCategory and primaryPublisher.
The complete raw dataset is not stored in this repository. A small cleaned sample is available at:
dataset/boardgames_sample.csv
More information is available in:
dataset/README.md
| No. | Attribute | Description |
|---|---|---|
| 1 | gameID |
Unique game identifier |
| 2 | name |
Game title |
| 3 | yearPublished |
Publication year |
| 4 | primaryPublisher |
First listed publisher |
| 5 | primaryCategory |
First listed category |
| 6 | minPlayers |
Minimum number of players |
| 7 | maxPlayers |
Maximum number of players |
| 8 | playingTime |
Playing time |
| 9 | usersRated |
Number of users who rated |
| 10 | averageRating |
Average rating |
| 11 | owned |
Ownership count |
| 12 | numComments |
Number of comments |
| 13 | averageWeight |
Average complexity value |
The raw CSV file was cleaned using Python.
The preprocessing script:
preprocessing/prepare_boardgames.py
Main preprocessing operations:
- Parsed the original CSV safely
- Extracted the first listed category as
primaryCategory - Extracted the first listed publisher as
primaryPublisher - Removed records with missing required values
- Cleaned embedded commas and line breaks
- Reduced the dataset from 56 columns to 13 required fields
- Preserved one output record per valid game
- Removed the CSV header before Hadoop processing
Final cleaned record count:
21,347
The project was executed using:
- Apache Hadoop: 3.4.1
- Apache Pig: 0.17.0
- OpenJDK: 11.0.32
- Operating System: macOS
- Hadoop mode: Pseudo-distributed
The Hadoop services used included:
- NameNode
- DataNode
- SecondaryNameNode
- ResourceManager
- NodeManager
- JobHistoryServer
The cleaned dataset was uploaded to HDFS at:
/boardgame/boardgames.csv
Find the board game with the highest owned value.
30549 Pandemic 168364
Pandemic is the most-owned game in the cleaned dataset with 168,364 owners.
MapReduce/MR1_MostOwned/
results/mapreduce/mr1_most_owned.txt
Generate one summary record for each game containing:
- Average rating
- Number of users who rated the game
- Number of comments
The output contains exactly:
21,347 records
Example for Pandemic:
30549 7.58896 109006 17305
MapReduce/MR2_RatingEngagement/
results/mapreduce/mr2_rating_engagement.txt
Count the number of games associated with each primaryPublisher.
The analysis produced:
3,824 primary publishers
| Publisher | Games |
|---|---|
| (Self-Published) | 543 |
| (Web published) | 380 |
| Hasbro | 379 |
| Decision Games (I) | 272 |
| GMT Games | 268 |
| Ravensburger | 255 |
| AMIGO | 249 |
| (Public Domain) | 214 |
| KOSMOS | 212 |
| 999 Games | 208 |
MapReduce/MR3_GamesPerPublisher/
results/mapreduce/mr3_games_per_publisher.txt
Games were grouped by primaryCategory, counted, and ranked by the number of games.
| Category | Games |
|---|---|
| Card Game | 4,615 |
| Abstract Strategy | 1,544 |
| Adventure | 1,144 |
| Action / Dexterity | 1,057 |
| Animals | 1,049 |
Card Game is the largest primary category with 4,615 games.
results/pig/pig1_top5_categories.txt
Games were sorted using usersRated.
Here, most-rated means the highest number of submitted user ratings, not the highest average rating.
30549|Pandemic|Medical|109006
Pandemic has the highest number of user ratings with 109,006 ratings.
results/pig/pig2_top10_most_rated.txt
Games were sorted according to the owned field.
30549|Pandemic|Medical|168364
Pandemic is the most-owned game with 168,364 owners.
This result matches MR-1.
results/pig/pig3_top10_most_owned.txt
Games were grouped by primaryCategory. Within each category, games were sorted by usersRated and the top 10 were retained.
The cleaned dataset contains:
81 primary categories
The analysis produced:
800 output records
The result is lower than 810 because some categories contain fewer than 10 games.
36218|Dominion|Card Game|81582
178900|Codenames|Card Game|74456
148228|Splendor|Card Game|64734
129622|Love Letter|Card Game|59484
39856|Dixit|Card Game|54525
28143|Race for the Galaxy|Card Game|48919
1927|Munchkin|Card Game|43759
98778|Hanabi|Card Game|41588
11|Bohnanza|Card Game|40113
50|Lost Cities|Card Game|39535
results/pig/pig4_top10_rated_by_category.txt
The chart below shows the Card Game category as a representative category-wise visualization.
Games were grouped by primaryCategory. Within each category, games were sorted by owned and the top 10 were retained.
The analysis produced:
800 output records across 81 primary categories
178900|Codenames|Card Game|119753
36218|Dominion|Card Game|106956
129622|Love Letter|Card Game|98395
148228|Splendor|Card Game|92366
1927|Munchkin|Card Game|78849
39856|Dixit|Card Game|76535
98778|Hanabi|Card Game|68643
133473|Sushi Go!|Card Game|65495
11|Bohnanza|Card Game|60282
28143|Race for the Galaxy|Card Game|59758
results/pig/pig5_top10_owned_by_category.txt
The chart below shows the Card Game category as a representative category-wise visualization.
Apache Pig 0.17.0 was executed in MapReduce mode with Hadoop 3.4.1.
During testing, a top-level ORDER BY operation caused a compatibility problem. The working Pig scripts therefore use nested ORDER BY operations inside grouped FOREACH blocks where required.
Execution evidence is stored in the screenshots/ directory.
| Screenshot | Description |
|---|---|
ss1_setup.png |
Hadoop environment, running services and HDFS files |
ss2_mr1.png |
MR-1 most-owned result and verification |
ss3_mr2.png |
MR-2 rating and engagement output |
ss4_mr3.png |
MR-3 publisher counts |
ss5_pig1.png |
Pig-1 top primary categories |
ss6_pig2.png |
Pig-2 top most-rated games |
ss7_pig3.png |
Pig-3 top most-owned games |
ss8_pig4.png |
Pig-4 category-wise most-rated analysis |
ss9_pig5.png |
Pig-5 category-wise most-owned analysis |
| Analysis | Final Result |
|---|---|
| MR-1: Most Owned Game | Pandemic — 168,364 owners |
| MR-2: Rating Summary | 21,347 output records |
| MR-3: Games per Publisher | (Self-Published) — 543 games; 3,824 publishers |
| Pig-1: Top 5 Categories | Card Game — 4,615 games |
| Pig-2: Top 10 Most-Rated | Pandemic — 109,006 users rated |
| Pig-3: Top 10 Most-Owned | Pandemic — 168,364 owners |
| Pig-4: Rated by Category | 800 output records across 81 categories |
| Pig-5: Owned by Category | 800 output records across 81 categories |
- Pandemic is the most-owned game with 168,364 owners.
- Pandemic has the largest number of user ratings with 109,006 ratings.
- Card Game is the largest primary category with 4,615 games.
- (Self-Published) is the largest primary publisher group with 543 games.
- The cleaned dataset contains 21,347 games.
- The cleaned dataset contains 81 primary categories and 3,824 primary publishers.
boardgame_lab/
├── MapReduce/
│ ├── MR1_MostOwned/
│ ├── MR2_RatingEngagement/
│ └── MR3_GamesPerPublisher/
├── PigAnalysis/
├── preprocessing/
│ └── prepare_boardgames.py
├── dataset/
│ ├── README.md
│ └── boardgames_sample.csv
├── results/
│ ├── mapreduce/
│ └── pig/
├── screenshots/
├── visualizations/
│ └── pig/
│ ├── pig1_top5_categories.png
│ ├── pig2_top10_most_rated.png
│ ├── pig3_top10_most_owned.png
│ ├── pig4_card_game_top10_rated.png
│ ├── pig5_card_game_top10_owned.png
│ └── visualize_pig_results.py
├── report/
│ └── BDAL_lab_report.pdf
├── README.md
└── .gitignore
The complete laboratory report is available here:
- BoardGameGeek Reviews Dataset - Kaggle
- BoardGameGeek XML API2
- Apache Hadoop 3.4.1 Documentation
- Apache Pig 0.17.0 Documentation
Tahsin Ahmed Rafi
Department of Computer Science and Engineering Premier University, Chittagong
Course: Big Data Analytics Laboratory (CSE 4346)