This project focuses on analyzing 311 service request data to uncover insights and patterns that can help improve city services and urban planning. The goal is to extract meaningful insights, such as identifying high-frequency complaint types, spatial and temporal trends, and potential correlations with external factors like demographics or infrastructure.
This project won the 1st price in the hackerthon.
The City-311-Insights-Challenge aims to leverage data science techniques to process and analyze 311 service request datasets.
It predicts future call volumes using multiple tree-based ML models: XGB regressor, Random Forest, Catboost regressor, LightGBM and Linear Regressor.
Before choosin a model, in order to understand the dataset, EDA is done. To capture the important seasonal features, seasonal patterns and periodogram were used. To recognize the relation between the features, correlation matrix were calculated.
The best performing model is selected using MSE, R-MSE and MAPE score, and later optimized using data optimization methods, such as log-transformation and winsorization, and feature selection using time importance.
Detecting anomaly is performed using statistical methods to filter out the irregular months. The top 10% anomaly values are replaced using the mean scores of normal months, balancing the dataset. Later, the model is retrained using the adjusted dataset, improving the final performace by 10%.
More detailed data visualization is performed on Power BI dashboard. This helps recognize additional points which we failed to cover in the notebook. For example, irregularities in data entry such as the closing date being earlier than the opening date.
- /data: Due to the large size, the datasets are not uploaded here.
- submission.ipynb: Jupyter notebooks with exploratory data analysis, visualizations, and model development.
- submission.csv
- anomalies.csv
- requirements.txt
- documentation.txt
- dashboard.pbix : The file is too large, please check it here
- Video demo
- Python 3.10
- Libraries:(install via
requirements.txt) - Jupyter Notebook (optional, for running
.ipynbfiles)
git clone https://github.com/chaw-thiri/City-311-Insights-Challenge.git
cd City-311-Insights-Challenge
# set up environment
python -m venv .venv
source .venv/bin/activate # On Windows, use `.venv\Scripts\activate`
# install dependencies
pip install -r requirements.txt
jupyter notebook submission.ipynb
Link for data retrieval : https://data.cityofnewyork.us/Social-Services/311-Service-Requests-from-2010-to-Present/erm2-nwe9/about_data