NLP News Insights & Automated Newsletter Generator

---

📖 Overview

This project is an end-to-end NLP pipeline designed to process raw news data, classify it into business-relevant categories, group related stories using semantic clustering, and automatically generate summarized newsletters with AI-generated titles.

It is specifically fine-tuned to handle the linguistic nuances of Portuguese (Portugal).

---

🚀 Key Features

The project is divided into two specialized pipelines that work together to transform raw data into actionable insights.

1. Model Training & Optimization (model_training.py)

• Intelligent Classification: Implements a robust Logistic Regression classifier paired with TF-IDF vectorization.
• Automated Hyperparameter Tuning: Powered by Optuna, the system automatically finds the best configuration for ngram_range, max_features, and regularization parameters.
• Handling Class Imbalance: Uses SMOTE (Synthetic Minority Over-sampling Technique) within an imbalanced-learn pipeline to ensure the model performs well even on rare news categories.
• Portuguese-Centric NLP: Pre-processing is optimized for Portuguese (PT-PT), including custom stopword removal and text cleaning via text-hammer.

2. Semantic Clustering & Newsletter Generation (newsletters_generator.py)

• Deep Semantic Embeddings: Leverages sentence-transformers to convert news text into high-dimensional vectors that capture the actual meaning of the articles.
• Advanced Topic Discovery: Uses UMAP for dimensionality reduction and HDBSCAN for density-based clustering, allowing the system to find news "groups" without knowing the number of topics in advance.
• Automated Summarization: Implements the TextRank algorithm to extract the most critical sentences from each news cluster, providing a quick-read summary.
• AI-Powered Title Generation: Integrates with Ollama (Gemma) to generate human-like, creative, and context-aware titles for each identified news topic.

📂 Input Data Structure

The system is designed to process news data in a specific JSON format. Each entry represents an entity (referred to as a "Flag") containing a list of related news articles.

Example Structure:

[
  {
    "flagId": 3825,
    "flagDescription": "Entity Name (e.g., Millennium bcp)",
    "news": [
      {
        "idTelenews": 44867414,
        "header": "Article Headline",
        "newsText": "Full text content of the news article..."
      }
    ]
  }
]

🏗️ Architecture & Workflow

The following diagram illustrates the data flow from raw JSON input to the final newsletter output:

graph TD
    A[Raw Data .json] --> B{Pre-processing}
    B --> C[Model Training & Optuna Optimization]
    B --> D[Sentence Embeddings Generation]
    D --> E[UMAP Dimensionality Reduction]
    E --> F[HDBSCAN Semantic Clustering]
    F --> G[TextRank Summarization]
    G --> H[LLM Title Generation]
    H --> I[Final Newsletter Output]
    style C fill:#979,stroke:#333,stroke-width:2px
    style I fill:#2ea44f,stroke:#24292e,stroke-width:2px,color:#fff

Loading