Skip to content

utils.md

Latest commit

 

History

History
115 lines (80 loc) · 3.43 KB

utils.md

File metadata and controls

115 lines (80 loc) · 3.43 KB
Raw

Utils

Shared infrastructure modules for the project. Handles database connectivity, engine creation, and logging in one place so all other scripts stay consistent and free of repeated setup code.

Folder Structure

utils/
├── __init__.py       — Marks the folder as a Python package
├── connection.py     — Loads and validates database credentials from environment variables
├── engine.py         — Builds the Postgres SQLAlchemy engine and MongoDB client
├── logger.py         — Creates stage-aware loggers with console and file output
└── __pycache__/      — Auto-generated Python bytecode cache (do not edit)

Modules

connection.py

Loads credentials for both PostgreSQL and MongoDB from a .env file using python-dotenv. On import, it validates that all required variables are present and raises an EnvironmentError immediately if any are missing — so connection problems surface at startup rather than mid-run.

Required environment variables:

Variable Database
POSTGRES_HOST PostgreSQL
POSTGRES_PORT PostgreSQL
POSTGRES_DATABASE PostgreSQL
POSTGRES_USERNAME PostgreSQL
POSTGRES_PASSWORD PostgreSQL
MONGO_URI MongoDB
MONGO_DB MongoDB

Also exposes get_mongo_db() which returns a connected MongoDB database instance directly.

engine.py

Builds ready-to-use database engine objects using the credentials from connection.py.

Function Returns
postgres_engine() A SQLAlchemy engine using psycopg2, with connection pooling (pool_size=5, max_overflow=10, pool_pre_ping=True)
mongo_client() A PyMongo database object connected to the configured MongoDB database

Both functions log a success message on connection and raise the original error on failure.

logger.py

Provides get_logger(stage, name) — a factory that returns a configured logging.Logger tied to a specific pipeline stage.

Parameters:

  • stage — must be one of "extraction", "transformation", or "loading"
  • name — a label for the specific script or task (e.g. "engines", "customers")

Behaviour:

  • Creates a logs/<stage>/ directory automatically
  • Writes a timestamped log file at logs/<stage>/<name>_YYYY-MM-DD_HH-MM.log
  • Console handler logs INFO and above
  • File handler logs DEBUG and above
  • Format: YYYY-MM-DD HH:MM:SS | LEVEL | message
  • Safe to call multiple times — duplicate handlers are skipped

How to Use

from utils.connection import get_mongo_db
from utils.engine import postgres_engine, mongo_client
from utils.logger import get_logger

# Logger
logger = get_logger("extraction", "customers")

# Postgres
engine = postgres_engine()

# MongoDB
db = mongo_client()
# or directly:
db = get_mongo_db()

Environment Setup

Create a .env file at the project root with the following:

POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DATABASE=your_db
POSTGRES_USERNAME=your_user
POSTGRES_PASSWORD=your_password

MONGO_URI=mongodb://localhost:XXXXX
MONGO_DB=your_mongo_db

Notes

  • Never hardcode credentials — all secrets must be defined in .env and excluded from version control via .gitignore.
  • Do not modify __pycache__ — it is auto-generated by Python and safe to delete if needed.
  • The logger stage must match exactly: "extraction", "transformation", or "loading".