feat(database): SQLAlchemy Models and Migration System (EPIC-1-10)

[deserat]

Summary

Implements comprehensive SQLAlchemy 2.0 ORM models and Alembic migration system for the Noosphere API service with full PostgreSQL + pgvector support.

Models Implemented (9 total)

• User: Multi-tenant base entity with JSONB settings storage
• UserConfig: Key-value configuration with composite primary key (user_id + key)
• Item: Core knowledge item entity with:
  • pgvector embeddings (1536 dimensions for OpenAI compatibility)
  • Full-text search via tsvector (auto-populated by PostgreSQL trigger)
  • PostgreSQL arrays for tags (GIN indexed)
  • ItemState enum (uncategorized, not-started, in-progress, completed, archived)
  • Comprehensive metadata (timestamps, confidence, cadence, content hash)
• ItemLink: Graph relationships between items with self-link prevention
• Conversation: RRD pattern (Retain-Reduce-Discard) for conversation history management
• Prompt: Versioned AI prompt templates with model configuration
• TokenUsage: AI API cost tracking with 6-decimal precision for micro-costs
• AuditLog: Complete audit trail with SET NULL on item delete to preserve history

All models use SQLAlchemy 2.0 patterns (DeclarativeBase, Mapped types, mapped_column).

Migration System

• Initial Alembic migration with all 9 tables, indexes, and constraints
• PostgreSQL trigger for auto-populating search_vector tsvector field
• Proper downgrade function with enum type cleanup for full reversibility
• Support for pgvector, JSONB, ARRAY, TSVECTOR, and custom enums

Key Technical Features

• Fixed circular import issue with late-import pattern for Alembic discovery
• Resolved B-tree index limitation on vector columns (deferred specialized indexes)
• Implemented proper enum cleanup in migration downgrade to enable re-upgrade
• Multi-tenant architecture with CASCADE deletes and SET NULL audit preservation

Test Plan

Database Setup

# Migration should already be applied, but to verify:
cd api-service
uv run alembic current
# Should show: 5a165fbf2690 (head)

Run Tests

# Run full test suite with coverage
uv run pytest tests/ --cov=app/models --cov-report=term

# Expected results:
# - 55 tests passing
# - 100% coverage on all models

Verify Migration Reversibility

# Test downgrade
uv run alembic downgrade -1

# Verify tables dropped
psql -U noosphere_user -d noosphere -c "\dt"

# Test upgrade
uv run alembic upgrade head

# Verify tables created
psql -U noosphere_user -d noosphere -c "\dt"

Test Vector Operations

# Run Python to test pgvector
uv run python -c "
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from app.models.item import Item
from app.models.user import User
import os

engine = create_engine(os.getenv('DATABASE_URL'))
Session = sessionmaker(bind=engine)
session = Session()

# Create test user
user = User(email='test@vector.com')
session.add(user)
session.commit()

# Create item with embedding
item = Item(
    title='Vector Test',
    file_path='/vector-test.md',
    user_id=user.id,
    embedding=[0.1] * 1536,  # 1536-dim vector
    no_ai=False
)
session.add(item)
session.commit()
print(f'✓ Successfully stored {len(item.embedding)}-dimensional vector')

# Cleanup
session.delete(item)
session.delete(user)
session.commit()
"

Test Full-Text Search Trigger

# Verify search_vector is auto-populated
psql -U noosphere_user -d noosphere -c "
SELECT title, search_vector IS NOT NULL as has_search_vector 
FROM items 
LIMIT 5;
"

Quality Checks

• ✅ Test Coverage: 100% (55 tests, all passing)
• ✅ Linting: ruff format + ruff check pass with 0 errors
• ✅ Type Checking: pyright passes with 0 errors on production code
• ✅ Migration Reversibility: Verified upgrade/downgrade cycle works

Implementation Plan

Complete implementation plan documented at:

• docs/dev/plans/EPIC-1-10-sqlalchemy-models-migrations.md

Related Issues

Closes #10

---

🤖 Generated with Claude Code

[gemini-code-assist]

Summary of Changes

Hello @deserat, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request lays the essential database groundwork for the Noosphere API service by introducing a complete set of SQLAlchemy ORM models and an Alembic migration system. It enables the application to store and manage complex data structures, including AI-related features like vector embeddings and prompt templates, while ensuring data integrity and scalability through PostgreSQL-specific types and robust foreign key management. The changes are critical for future development, providing a solid and well-tested data layer.

Highlights

• New SQLAlchemy ORM Models: Introduced 9 comprehensive SQLAlchemy 2.0 ORM models for core entities like User, Item, Conversation, Prompt, TokenUsage, AuditLog, UserConfig, and ItemLink, establishing the foundational data schema for the Noosphere API service.
• Alembic Migration System: Implemented a robust Alembic migration system, including an initial migration script that sets up all 9 tables with appropriate indexes, constraints, and PostgreSQL-specific features like pgvector and TSVECTOR.
• Advanced Database Features: Integrated pgvector for AI vector embeddings (1536 dimensions for OpenAI compatibility) in the Item model, and full-text search capabilities using PostgreSQL TSVECTOR with an auto-populating trigger. Also utilized JSONB for flexible settings and ARRAY for tags.
• Multi-Tenant Architecture & Audit: Designed models with multi-tenancy in mind, using CASCADE deletes for user-related data and SET NULL for AuditLog entries to preserve historical records even if the associated item is deleted.
• Circular Import Resolution: Addressed potential circular import issues for Alembic autogenerate discovery by implementing a late-import pattern within app.db.base.
• Comprehensive Testing: Added extensive unit tests for all new models and database fixtures to ensure correctness, data integrity, and proper functionality of constraints and relationships.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This is an excellent and comprehensive pull request that lays a solid foundation for the application's database layer. The SQLAlchemy models are well-designed using modern 2.0 patterns, and the use of advanced PostgreSQL features is well-executed. The Alembic migration is robust and reversible. The test suite is outstanding, covering not just model logic but also database constraints and relationships, which is crucial for a change of this nature. I've left a couple of minor suggestions related to test assertion strictness and documentation consistency. Fantastic work!

[deserat]

/gemini review

[gemini-code-assist]

Code Review

This is an excellent and comprehensive pull request that lays a solid foundation for the application's database layer. The SQLAlchemy 2.0 models are well-structured, using modern patterns and leveraging PostgreSQL-specific features like pgvector, JSONB, and TSVECTOR effectively. The Alembic migration is clean and the strategy for handling model discovery to avoid circular imports is well-implemented. The accompanying test suite is thorough and demonstrates a strong commitment to quality.

I have a few suggestions to further improve the robustness and correctness of the implementation, primarily concerning data types for financial data, handling of timestamps with timezones, and test environment isolation. These are detailed in the specific comments.

[deserat]

/gemini review

[gemini-code-assist]

Code Review

This is an impressive pull request that establishes a comprehensive and robust database foundation for the application. The implementation of 9 SQLAlchemy models using modern 2.0 patterns, along with a full Alembic migration system and extensive test coverage, is excellent. The use of advanced PostgreSQL features like pgvector and tsvector is well-executed. My review includes one high-severity comment regarding the initial use of timezone-naive datetimes in the first migration, which is a critical point for data integrity, and one medium-severity suggestion to improve a test assertion. Overall, this is a high-quality contribution.

[gemini-code-assist]

The initial migration creates all DateTime columns as timezone-naive (e.g., sa.DateTime()), which poses a significant risk for server-side applications by potentially causing hard-to-debug issues related to timezones. All timestamps should be stored with timezone information from the very beginning.

While a subsequent migration (35617fbafeb4...) corrects this by altering the columns to use sa.DateTime(timezone=True), the best practice is to define the models correctly from the start. This would ensure the initial migration is correct and avoid the need for immediate "fixup" migrations. This comment on the users.created column applies to all other timestamp columns defined in this initial migration file.

[gemini-code-assist]

The assertion assert item.modified >= original_modified is not strict enough to verify that the onupdate trigger has actually changed the timestamp, as it would pass even if the timestamp remained the same. To make this test more robust and ensure the value was updated, you should assert that the modified timestamp is strictly greater than the original one.

Suggested change

	assert item.modified >= original_modified
	assert item.modified > original_modified