Skip to content

AGENTS.md

Latest commit

 

History

History
422 lines (320 loc) · 21.9 KB

AGENTS.md

File metadata and controls

422 lines (320 loc) · 21.9 KB
Raw

Agent instructions

Before editing or inspecting a target path, read every applicable AGENTS.md from the repository root through the target directory. Do not load instructions from unrelated subtrees.

This file provides guidance to coding agents working in this repository.

Version Control

  • This repository uses jj. Before changing files, load working-with-jj and codealive-jj-development, stay in the current or assigned root, and use jj—not raw Git—for VCS mutations.

Development Commands

Installation and Setup

# Using uv (recommended)
uv venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
uv pip install -e .

# Using pip
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
pip install -e .

Dependency Rules

  • Keep Python dependency specifiers exact where this repository already pins them, including test extras and build-system.
  • If pyproject.toml changes affect resolved packages, verify the result through the lock/install path used by CI.
  • Do not introduce floating versions in CI or release automation when exact pins are practical.
  • Do not adopt a versioned dependency, runtime, build tool, or GitHub Action until its release is more than seven days old. Keep tool.uv.exclude-newer = "7 days" and audit release dates for non-Python tooling separately.

Running the MCP Server

# Basic run with stdio transport
python src/codealive_mcp_server.py

# With debug mode enabled
python src/codealive_mcp_server.py --debug

# With SSE transport
python src/codealive_mcp_server.py --transport sse --host 0.0.0.0 --port 8000

# With custom API key and base URL
python src/codealive_mcp_server.py --api-key YOUR_KEY --base-url https://custom.url

Docker Usage

# Build Docker image
docker build -t codealive-mcp .

# Run with Docker
docker run --rm -i -e CODEALIVE_API_KEY=your_key_here codealive-mcp

Testing

Quick Smoke Test

After making local changes, quickly verify everything works:

# Using make (recommended)
make smoke-test

# Or directly
python smoke_test.py

# With valid API key for full testing
CODEALIVE_API_KEY=your_key python smoke_test.py

The smoke test:

  • ✓ Verifies server starts and connects via stdio
  • ✓ Checks all tools are registered correctly
  • ✓ Tests each tool responds appropriately
  • ✓ Validates parameter handling
  • ✓ Runs in ~5 seconds

Unit Tests

Run comprehensive unit tests with pytest:

# Using make
make unit-test

# Or directly
pytest src/tests/ -v

# With coverage
pytest src/tests/ -v --cov=src

All Tests

Run both smoke tests and unit tests:

make test

Architecture

This is a Model Context Protocol (MCP) server that provides AI clients with access to CodeAlive's semantic code search and analysis capabilities.

Core Components

  • codealive_mcp_server.py: Main entry point — bootstraps logging, tracing, registers tools and middleware
  • Eleven public tools: get_data_sources, semantic_search, grep_search, get_repository_ontology, get_file_tree, read_file, fetch_artifacts, get_artifact_relationships, get_artifact_query_schema, query_artifact_metadata, chat. Review capabilities additionally advertise ask_codebase (routed to /api/tools/ask_codebase) and hide chat; reviewers also hide ontology. Web Server remains the authorization boundary.
  • core/client.py: CodeAliveContext dataclass + codealive_lifespan (httpx.AsyncClient lifecycle, _server_ready flag)
  • core/logging.py: loguru structured JSON logging + PII masking + OTel context injection
  • core/observability.py: OpenTelemetry TracerProvider setup with OTLP export
  • middleware/: N8NRemoveParametersMiddleware (strips n8n extra params) + ObservabilityMiddleware (OTel spans per tool call)

Key Architectural Patterns

  1. FastMCP Framework: Uses FastMCP 3.x with lifespan context, middleware hooks, and built-in Client for testing
  2. HTTP Auth via get_http_headers: FastMCP 3.x strips the authorization header by default (to prevent accidental credential forwarding to downstream services). Our get_api_key_from_context() in core/client.py must use get_http_headers(include={"authorization"}) to read Bearer tokens from HTTP/streamable-http clients. Do not remove the include= parameter — without it, all HTTP-transport clients (LibreChat, n8n, etc.) will fail with a misleading STDIO-mode error.
  3. HTTP Client Management: Single persistent httpx.AsyncClient with connection pooling, created in lifespan
  4. Tool API v3 Backend Contract: every MCP tool delegates to POST /api/tools/{name} and requests output_format=agentic
  5. Environment Configuration: Supports both .env files and command-line arguments with precedence
  6. Error Handling: Centralized in utils/errors.py — all tools use handle_api_error() with method= prefix
  7. N8N Middleware: Strips extra parameters (sessionId, action, chatInput, toolCallId) from n8n tool calls before validation
  8. Observability Middleware: Wraps every tools/call in an OTel span with GenAI semantic conventions

Data Flow

  1. AI client connects to MCP server via stdio/HTTP transport
  2. Client calls tools (get_data_sources → semantic_search / grep_search → fetch_artifacts / get_artifact_relationships → chat only if synthesis is still needed)
  3. Middleware chain runs: N8N cleanup → ObservabilityMiddleware (OTel span + log correlation)
  4. Tool translates MCP call to CodeAlive API request (with X-CodeAlive-* headers)
  5. Response parsed and returned to the AI client — as a dict for metadata/discovery tools, as an XML string for fetch_artifacts, or as plain text for chat

Environment Variables

  • CODEALIVE_API_KEY: Required API key for CodeAlive service
  • CODEALIVE_BASE_URL: API base URL (defaults to https://app.codealive.ai)
  • CODEALIVE_IGNORE_SSL: Set to disable SSL verification (debug mode)
  • DEBUG_MODE: Set to true to enable DEBUG-level logging
  • OTEL_EXPORTER_OTLP_ENDPOINT: If set, traces are exported via OTLP/HTTP to this endpoint

Data Source Types

  • Repository: Individual code repositories with URL and repository ID
  • Workspace: Collections of repositories accessible via workspace ID
  • Tool calls can target specific repositories or entire workspaces for broader context

Integration Patterns

The server is designed to integrate with:

  • Claude Desktop/Code (via settings.json configuration)
  • Cursor (via MCP settings panel)
  • VS Code with GitHub Copilot (via settings.json)
  • Continue (via config.yaml)
  • n8n (via AI Agent node with MCP tools)
  • Any MCP-compatible AI client

Key integration considerations:

  • AI clients should use get_data_sources first to discover available repositories/workspaces, then default to semantic_search and grep_search for evidence gathering; use chat only as a slower synthesis fallback
  • n8n Integration: The server includes middleware to automatically strip n8n's extra parameters (sessionId, action, chatInput, toolCallId) from tool calls, so n8n works out of the box without any special configuration

Logging Best Practices

This project uses loguru for structured JSON logging. All logs go to stderr (safe for stdio MCP transport). Follow these rules strictly when writing or modifying code.

Rules

  1. Always use loguru, never print() or stdlib logging. Import: from loguru import logger. No import logging in application code — stdlib is intercepted via _InterceptHandler and routed through loguru automatically.

  2. All logs go to stderr. The stdio MCP transport uses stdout for protocol messages. Any stray print() or stdout write will corrupt the MCP protocol and break the client. If you add a new log sink, it must target sys.stderr.

  3. Never call response.text without a debug guard. log_api_response() is protected by _is_debug_enabled() because reading response.text consumes the response body. If you add new response logging, always check _is_debug_enabled() first:

    if not _is_debug_enabled():
    return  # Do NOT touch response body at INFO level

  4. Mask PII in logs. User queries, questions, messages, and response bodies must never appear in full in logs. Use _sanitize_body() for request bodies and truncate response bodies to _RESPONSE_BODY_MAX_LEN. The PII fields list is in _PII_FIELDS — extend it if you add tools that accept user content.

  5. Mask Authorization headers. Always replace Authorization header values with "Bearer ***" in logs. See the pattern in log_api_request().

  6. Use structured logging fields, not string interpolation. Prefer logger.bind(request_id=rid).debug("message") or logger.info("msg {key}", key=value) over f-strings in the message. This makes logs machine-parseable.

  7. Use logger.configure(patcher=...) for global context injection (like OTel trace_id). Do NOT pass patcher to logger.add() — loguru 0.7.x does not support it there.

  8. Tool-call failures are warnings with full structured arguments. Do not log MCP tool-call failures as error unless the whole server process is failing. A bad tool call is recoverable input for the model loop, same as backend agent tools. Log it as logger.warning(..., tool_arguments={...}) or with .bind(tool_arguments=...). Do not redact tool_arguments in the logger path; the purpose is to recover the exact failing invocation. Authorization headers remain masked via log_api_request().

  9. Tool-call lifecycle logs are debug. The per-tool "started" and "completed" messages from ObservabilityMiddleware must be logger.debug, not logger.info. Info-level logs should not be emitted for every agent step/tool step.

OTel Trace Correlation

Every log record automatically gets trace_id and span_id injected by _otel_patcher (registered via logger.configure). The ObservabilityMiddleware also uses logger.contextualize(trace_id=..., tool=...) so all logs within a tool call carry the correlation ID. Do not duplicate this — it's automatic.

Observability Best Practices

OpenTelemetry Tracing

  • TracerProvider is initialized once in core/observability.py via init_tracing(), called at startup in main().
  • If OTEL_EXPORTER_OTLP_ENDPOINT is set, traces export via OTLP/HTTP; otherwise a no-op provider is used (trace IDs still appear in logs).
  • atexit.register(provider.shutdown) ensures pending spans are flushed on process exit. Do not skip this if modifying the init logic.
  • HTTPX auto-instrumentation (HTTPXClientInstrumentor) injects traceparent headers into all outbound HTTP calls. Do not add manual propagation.

Middleware Spans

The ObservabilityMiddleware creates a span per tool call with these attributes:

  • gen_ai.operation.name = "execute_tool"
  • gen_ai.tool.name / mcp.tool.name = tool name
  • mcp.method = "tools/call"

On errors, the span gets StatusCode.ERROR + record_exception(). Do not add redundant span creation inside tool functions — the middleware handles it.

Required MCP observability fix pattern

When touching MCP tool observability, update both the generic middleware and the tool-specific body:

  • In src/middleware/observability_middleware.py, extract tool arguments from the incoming tools/call message (FastMCP currently exposes this through the message payload; keep the extraction defensive). Add them to the log context, e.g. with logger.contextualize(trace_id=trace_id, tool=tool_name, tool_arguments=tool_arguments): ....
  • Change middleware lifecycle logs:
    • logger.info("Tool call started...") -> logger.debug(...)
    • logger.info("Tool call completed...") -> logger.debug(...)
    • logger.error("Tool call failed...") -> logger.warning(..., tool_arguments=tool_arguments)
  • Keep OTel span semantics unchanged: failed tool calls should still set StatusCode.ERROR and record_exception(exc) because tracing represents the tool invocation outcome, while logs use Warning to avoid misclassifying a recoverable model/tool-call error as a server crash.
  • In each tool body, log in-band validation failures before raising ToolError. Include all tool parameters in tool_arguments.

Concrete example: src/tools/artifact_relationships.py::get_artifact_relationships must log these branches as Warning with full arguments:

tool_arguments = {
    "identifier": identifier,
    "profile": profile,
    "max_count_per_type": max_count_per_type,
}
  • missing/empty identifier
  • max_count_per_type outside 1..1000
  • unsupported profile fallback branch
  • backend HTTPStatusError / unexpected exception before delegating to handle_api_error(...)

The API request/response helpers stay Debug and keep their existing masking rules. Do not put raw response bodies into warning logs.

Adding New Tools — Observability Checklist

When adding a new tool, ensure:

  1. The tool receives ctx: Context as its first argument (required for lifespan context and logging)
  2. API requests include all four X-CodeAlive-* headers: Integration, Tool, Client, plus Authorization
  3. Call log_api_request() before and log_api_response() after the HTTP call
  4. Errors are logged as Warning with full tool_arguments before they go through handle_api_error(ctx, e, "description", method=_TOOL_NAME) — this ensures the [tool_name] prefix in error messages and preserves the exact failed call in logs
  5. The middleware automatically wraps the tool in an OTel span — no manual span creation needed

Tool Response Conventions

Response format: dict for metadata/discovery, XML only for source code

Tools that return structured metadata (identifiers, match counts, line numbers, relationship groups, data source listings) return a dict (or list of dicts). FastMCP serializes it automatically via pydantic_core.to_json, which preserves Unicode — no manual json.dumps() needed. Examples: semantic_search, grep_search, get_data_sources, get_repository_ontology, get_file_tree, read_file, get_artifact_relationships, get_artifact_query_schema, and query_artifact_metadata.

Never call json.dumps(...) from a tool's return path. Python's json.dumps defaults to ensure_ascii=True and escapes Cyrillic/CJK/etc. to \uXXXX. Returning a dict lets FastMCP route through pydantic_core.to_json, which emits UTF-8. If you must serialize manually for some reason, pass ensure_ascii=False explicitly.

Only fetch_artifacts returns an XML string. XML tags give the LLM clear structural boundaries between artifacts, content blocks, and inline relationships when streaming source code — this is critical for accurate reasoning over multi-artifact responses. Do not convert fetch_artifacts to dict/JSON — the XML structure is intentional.

Hint other MCP tools when the response implies a follow-up call

If a tool's response is meant to be used as input to another MCP tool, the response itself MUST embed a hint (or equivalent) directing the agent to that follow-up tool. The hint should explain what to call next, with what value, and why. Do NOT rely on the agent to remember workflow rules from the tool description alone — descriptions are not always re-read mid-conversation, but the response is always in front of the model when it decides what to do next.

Examples in this repo:

  • semantic_search and grep_search return a hint field telling the agent that description is a triage pointer only and that real understanding must come from fetch_artifacts(identifier) or a local Read(path). Implementation: _SEARCH_HINT in src/utils/response_transformer.py.
  • fetch_artifacts emits a <hint>…get_artifact_relationships…</hint> element whenever an artifact has call relationships, telling the agent it can drill down further. Implementation: _build_artifacts_xml in src/tools/fetch_artifacts.py.
  • fetch_artifacts also emits a <not_found count="N"> block plus a <hint> (_NOT_FOUND_HINT) whenever a requested identifier is missing or inaccessible — it relies on the backend found flag, falling back to content is null and a requested-vs-returned diff so it never silently drops a requested artifact. Implementation: _build_artifacts_xml in src/tools/fetch_artifacts.py.

When you add or change a tool whose output is structurally a "pointer" to data held by another tool (identifiers, IDs, references), add or update the hint in the same change. If you remove a follow-up workflow, remove the stale hint too.

Testing Best Practices

The project has tests across four tiers: unit tests, e2e tool tests, smoke tests, and integration tests.

Test Tiers

Tier Files What it tests How to run
Unit test_*.py (except test_e2e_*) Individual functions, XML builders, error handling, PII masking, middleware pytest src/tests/ -v
E2E test_e2e_tools.py Full MCP protocol: Client → FastMCP → tool → mock HTTP → response pytest src/tests/test_e2e_tools.py -v
Smoke smoke_test.py Real server startup via stdio, tool registration, basic invocations python smoke_test.py
Integration integration_test.py All tools against live backend — verifies filtering params (max_results, paths, extensions, regex), relationship profiles, validation edges, full agent workflow CODEALIVE_API_KEY=... python integration_test.py

Integration tests require a valid CODEALIVE_API_KEY and network access. They auto-select the CodeAlive backend repo as target, or accept --target <name>. Use make integration-test as a shortcut.

E2E Test Pattern (FastMCP Client)

E2E tests use FastMCP's built-in Client class with httpx.MockTransport for the backend. This is the canonical pattern — use it for all new tool tests:

from fastmcp import Client, FastMCP

def _server(routes: dict) -> FastMCP:
    @asynccontextmanager
    async def lifespan(server):
        transport = httpx.MockTransport(handler_dispatching_by_path)
        async with httpx.AsyncClient(transport=transport, base_url="https://test.local") as client:
            yield CodeAliveContext(client=client, api_key="", base_url="https://test.local")

    mcp = FastMCP("Test", lifespan=lifespan)
    mcp.tool()(your_tool)
    return mcp

async def test_tool():
    mcp = _server({"/api/endpoint": lambda r: httpx.Response(200, json={...})})
    async with Client(mcp) as client:
        result = await client.call_tool("tool_name", {"arg": "value"})
    assert "expected" in result.content[0].text

Key points:

  • httpx.MockTransport — no network, no external dependencies, tests run in < 1 second
  • Custom lifespan yields a real CodeAliveContext with a mock-backed httpx client
  • monkeypatch.setenv("CODEALIVE_API_KEY", ...) for get_api_key_from_context fallback
  • Use raise_on_error=False when testing error paths, then assert on result.content[0].text
  • For chat-style buffered responses, return httpx.Response(200, json=payload) and assert against the Tool API v3 envelope content

Unit Test Patterns

  • OTel tests: Do NOT use trace.set_tracer_provider() in tests — it's global and can only be called once per process. Instead, patch the module-level _tracer variable: 
    with patch("middleware.observability_middleware._tracer", test_tracer):
    ...
  • Logging level tests: Set logging_module._current_level = "DEBUG" in setup_method, restore to "INFO" in teardown_method.
  • Avoid InMemorySpanExporter — it doesn't exist in current OTel SDK. Use a custom collector: 
    class _CollectingExporter(SpanExporter):
    def __init__(self):
        self.spans = []
    def export(self, spans):
        self.spans.extend(spans)
        return SpanExportResult.SUCCESS

Testing Pitfalls to Avoid

  1. Never mock what you can test through the protocol. Prefer e2e tests with Client(mcp) over mocking ctx, ctx.request_context, etc. The e2e approach catches middleware issues, lifespan bugs, and serialization problems that mocks hide.
  2. Never consume response.text in tests and then test streaming. The aiter_lines() method works on MockTransport responses because httpx buffers the content, but calling .text first would consume it.
  3. Always test both success and error paths. Every tool should have at least: happy path, empty/invalid input, backend HTTP error.
  4. Use monkeypatch for env vars, not os.environ directly. This ensures cleanup even if a test fails.
  5. Mark async tests with @pytest.mark.asyncio. The project uses asyncio_mode = "strict" — unmarked async tests will be silently skipped.

Publishing and Releases

Before any authorized jj git push, run .githooks/pre-push manually from the canonical checkout. Secondary jj workspaces have no .git, and Git hooks are not a jj policy boundary.

Version Management — Keep All Three Files in Sync

The version is declared in three places that MUST stay consistent:

File Field Role
pyproject.toml [tool.setuptools_scm] fallback_version Source of truth. Used by setuptools-scm when no git tag is present.
manifest.json "version" MCP Registry manifest (Claude Desktop discovery).
server.json "version" MCP Registry server schema (registry listing).

When bumping the version, update all three files in the same commit.

Automated Publishing

The project uses automated publishing:

  • Trigger: Push version change to main branch
  • Process: Tests → Build → Docker → MCP Registry → GitHub Release
  • Result: Available at io.github.codealive-ai/codealive-mcp in MCP Registry

Version Guidelines

  • Patch (0.2.0 → 0.2.1): Bug fixes, minor improvements
  • Minor (0.2.0 → 0.3.0): New features, enhancements
  • Major (0.2.0 → 1.0.0): Breaking changes, major releases

When implementing features or fixes, evaluate if they warrant a version bump for users to benefit from the changes through the MCP Registry.