more module 3

Author: jeffheaton

module_3/.memory/reference/api-spreadsheet-library.md

@@ -0,0 +1,24 @@
+---
+classification: internal
+project: proj-csv
+doc_type: reference
+---
+
+## Spreadsheet Library Reference
+
+The CSV export service uses **csv-stream-writer** (version 2.x) as its spreadsheet and CSV generation library. This library was selected because it supports true streaming output, has no transitive dependencies, and produces RFC 4180-compliant CSV without requiring the caller to manage quoting or escaping manually.
+
+The library is initialized once per export job with a target writable stream. Column headers are declared at initialization time and cannot be changed after the first row is written. Each row is passed to the library as a plain object whose keys match the declared header names; the library handles type coercion, special-character escaping, and line termination automatically.
+
+The library does not support XLSX or ODS output. Any future requirement to generate spreadsheet formats other than CSV will require either adding a second library or replacing csv-stream-writer with a multi-format library. That decision must go through the standard format-change approval process described in the export format decision document.
+
+Usage example:
+
+```python
+writer = CsvStreamWriter(stream, columns=["id", "title", "status", "due_date"])
+for task in task_batch:
+    writer.write_row(task)
+writer.close()
+```
+
+The library is pinned to a minor version in the service's dependency manifest. Patch updates may be applied without review. Minor or major version upgrades require a changelog review and a regression run against the export integration test suite before they are merged.

module_3/.memory/reference/cost-breakdown.md

@@ -0,0 +1,15 @@
+---
+classification: confidential
+project: proj-csv
+doc_type: finance
+---
+
+## Internal Cost Figures: CSV Export Feature
+
+This document contains confidential financial projections and actual cost data for the CSV export feature. It is restricted to finance leads and senior engineering management.
+
+The infrastructure cost for the export feature in the first quarter of operation was $4,200, broken down as follows: compute for the export workers accounted for $1,800, object storage for generated export files accounted for $900, egress bandwidth for file downloads accounted for $1,100, and monitoring and alerting overhead accounted for $400. These figures are based on an average of 3,200 export jobs per month across all projects.
+
+Projected annual cost at current growth rates is $67,000, assuming a 40 percent increase in export volume driven by new enterprise customer onboarding. The unit cost per export job is expected to decrease from $1.31 to $0.94 as batch-processing optimizations ship in Q3. Cost reduction proposals under review include moving completed export files to a cheaper storage tier after 72 hours and capping retention at 30 days, which would reduce storage costs by an estimated 35 percent.
+
+Do not share these figures outside approved channels. All cost discussions in public project documents must reference only the feature's relative priority tier, not dollar amounts.

module_3/.memory/reference/decision-csv-format.md

@@ -0,0 +1,13 @@
+---
+classification: internal
+project: proj-csv
+doc_type: decision
+---
+
+## Decision: CSV as the Export File Format for the Task List
+
+After evaluating several candidate formats including JSON, XLSX, and plain CSV, the team decided to use CSV as the standard file format for exporting the task list. CSV was chosen because it is universally supported by spreadsheet applications, requires no special libraries to open, and produces compact output that is easy to diff in version control. The format aligns with what our primary users—project managers and team leads—already use in their day-to-day tooling.
+
+Alternative formats were considered and rejected for the following reasons. JSON was ruled out because non-technical stakeholders cannot open it without additional tooling. XLSX was ruled out due to binary format complexity, licensing concerns around third-party spreadsheet libraries, and the additional dependency weight it would add to the export service. Plain text was too unstructured to be useful for downstream import workflows.
+
+The decision is considered stable. Any future proposal to change the export format must include a migration plan for existing integrations and must be approved by the product lead before implementation begins.

module_3/.memory/reference/error-codes.md

@@ -0,0 +1,21 @@
+---
+classification: internal
+project: proj-csv
+doc_type: reference
+---
+
+## Export Error Code Reference
+
+This document lists error codes produced by the CSV export service and describes their meaning and recommended remediation steps.
+
+**E_EXPORT_400** — Invalid export request. The request body failed schema validation. Check that all required fields are present and that field values match the expected types. No export job was created.
+
+**E_EXPORT_403** — Permission denied. The requesting user does not have export rights for the specified project. Contact your project administrator to have the export permission granted to your role.
+
+**E_EXPORT_404** — Project not found. The project identifier supplied in the export request does not match any known project. Verify the project ID and retry.
+
+**E_EXPORT_417** — Expectation failed during export generation. This error indicates that the export service received a request it accepted but could not fulfil because an internal precondition was not met at generation time. Common causes include a task filter that returns zero rows, a missing template configuration, or a column mapping that references a field that no longer exists in the task schema. Inspect the job detail record for the specific precondition message and correct the export configuration before retrying.
+
+**E_EXPORT_500** — Internal server error. An unexpected failure occurred inside the export service. The error has been logged automatically. If the error persists after retrying, open a support ticket and include the job ID from the error response.
+
+**E_EXPORT_503** — Export service temporarily unavailable. The service is under maintenance or experiencing high load. Retry after the interval specified in the Retry-After response header.

module_3/.memory/reference/feature-csv-export.md

@@ -0,0 +1,11 @@
+---
+classification: internal
+project: proj-csv
+doc_type: feature
+---
+
+## Feature: CSV Export
+
+The CSV export feature builds its output file using a streaming writer that processes tasks row by row without loading the entire dataset into memory. When a user requests an export, the export service opens a writable stream, writes the header row containing column names, then iterates over the filtered task set in batches of 500 records. Each task is serialized to a CSV row and flushed to the stream immediately. Once all records are written the stream is closed and the completed file is handed off to the download handler. This streaming approach keeps memory consumption flat regardless of how many tasks are exported.
+
+If CSV export generation fails at any stage, the service applies an exponential backoff retry policy. The first retry occurs after two seconds, the second after four seconds, and the third after eight seconds. After three failed attempts the job is marked as permanently failed and the user receives an error notification. Transient network errors and temporary storage unavailability are retried automatically. Validation errors and permission errors are not retried because they indicate a problem that will not resolve itself without user intervention.

module_3/.memory/reference/feature-csv-import.md

@@ -0,0 +1,13 @@
+---
+classification: internal
+project: proj-csv
+doc_type: feature
+---
+
+## Feature: CSV Import
+
+The CSV import feature allows users to bulk-load tasks into the system from a CSV file. When a user uploads a file, the import service reads it line by line, validates each row against the task schema, and inserts valid rows into the database. Rows that fail validation are collected into an error report that the user can download after the import completes.
+
+The import service enforces a maximum file size of 10 MB and a maximum row count of 5,000 tasks per import operation. Files that exceed either limit are rejected immediately with an informative error message before any rows are processed. Duplicate detection is based on the external task ID field. If a row shares an external ID with an existing task, the import service updates the existing record rather than creating a new one.
+
+Column mapping is configurable. Users may upload a column-map JSON file alongside the CSV to specify which CSV column corresponds to which task field. If no column map is provided, the import service expects the CSV header row to use the canonical field names defined in the task schema documentation.

module_3/.memory/reference/security-export-visibility.md

@@ -0,0 +1,15 @@
+---
+classification: internal
+project: proj-csv
+doc_type: decision
+---
+
+## Decision: Task Visibility Rules for CSV Export
+
+This document records the access-control decision governing which tasks are permitted to appear in a CSV export file. The rules apply to all export jobs regardless of who initiates them.
+
+Only tasks that the requesting user is already permitted to read within the application may be included in an exported CSV file. The export service re-evaluates row-level read permissions for every task at export generation time using the same permission engine as the task list API. Tasks the user cannot read in the UI will not appear in the export even if the user constructs a filter that would otherwise match them. This ensures that exporting does not bypass any visibility restriction already enforced elsewhere in the system.
+
+Archived tasks are excluded from all exports by default. A user may opt in to including archived tasks by enabling the include_archived flag in the export request, provided they hold the archive-viewer permission. Deleted tasks are permanently excluded and cannot be included in any export regardless of permissions or flags.
+
+Tasks belonging to private sub-projects are excluded unless the requesting user is an explicit member of that sub-project. Project-level export permission does not grant access to tasks in private sub-projects. This rule was established to prevent accidental disclosure of tasks that project members have intentionally scoped to a smaller audience within the same project.

module_3/.memory/reference/standards-review.md

@@ -0,0 +1,15 @@
+---
+classification: internal
+project: proj-csv
+doc_type: standard
+---
+
+## Review Standards for the CSV Export Implementation
+
+All pull requests that touch the CSV export implementation must satisfy the following review standards before they can be merged. These standards apply to the export service, the export worker, the column-mapping layer, and any shared libraries used exclusively by the export pipeline.
+
+Code reviewers must verify that streaming is used throughout the output path. No implementation may buffer the full task set in memory before writing. Reviewers should check that the batch size constant is configurable via environment variable and that the default value is documented in the service README. Any change that introduces a new dependency on a third-party library requires sign-off from a senior engineer in addition to the standard two-reviewer requirement.
+
+Security review is mandatory for any change that modifies access-control checks, changes which task fields are included in export output, or alters the authentication path for export download URLs. Security reviews must be completed by a team member who holds the security-reviewer role and must be documented with a checklist comment on the pull request.
+
+Performance review is required for changes that affect the main export loop or the streaming write path. The reviewer must confirm that the change has been benchmarked against the baseline export throughput figure recorded in the performance log. A regression of more than five percent in throughput requires a follow-up task before the change can be merged to the main branch.

module_3/CLAUDE.md

@@ -24,6 +24,7 @@ docker run -it --rm -p 8501:8501 -p 8502:8502 \
   -e SLACK_BOT_TOKEN=your-token \
   -e SLACK_TEAM_ID=your-team-id \
   -v "$PWD":/workspace \
+  -v "$PWD/.memory":/memory \
   agentic_engineer_3
 ```
 

module_3/Dockerfile

@@ -12,7 +12,9 @@ ENV PYTHONUNBUFFERED=1 \
     # These dirs are created below; if /workspace is bind-mounted the host
     # directory will shadow them, but HF will re-create subdirs as needed.
     HF_HOME=/workspace/.cache/huggingface \
-    SENTENCE_TRANSFORMERS_HOME=/workspace/.cache/sentence-transformers
+    SENTENCE_TRANSFORMERS_HOME=/workspace/.cache/sentence-transformers \
+    HOST=0.0.0.0 \
+    CLIENT_PORT=8502
 
 # --- OS packages ---
 # Core tools (curl, git, etc.) plus Module 3.2 diagnostics/build dependencies:
@@ -63,6 +65,7 @@ import sklearn; \
 import pydantic; \
 import httpx; \
 import rank_bm25; \
+import sqlite_vec; \
 print('All Module 3.2 imports OK')"
 
 # --- Node/npm agent tooling ---
@@ -104,11 +107,16 @@ COPY agents/ /root/.claude/agents/
 RUN mkdir -p \
     /workspace/mcp-servers/storage \
     /workspace/mcp-servers/retrieval \
-    /workspace/.memory/reference \
     /workspace/.cache/huggingface \
     /workspace/.cache/sentence-transformers \
     /workspace/docs \
-    /workspace/tests
+    /workspace/tests \
+    /memory/reference
+
+# Bake the reference corpus into the image so the retrieval server works
+# without a bind-mount. Mounting -v "$PWD/.memory":/memory at runtime
+# will shadow this with a live corpus for development use.
+COPY .memory/reference/ /memory/reference/
 
 # --- Entrypoint ---
 COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh