todo.md

Project TODO

• Inspect the existing repository structure and preserve all working components.
• Verify the official OpenStax "Introduction to Business" source, authorship, and Creative Commons reuse license.
• Download the official PDF to data/raw/ without bypassing restrictions.
• Validate that the downloaded PDF opens successfully and record its exact filename and file size.
• Update README.md with the verified book provenance, license, download date, local file location, and Phase 01 limitations.
• Exclude data/raw/introduction-to-business-openstax.pdf from Git while retaining the local source file.
• Add data/raw/SOURCE.md with verified textbook provenance, local filename, size, and SHA-256 metadata.
• Update README.md to explain the local-only source PDF policy and official reacquisition path.
• Revalidate the local PDF’s existence, checksum, and readability after Git exclusion.
• Check whether Phase 01 artifacts are present in the user-connected Business Knowledge AI folder.
• Synchronize any missing Phase 01 artifacts to the user-connected Business Knowledge AI folder without starting Phase 02.
• Inspect the retained source PDF and existing connected-folder structure for Phase 02.
• Define structured page records that preserve page number, chapter, section, source, extracted text, cleaned text, and quality flags.
• Create notebooks/01_document_ingestion.ipynb with a visible PyMuPDF extraction and cleaning workflow.
• Save processed page records under data/processed/ and validate output completeness and metadata preservation.
• Document the Phase 02 notebook-only scope without adding embeddings, retrieval, LLM, or LangGraph components.
• Inspect Phase 02 page records for chunking readiness and required provenance fields.
• Compare multiple RecursiveCharacterTextSplitter chunk-size and overlap configurations in notebooks/02_chunking.ipynb.
• Select and document a final configuration that preserves chunk ID, text, source, page, chapter, and section metadata.
• Save and validate a reusable final chunk artifact under data/processed/.
• Document the Phase 03 no-embeddings, no-retrieval, no-reranking, no-LLM, and no-LangGraph scope boundary.
• Inspect Phase 03 final chunks for embedding readiness and retained provenance fields.
• Verify whether the exact BAAI/bge-m3 model can be downloaded and loaded without substituting another embedding model.
• Create notebooks/03_embeddings.ipynb with real BGE-M3 batching, dimension inspection, normalization, and metadata preservation.
• Save and validate a reusable embedding artifact, or record the verified model-availability limitation without fabricating embeddings.
• Document the Phase 04 no-Qdrant boundary and synchronize verified deliverables to the connected project folder.
• Inspect the Phase 04 embedding status and confirm whether a real BGE-M3 embedding artifact exists.
• Define the in-memory Qdrant collection, payload schema, metadata filter, and idempotent re-indexing validation for real vectors.
• Create notebooks/04_qdrant_indexing.ipynb with a strict no-fabrication embedding preflight.
• Execute or limitation-test the Qdrant notebook and verify collection statistics, filtering, and duplicate-free re-indexing when real embeddings are available.
• Document the Phase 05 outcome and synchronize verified deliverables to the connected project folder without implementing retrieval.
• Inspect Phase 04 embedding and Phase 05 Qdrant status records for real-artifact readiness.
• Define sample business questions, Top-K configurations, displayed result fields, and manual quality-review criteria.
• Create notebooks/05_basic_retrieval.ipynb with exact BGE-M3 query embedding and Qdrant Top-K retrieval logic protected by a no-fabrication preflight.
• Execute or limitation-test the basic-retrieval notebook without implementing BM25, hybrid retrieval, reranking, LLM, or LangGraph.
• Document the Phase 06 outcome and synchronize verified deliverables to the connected project folder.
• Inspect the Phase 04–06 embedding, indexing, and basic-retrieval status records for hybrid-retrieval readiness.
• Define real BGE-M3/Qdrant dense retrieval, BM25 sparse retrieval, reciprocal-rank fusion, comparison displays, and provenance validation.
• Create notebooks/06_hybrid_retrieval.ipynb with strict no-fabrication preflight for unavailable real vectors.
• Execute or limitation-test dense-only, BM25-only, and hybrid candidate-set paths without adding reranking or LangGraph.
• Document the Phase 07 outcome and synchronize verified deliverables to the connected project folder.
• Inspect the Phase 07 hybrid-retrieval status and candidate artifact for real reranking readiness.
• Verify exact BAAI/bge-reranker-v2-m3 availability and define a no-substitution reranking preflight.
• Create notebooks/07_reranking.ipynb with Top-N hybrid candidates, BGE reranking, Top-K context selection, and before-after ordering displays.
• Execute or limitation-test reranking without fabricating candidate scores or rankings and without adding LLM generation or LangGraph.
• Document the Phase 08 outcome and synchronize verified deliverables to the connected project folder.
• Inspect available retrieved-context artifacts and model access for grounded Phase 09 experiments.
• Create notebooks/08_prompt_engineering.ipynb covering the eight requested grounded prompting techniques without LangGraph.
• Execute the notebook or record verified limitations without fabricating retrieved context, prompts, model outputs, or comparisons.
• Update Phase 09 documentation and synchronize verified deliverables to the connected project folder.
• Verify exact Qwen2.5-7B-Instruct availability, OpenStax permission state, and reusable retrieved-context readiness for Phase 10.
• Create notebooks/09_llm_generation.ipynb with grounded Qwen context-to-answer generation and temperature, max-token, and prompt-structure experiments without LangGraph.
• Execute the notebook or record verified limitations without substituting a model or fabricating answers, citations, or parameter comparisons.
• Update Phase 10 documentation and synchronize verified deliverables to the connected project folder.
• Verify LangGraph installation and determine a deterministic, non-agentic learning-example scope for Phase 11.
• Create notebooks/10_langgraph_basics.ipynb teaching state, StateGraph, nodes, edges, START, END, state updates, conditional edges, and checkpointing without RAG, agents, or tools.
• Execute the toy graphs and persist only verifiable Phase 11 learning outputs.
• Update Phase 11 documentation and synchronize verified deliverables to the connected project folder.
• Inspect real retrieval, reranking, model, and permission artifacts to define truthful deterministic Phase 12 preflights.
• Create notebooks/11_langgraph_rag.ipynb with the fixed process_query → retrieve → rerank → build_prompt → generate_answer → format_response pipeline and graph visualization.
• Execute the graph on real questions or record verified stage limitations without fabricating retrieval, reranking, generation, citations, or state transitions.
• Update Phase 12 documentation and synchronize verified deliverables to the connected project folder.
• Verify LangGraph checkpointing availability and define separate conversation-history and retrieved-document-context state fields for Phase 13.
• Create notebooks/12_memory.ipynb demonstrating two deterministic conversational turns with MemorySaver checkpointing and a contextual follow-up answer without agents.
• Execute the checkpointed memory graph and persist verifiable thread state, history, and document-context separation evidence.
• Update Phase 13 documentation and synchronize verified deliverables to the connected project folder.
• Inspect OpenStax provenance, verified source artifacts, and all retrieval/generation pipeline statuses to determine which Phase 14 metrics can be truthfully computed.
• Create notebooks/13_evaluation.ipynb with a source-grounded evaluation dataset, metric definitions, pipeline-variant comparison matrix, and strict no-fabrication guards.
• Execute only eligible retrieval/generation evaluation calculations and persist actual metrics or detailed unavailable-artifact records without inventing ground truth or scores.
• Update Phase 14 documentation and synchronize verified deliverables to the connected project folder.
• Preserve Phase 14 evaluation-artifact checksums before correcting only the validation test’s notebook cell-location assertion.
• Correct the Phase 14 artifact-validation test to inspect Markdown and code cells without changing evaluation methodology or artifacts.
• Run focused, Phase 14, and full regression tests, then verify notebook JSON and stored evaluation-artifact integrity.
• Inspect FastAPI deployment requirements, existing service boundaries, and the validated deterministic LangGraph pipeline artifacts.
• Define a modular /api/health and /api/chat FastAPI contract that preserves the fixed process_query → retrieve → rerank → build_prompt → generate_answer → format_response workflow.
• Implement the production FastAPI service and deterministic LangGraph adapter without agents, tool calling, retrieval loops, or self-correction.
• Add endpoint tests, document real-artifact/model preflight behavior, and synchronize verified Phase 15 deliverables to the connected project folder.
• Inspect the existing React structure, reusable AI chat component, and /api/chat response schema for Phase 16 integration.
• Define a responsive grounded-chat UX with conversation history, loading/error states, citations, source cards, and page/chapter provenance.
• Implement the React chat interface, local conversation state, and POST /api/chat integration without agent-specific UI.
• Add frontend tests, visually verify desktop and mobile layouts, document Phase 16, and synchronize verified deliverables to the connected project folder.
• Add rendered-UI behavior tests for the API send flow, loading state, unavailable-answer state, error state, conversation history, and citation/source-card rendering.
• Run the rendered-UI behavior tests and retain static artifact checks as supplemental Phase 16 coverage.
• Diagnose and fix the chat request failure reporting that the deterministic FastAPI RAG service is unavailable after startup.
• Verify /api/health and POST /api/chat through the deployed proxy, then synchronize and publish the corrected behavior.
• Verify the live catalog for the exact Qwen2.5-7B-Instruct model and investigate official OpenStax generative-AI authorization terms.
• Enable generated answers only if both exact-model availability and a lawful authorization basis are verified; otherwise retain the explicit unavailable state.
• Test, document, synchronize, and publish the verification outcome without fabricating approval, model access, or generated answers.
• Diagnose and fix the renewed production FastAPI lifecycle failure that returned HTTP 503 during the generation-preflight health check.
• Capture and store an auditable live model-catalog result showing whether the exact required Qwen identifier is present.
• Inspect only the Qwen generation configuration, deployment runtime, provider path, and precise failure condition.
• Verify whether Qwen2.5-7B-Instruct can run locally or through a configured provider in the current environment without substituting a model.
• Investigate the official OpenStax generative-AI authorization condition separately from model availability.
• Deliver a generation-layer report with the minimal supported configuration and exact files to modify, without changing retrieval, citations, or unrelated pipeline stages.
• Inspect the live provider catalog and existing isolated generation abstraction to select a verified available instruction model.
• Record the user's confirmation of written OpenStax authorization and enable the existing authorization guard for generated answers.
• Implement a configurable gpt-5-mini generator behind the existing deterministic prompt and citation interfaces without touching retrieval stages.
• Add generation-layer contract tests and run a real grounded API generation test using retrieved OpenStax context.
• Document, synchronize, and publish the verified available-model replacement.
• Verify the published /api/chat endpoint returns a real gpt-5-mini grounded answer with preserved OpenStax citations.
• Inspect the published response citations array directly for count and OpenStax source, page, chapter, and official-book URL fields.
• Replace only the unavailable Qwen generation configuration with the selected available model while preserving prompts, grounded-answer behavior, and citation interfaces.
• Execute real API generation tests against retrieved OpenStax context and verify a non-fabricated generated answer with citations.
• Document the model change, synchronize verified generation-layer deliverables to the connected folder, and publish the configuration.
• Save a final checkpoint that includes the production citation-array verification record and completed available-model tracker state.
• Compare the complete managed workspace against the connected local project folder, excluding only generated dependencies, caches, logs, and Git internals.
• Synchronize all remaining project directories and files to the connected local folder and verify complete tree and content parity.
• Document and publish the completed full-workspace synchronization record.
• Perform a read-only final comparison of all 196 managed files for source/destination existence, size, SHA-256 checksum, and extra destination paths.
• Report the final checksum-verification result with exact discrepancy paths if any are found.
• Review the current rendered chat workspace and prepare a prioritized frontend/UI–UX redesign recommendation without changing the RAG pipeline.
• Read the attached prompt, extract its detailed requirements, and add the resulting implementation tasks to this tracker.
• Create a complete visual mockup for Option A — Dark Executive, without changing the application frontend.
• Create a complete visual mockup for Option B — Modern Blue Intelligence, without changing the application frontend.
• Create a complete visual mockup for Option C — Light Editorial Knowledge, without changing the application frontend.
• Verify that all three generated mockup assets have completed and are visually reviewable before presenting them.
• Present the three distinct concept mockups with palette, layout, UX philosophy, components, and Business Knowledge AI fit.
• Wait for and record the user's explicit A, B, or C selection before implementing any concept; the first selection cycle was superseded by the user's additional concept request.
• Create and visually verify three additional non-implemented UI/UX mockups using the requested black/red, white/red, and purple/black palettes.
• Present the three new concept directions and wait for an explicit selection before changing any application UI; the user selected Option E — Vermillion Ledger.
• Inspect the existing chat workspace and reusable UI components before applying the approved Vermillion Ledger redesign.
• Implement the Vermillion Ledger paper-and-vermilion visual system, editorial masthead, research index, answer document, and evidence register without changing application behavior.
• Preserve and refine responsive chat sending, loading, error, citation, source-card, conversation-history, and accessibility behavior within the approved layout.
• Add and run focused frontend tests, TypeScript validation, and desktop/mobile visual checks for the approved redesign.
• Document, synchronize to the connected local folder, checkpoint, and publish the approved Vermillion Ledger redesign.
• Verify the automatically published Vermillion Ledger redesign on the live production URL.
• Record production verification and save the final published tracker state.
• Remove only the visible “Vermillion Ledger” text from the frontend while retaining the approved visual system, layout, and behavior.
• Validate, synchronize, checkpoint, and publish the narrow visible-title removal.
• Identify and remove only the obsolete Qwen-specific notebooks/09_llm_generation.ipynb notebook and direct references.
• Verify the active gpt-5-mini generation path, synchronize the cleanup, checkpoint, and publish it.
• Audit GitHub access and repository-ready project files before external publication.
• Prepare a professional GitHub README and repository metadata while preserving exclusions for large, generated, and secret-bearing files.
• Create a GitHub repository and push the reviewed project history and current source.
• Verify the remote repository presentation and record the published URL: https://github.com/ms467154-coder/business-knowledge-ai
• Assess genuinely free LLM deployment options that can preserve the existing grounded generation interface; recommend Qwen2.5 7B Instruct through Ollama for local use.
• Audit the current generation abstraction and document the local Ollama runtime boundary.
• Add configurable Qwen2.5 7B Ollama generation support while retaining gpt-5-mini as the deployed default.
• Add tests and local setup documentation for the Ollama-compatible provider path.
• Validate, synchronize, push to GitHub, checkpoint, and publish the configurable local-LLM update.
• Audit local Ollama availability and the current dual-provider generation implementation before the local-only migration; the connected Windows runtime was not reachable for a live Ollama check.
• Remove Forge and gpt-5-mini generation support; make Qwen2.5 7B through Ollama the only supported generator.
• Update tests and documentation, then perform contract validation; real local-runtime validation remains dependent on starting Ollama on the Windows computer.
• Synchronize, push to GitHub, checkpoint, and publish the local-only generation migration.
• Inventory and remove approved Phase/phase-status Markdown documents and clearly obsolete reports while preserving core repository files.
• Validate retained project references, synchronize the cleanup, push it to GitHub, checkpoint, and publish it.
• Capture the reported Qwen through Ollama generation failure and verify runtime/model reachability.
• Correct only the local generation configuration or integration issue found, then validate grounded generation where reachable.
• Synchronize, push, checkpoint, publish, and document the Ollama generation diagnosis or fix.
• Download and verify the official Ollama Windows installer outside of Forge for local Qwen runtime setup.
• Verify the installed Windows Ollama executable, local API service, and current qwen2.5:7b model inventory.
• Pull qwen2.5:7b locally if it is absent, then verify direct Ollama generation.
• Run the Windows-local Business Knowledge AI service against Ollama and verify an actual grounded /api/chat answer with citations.
• Compare the Windows workspace against the current local-only Ollama generator code and synchronize any stale generator files before validation.
• Diagnose the newly reported local LLM failure by checking the live Windows Ollama service, Qwen model inventory, and application processes.
• Reproduce and resolve the reported local generation failure with the minimum verified runtime or configuration correction.