Skip to content

Latest commit

 

History

History
88 lines (72 loc) · 5.28 KB

File metadata and controls

88 lines (72 loc) · 5.28 KB

Repository Instructions

Project Goal

Build a COM-480 Milestone 3 interactive D3 dashboard that helps investors compare AI value-chain companies across infrastructure, compute, AI infrastructure software, foundation models, and AI-enabled applications.

Layout Rules

  • src/ contains application source code and data-pipeline source code.
  • src/data/ contains data-pipeline code, seed files, collectors, source adapters, validators, LLM extraction logic, and pipeline CLI entrypoints.
  • data/ contains generated machine-readable outputs consumed by the visualization.
  • doc/ contains human-facing documentation, milestone PDFs, process-book material, screencast planning, data explanations, and pipeline explanations.
  • agents/ contains agent-facing notes and plans that are not the root instruction file.
  • agents/impl.md is the implementation handoff for the next pipeline work. Keep it current when implementation direction changes.
  • README.md is the public GitHub entry point required by Milestone 3.
  • .env_example documents expected API keys and provider toggles. Never commit .env or real secrets.
  • Do not put generated app data under doc/; explain the data there instead.

Data Pipeline Rules

  • Select companies by strategic relevance and source availability, not by layer alone.
  • Preserve each company's AI value-chain layer as a visualization/story field.
  • Route collection by company-level source coverage across SEC EDGAR, Financial Datasets MCP, FMP MCP, TradingView MCP, Qieman MCP, yfinance, FinanceToolkit, public web, and LLM extraction.
  • Treat 36Kr PitchHub (https://pitchhub.36kr.com/) as a China startup discovery and enrichment source for project pages, project sets, financing events, and investor-interest lists when pages are publicly accessible.
  • Treat 18.com.cn / 东方财富 as a China public-market news and theme-discovery source for listed companies, sector/concept pages, financing news, and market articles; do not treat it as a private startup database.
  • Use public structured sources before public web extraction.
  • For public companies, enrich valuation, quality, safety, growth/stability, and timing-overlay metrics before frontend integration. Missing metrics must remain null with review issues rather than guessed.
  • Store acquired metrics as time-indexed observations in data/metrics.sqlite, keyed by company, metric name, provider/source, period, and ISO fetched_at. Export JSON snapshots for the frontend, but do not treat JSON as the metric source of truth.
  • Use LLMs only to extract structured facts from public text when structured APIs are unavailable or incomplete.
  • Cache source fetches and LLM extraction results by source URL/content hash and prompt version.
  • Log LLM provider, model, prompt version, source, extracted fields, confidence, and timestamp.
  • Do not impose an arbitrary LLM call limit; control usage with caching and source discipline.
  • Seed discovery may propose candidates, but it must not silently rewrite seed CSVs.
  • Keep src/data/scripts/pipeline.py as a thin CLI wrapper. Put implementation modules under src/data/scripts/<module>/, such as schema/, sources/, llms/, prompts/, and io/.
  • Do not wire the dashboard to raw seed exports as if they are final investment data. Build enrichment first, then connect src/main.ts to enriched generated data.

Company Seed Rules

  • Use acquisition-class seed files:
    • src/data/seeds/public_listed.csv
    • src/data/seeds/private_public_web.csv
    • src/data/seeds/manual_review.csv
  • Keep personal_interest for companies the team specifically wants to include.
  • Use priority to distinguish must-have, strong, normal, and candidate companies.
  • Keep native_name for Chinese, Hong Kong, Taiwan, Japan, and Korea companies when a local-language/common local name is available.
  • For Layer 0 companies, populate layer_substage and value_chain_order using the physical AI infrastructure chain:
    • 0A Materials & wafers
    • 0B Fab equipment
    • 0C Memory & storage
    • 0D Packaging & substrates
    • 0E Optical & electrical interconnect
    • 0F Server & rack manufacturing
    • 0G Data-center power
    • 0H Cooling & thermal
    • 0I Construction & deployment
  • For listed companies, keep both machine fields (exchange, ticker) and the display field stock_code, formatted like NASDAQ:MRVL.
  • Keep source URLs and include reasons close to the seed rows.

Commit Convention

Use this format:

[<component>]: <summary>

- <change description line 1>
- <change description line 2>

Rules:

  • Summary is imperative mood, lowercase, no trailing period, and no more than 72 characters.
  • Body uses - bullet lines describing what changed and why.
  • Keep one logical change per commit.
  • Reference issues at the end of the body when applicable, for example Refs: #42.

Common component tags:

  • misc - minor changes, formatting, config tweaks.
  • bugfix - bug fixes.
  • refactor - restructuring without behavior changes.
  • docs - documentation-only changes.
  • test - tests or benchmarks.
  • ci - CI/CD pipeline changes.
  • agent - agent-side orchestration and notes.
  • scripts - utility scripts, benchmarking, data prep, deployment helpers.
  • data - data pipeline, seed files, generated data schemas, or data docs.
  • viz - dashboard and visualization changes.