Skip to content

ROADMAP.md

Latest commit

 

History

History
146 lines (120 loc) · 6.72 KB

ROADMAP.md

File metadata and controls

146 lines (120 loc) · 6.72 KB
Raw

Roadmap & Next Steps

Status: v0.3.1 — alpha. CI live at .github/workflows/ci.yml. Public MCP: docs/DEPLOY_WORKERS_MCP.md. Internal FastAPI/gVisor: docs/DEPLOY_GVISOR.md.

🟢 Ready to use right now

  • pip install -e '.[dev]' → venv with everything.
  • findata serve --host 0.0.0.0 --port 8000 → REST + MCP server.
  • findata bcb get selic -n 10 → CLI access to BCB.
  • All unit tests pass (pytest), ruff and mypy --strict clean.

🟡 Immediate next steps (pick them up when you hit the WSL server)

  1. Run the server on WSL

    # in WSL
git clone https://github.com/robertoecf/openfindata.git
cd openfindata
python3 -m venv .venv && . .venv/bin/activate
pip install -e .
findata serve --host 0.0.0.0 --port 8000

    Or Docker: docker compose up -d.

  2. GitHub Actions CI CI is live at .github/workflows/ci.yml.

  3. Publish to PyPI

  4. Systemd / process manager on WSL Minimal findata.service unit:

    [Unit]
Description=Dados Financeiros Abertos
After=network.target

[Service]
Type=simple
User=yourself
WorkingDirectory=/srv/openfindata
ExecStart=/srv/openfindata/.venv/bin/findata serve --host 0.0.0.0 --port 8000 --no-banner
Restart=on-failure

[Install]
WantedBy=multi-user.target

  5. Expose behind nginx or Caddy if you want HTTPS. Caddy one-liner:

    findata.yourdomain.com { reverse_proxy localhost:8000 }

🔵 Feature roadmap (0.2+)

  • Rate limiting (slowapi) when exposing publicly.
  • Observability — structured JSON logs, /metrics (Prometheus exporter), optional OpenTelemetry via env vars.
  • Redis cache — drop-in replacement for the in-memory LRU for multi-replica deploys.
  • ANBIMA indexes — IMA, IMA-B, IDkA, IHFA.
  • B3 native — scrape official CSVs/COTAHIST to remove the yfinance dep.
  • IBGE expansion — PNAD Contínua, produção industrial, comércio varejista.
  • TypeScript SDK — generate from the OpenAPI spec.
  • Webhooks / streaming — SSE for "give me the new PTAX the moment BCB publishes it".

📚 Lessons from adjacent projects

From OpenBB-finance/OpenBB (Python, global) — the reference 🐐

The closest large-scale analogue to what we're building, and the bar to measure against. OpenBB's Open Data Platform is a "connect once, consume everywhere" layer that exposes the same standardized data across a Python SDK, a CLI, a FastAPI REST server, an MCP server for AI agents, Excel, and the OpenBB Workspace UI. That is precisely our thesis (lib + REST + CLI + MCP over a single normalized core), validated at scale.

Evaluation:

  • ✅ Single normalized core, many surfaces. Our sources/<source>/ feeding one model that the Python lib, REST, CLI and /mcp all reuse mirrors their architecture — we are directionally correct, keep it that way.
  • ✅ MCP as a first-class surface. OpenBB ships an MCP server alongside REST; validates our early bet on /mcp instead of treating it as an afterthought.
  • 🟡 Provider/extension system as installable packages. OpenBB lets third parties add data sources without forking core — separate extension repos, discovered at runtime. Our sources/ are in-tree today; a plugin/entry-point mechanism is the natural path once external contributors want their own source without a PR to core.
  • 🟡 "Build once, deploy everywhere" templates. A scaffolding command for a new source (findata new-source <name>) would lower the contribution bar the way their extension template does.
  • 🟡 Spreadsheet surface. Their Excel add-in is a reminder that many BR analysts live in spreadsheets; a thin =FINDATA(...) bridge over the REST API is a cheap, high-leverage future surface.
  • ❌ AGPLv3 + enterprise Workspace/Hub. OpenBB is AGPLv3 with a hosted pro tier. We are deliberately MIT and self-host-first — do not import the account/Hub/proprietary-provider machinery; it cuts against the manifesto.
  • ❌ Global/paid-provider breadth (FMP, Polygon, etc.). Our scope is BR public sources with no API keys; chasing global paid providers would dilute the "if the data is public, the infra should be too" thesis.

From Tpessia/dados-findanceiros (TS/NestJS, BR)

Already absorbed:

  • ✅ IPEA Data (OData v4) — unique macro series with 1940s+ history, ported in v0.1.0.
  • ❌ Tesouro Direto D0 JSON — endpoint has been retired (HTTP 410).
  • ❌ BCB SGS duplicates — already covered, no action.

From gprossignoli/findata (Python, global)

Student project, inactive, sync requests/RabbitMQ/MongoDB — most of the stack is in the opposite direction of ours (async httpx + FastAPI + MCP). But two ideas are worth copying:

  • ✅ Per-source top-level packages. Our src/findata/sources/<source>/ already follows this; codified in CONTRIBUTING.md.
  • ✅ *_adapter.py naming for external deps. Consider renaming internal clients (e.g., future yfinance_adapter.py, anbima_adapter.py) to make the boundary explicit and greppable. Low-priority refactor.
  • 🟡 Use-cases as classes. Not critical today (our functions are already tiny), but if a flow grows to orchestrate multiple adapters it should graduate into a UseCase class so CLI / HTTP / MCP can all reuse it.
  • ❌ Clean-Architecture three-folder ceremony (domain/application/ infraestructure/). Overkill for stateless wrappers — skip.
  • ❌ APScheduler / RabbitMQ / MongoDB. A library shouldn't embed a scheduler or a broker; let callers (cron, Airflow, GH Actions) drive it.
  • ❌ yfinance fork. They fork to fix non-US tickers; we already use mainline yfinance for B3 without modification.

🧪 Known caveats

  • CVM financial statements (DFP/ITR) download a multi-hundred-MB ZIP for a full year — always pass cnpj= when hitting /cvm/financials/*.
  • Fund daily NAV files are ~50 MB/month — same recommendation: filter by cnpj=.
  • yfinance is a core dependency since v0.1.0; if you use pip install openfindata --no-deps and skip it, /b3/* returns 503.
  • fastapi-mcp is pinned at a minimum version; if your deployment picks up a major-version break, /mcp is silently disabled but the REST API keeps serving.