chore: Add more tasks and delete unused fules.

Author: Dayan Graham

.cursor/rules/python.mdc

@@ -1,8 +1,3 @@
----
-description: 
-globs: 
-alwaysApply: true
----
 # Python Style Guide (Stream Viz Backend)
 
 ## Design Philosophy
@@ -21,7 +16,7 @@ alwaysApply: true
     - API request/response validation (FastAPI integration).
     - WebSocket message validation.
     - Complex internal data structures for clarity and validation.
-- **Validation:** Check pyright regularly to ensure types are correct.
+- **Validation:** Run `mypy` regularly to check type consistency (`mypy src/`).
 
 ## Asynchronous Programming (Asyncio)
 - **Default:** Use `async def` for all I/O-bound operations (network calls, database access, etc.).
@@ -61,11 +56,6 @@ alwaysApply: true
     - Implement structured logging (e.g., using `structlog` or Python's JSON formatter).
     - Include relevant context in log messages (e.g., `query_id`, client address).
 
-## Code Style
-- Do not put whitespace on blank lines
-- Generally follow Ruff rules
-- Be SUPER obsessive about not leaving unused variables, params or imports
-
 ## Testing
 - **Framework:** Use `pytest`.
 - **Async Testing:** Use `httpx` for testing FastAPI endpoints asynchronously.
@@ -84,3 +74,8 @@ alwaysApply: true
 - **Run Scripts:** `uv run path/to/script.py` (add `--with <package>` for temporary dependencies).
 - **Format/Lint:** `ruff format .` and `ruff check . --fix`.
 - **Type Check:** `mypy src/` (or relevant paths).
+
+### Common Patterns 
+
+- **Dependency Injection:** Use dependency injection to decouple components and improve testability rather than app.state
+

IMPROVEMENTS.md

@@ -0,0 +1,56 @@
+# Stream Viz Backend Code Review & Improvement Suggestions
+
+This document outlines findings and suggestions for improving the `stream-viz` backend codebase based on a review focusing on SOLID principles, clarity, and best practices.
+
+## Overall Structure & Design
+
+*   **Good Separation:** The project follows a reasonable structure separating concerns into `api`, `core`, `utils`, and `main.py`. This promotes modularity.
+*   **Dependency Injection:** Good use of FastAPI's dependency injection (`Depends`) for `TimeplusQueryManager` and `PerspectiveManager` in the WebSocket endpoint (`api/websocket/endpoint.py`). This improves testability and decoupling.
+*   **Manager Classes:** The use of `TimeplusQueryManager` and `PerspectiveManager` centralizes logic for interacting with these external services, which is good. They act as facades and manage resource lifecycles (like queries and tables) using reference counting.
+*   **Configuration:** Using Pydantic's `BaseSettings` (`config.py`) for configuration management is a standard and effective practice.
+*   **Asyncio Usage:** The codebase heavily relies on `asyncio`, which is appropriate for I/O-bound tasks like handling WebSockets and interacting with external streaming services.
+
+## SOLID Principles & Clean Code Analysis
+
+1.  **Single Responsibility Principle (SRP):**
+    *   **Mostly Good:** Classes like `WebSocketConnectionManager`, `TimeplusQueryManager`, and `PerspectiveManager` generally adhere well to SRP.
+    *   **`WebSocketMessageHandler`:** (`api/websocket/message_handler.py`) The `handle_start_query` method is quite long and complex. Consider breaking it down into smaller helper methods.
+    *   **`TimeplusQueryManager._run_query`:** This method is very long (~100 lines) and handles many tasks within the stream processing loop. Refactor into smaller, focused async methods (e.g., `_process_schema`, `_process_data_chunk`).
+    *   **`websocket_endpoint`:** (`api/websocket/endpoint.py`) The main `try...except` block is large. Some error/cleanup logic could potentially move to `WebSocketConnectionManager` or `WebSocketMessageHandler`.
+
+2.  **Open/Closed Principle (OCP):**
+    *   **Message Handling:** Dispatching messages via `isinstance` is acceptable for now but consider a more scalable pattern (e.g., command pattern or dictionary mapping) if more message types are added.
+
+3.  **Liskov Substitution Principle (LSP):**
+    *   **Opportunity:** If supporting alternative streaming sources or visualization backends becomes necessary, define abstract base classes/interfaces for managers (`QueryManager`, `VizManager`) to enable substitution.
+
+4.  **Interface Segregation Principle (ISP):**
+    *   **Mostly Good:** Manager interfaces are generally focused.
+    *   **`TimeplusQueryManager` Callbacks:** The specific set of callbacks required by `start_or_join_query` tightly couples it to `WebSocketMessageHandler`. Consider alternatives if other consumers need different interaction patterns.
+
+5.  **Dependency Inversion Principle (DIP):**
+    *   **Good (FastAPI):** FastAPI's `Depends` inverts control effectively.
+    *   **Managers & Clients:** Managers (`TimeplusQueryManager`, `PerspectiveManager`) create their underlying clients (`proton_driver`, `perspective`) directly. Injecting these clients (or interfaces) would improve testability and flexibility.
+
+## Clarity, Messiness & Specific Issues
+
+*   **Error Handling:**
+    *   Extensive but complex, especially in WebSocket endpoint and message handler cleanup logic. Centralize where possible.
+    *   Unclear error propagation strategy from Timeplus callbacks in `WebSocketMessageHandler`. Decide whether errors should stop the upstream query.
+*   **Async/Sync Mix:**
+    *   Verify potentially blocking calls (e.g., `PerspectiveManager.update_table`) and use `asyncio.to_thread` if needed.
+    *   Simplify unnecessary async wrappers around non-blocking calls in `WebSocketMessageHandler`.
+*   **Missing Tests:** The `src/stream_viz/tests` directory is empty. This is a critical gap.
+*   **Empty Files:** `src/stream_viz/utils/hashing.py` is empty. Implement or remove.
+*   **Magic Strings:** Replace string literals for WebSocket message types (e.g., "table_ready") with Enums or constants for robustness.
+*   **Resource Cleanup:** Double-check cleanup logic (`cleanup_query_resources`, `shutdown`) for edge cases and completeness.
+*   **Perspective Client Loop:** Ensure the event loop handling in `PerspectiveManager` is robust for deployment scenarios.
+
+## High-Priority Improvement Actions
+
+1.  **Implement Comprehensive Tests:** Add unit tests (mocking external deps) and integration tests. Start with managers and critical WebSocket logic.
+2.  **Refactor Large Methods:** Break down `handle_start_query` (in `WebSocketMessageHandler`), `_run_query` (in `TimeplusQueryManager`), and `websocket_endpoint`.
+3.  **Clarify & Standardize Error Handling:** Define clear error propagation rules (especially for callbacks) and simplify cleanup logic.
+4.  **Address Async/Sync:** Use `asyncio.to_thread` for confirmed blocking calls. Remove unnecessary wrappers.
+5.  **Inject Dependencies:** Inject `proton_driver` and `perspective` clients into their respective managers.
+6.  **Populate/Remove Empty Files:** Add tests to `tests/`. Implement or remove `utils/hashing.py`. 

TODO.md

@@ -0,0 +1,10 @@
+"""
+This type stub file was generated by pyright.
+"""
+
+from .client import Client
+from .dbapi import connect
+
+VERSION = ...
+__version__ = ...
+__all__ = ['Client', 'connect']

src/stream_viz/utils/hashing.py

@@ -0,0 +1,93 @@
+"""
+This type stub file was generated by pyright.
+"""
+
+class BlockInfo:
+    is_overflows = ...
+    bucket_num = ...
+    def write(self, buf): # -> None:
+        ...
+    
+    def read(self, buf): # -> None:
+        ...
+    
+
+
+class BaseBlock:
+    def __init__(self, columns_with_types=..., data=..., info=..., types_check=...) -> None:
+        ...
+    
+    def normalize(self, data):
+        ...
+    
+    @property
+    def num_columns(self):
+        ...
+    
+    @property
+    def num_rows(self):
+        ...
+    
+    def get_columns(self):
+        ...
+    
+    def get_rows(self):
+        ...
+    
+    def get_column_by_index(self, index):
+        ...
+    
+    def transposed(self): # -> list[tuple[Any, ...]]:
+        ...
+    
+
+
+class ColumnOrientedBlock(BaseBlock):
+    def normalize(self, data): # -> list[Any]:
+        ...
+    
+    @property
+    def num_columns(self): # -> int:
+        ...
+    
+    @property
+    def num_rows(self): # -> int:
+        ...
+    
+    def get_columns(self): # -> list[Any]:
+        ...
+    
+    def get_rows(self): # -> list[tuple[Any, ...]]:
+        ...
+    
+    def get_column_by_index(self, index):
+        ...
+    
+
+
+class RowOrientedBlock(BaseBlock):
+    dict_row_types = ...
+    tuple_row_types = ...
+    supported_row_types = ...
+    def normalize(self, data): # -> list[Any]:
+        ...
+    
+    @property
+    def num_columns(self): # -> int:
+        ...
+    
+    @property
+    def num_rows(self): # -> int:
+        ...
+    
+    def get_columns(self): # -> list[tuple[Any, ...]]:
+        ...
+    
+    def get_rows(self): # -> list[Any]:
+        ...
+    
+    def get_column_by_index(self, index): # -> list[Any]:
+        ...
+    
+
+

typings/proton_driver/__init__.pyi

@@ -0,0 +1,13 @@
+"""
+This type stub file was generated by pyright.
+"""
+
+class BlockStreamProfileInfo:
+    def __init__(self) -> None:
+        ...
+    
+    def read(self, fin): # -> None:
+        ...
+    
+
+