From 324d88d42846262e32cbca3b4c3c0b9b2b0bcaa4 Mon Sep 17 00:00:00 2001 From: Kastier1 <40179067+Kastier1@users.noreply.github.com.> Date: Wed, 14 Oct 2026 12:32:15 -0700 Subject: [PATCH 1/5] ENG-12238: document Reflex app scaling and recovery --- .../performance_and_execution.md | 3 + docs/ai_builder/integrations/mcp_overview.md | 11 +- docs/app/news/+scaling-guide.docs.md | 1 + .../docpage/sidebar/sidebar_items/learn.py | 1 + docs/hosting/scaling.md | 185 ++++++++++++++++++ docs/hosting/self-hosting.md | 3 + 6 files changed, 201 insertions(+), 3 deletions(-) create mode 100644 docs/app/news/+scaling-guide.docs.md create mode 100644 docs/hosting/scaling.md diff --git a/docs/advanced_onboarding/performance_and_execution.md b/docs/advanced_onboarding/performance_and_execution.md index b86d0227561..3a26e3c259e 100644 --- a/docs/advanced_onboarding/performance_and_execution.md +++ b/docs/advanced_onboarding/performance_and_execution.md @@ -5,6 +5,9 @@ meta_description: Compare Reflex performance with Streamlit and Dash using publi # Performance and execution +For deployment-level capacity and recovery, see [Scaling Reflex Apps](/docs/hosting/scaling/), +including shared state, independent job workers, and Redis diagnostics. + Reflex combines a customizable React interface with explicit Python state and event handlers. Published comparisons show concrete performance gains over Streamlit and Dash: faster initial data visibility in an NBA dashboard and faster bulk updates in a component-heavy form. The measurements below explain where those gains appeared and how to apply the same rendering and state-management techniques in your app. ## Reflex vs Streamlit: faster time to visible data diff --git a/docs/ai_builder/integrations/mcp_overview.md b/docs/ai_builder/integrations/mcp_overview.md index 23208baa4d5..1a6b02170ca 100644 --- a/docs/ai_builder/integrations/mcp_overview.md +++ b/docs/ai_builder/integrations/mcp_overview.md @@ -12,11 +12,16 @@ The hosted server is available at: https://build.reflex.dev/mcp ``` -## Available tool +## Documentation tools -The server currently exposes one read-only tool: +Check your server's `tools/list` response for the tools available in its deployed version. Releases with the knowledge tools provide two read-only documentation operations: -- `search_docs(query)` searches the Reflex documentation index and returns the relevant documentation sections and their source paths. +- `search_reflex_knowledge(queries)` accepts up to five short keyword queries and returns candidate keys, summaries, and source labels from the bundled documentation and curated knowledge. +- `get_reflex_knowledge(keys)` fetches the full text for the exact keys returned by search. Fetch the relevant entries before relying on a summary; the server reports unknown keys separately. + +For example, search with `{"queries": ["horizontal scaling", "durable jobs", "Redis Sentinel"]}`, then pass the relevant returned key to `get_reflex_knowledge`. The [scaling guide](/docs/hosting/scaling/) explains version compatibility, regional session state, and recovery assumptions. Split more than five topics into separate search calls. + +Older deployments may expose `search_docs` instead. Follow the tool schema advertised by the connected server. The knowledge tools use a packaged documentation snapshot, so a website update alone does not refresh the server's corpus; the server release must also update its locked documentation bundle. Use it for questions about Reflex APIs, components, configuration, and development workflows. Review the returned source before applying a change that depends on version-specific behavior. diff --git a/docs/app/news/+scaling-guide.docs.md b/docs/app/news/+scaling-guide.docs.md new file mode 100644 index 00000000000..653fcee4a86 --- /dev/null +++ b/docs/app/news/+scaling-guide.docs.md @@ -0,0 +1 @@ +Add a scaling guide covering horizontal backends, state-store version compatibility, durable jobs, regional data routing, telemetry, and Redis recovery checks. diff --git a/docs/app/reflex_docs/templates/docpage/sidebar/sidebar_items/learn.py b/docs/app/reflex_docs/templates/docpage/sidebar/sidebar_items/learn.py index f03b8315a29..14f407d03a4 100644 --- a/docs/app/reflex_docs/templates/docpage/sidebar/sidebar_items/learn.py +++ b/docs/app/reflex_docs/templates/docpage/sidebar/sidebar_items/learn.py @@ -260,6 +260,7 @@ def get_sidebar_items_hosting(): "Self Hosting", children=[ hosting.self_hosting, + hosting.scaling, hosting.databricks, hosting.cloud_providers, hosting.bring_your_own_cloud, diff --git a/docs/hosting/scaling.md b/docs/hosting/scaling.md new file mode 100644 index 00000000000..0d350704e24 --- /dev/null +++ b/docs/hosting/scaling.md @@ -0,0 +1,185 @@ +--- +title: Scaling Reflex Apps +meta_description: Horizontal scaling for Reflex apps, durable jobs, regional state, Redis Sentinel support limits, database read routing, OpenTelemetry compatibility, Redis spikes, and scale-down recovery. +--- + +# Scaling Reflex Apps + +Scale Reflex horizontally: run many small, replaceable instances of the same application. Keep interactive events quick, move long-running work to independent workers, and keep session state close to the region serving the user. + +This guide describes an architecture for self-hosted Reflex applications. Choose instance sizes, replica counts, and operational settings from production-mode load tests on the supported Reflex version. Managed hosting may expose different controls. + +## Prefer horizontal scaling + +Horizontal scaling adds application instances behind a load balancer. It provides capacity in small increments and lets an instance be replaced or removed without making it the permanent owner of a user’s work. Each replica runs the same application code and uses the shared services for its region. + +Vertical scaling gives an existing instance more CPU or memory. It can help a resource-heavy process, but it has a machine-size ceiling and does not remove a single-instance failure risk. Use it selectively when measurements justify it; prefer horizontal growth for the interactive backend. + +### Design for small replaceable instances + +- Serve the frontend independently, such as a static frontend through a CDN, and place the Python backends behind a WebSocket-capable load balancer. +- Share one region-local session Redis service across the backend replicas in that region. Store durable records, files, and jobs outside individual processes and local disks. +- Set process counts deliberately. Increasing replicas should not accidentally multiply an oversized worker pool and exhaust database or Redis connections. +- Scale down gracefully: stop routing new connections to a retiring instance, drain work that can finish quickly, and test reconnects to another healthy instance. + +For a small backend instance, one server process can be a useful starting point for measurement. This is a per-instance command; the deployment platform controls the number of replicas: + +```bash +REFLEX_USE_GRANIAN=true GRANIAN_WORKERS=1 reflex run --env prod \ + --backend-only --backend-port 8000 +``` + +Configure the exported frontend’s API URL for the load-balanced backend. One process is an example, not a universal sizing rule. See the [self-hosting guide](/docs/hosting/self-hosting/) for production commands and networking. + +## Pin versions and plan state-store upgrades + +Use the same application code, dependency lockfile, and compatible state schema on every replica sharing a state store. This guide's source checks are pinned to [Reflex revision d02ba5409579e24e67401f9dd6eb2fc4d3ef5e1b](https://github.com/reflex-dev/reflex/tree/d02ba5409579e24e67401f9dd6eb2fc4d3ef5e1b). They establish implementation and dependency constraints, not a load-test or failover certification for every release. + +| Deployment | Verified constraint | Deployment action | +| --- | --- | --- | +| Reflex 0.9 and 0.10 together | Reflex 0.9 and 0.10 cannot share a state store. 0.10 can load 0.9 state; 0.9 discards 0.10 state. | Do not use a mixed-version rolling deployment against the same Redis store or disk state directory. Upgrade all instances together, or use separate stores and accept session reset at cutover. | +| Rollback from 0.10 to 0.9 | Sessions saved by 0.10 reset when they reach 0.9. | Use a fresh state store, or clear only the app's dedicated state store as part of an approved rollback procedure. Keep durable records outside session state. | +| `reflex-otel==0.1.0` | Its release manifest declares Python >=3.10 and `reflex-base >=0.9.11`. | This is a release-specific dependency floor, not a guarantee for every 0.9 deployment. Pin the compatible framework, SDK, and exporter, and test the features used by that release. | +| `reflex-otel==0.2.0` and the reviewed source | Python >=3.11 and `reflex-base >=0.10.0`. Both instrumentor releases declare `opentelemetry-api >=1.30.0,<2.0` and instrumentation/ASGI instrumentation >=0.49b0,<1.0. | Resolve and pin the exact instrumentor, SDK, exporter, and compatible framework versions in your lockfile. Verify backend and optional browser export together before rollout. Do not install 0.2.0 against 0.9. | + +The table records constraints from [self-hosting](https://github.com/reflex-dev/reflex/blob/d02ba5409579e24e67401f9dd6eb2fc4d3ef5e1b/docs/hosting/self-hosting.md), the [0.1.0 instrumentor manifest](https://github.com/reflex-dev/reflex/blob/3d47357ac158c51a0c131ec48c41232830bd9031/packages/reflex-otel/pyproject.toml), and the [0.2.0 instrumentor manifest](https://github.com/reflex-dev/reflex/blob/40ffab8124ba257053c395309c87f1c818b4e0a2/packages/reflex-otel/pyproject.toml). For another release, recheck those files at that exact tag or commit. Record the resolved Python, Reflex, reflex-base, reflex-otel, Redis client/server, broker, and result-store versions with your load and recovery test results. A dependency lower bound is not evidence that an arbitrary combination has been tested. + +## Keep interactive events quick + +An event should validate the request, perform bounded work, update a small amount of UI state, and return. Paginate large result sets, separate unrelated views into their own state classes, and avoid blocking I/O or long computation while holding a state lock. + +Adding replicas will not shorten one oversized event or remove contention within one session. Measure handler duration and state acquisition separately, then address the work responsible. The [performance and execution guide](/docs/advanced-onboarding/performance-and-execution/) explains these boundaries. + +## Run long work in independent workers + +Send long-running work to a queue system such as Taskiq or Celery, with workers deployed separately from the Reflex backend. Design the handoff so the backend that accepts a job can be removed after durable acceptance. Another backend must still be able to find the job, show its progress, and retrieve its result; verify this behavior with the selected broker, workers, and result store. + +Reflex background events run inside an application backend. They can keep short-lived UI work responsive, but they do not give a job an independent lifetime. Use an external worker whenever the task must survive removal of its originating backend. See [background event lifecycle](/docs/events/background-events/). + +### Hand off jobs durably + +Consider a report export. The UI needs confirmation that the job was accepted; it should not need its original backend to stay alive until the report finishes. + +1. **Accept the request.** Check authorization and validate inputs. Create a durable job record with a stable job ID, owner or tenant, input reference, and initial status. +2. **Persist the handoff.** Record the enqueue intent reliably. For example, write the job and an outbox entry in one database transaction. A dispatcher publishes the outbox entry to the broker and marks it delivered only after broker confirmation, retrying with the same job ID. A crash after publication can cause duplicate delivery, so the consumer must be idempotent. Monitor and retry pending outbox records so recorded work is not silently left unqueued. +3. **Return to the UI.** Keep the job ID and a short status in session state, then return from the event. Do not wait for the worker’s result inside the event. +4. **Process independently.** The worker receives the job ID, loads authorized inputs from shared storage, and executes the task. It must not depend on the original backend’s memory, local files, or WebSocket. +5. **Persist progress and results.** Write completion, failure, and result references to shared durable storage. Store large outputs in shared object storage. +6. **Read from any backend.** After checking access, the backend handling the next request reads the job record by ID. Use bounded polling or a notification that triggers a fresh read. Reopening the app should recover outstanding jobs from durable records. + +An illustrative record makes the ownership boundary explicit: + +```text +job_id: stable identifier +owner_id: authenticated user or tenant +input_ref: shared data or object reference +status: queued | running | succeeded | failed +result_ref: shared output reference when complete +``` + +### Configure recovery rather than assuming it + +A queue library alone does not guarantee durability. Choose a broker and result store with the required persistence, configure acknowledgements and redelivery, and test worker crashes. Use a tenant-scoped idempotency key with a database uniqueness constraint for repeated submissions, and idempotent or transactional worker effects so redelivery does not duplicate side effects. Persist the outcome before acknowledging success. If an external effect cannot be committed atomically with the outcome, use that service's idempotency support or a reconciliation procedure; there is no general exactly-once guarantee. Bound retries, add timeouts, and make permanently failed jobs visible. + +Celery acknowledgement timing and worker-loss behavior affect whether work is redelivered. Taskiq likewise needs a suitable broker and an actual result backend or application job store. Specify result retention, backups, and expiry independently of broker delivery; keep the authoritative job status in the application store if broker results expire sooner than users need them. Avoid process-local queues and in-memory-only results for work that must survive scale-down. Scale workers by backlog and oldest-job age independently of app replicas. + +References: [Celery task semantics](https://docs.celeryq.dev/en/stable/userguide/tasks.html), [Taskiq architecture](https://taskiq-python.github.io/guide/architecture-overview.html), and [Celery worker shutdown](https://docs.celeryq.dev/en/stable/userguide/workers.html). + +## Keep session state local to each region + +For a multi-region application, use region-local Redis for Reflex session state. Put more persistent or cross-region data in a database or a separately designed global Redis service. Frequent session-state access stays close to the backend, while important records remain independent of a UI session. + +- **Regional sessions:** each region’s backends share that region’s session Redis. Route an active session consistently to its region; individual backend instances remain replaceable. +- **Persistent records:** business data, preferences, and jobs belong in a store available to the regions that need them. Define consistency, write ownership, replication, residency, and recovery expectations. +- **Global Redis:** configure persistence, replication, expiry, and recovery for the items it holds. A globally reachable endpoint alone does not guarantee durability. +- **Regional failover:** moving a user to another region may create a fresh UI session. Reconstruct useful state from persistent records, including outstanding jobs. Do not promise seamless transfer of region-local session state. + +For example, a user starts an export in Region A and reconnects in Region B. The independent worker continues. Region B checks authorization and reads the durable job and result without needing the original backend or Region A’s session Redis. + +See [Redis persistence and recovery tradeoffs](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/). + +### Redis Sentinel availability versus Redis Cluster sharding + +Redis Sentinel monitors non-clustered Redis and coordinates promotion of a replica when the primary fails; it does not shard data. Redis Cluster partitions data across shards and requires a cluster-aware client. Horizontal application scaling and Redis sharding are separate decisions: adding backend instances does not itself require Cluster. + +**No built-in Sentinel adapter has been verified for the Reflex source reviewed here.** The Redis factory accepts `redis://`, `rediss://`, and `unix://` URLs and creates ordinary synchronous or asynchronous Redis clients. It does not construct a Sentinel client, discover a primary through Sentinel, or construct a Redis Cluster client. Do not point `REFLEX_REDIS_URL` at a Sentinel control port or invent a Sentinel URL scheme. Redis Cluster is not a supported session-state configuration in this reviewed client path. + +A provider may expose a stable ordinary Redis primary endpoint backed by its own failover system. That is a provider-specific deployment choice, not native Sentinel integration. Validate that endpoint's primary switching, connection recovery, state locks, and persistence with your exact Reflex and Redis client versions before relying on it. A custom Sentinel adapter needs its own implementation and integration tests before it can be documented as supported. + +Sentinel's Redis replication is asynchronous, so failover can lose recent acknowledged writes. Define recovery behavior and capacity on the remaining nodes. Persistent business records and jobs need their own durability guarantees. See [Redis Sentinel requirements](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/) and [Redis Cluster specification](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/). + +### Separate caching from session state + +Use separate Redis services for application caching and Reflex session state. Reflex reads `REFLEX_REDIS_URL` for state. Application cache code can read a separate variable such as `APP_CACHE_REDIS_URL`; that second name is an application convention, not another built-in Reflex cache setting. + +Different key prefixes or logical database numbers on one server do not isolate resource usage or failure. + +```bash +# Illustrative ordinary regional endpoints, not Sentinel setup. +REFLEX_REDIS_URL=rediss://state.eu.example:6379/0 +APP_CACHE_REDIS_URL=rediss://cache.eu.example:6379/0 +``` + +Supply real endpoints and credentials through deployment configuration. Use bounded cache TTLs, reuse clients, and prevent cache misses from causing unlimited database work. Keep cache clients outside serialized session state. + +## Route database queries explicitly + +For a globally distributed app, consider a read-only database replica in each region that hosts Reflex backends. Eligible reads can stay local, reducing cross-region latency and read load on the primary. Make routing an explicit policy in the data-access layer that covers every query. + +- **Primary:** route writes, migrations, locking reads, and read-write transactions to the current primary. Keep the operations in a transaction on its owning connection. +- **Regional replicas:** route read-only queries that tolerate replication lag to a healthy nearby replica. A `SELECT` that locks rows or participates in a write transaction is not an eligible replica read. +- **Consistency exceptions:** use the primary for read-after-write and freshness-critical reads unless the replica is verified to have caught up. For example, read a newly created job from the primary when immediately confirming submission. +- **Failures:** monitor lag and health. Use another healthy replica or a bounded fallback to the primary, with capacity protection. Never send writes to a replica outside a controlled promotion or failover process. + +Test delayed replication, replica unavailability, and primary failover. Confirm that routing preserves transaction consistency and does not turn a regional outage into unlimited load on the primary. See [PostgreSQL replication tradeoffs](https://www.postgresql.org/docs/current/high-availability.html). + +## Use OpenTelemetry to locate bottlenecks + +Use Reflex's built-in OpenTelemetry trace points and metrics for a compatible release. They remain inactive until the optional `reflex-otel` package is enabled and an SDK/exporter is configured. The reviewed `reflex-otel` source requires Python >=3.11 and `reflex-base >=0.10.0`; it is not a universal add-on for 0.9 deployments. Pin the framework, instrumentor, OpenTelemetry SDK, and exporter together using the version table above, and verify that telemetry reaches the collector. + +- **Event latency:** use event-handler spans and `reflex.event.duration` to identify slow handlers and failures. Use `reflex.state.acquire.duration` to distinguish state-acquisition waits from the rest of event execution. +- **Connection and payload load:** correlate `reflex.websocket.connections` and `reflex.websocket.message.size` with instance count and event latency. The instrumentor also provides ASGI request and connection tracing. +- **Browser visibility:** optional browser instrumentation connects frontend interactions to backend event traces. + +See the [Reflex OpenTelemetry guide](/docs/api-reference/observability/) for the instrumentation scope and setup. + +Database queries, queue publication, and external Taskiq/Celery workers need their own compatible instrumentation or application spans. Propagate trace context through the broker and restore it in the worker so enqueueing, execution, and database work can be correlated. Measure queue wait separately from execution; Reflex’s instrumentor does not automatically supply these external spans. + +Identify service, region, deployment version, and database role in telemetry. Follow a slow event or job to determine whether time is spent acquiring state, crossing regions, querying a database, or waiting for worker capacity. Verify a representative end-to-end trace, choose sampling and redaction rules, and avoid sensitive payloads and high-cardinality job IDs in metric labels. See [OpenTelemetry context propagation](https://opentelemetry.io/docs/languages/python/propagation/). + +## Validate scale down as well as scale out + +Load-test realistic sessions and events in production mode. Record the Reflex version, instance size, process and replica counts, event rate, payload sizes, p50/p95 latency, error rate, and backend, database, and Redis resources. Measure queue delay and job completion separately from interactive latency. + +- Add replicas under load and verify latency and errors stay within target. Watch for a downstream database or Redis bottleneck. +- Remove the backend that accepted a long job. Reconnect through another replica and confirm the job continues and its result remains accessible. +- Restart a worker mid-job. Verify the configured recovery behavior and that retries do not duplicate effects. Test worker drain separately from app scale-down. +- Repeat a submission and interrupt enqueueing. Verify idempotency and that accepted work is not silently lost between the job store and broker. +- Fail over the tested regional Redis endpoint or explicitly validated custom Sentinel integration. Measure recovery and exercise active sessions, locks, and reconnects; an unverified adapter is not a supported failover setup. +- Move a user to another region. Verify session-reset behavior, authorization, and retrieval of persistent records and jobs. +- Delay or disable a database replica. Verify freshness-sensitive reads, transaction routing, and bounded primary fallback. +- Follow an event-to-worker-to-database trace and check that missing telemetry does not hide a bottleneck. + +Scale backends using resource pressure, event latency, connection load, and errors. Scale workers using queue backlog and oldest-job age. Allow enough time for the chosen drain and recovery policies. There is no universal sessions-per-instance figure. + +## Diagnose Redis spikes before resizing + +Record the affected region and endpoint, time window, deployment changes, active sessions, and event rate. Compare state and cache services separately. Start with provider metrics and bounded read-only diagnostics using approved operator access: + +```text +INFO memory +INFO stats +INFO clients +INFO commandstats +INFO keyspace +SLOWLOG GET 20 +``` + +- **Memory growth:** compare key counts, state sizes, expirations, and evictions. Look for full datasets copied into each session, unbounded histories, and cache entries without useful TTLs. +- **CPU and commands:** correlate event frequency, polling, serialization, and refill bursts. Reduce unnecessary work before adding instances that repeat it. +- **Connections:** check replica and process counts, pool sizes, client reuse, and retry or reconnect bursts. More app instances also mean more connections. +- **Latency and locks:** inspect slow I/O and lock-held time. Move long work to the external worker path; increasing a timeout does not add throughput. + +`SLOWLOG` measures server execution, excludes network latency, and can expose private arguments. Redact before sharing. Alert on state eviction and memory pressure. A `noeviction` policy prevents eviction but rejects new writes at the limit. + +References: [Redis INFO](https://redis.io/docs/latest/commands/info/), [slow log](https://redis.io/docs/latest/commands/slowlog-get/), and [eviction policies](https://redis.io/docs/latest/develop/reference/eviction/). diff --git a/docs/hosting/self-hosting.md b/docs/hosting/self-hosting.md index 8ef5b76e006..b6b81440953 100644 --- a/docs/hosting/self-hosting.md +++ b/docs/hosting/self-hosting.md @@ -18,6 +18,9 @@ Use `reflex deploy` for the managed workflow. Follow this page when you need to - Moving from managed hosting also requires transferring persistent data, secrets, domains, and any external service configuration. - You operate the servers, TLS, monitoring, backups, and updates in a self-hosted environment. +For multiple backend replicas, regional state, durable jobs, and recovery tests, +see [Scaling Reflex Apps](/docs/hosting/scaling/). + ## Managed vs. self-hosted | Deployment choice | Managed (`reflex deploy`) | Self-hosted | From e29bd80cd5226c8ea4ac376639630d2a22be8a60 Mon Sep 17 00:00:00 2001 From: Kastier1 <40179067+Kastier1@users.noreply.github.com.> Date: Wed, 14 Oct 2026 14:11:07 -0700 Subject: [PATCH 2/5] ENG-12238: illustrate scaling architecture and job recovery --- .../assets/hosting/scaling-architecture.svg | 148 +++++++++ .../assets/hosting/scaling-job-lifecycle.svg | 75 +++++ docs/app/news/+scaling-guide.docs.md | 2 +- docs/hosting/scaling.md | 281 ++++++++++-------- 4 files changed, 379 insertions(+), 127 deletions(-) create mode 100644 docs/app/assets/hosting/scaling-architecture.svg create mode 100644 docs/app/assets/hosting/scaling-job-lifecycle.svg diff --git a/docs/app/assets/hosting/scaling-architecture.svg b/docs/app/assets/hosting/scaling-architecture.svg new file mode 100644 index 00000000000..019b94b076b --- /dev/null +++ b/docs/app/assets/hosting/scaling-architecture.svg @@ -0,0 +1,148 @@ + + Reference architecture for a scaled Reflex application + A browser loads a static frontend and exchanges events and updates through a WebSocket-capable load balancer with replaceable backend replicas. Backends use a primary database for writes and fresh reads, lagging read replicas only for eligible reads, regional Redis for session state, and an optional separate application cache. Choose one state Redis connection path: a direct ordinary Redis URL, or an optional independently validated Sentinel-aware endpoint or adapter. That integration discovers the primary through Sentinel control traffic and sends application data to Redis. Three example Sentinel instances monitor Redis with a quorum of two; a majority authorizes failover. Native Sentinel discovery is not verified in Reflex. A reliable job handoff publishes to a task broker. Independent workers persist outcomes in shared durable storage before acknowledging success. Queue persistence, recovery, result storage, and idempotency must be configured and tested. + + + + + + + + + A production app, split by responsibility + Illustrative topology. Add services only when the application needs them. + + Request / response + + Async job + + Replication + + Control plane + + + Browser + React UI + + Static frontend / CDN + HTML, JavaScript, assets + + HTTPS: load frontend + + + Load balancer + WebSocket-capable + + HTTP / WebSocket + events ↔ updates + + Reflex backend replicas + + + + 1 + 2 + N + Same code • compatible state schema + + Route connections + + + + + + + + Writes / fresh reads + Lag-tolerant reads + Choose one Redis path + Direct + ordinary + Redis URL + + Optional, validated + Sentinel-aware + endpoint / adapter + Separate integration + + State / + locks + + + Discover current primary + + + Primary database + Business records + Jobs + result references + + Read replica DBs + Read-only routing + May lag behind primary + + Asynchronous DB replication + + + Regional state Redis + Primary data endpoint + Shared by this region + + State Redis replica + + Async + replication + + + + Optional cache + Separate Redis service + Cache get / set + TTLs + + + Optional Sentinel control plane + + + + Sentinel 1 + Sentinel 2 + Sentinel 3 + Example quorum: 2 of 3; majority authorizes failover. + Requires validated Sentinel-aware integration. + No verified native Reflex Sentinel discovery. + + + Monitor / coordinate failover + + + Publish after reliable handoff (see job lifecycle) + + Task queue / broker + Configure persistence + and delivery / recovery + + Independent workers + Idempotent task effects + Shared inputs + result storage + + Deliver / redeliver + + Ack after persistence + + Persist outcomes in shared durable storage + Storage persistence, replication lag, retries and failover need explicit policies. + Validate recovery for the selected versions; this topology is not a guarantee. + diff --git a/docs/app/assets/hosting/scaling-job-lifecycle.svg b/docs/app/assets/hosting/scaling-job-lifecycle.svg new file mode 100644 index 00000000000..b427681c2da --- /dev/null +++ b/docs/app/assets/hosting/scaling-job-lifecycle.svg @@ -0,0 +1,75 @@ + + A durable job handoff and recovery lifecycle + An authorized request creates a durable job and outbox record in one transaction using a stable job identifier and a tenant-scoped idempotency key enforced by database uniqueness. The backend can return the job identifier while a dispatcher retries publication until broker confirmation. The broker delivers work to an independent worker. The worker applies effects that are idempotent or transactionally deduplicated by job ID and persists the outcome and result reference before acknowledging success. A worker crash before acknowledgement can lead to redelivery under the selected broker policy, so duplicate delivery must be safe. Any backend checks authorization and reads progress or results from durable records, using a primary or verified caught-up replica for freshness-sensitive reads. This is a design pattern requiring configured and tested persistence, recovery and retention, not an exactly-once guarantee. + + + + + + + + A job can outlive its accepting backend + Example: transactional outbox + configured broker + shared durable results. + + + 1. Authorize and record the request + Commit job + outbox entry in one database transaction. + Stable job ID; enforce a tenant-scoped idempotency key in the database. + + + After commit + + Return the job ID to the UI + Return from the event; show status. + The original backend can retire. + + + Independently + + 2. Dispatch the outbox entry + Publish the same job ID on retry. + Mark sent after broker confirms. + + + Publish / confirm + + 3. Broker delivers the job + Persistence + redelivery configured + + Deliver + + + 4. Worker runs independently + Idempotent / deduplicated effects + Bound retries; surface failed jobs. + + Recovery is explicit + Crash before acknowledgement? + Recover / redeliver under the + selected broker and worker policy. + Duplicate delivery is possible + Retry must not duplicate effects. + External effects may need their + own idempotency or reconciliation. + + + Persist first + + 5. Store outcome + result reference, then acknowledge success + Any backend can authorize and read the job; use the primary for fresh reads. + Store large outputs in shared object storage; set retention and recovery policy. + + A queue library alone does not supply durability or an exactly-once guarantee. + Test interrupted publication, worker loss, duplicate delivery and result recovery. + diff --git a/docs/app/news/+scaling-guide.docs.md b/docs/app/news/+scaling-guide.docs.md index 653fcee4a86..c43cc954bea 100644 --- a/docs/app/news/+scaling-guide.docs.md +++ b/docs/app/news/+scaling-guide.docs.md @@ -1 +1 @@ -Add a scaling guide covering horizontal backends, state-store version compatibility, durable jobs, regional data routing, telemetry, and Redis recovery checks. +Add a scaling guide with production architecture and job lifecycle diagrams, covering horizontal backends, state-store version compatibility, durable jobs, regional data routing, telemetry, and Redis recovery checks. diff --git a/docs/hosting/scaling.md b/docs/hosting/scaling.md index 0d350704e24..450996e27e2 100644 --- a/docs/hosting/scaling.md +++ b/docs/hosting/scaling.md @@ -5,181 +5,210 @@ meta_description: Horizontal scaling for Reflex apps, durable jobs, regional sta # Scaling Reflex Apps -Scale Reflex horizontally: run many small, replaceable instances of the same application. Keep interactive events quick, move long-running work to independent workers, and keep session state close to the region serving the user. +Scale the interactive backend by adding small, replaceable replicas behind a load balancer. Keep UI events quick and give long-running jobs their own workers and durable storage. -This guide describes an architecture for self-hosted Reflex applications. Choose instance sizes, replica counts, and operational settings from production-mode load tests on the supported Reflex version. Managed hosting may expose different controls. +This guide uses a report-export application to explain the boundaries: a user requests a report, leaves the page, and later retrieves the result through any healthy backend. Choose capacity and recovery settings from production-mode tests of your exact versions. Managed hosting may expose different controls. -## Prefer horizontal scaling +## Reference production architecture -Horizontal scaling adds application instances behind a load balancer. It provides capacity in small increments and lets an instance be replaced or removed without making it the permanent owner of a user’s work. Each replica runs the same application code and uses the shared services for its region. +This is an illustrative deployment, not a list of required components. Start with the request path and stores your app needs. Add read replicas for lag-tolerant read load, a separate cache for expensive repeatable reads, and a queue when jobs must outlive a backend. The optional Sentinel panel requires a separately implemented and validated integration. -Vertical scaling gives an existing instance more CPU or memory. It can help a resource-heavy process, but it has a machine-size ceiling and does not remove a single-instance failure risk. Use it selectively when measurements justify it; prefer horizontal growth for the interactive backend. - -### Design for small replaceable instances - -- Serve the frontend independently, such as a static frontend through a CDN, and place the Python backends behind a WebSocket-capable load balancer. -- Share one region-local session Redis service across the backend replicas in that region. Store durable records, files, and jobs outside individual processes and local disks. -- Set process counts deliberately. Increasing replicas should not accidentally multiply an oversized worker pool and exhaust database or Redis connections. -- Scale down gracefully: stop routing new connections to a retiring instance, drain work that can finish quickly, and test reconnects to another healthy instance. - -For a small backend instance, one server process can be a useful starting point for measurement. This is a per-instance command; the deployment platform controls the number of replicas: - -```bash -REFLEX_USE_GRANIAN=true GRANIAN_WORKERS=1 reflex run --env prod \ - --backend-only --backend-port 8000 +```python exec +import reflex as rx ``` -Configure the exported frontend’s API URL for the load-balanced backend. One process is an example, not a universal sizing rule. See the [self-hosting guide](/docs/hosting/self-hosting/) for production commands and networking. - -## Pin versions and plan state-store upgrades - -Use the same application code, dependency lockfile, and compatible state schema on every replica sharing a state store. This guide's source checks are pinned to [Reflex revision d02ba5409579e24e67401f9dd6eb2fc4d3ef5e1b](https://github.com/reflex-dev/reflex/tree/d02ba5409579e24e67401f9dd6eb2fc4d3ef5e1b). They establish implementation and dependency constraints, not a load-test or failover certification for every release. - -| Deployment | Verified constraint | Deployment action | -| --- | --- | --- | -| Reflex 0.9 and 0.10 together | Reflex 0.9 and 0.10 cannot share a state store. 0.10 can load 0.9 state; 0.9 discards 0.10 state. | Do not use a mixed-version rolling deployment against the same Redis store or disk state directory. Upgrade all instances together, or use separate stores and accept session reset at cutover. | -| Rollback from 0.10 to 0.9 | Sessions saved by 0.10 reset when they reach 0.9. | Use a fresh state store, or clear only the app's dedicated state store as part of an approved rollback procedure. Keep durable records outside session state. | -| `reflex-otel==0.1.0` | Its release manifest declares Python >=3.10 and `reflex-base >=0.9.11`. | This is a release-specific dependency floor, not a guarantee for every 0.9 deployment. Pin the compatible framework, SDK, and exporter, and test the features used by that release. | -| `reflex-otel==0.2.0` and the reviewed source | Python >=3.11 and `reflex-base >=0.10.0`. Both instrumentor releases declare `opentelemetry-api >=1.30.0,<2.0` and instrumentation/ASGI instrumentation >=0.49b0,<1.0. | Resolve and pin the exact instrumentor, SDK, exporter, and compatible framework versions in your lockfile. Verify backend and optional browser export together before rollout. Do not install 0.2.0 against 0.9. | - -The table records constraints from [self-hosting](https://github.com/reflex-dev/reflex/blob/d02ba5409579e24e67401f9dd6eb2fc4d3ef5e1b/docs/hosting/self-hosting.md), the [0.1.0 instrumentor manifest](https://github.com/reflex-dev/reflex/blob/3d47357ac158c51a0c131ec48c41232830bd9031/packages/reflex-otel/pyproject.toml), and the [0.2.0 instrumentor manifest](https://github.com/reflex-dev/reflex/blob/40ffab8124ba257053c395309c87f1c818b4e0a2/packages/reflex-otel/pyproject.toml). For another release, recheck those files at that exact tag or commit. Record the resolved Python, Reflex, reflex-base, reflex-otel, Redis client/server, broker, and result-store versions with your load and recovery test results. A dependency lower bound is not evidence that an arbitrary combination has been tested. - -## Keep interactive events quick - -An event should validate the request, perform bounded work, update a small amount of UI state, and return. Paginate large result sets, separate unrelated views into their own state classes, and avoid blocking I/O or long computation while holding a state lock. - -Adding replicas will not shorten one oversized event or remove contention within one session. Measure handler duration and state acquisition separately, then address the work responsible. The [performance and execution guide](/docs/advanced-onboarding/performance-and-execution/) explains these boundaries. - -## Run long work in independent workers +```python eval +rx.el.figure( + rx.el.div( + rx.image( + src=rx.asset("hosting/scaling-architecture.svg"), + alt="Production architecture: browser and frontend, load balancer, backend replicas, databases, separate Redis roles, optional Sentinel integration, queue and workers. Text description follows.", + width="100%", + min_width="760px", + height="auto", + ), + overflow_x="auto", + width="100%", + tab_index=0, + role="region", + custom_attrs={ + "aria-label": "Production architecture; scroll horizontally on narrow screens" + }, + ), + rx.el.figcaption( + rx.el.a( + "Open architecture diagram at full size", + href=rx.asset("hosting/scaling-architecture.svg"), + target="_blank", + rel="noopener noreferrer", + text_decoration="underline", + ), + ), + width="100%", + margin_y="1.5em", +) +``` -Send long-running work to a queue system such as Taskiq or Celery, with workers deployed separately from the Reflex backend. Design the handoff so the backend that accepts a job can be removed after durable acceptance. Another backend must still be able to find the job, show its progress, and retrieve its result; verify this behavior with the selected broker, workers, and result store. +### Architecture text alternative -Reflex background events run inside an application backend. They can keep short-lived UI work responsive, but they do not give a job an independent lifetime. Use an external worker whenever the task must survive removal of its originating backend. See [background event lifecycle](/docs/events/background-events/). +The labeled arrows distinguish traffic types. On narrow screens, scroll the diagram or open it at full size. -### Hand off jobs durably +- **Request/response:** browser ↔ static frontend/CDN for assets; browser frontend ↔ load balancer ↔ one healthy backend replica for events and state updates over WebSockets. Backends share the same application build. +- **Application data:** backend replicas ↔ regional Redis session state; optional application cache uses a separate Redis service. Backends and workers write durable records to the primary database. Only lag-tolerant reads go to read replicas; read-after-write and other freshness-critical reads use the primary. +- **Asynchronous jobs:** backend → durable job record/outbox → task queue → independent workers. Workers save status and results in shared storage; any authorized backend can read them. The job lifecycle below explains confirmation, retry and acknowledgement boundaries. +- **Replication:** primary database → read replica databases; Redis primary → Redis replica. These arrows copy data, not application requests. Asynchronous replication can lag and can lose recent writes on failover. +- **Sentinel monitoring/failover:** optional Sentinel instances monitor the Redis primary/replica and coordinate promotion. A validated Sentinel-aware integration discovers the current primary; application data still goes to Redis, never to a Sentinel control port. This is not native Reflex Sentinel support. -Consider a report export. The UI needs confirmation that the job was accepted; it should not need its original backend to stay alive until the report finishes. +## Serve requests with replaceable backends -1. **Accept the request.** Check authorization and validate inputs. Create a durable job record with a stable job ID, owner or tenant, input reference, and initial status. -2. **Persist the handoff.** Record the enqueue intent reliably. For example, write the job and an outbox entry in one database transaction. A dispatcher publishes the outbox entry to the broker and marks it delivered only after broker confirmation, retrying with the same job ID. A crash after publication can cause duplicate delivery, so the consumer must be idempotent. Monitor and retry pending outbox records so recorded work is not silently left unqueued. -3. **Return to the UI.** Keep the job ID and a short status in session state, then return from the event. Do not wait for the worker’s result inside the event. -4. **Process independently.** The worker receives the job ID, loads authorized inputs from shared storage, and executes the task. It must not depend on the original backend’s memory, local files, or WebSocket. -5. **Persist progress and results.** Write completion, failure, and result references to shared durable storage. Store large outputs in shared object storage. -6. **Read from any backend.** After checking access, the backend handling the next request reads the job record by ID. Use bounded polling or a notification that triggers a fresh read. Reopening the app should recover outstanding jobs from durable records. +Host the exported frontend independently, for example on a CDN. Configure its API URL for the WebSocket-capable load balancer in front of the Python backends. Each region's replicas share that region's session Redis; durable records, files and jobs live outside individual processes and local disks. -An illustrative record makes the ownership boundary explicit: +Start with deliberate process counts so adding replicas does not multiply oversized database or Redis connection pools. One Granian process per small instance is a measurement starting point: -```text -job_id: stable identifier -owner_id: authenticated user or tenant -input_ref: shared data or object reference -status: queued | running | succeeded | failed -result_ref: shared output reference when complete +```bash +GRANIAN_WORKERS=1 reflex run --env prod \ + --backend-only --backend-port 8000 ``` -### Configure recovery rather than assuming it +This command assumes Granian. If the environment also includes Uvicorn and Gunicorn, set `REFLEX_USE_GRANIAN=true` explicitly. The deployment platform controls replica count. See [self-hosting](/docs/hosting/self-hosting/) for production commands and networking. + +More replicas do not shorten one oversized event or remove contention within one session. Paginate results, keep session state small, and measure handler duration separately from state acquisition. Use async database/API clients in [background events](/docs/events/background-events/), holding the state lock only briefly to update UI state. For unavoidable blocking I/O, use `await rx.run_in_thread(...)` from a background event; use process workers or a process executor for CPU-heavy Python work. See [performance and execution](/docs/advanced-onboarding/performance-and-execution/). + +## Let jobs outlive a backend + +A report export may take minutes. The UI needs a job ID and progress, while the worker needs to finish even if the accepting backend is scaled down. Use an independent queue/worker system such as [Taskiq](https://taskiq-python.github.io/guide/architecture-overview.html) or [Celery](https://docs.celeryq.dev/en/stable/userguide/tasks.html) for that requirement. Reflex background events run inside a backend and end with that process. + +```python eval +rx.el.figure( + rx.el.div( + rx.image( + src=rx.asset("hosting/scaling-job-lifecycle.svg"), + alt="Job lifecycle: commit a job and outbox, confirm queue publication, process idempotently, save results before acknowledgement, and recover through bounded retries. Text description follows.", + width="100%", + min_width="760px", + height="auto", + ), + overflow_x="auto", + width="100%", + tab_index=0, + role="region", + custom_attrs={ + "aria-label": "Job lifecycle; scroll horizontally on narrow screens" + }, + ), + rx.el.figcaption( + rx.el.a( + "Open job lifecycle diagram at full size", + href=rx.asset("hosting/scaling-job-lifecycle.svg"), + target="_blank", + rel="noopener noreferrer", + text_decoration="underline", + ), + ), + width="100%", + margin_y="1.5em", +) +``` -A queue library alone does not guarantee durability. Choose a broker and result store with the required persistence, configure acknowledgements and redelivery, and test worker crashes. Use a tenant-scoped idempotency key with a database uniqueness constraint for repeated submissions, and idempotent or transactional worker effects so redelivery does not duplicate side effects. Persist the outcome before acknowledging success. If an external effect cannot be committed atomically with the outcome, use that service's idempotency support or a reconciliation procedure; there is no general exactly-once guarantee. Bound retries, add timeouts, and make permanently failed jobs visible. +### Job lifecycle text alternative -Celery acknowledgement timing and worker-loss behavior affect whether work is redelivered. Taskiq likewise needs a suitable broker and an actual result backend or application job store. Specify result retention, backups, and expiry independently of broker delivery; keep the authoritative job status in the application store if broker results expire sooner than users need them. Avoid process-local queues and in-memory-only results for work that must survive scale-down. Scale workers by backlog and oldest-job age independently of app replicas. +1. **Accept and record.** Authorize the request, validate inputs, and commit a job record and enqueue intent together, such as an outbox entry in the same database transaction. Store a stable job ID, owner/tenant, input reference and status. A tenant-scoped idempotency key with a uniqueness constraint handles repeated submissions. +2. **Return and dispatch.** Return the job ID after the transaction commits. A dispatcher publishes pending outbox entries, marks them delivered only after broker confirmation, and retries with the same job ID. Monitor pending entries: an accepted record does not itself prove the broker received it. +3. **Process independently.** Workers load authorized inputs from shared storage. They must not need the original backend's memory, local files or WebSocket. A crash after publication can cause duplicate delivery, so worker effects must be idempotent or transactionally deduplicated by job ID. +4. **Save, then acknowledge.** Persist progress, completion/failure and result references in shared durable storage before acknowledging success to the broker. Keep large outputs in object storage. For external effects that cannot commit atomically with the outcome, use the service's idempotency support or reconciliation; there is no general exactly-once guarantee. +5. **Recover and read.** Configure broker acknowledgements and redelivery, timeouts and bounded retries. Expose permanent failures. Any healthy backend can authorize access and read the job by ID; reopening the app can recover outstanding jobs from durable records. -References: [Celery task semantics](https://docs.celeryq.dev/en/stable/userguide/tasks.html), [Taskiq architecture](https://taskiq-python.github.io/guide/architecture-overview.html), and [Celery worker shutdown](https://docs.celeryq.dev/en/stable/userguide/workers.html). +A queue library alone does not guarantee durability. Select broker persistence and worker-loss behavior, then test them. Taskiq needs an actual result backend or application job store; Celery acknowledgement and worker-loss settings affect redelivery. Define result retention, backups and expiry independently of queue delivery. Avoid process-local queues and in-memory-only results for jobs that must survive scale-down. Scale workers by backlog and oldest-job age, independently of backend replicas. -## Keep session state local to each region +## Keep regional state and durable data separate -For a multi-region application, use region-local Redis for Reflex session state. Put more persistent or cross-region data in a database or a separately designed global Redis service. Frequent session-state access stays close to the backend, while important records remain independent of a UI session. +Keep frequent session-state access close to the backends: each region uses its own Redis session store, and active sessions are routed consistently to that region. Business records, preferences and jobs belong in stores with explicit write ownership, consistency, replication and recovery policies. -- **Regional sessions:** each region’s backends share that region’s session Redis. Route an active session consistently to its region; individual backend instances remain replaceable. -- **Persistent records:** business data, preferences, and jobs belong in a store available to the regions that need them. Define consistency, write ownership, replication, residency, and recovery expectations. -- **Global Redis:** configure persistence, replication, expiry, and recovery for the items it holds. A globally reachable endpoint alone does not guarantee durability. -- **Regional failover:** moving a user to another region may create a fresh UI session. Reconstruct useful state from persistent records, including outstanding jobs. Do not promise seamless transfer of region-local session state. +Moving a user to another region may create a fresh UI session. Rebuild useful state from durable records rather than promising transfer of regional Redis state. For example, after an export starts in Region A, a backend in Region B can authorize the user and read its job/result without the original session. A globally reachable Redis endpoint alone does not make data durable; review its [persistence and recovery tradeoffs](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/). -For example, a user starts an export in Region A and reconnects in Region B. The independent worker continues. Region B checks authorization and reads the durable job and result without needing the original backend or Region A’s session Redis. +### Use the primary for writes and fresh reads -See [Redis persistence and recovery tradeoffs](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/). +Read replicas are useful when nearby, read-only copies reduce latency or primary load. Make routing explicit in the data-access layer: -### Redis Sentinel availability versus Redis Cluster sharding +| Operation | Destination | +| --- | --- | +| Writes, migrations, locking reads and read-write transactions | Current primary; keep each transaction on its owning connection. | +| Read-after-write or other freshness-critical reads | Primary unless a replica is verified to have caught up. Confirm a newly submitted job here. | +| Read-only queries that tolerate replication lag | Healthy read replica; a locking `SELECT` is not eligible. | +| Replica unavailable or too far behind | Another healthy replica or bounded primary fallback, with capacity protection. | -Redis Sentinel monitors non-clustered Redis and coordinates promotion of a replica when the primary fails; it does not shard data. Redis Cluster partitions data across shards and requires a cluster-aware client. Horizontal application scaling and Redis sharding are separate decisions: adding backend instances does not itself require Cluster. +Test lag and outages before relying on regional routing. Never send writes to a replica outside a controlled promotion process. See [PostgreSQL replication tradeoffs](https://www.postgresql.org/docs/current/high-availability.html). -**No built-in Sentinel adapter has been verified for the Reflex source reviewed here.** The Redis factory accepts `redis://`, `rediss://`, and `unix://` URLs and creates ordinary synchronous or asynchronous Redis clients. It does not construct a Sentinel client, discover a primary through Sentinel, or construct a Redis Cluster client. Do not point `REFLEX_REDIS_URL` at a Sentinel control port or invent a Sentinel URL scheme. Redis Cluster is not a supported session-state configuration in this reviewed client path. +### Add a separate cache only for a measured need -A provider may expose a stable ordinary Redis primary endpoint backed by its own failover system. That is a provider-specific deployment choice, not native Sentinel integration. Validate that endpoint's primary switching, connection recovery, state locks, and persistence with your exact Reflex and Redis client versions before relying on it. A custom Sentinel adapter needs its own implementation and integration tests before it can be documented as supported. +For example, cache an expensive, repeatable lookup when bounded staleness is acceptable. Use a separate Redis service so cache churn, eviction and refill bursts do not compete with Reflex session state. Key prefixes or logical database numbers on one server do not isolate resources or failures. -Sentinel's Redis replication is asynchronous, so failover can lose recent acknowledged writes. Define recovery behavior and capacity on the remaining nodes. Persistent business records and jobs need their own durability guarantees. See [Redis Sentinel requirements](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/) and [Redis Cluster specification](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/). +`REFLEX_REDIS_URL` configures Reflex state. Application caching uses your own client/configuration, with bounded TTLs, reused clients and limits on cache-miss database work. Keep cache clients outside serialized session state. No cache is required by this architecture. -### Separate caching from session state +### Sentinel is an optional integration, not a Reflex setting -Use separate Redis services for application caching and Reflex session state. Reflex reads `REFLEX_REDIS_URL` for state. Application cache code can read a separate variable such as `APP_CACHE_REDIS_URL`; that second name is an application convention, not another built-in Reflex cache setting. +**Reflex's built-in Redis configuration does not support connecting directly through Sentinel.** In the reviewed source, the factory creates ordinary Redis clients from `redis://`, `rediss://` or `unix://` URLs. It does not discover a primary through Sentinel or construct a Redis Cluster client. Do not point `REFLEX_REDIS_URL` at a Sentinel control port or invent a Sentinel URL scheme. Redis Cluster is not supported by this session-state client path. -Different key prefixes or logical database numbers on one server do not isolate resource usage or failure. +The diagram's optional Sentinel panel illustrates a deployment you must implement and validate separately. It needs a Sentinel-aware proxy/endpoint or custom integration that connects application traffic to the current Redis primary. A provider's stable ordinary Redis endpoint may handle failover internally; that does not make Reflex Sentinel-aware. Test primary switching, connection recovery, state locks and persistence with your exact versions before relying on either approach. -```bash -# Illustrative ordinary regional endpoints, not Sentinel setup. -REFLEX_REDIS_URL=rediss://state.eu.example:6379/0 -APP_CACHE_REDIS_URL=rediss://cache.eu.example:6379/0 -``` +Sentinel monitors non-clustered Redis and promotes replicas; it does not shard data. The illustration uses three Sentinel instances in independent failure domains and an example failure-detection quorum of two. Failover also requires majority authorization. Redis replication is asynchronous, so failover can lose recent acknowledged writes. Durable business records and jobs need their own recovery guarantees. See [Redis Sentinel requirements](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/). -Supply real endpoints and credentials through deployment configuration. Use bounded cache TTLs, reuse clients, and prevent cache misses from causing unlimited database work. Keep cache clients outside serialized session state. +## Keep builds and state versions compatible -## Route database queries explicitly +Use `uv` or another locking package manager to freeze dependencies across builds. Replicas sharing a state store must run compatible application code and state schemas. -For a globally distributed app, consider a read-only database replica in each region that hosts Reflex backends. Eligible reads can stay local, reducing cross-region latency and read load on the primary. Make routing an explicit policy in the data-access layer that covers every query. +- Compatible framework upgrades can preserve saved state; plan for session reset on downgrade/rollback and read the release notes for migration instructions. +- Changing stored field names/types makes saved state incompatible. Changing defaults or method bodies does not change the current saved-state schema. +- Rebuild and deploy the frontend alongside changes to fields, computed-variable definitions or handler names. Frontend compatibility checks and saved-state compatibility checks are distinct. -- **Primary:** route writes, migrations, locking reads, and read-write transactions to the current primary. Keep the operations in a transaction on its owning connection. -- **Regional replicas:** route read-only queries that tolerate replication lag to a healthy nearby replica. A `SELECT` that locks rows or participates in a write transaction is not an eligible replica read. -- **Consistency exceptions:** use the primary for read-after-write and freshness-critical reads unless the replica is verified to have caught up. For example, read a newly created job from the primary when immediately confirming submission. -- **Failures:** monitor lag and health. Use another healthy replica or a bounded fallback to the primary, with capacity protection. Never send writes to a replica outside a controlled promotion or failover process. +**0.9 → 0.10 migration:** Reflex 0.9 and 0.10 cannot share a state store. 0.10 can load compatible 0.9 state; 0.9 discards 0.10 state. Do not roll mixed versions against the same Redis store or disk state directory. Upgrade all instances together, or use separate stores and accept session reset at cutover. Use a fresh dedicated state store for rollback; keep durable records outside session state. See [self-hosting compatibility guidance](/docs/hosting/self-hosting/). -Test delayed replication, replica unavailability, and primary failover. Confirm that routing preserves transaction consistency and does not turn a regional outage into unlimited load on the primary. See [PostgreSQL replication tradeoffs](https://www.postgresql.org/docs/current/high-availability.html). +## Use telemetry to choose the next change -## Use OpenTelemetry to locate bottlenecks +Enable the optional `reflex-otel` instrumentor with a compatible SDK/exporter and verify data reaches the collector. Trace points and metrics remain inactive until configured. Start with the question you need to answer: -Use Reflex's built-in OpenTelemetry trace points and metrics for a compatible release. They remain inactive until the optional `reflex-otel` package is enabled and an SDK/exporter is configured. The reviewed `reflex-otel` source requires Python >=3.11 and `reflex-base >=0.10.0`; it is not a universal add-on for 0.9 deployments. Pin the framework, instrumentor, OpenTelemetry SDK, and exporter together using the version table above, and verify that telemetry reaches the collector. +| Question | Measure and act | +| --- | --- | +| Is one handler slow, or is it waiting for state? | Compare event spans/`reflex.event.duration` with `reflex.state.acquire.duration`; shorten the handler or its lock-held work. | +| Are connections or payloads driving load? | Correlate `reflex.websocket.connections` and `reflex.websocket.message.size` with latency and replica count; reduce unnecessary events/payloads. | +| Is the report waiting or computing? | Measure queue wait and worker execution separately; increase worker capacity only when that is the bottleneck. | -- **Event latency:** use event-handler spans and `reflex.event.duration` to identify slow handlers and failures. Use `reflex.state.acquire.duration` to distinguish state-acquisition waits from the rest of event execution. -- **Connection and payload load:** correlate `reflex.websocket.connections` and `reflex.websocket.message.size` with instance count and event latency. The instrumentor also provides ASGI request and connection tracing. -- **Browser visibility:** optional browser instrumentation connects frontend interactions to backend event traces. +Use service, region, deployment version and database role to locate the slow boundary. Database queries, queue publication and external workers need their own instrumentation; propagate trace context through the broker. Reflex does not automatically create those external spans. Verify one event-to-worker-to-database trace, set sampling/redaction rules, and keep sensitive payloads and high-cardinality job IDs out of metric labels. See [Reflex OpenTelemetry](/docs/api-reference/observability/) and [context propagation](https://opentelemetry.io/docs/languages/python/propagation/). -See the [Reflex OpenTelemetry guide](/docs/api-reference/observability/) for the instrumentation scope and setup. +### Pin the telemetry combination -Database queries, queue publication, and external Taskiq/Celery workers need their own compatible instrumentation or application spans. Propagate trace context through the broker and restore it in the worker so enqueueing, execution, and database work can be correlated. Measure queue wait separately from execution; Reflex’s instrumentor does not automatically supply these external spans. +| Instrumentor release | Declared dependency floors | +| --- | --- | +| `reflex-otel==0.1.0` | Python >=3.10; `reflex-base >=0.9.11`. | +| `reflex-otel==0.2.0` and the reviewed source | Python >=3.11; `reflex-base >=0.10.0`. Do not install 0.2.0 against 0.9. | -Identify service, region, deployment version, and database role in telemetry. Follow a slow event or job to determine whether time is spent acquiring state, crossing regions, querying a database, or waiting for worker capacity. Verify a representative end-to-end trace, choose sampling and redaction rules, and avoid sensitive payloads and high-cardinality job IDs in metric labels. See [OpenTelemetry context propagation](https://opentelemetry.io/docs/languages/python/propagation/). +Both releases declare `opentelemetry-api >=1.30.0,<2.0` and instrumentation/ASGI instrumentation >=0.49b0,<1.0. These are dependency constraints, not certification of arbitrary SDK/exporter combinations. Lock the exact framework, instrumentor, SDK and exporter; verify backend and optional browser export together. -## Validate scale down as well as scale out +Source checks use [Reflex d02ba540](https://github.com/reflex-dev/reflex/tree/d02ba5409579e24e67401f9dd6eb2fc4d3ef5e1b), the [0.1.0 manifest](https://github.com/reflex-dev/reflex/blob/3d47357ac158c51a0c131ec48c41232830bd9031/packages/reflex-otel/pyproject.toml), and the [0.2.0 manifest](https://github.com/reflex-dev/reflex/blob/40ffab8124ba257053c395309c87f1c818b4e0a2/packages/reflex-otel/pyproject.toml). Recheck another release at its exact tag/commit, and record resolved Python, Reflex, reflex-base, reflex-otel, Redis client/server, broker and result-store versions with test results. -Load-test realistic sessions and events in production mode. Record the Reflex version, instance size, process and replica counts, event rate, payload sizes, p50/p95 latency, error rate, and backend, database, and Redis resources. Measure queue delay and job completion separately from interactive latency. +## Validate recovery under load -- Add replicas under load and verify latency and errors stay within target. Watch for a downstream database or Redis bottleneck. -- Remove the backend that accepted a long job. Reconnect through another replica and confirm the job continues and its result remains accessible. -- Restart a worker mid-job. Verify the configured recovery behavior and that retries do not duplicate effects. Test worker drain separately from app scale-down. -- Repeat a submission and interrupt enqueueing. Verify idempotency and that accepted work is not silently lost between the job store and broker. -- Fail over the tested regional Redis endpoint or explicitly validated custom Sentinel integration. Measure recovery and exercise active sessions, locks, and reconnects; an unverified adapter is not a supported failover setup. -- Move a user to another region. Verify session-reset behavior, authorization, and retrieval of persistent records and jobs. -- Delay or disable a database replica. Verify freshness-sensitive reads, transaction routing, and bounded primary fallback. -- Follow an event-to-worker-to-database trace and check that missing telemetry does not hide a bottleneck. +When scaling down, stop routing new connections to retiring backends, drain bounded work, and test reconnects to another healthy replica. Allow time for worker drain and recovery separately. There is no universal sessions-per-instance figure. -Scale backends using resource pressure, event latency, connection load, and errors. Scale workers using queue backlog and oldest-job age. Allow enough time for the chosen drain and recovery policies. There is no universal sessions-per-instance figure. +Here are example validation tasks for the components you choose to deploy. Run realistic sessions in production mode and record p50/p95 latency, errors, resource use, event rate, payload sizes, instance/process counts, queue delay and job completion: -## Diagnose Redis spikes before resizing +- Add backends under load; check whether the database or Redis becomes the bottleneck. +- Remove the backend that accepted a report; reconnect elsewhere and retrieve the job/result. +- Restart a worker mid-job; verify redelivery/retry behavior and absence of duplicate side effects. +- Repeat a submission and interrupt queue publication; verify idempotency and recovery of pending outbox entries. +- Fail over the tested Redis endpoint or validated Sentinel integration; exercise locks, active sessions and reconnects. An unvalidated integration is not a supported failover setup. +- Move a user between regions; verify session reset, authorization and recovery of durable jobs. +- Delay or disable a read replica; verify primary routing for fresh reads and bounded fallback. -Record the affected region and endpoint, time window, deployment changes, active sessions, and event rate. Compare state and cache services separately. Start with provider metrics and bounded read-only diagnostics using approved operator access: - -```text -INFO memory -INFO stats -INFO clients -INFO commandstats -INFO keyspace -SLOWLOG GET 20 -``` +### Turn Redis spikes into an action -- **Memory growth:** compare key counts, state sizes, expirations, and evictions. Look for full datasets copied into each session, unbounded histories, and cache entries without useful TTLs. -- **CPU and commands:** correlate event frequency, polling, serialization, and refill bursts. Reduce unnecessary work before adding instances that repeat it. -- **Connections:** check replica and process counts, pool sizes, client reuse, and retry or reconnect bursts. More app instances also mean more connections. -- **Latency and locks:** inspect slow I/O and lock-held time. Move long work to the external worker path; increasing a timeout does not add throughput. +Compare session Redis and cache separately over the affected time window. Use provider metrics or bounded diagnostics such as `INFO memory`, `INFO clients`, `INFO commandstats` and `SLOWLOG GET 20` through authorized operator access. -`SLOWLOG` measures server execution, excludes network latency, and can expose private arguments. Redact before sharing. Alert on state eviction and memory pressure. A `noeviction` policy prevents eviction but rejects new writes at the limit. +| Signal | Next action | +| --- | --- | +| Memory, key count or evictions growing | Reduce per-session datasets/history; bound cache TTLs and monitor state eviction. | +| CPU/commands rise with event or polling rate | Reduce duplicate events/polling and limit cache refill concurrency before adding more clients. | +| Connections rise after scaling | Check process counts, pool limits, client reuse and reconnect backoff. | +| Latency tracks long-held state locks | Move slow work to the background/worker path; increasing timeouts does not add throughput. | -References: [Redis INFO](https://redis.io/docs/latest/commands/info/), [slow log](https://redis.io/docs/latest/commands/slowlog-get/), and [eviction policies](https://redis.io/docs/latest/develop/reference/eviction/). +`SLOWLOG` excludes network latency and may expose private arguments; redact before sharing. A `noeviction` policy rejects new writes at the memory limit. See [Redis diagnostics](https://redis.io/docs/latest/commands/info/) and [eviction policies](https://redis.io/docs/latest/develop/reference/eviction/). From c553a8d1f703f46a97ec9b7484f14213f9db5ef8 Mon Sep 17 00:00:00 2001 From: Kastier1 <40179067+Kastier1@users.noreply.github.com.> Date: Mon, 19 Oct 2026 12:08:14 -0700 Subject: [PATCH 3/5] ENG-12238: address scaling guide review and test job recovery --- .../assets/hosting/scaling-architecture.svg | 135 ++++----- docs/app/news/+scaling-guide.docs.md | 2 +- docs/app/tests/test_routes.py | 53 +++- docs/app/tests/test_scaling_job_example.py | 275 ++++++++++++++++++ docs/hosting/scaling.md | 116 ++++++-- 5 files changed, 476 insertions(+), 105 deletions(-) create mode 100644 docs/app/tests/test_scaling_job_example.py diff --git a/docs/app/assets/hosting/scaling-architecture.svg b/docs/app/assets/hosting/scaling-architecture.svg index 019b94b076b..ddd936108df 100644 --- a/docs/app/assets/hosting/scaling-architecture.svg +++ b/docs/app/assets/hosting/scaling-architecture.svg @@ -1,6 +1,6 @@ - + Reference architecture for a scaled Reflex application - A browser loads a static frontend and exchanges events and updates through a WebSocket-capable load balancer with replaceable backend replicas. Backends use a primary database for writes and fresh reads, lagging read replicas only for eligible reads, regional Redis for session state, and an optional separate application cache. Choose one state Redis connection path: a direct ordinary Redis URL, or an optional independently validated Sentinel-aware endpoint or adapter. That integration discovers the primary through Sentinel control traffic and sends application data to Redis. Three example Sentinel instances monitor Redis with a quorum of two; a majority authorizes failover. Native Sentinel discovery is not verified in Reflex. A reliable job handoff publishes to a task broker. Independent workers persist outcomes in shared durable storage before acknowledging success. Queue persistence, recovery, result storage, and idempotency must be configured and tested. + A browser loads a static frontend and exchanges events and updates through a WebSocket-capable load balancer with replaceable backend replicas. Backends use a primary database for writes and fresh reads, lagging read replicas only for eligible reads, regional Redis for session state, and an optional separate application cache. An ordinary Redis URL connects backends through a provider-managed failover endpoint to the regional Redis current primary. Redis replication to a replica is asynchronous. Endpoint switching, connection recovery and persistence depend on the selected provider and configuration; the endpoint alone does not prevent data loss. A reliable job handoff publishes to a task broker. Independent workers persist outcomes in shared durable storage before acknowledging success. Queue persistence, recovery, result storage, and idempotency must be configured and tested. - - + A production app, split by responsibility Illustrative topology. Add services only when the application needs them. Request / response - - Async job - - Replication - - Control plane + + Async job + + Replication Browser @@ -61,44 +57,36 @@ Route connections - - - - + + + - Writes / fresh reads - Lag-tolerant reads - Choose one Redis path - Direct - ordinary - Redis URL - - Optional, validated - Sentinel-aware - endpoint / adapter - Separate integration - - State / - locks - - - Discover current primary + Writes / fresh reads + Lag-tolerant reads + Ordinary Redis URL + + Provider-managed + failover endpoint + Provider-specific behavior + + State / + locks - - Primary database - Business records - Jobs + result references - - Read replica DBs - Read-only routing - May lag behind primary - - Asynchronous DB replication + + Primary database + Business records + Jobs + result references + + Read replica DBs + Read-only routing + May lag behind primary + + Asynchronous DB replication Regional state Redis - Primary data endpoint + Current primary Shared by this region State Redis replica @@ -106,43 +94,28 @@ Async replication - - - Optional cache - Separate Redis service - Cache get / set + TTLs + + + Optional cache + Separate Redis service + Cache get / set + TTLs - - Optional Sentinel control plane - - - - Sentinel 1 - Sentinel 2 - Sentinel 3 - Example quorum: 2 of 3; majority authorizes failover. - Requires validated Sentinel-aware integration. - No verified native Reflex Sentinel discovery. - - - Monitor / coordinate failover - - - Publish after reliable handoff (see job lifecycle) - - Task queue / broker - Configure persistence - and delivery / recovery - - Independent workers - Idempotent task effects - Shared inputs + result storage - - Deliver / redeliver - - Ack after persistence - - Persist outcomes in shared durable storage - Storage persistence, replication lag, retries and failover need explicit policies. - Validate recovery for the selected versions; this topology is not a guarantee. + + Publish after reliable handoff (see job lifecycle) + + Task queue / broker + Configure persistence + and delivery / recovery + + Independent workers + Idempotent task effects + Shared inputs + result storage + + Deliver / redeliver + + Ack after persistence + + Persist outcomes in shared durable storage + Storage persistence, replication lag and job recovery depend on configuration. + A provider endpoint alone does not guarantee recovery or prevent data loss. diff --git a/docs/app/news/+scaling-guide.docs.md b/docs/app/news/+scaling-guide.docs.md index c43cc954bea..93176c2de7d 100644 --- a/docs/app/news/+scaling-guide.docs.md +++ b/docs/app/news/+scaling-guide.docs.md @@ -1 +1 @@ -Add a scaling guide with production architecture and job lifecycle diagrams, covering horizontal backends, state-store version compatibility, durable jobs, regional data routing, telemetry, and Redis recovery checks. +Add a scaling guide with production architecture and job lifecycle diagrams and a Reflex report-job example, covering horizontal backends, state-store version compatibility, durable jobs, regional data routing, telemetry, and Redis recovery checks. diff --git a/docs/app/tests/test_routes.py b/docs/app/tests/test_routes.py index 195e4d1b1d8..f0e46936b4d 100644 --- a/docs/app/tests/test_routes.py +++ b/docs/app/tests/test_routes.py @@ -5,9 +5,12 @@ import sys from collections import Counter from pathlib import Path +from urllib.parse import urlsplit import pytest import reflex as rx +from reflex_base.components.component import BaseComponent +from reflex_site_shared.docs.markdown import render_docgen_document from reflex_docs.pages.docs.metadata import docs_metadata @@ -318,6 +321,46 @@ def test_python_block_links_do_not_repeat_docs_basename(): ) +@pytest.mark.parametrize("frontend_path", ["", "/docs"]) +@pytest.mark.parametrize( + "asset_name", ["scaling-architecture.svg", "scaling-job-lifecycle.svg"] +) +def test_scaling_full_size_links_match_the_displayed_assets( + monkeypatch, frontend_path, asset_name +): + """Full-size diagram links must not add the router basename a second time.""" + docs_app = Path(__file__).resolve().parent.parent + monkeypatch.chdir(docs_app) + config = rx.Config(app_name="asset_links", frontend_path=frontend_path) + monkeypatch.setattr("reflex.assets.get_config", lambda: config) + body, _ = render_docgen_document( + "docs/hosting/scaling.md", docs_app.parent / "hosting/scaling.md" + ) + pending: list[BaseComponent] = [body] + images = [] + links = [] + while pending: + component = pending.pop() + pending.extend(component.children) + if component.tag == "img" and asset_name in str(getattr(component, "src", "")): + images.append(component) + elif component.tag in {"a", "Link"} and asset_name in str(component): + links.append(component) + + assert len(images) == len(links) == 1 + image_url = str(getattr(images[0], "src", "")).strip('"') + link = links[0] + if link.tag == "Link": + destination = str(link.to).strip('"') + assert urlsplit(destination).path == f"/hosting/{asset_name}", ( + "React Router adds frontend_path to the destination; " + f"an already-prefixed asset URL would repeat it: {destination}" + ) + else: + assert urlsplit(str(link.href).strip('"')).path == urlsplit(image_url).path + assert urlsplit(image_url).path == f"{frontend_path}/hosting/{asset_name}" + + # Router links that survive the basename check still have to name a real page. # A path reached through `href=`, `to=` or `rx.redirect(...)` is navigation and is # always checked. Bare path literals are checked only when their first segment is a @@ -328,19 +371,25 @@ def test_python_block_links_do_not_repeat_docs_basename(): def test_python_block_links_resolve_to_real_routes(routes_fixture): - """Router links inside ```python blocks must name a registered docs route. + """Python block links must name a registered docs route or an existing asset. `test_doc_links.py` validates prose links off the markdown AST, which ignores fenced code blocks by design — so links written in ```python blocks are checked here instead. """ known = {route.path.rstrip("/") for route in routes_fixture if route.path} + asset_root = Path(__file__).resolve().parent.parent / "assets" + asset_paths = { + "/" + asset.relative_to(asset_root).as_posix() + for asset in asset_root.rglob("*") + if asset.is_file() + } sections = {path.split("/")[1] for path in known if path.count("/") > 1} def is_broken(link: str, navigation: bool) -> bool: path = link.split("#")[0].rstrip("/") segments = path.split("/") - if len(segments) < 3 or path in known: + if len(segments) < 3 or path in known or path in asset_paths: return False # A misspelled section (/enterprize/...) still has to fail, so navigation # is validated whatever its first segment says. diff --git a/docs/app/tests/test_scaling_job_example.py b/docs/app/tests/test_scaling_job_example.py new file mode 100644 index 00000000000..d0b082ec001 --- /dev/null +++ b/docs/app/tests/test_scaling_job_example.py @@ -0,0 +1,275 @@ +"""Exercise the published handlers; fake services do not establish durability.""" + +import asyncio +import re +import sys +from contextlib import asynccontextmanager +from pathlib import Path +from types import ModuleType, SimpleNamespace +from unittest.mock import AsyncMock +from uuid import uuid4 + +import pytest +import reflex as rx +from reflex.compiler.compiler import compile_page +from reflex.istate.manager.memory import StateManagerMemory +from reflex.istate.manager.token import BaseStateToken +from reflex.istate.proxy import StateProxy +from reflex_base.event.context import EventContext +from reflex_base.registry import RegistrationContext + +GUIDE = Path(__file__).resolve().parents[2] / "hosting/scaling.md" + + +class ReportServiceError(Exception): + """The adapter could not confirm the requested operation.""" + + +class ReportService: + """A deterministic adapter double, never a production job store.""" + + def __init__(self): + self.jobs = {} + self.outbox = [] + self.calls = [] + self.fail_accept = False + self.lose_response = False + self.pause = "" + self.started = asyncio.Event() + self.release = asyncio.Event() + + async def boundary(self, operation): + """Assert service I/O happens after releasing the real state lock.""" + manager = EventContext.get().state_manager + assert isinstance(manager, StateManagerMemory) + assert not any(lock.locked() for lock in manager._states_locks.values()) + if self.pause == operation: + self.started.set() + await self.release.wait() + + async def accept_report(self, owner, request_key, report): + """Emulate the documented owner-scoped acceptance contract.""" + await self.boundary("accept") + self.calls.append(("accept", owner, request_key, report)) + if self.fail_accept: + raise ReportServiceError + key = (owner, request_key) + if key in self.jobs: + if self.jobs[key].report != report: + raise ReportServiceError + else: + self.jobs[key] = SimpleNamespace( + id=uuid4().hex, request_key=request_key, report=report, status="queued" + ) + self.outbox.append(self.jobs[key].id) + if self.lose_response: + raise ReportServiceError + return self.jobs[key] + + async def read_report(self, owner, job_id, request_key=""): + """Return only the owner's selected or latest job.""" + await self.boundary("read") + self.calls.append(("read", owner, job_id, request_key)) + owned = [job for (who, _), job in self.jobs.items() if who == owner] + if job_id: + return next((job for job in owned if job.id == job_id), None) + if request_key: + return self.jobs.get((owner, request_key)) + return owned[-1] if owned else None + + +@pytest.fixture +def example(monkeypatch): + """Import exactly the code a reader copies, with app-owned adapters.""" + source = GUIDE.read_text() + blocks = re.findall(r"^```python\n(.*?)^```", source, re.MULTILINE | re.DOTALL) + matches = [block for block in blocks if "class ReportState(rx.State):" in block] + assert len(matches) == 1 + with RegistrationContext(_config=rx.Config(app_name="scaling_example_test")): + auth = ModuleType("report_app.auth") + monkeypatch.setitem(sys.modules, auth.__name__, auth) + exec( + "import reflex as rx\n" + "class AuthState(rx.State):\n" + " _owner_id: str = ''\n" + " _session_revision: int = 0\n", + auth.__dict__, + ) + service = ReportService() + adapter = ModuleType("report_app.reports") + adapter.__dict__.update( + accept_report=service.accept_report, + read_report=service.read_report, + ReportServiceError=ReportServiceError, + ) + monkeypatch.setitem(sys.modules, adapter.__name__, adapter) + module = ModuleType(f"scaling_report_example_{uuid4().hex}") + monkeypatch.setitem(sys.modules, module.__name__, module) + exec(matches[0], module.__dict__) + assert issubclass(module.ReportState, rx.State) + assert module.ReportState.event_handlers["sync_report"].is_background + yield SimpleNamespace(module=module, service=service, auth=auth.AuthState) + + +def test_report_panel_compiles_frontend_event_and_state_bindings(example): + """Compile the copied UI itself, including real event arguments and Vars.""" + _, code = compile_page("report-example", example.module.report_panel()) + assert "export default" in code + assert "sync_report" in code and "new_report" in code + assert "job_id" in code and "status" in code + + +@asynccontextmanager +async def session(example, owner="alice"): + """Use real Reflex state, StateProxy and state-manager lock semantics.""" + manager = StateManagerMemory() + token = uuid4().hex + try: + with EventContext(token=token, state_manager=manager, enqueue_impl=AsyncMock()): + cls = example.module.ReportState + root = await manager.get_state(BaseStateToken(ident=token, cls=cls)) + state = root.get_substate(cls.get_full_name().split(".")) + proxy = StateProxy(state) + async with proxy: + auth = await proxy.get_state(example.auth) + auth._owner_id = owner + yield proxy + finally: + await manager.close() + + +@pytest.mark.asyncio +@pytest.mark.parametrize("lost_response", [False, True]) +async def test_retries_reuse_the_request_key_and_committed_job(example, lost_response): + """A repeated submission, including lost acknowledgement, keeps one job.""" + async with session(example) as state: + example.service.lose_response = lost_response + await example.module.ReportState.sync_report.fn(state, True) + request_key = state._request_key + example.service.lose_response = False + await example.module.ReportState.sync_report.fn(state, True) + assert state._request_key == request_key + assert len(example.service.jobs) == len(example.service.outbox) == 1 + assert state.job_id == example.service.outbox[0] + assert state.status == "queued" and not state.error + async with state: + example.module.ReportState.new_report.fn(state) + assert state._request_key != request_key + assert not state.job_id and not state.status + + +@pytest.mark.asyncio +async def test_failed_accept_does_not_claim_an_accepted_job(example): + """A failed acceptance leaves a retryable key without a job ID/status.""" + example.service.fail_accept = True + async with session(example) as state: + await example.module.ReportState.sync_report.fn(state, True) + assert state._request_key and state.error + assert not state.job_id and not state.status + assert not example.service.jobs and not example.service.outbox + + +@pytest.mark.asyncio +async def test_status_read_uses_owner_and_selected_job_id(example): + """Possessing another owner's job ID does not return its status.""" + async with session(example) as alice: + await example.module.ReportState.sync_report.fn(alice, True) + job_id = alice.job_id + await example.module.ReportState.sync_report.fn(alice) + assert example.service.calls[-1] == ( + "read", + "alice", + job_id, + alice._request_key, + ) + async with session(example, "bob") as bob: + async with bob: + bob.job_id = job_id + await example.module.ReportState.sync_report.fn(bob) + assert example.service.calls[-1] == ("read", "bob", job_id, "") + assert not bob.job_id and not bob.status + + +@pytest.mark.asyncio +@pytest.mark.parametrize("operation", ["submit", "read"]) +@pytest.mark.parametrize("change", ["logout", "same_owner_relogin", "selection"]) +async def test_stale_service_response_does_not_update_the_ui( + example, operation, change +): + """Responses cannot overwrite a changed authentication or selection.""" + async with session(example) as state: + if operation == "read": + await example.module.ReportState.sync_report.fn(state, True) + example.service.pause = "accept" if operation == "submit" else "read" + task = asyncio.create_task( + example.module.ReportState.sync_report.fn(state, operation == "submit") + ) + await asyncio.wait_for(example.service.started.wait(), 2) + async with state: + if change == "selection": + example.module.ReportState.new_report.fn(state) + else: + auth = await state.get_state(example.auth) + auth._session_revision += 1 + auth._owner_id = "" if change == "logout" else "alice" + state.job_id = state.status = state.error = "" + example.service.release.set() + await asyncio.wait_for(task, 2) + assert not state.job_id and not state.status and not state.error + + +@pytest.mark.asyncio +async def test_fresh_state_recovers_job_and_retry_key(example): + """A new session uses an authorized durable lookup, not old UI state.""" + async with session(example) as original: + await example.module.ReportState.sync_report.fn(original, True) + job_id, request_key = original.job_id, original._request_key + async with session(example) as fresh: + assert not fresh.job_id and not fresh._request_key + await example.module.ReportState.sync_report.fn(fresh) + assert example.service.calls[-1] == ("read", "alice", "", "") + assert fresh.job_id == job_id and fresh._request_key == request_key + await example.module.ReportState.sync_report.fn(fresh, True) + assert len(example.service.jobs) == len(example.service.outbox) == 1 + + +@pytest.mark.asyncio +@pytest.mark.parametrize("pending_submit", [False, True]) +async def test_refresh_does_not_replace_a_new_request_with_an_older_job( + example, pending_submit +): + """Refresh keeps a new request's key even before its acceptance returns.""" + async with session(example) as state: + await example.module.ReportState.sync_report.fn(state, True) + old_job = state.job_id + async with state: + example.module.ReportState.new_report.fn(state) + new_key = state._request_key + task = None + if pending_submit: + example.service.pause = "accept" + task = asyncio.create_task( + example.module.ReportState.sync_report.fn(state, True) + ) + await asyncio.wait_for(example.service.started.wait(), 2) + await example.module.ReportState.sync_report.fn(state) + assert state._request_key == new_key + assert not state.job_id and not state.status + if task is not None: + example.service.release.set() + await asyncio.wait_for(task, 2) + assert not state.job_id + example.service.pause = "" + await example.module.ReportState.sync_report.fn(state, True) + assert state._request_key == new_key and state.job_id != old_job + assert len(example.service.jobs) == len(example.service.outbox) == 2 + + +@pytest.mark.asyncio +async def test_signed_out_state_does_not_call_job_services(example): + """An absent trusted identity prevents both reads and submissions.""" + async with session(example, "") as state: + await example.module.ReportState.sync_report.fn(state, True) + await example.module.ReportState.sync_report.fn(state) + assert not example.service.calls + assert not state.job_id and not state.status diff --git a/docs/hosting/scaling.md b/docs/hosting/scaling.md index 450996e27e2..9005bd54a0f 100644 --- a/docs/hosting/scaling.md +++ b/docs/hosting/scaling.md @@ -1,6 +1,6 @@ --- title: Scaling Reflex Apps -meta_description: Horizontal scaling for Reflex apps, durable jobs, regional state, Redis Sentinel support limits, database read routing, OpenTelemetry compatibility, Redis spikes, and scale-down recovery. +meta_description: Horizontal scaling for Reflex apps, durable jobs, regional state, Redis failover, database read routing, OpenTelemetry compatibility, Redis spikes, and scale-down recovery. --- # Scaling Reflex Apps @@ -11,7 +11,7 @@ This guide uses a report-export application to explain the boundaries: a user re ## Reference production architecture -This is an illustrative deployment, not a list of required components. Start with the request path and stores your app needs. Add read replicas for lag-tolerant read load, a separate cache for expensive repeatable reads, and a queue when jobs must outlive a backend. The optional Sentinel panel requires a separately implemented and validated integration. +This is an illustrative deployment, not a list of required components. Start with the request path and stores your app needs. Add read replicas for lag-tolerant read load, a separate cache for expensive repeatable reads, and a queue when jobs must outlive a backend. ```python exec import reflex as rx @@ -22,7 +22,7 @@ rx.el.figure( rx.el.div( rx.image( src=rx.asset("hosting/scaling-architecture.svg"), - alt="Production architecture: browser and frontend, load balancer, backend replicas, databases, separate Redis roles, optional Sentinel integration, queue and workers. Text description follows.", + alt="Production architecture: browser and frontend, load balancer, backend replicas, databases, separate Redis roles, provider-managed failover endpoint, queue and workers. Text description follows.", width="100%", min_width="760px", height="auto", @@ -38,7 +38,7 @@ rx.el.figure( rx.el.figcaption( rx.el.a( "Open architecture diagram at full size", - href=rx.asset("hosting/scaling-architecture.svg"), + href="/hosting/scaling-architecture.svg", target="_blank", rel="noopener noreferrer", text_decoration="underline", @@ -57,7 +57,7 @@ The labeled arrows distinguish traffic types. On narrow screens, scroll the diag - **Application data:** backend replicas ↔ regional Redis session state; optional application cache uses a separate Redis service. Backends and workers write durable records to the primary database. Only lag-tolerant reads go to read replicas; read-after-write and other freshness-critical reads use the primary. - **Asynchronous jobs:** backend → durable job record/outbox → task queue → independent workers. Workers save status and results in shared storage; any authorized backend can read them. The job lifecycle below explains confirmation, retry and acknowledgement boundaries. - **Replication:** primary database → read replica databases; Redis primary → Redis replica. These arrows copy data, not application requests. Asynchronous replication can lag and can lose recent writes on failover. -- **Sentinel monitoring/failover:** optional Sentinel instances monitor the Redis primary/replica and coordinate promotion. A validated Sentinel-aware integration discovers the current primary; application data still goes to Redis, never to a Sentinel control port. This is not native Reflex Sentinel support. +- **Redis failover:** backends use an ordinary Redis URL through a provider-managed failover endpoint, which routes application traffic to the current primary. Test connection recovery and the provider's persistence behavior during failover. ## Serve requests with replaceable backends @@ -99,7 +99,7 @@ rx.el.figure( rx.el.figcaption( rx.el.a( "Open job lifecycle diagram at full size", - href=rx.asset("hosting/scaling-job-lifecycle.svg"), + href="/hosting/scaling-job-lifecycle.svg", target="_blank", rel="noopener noreferrer", text_decoration="underline", @@ -120,6 +120,87 @@ rx.el.figure( A queue library alone does not guarantee durability. Select broker persistence and worker-loss behavior, then test them. Taskiq needs an actual result backend or application job store; Celery acknowledgement and worker-loss settings affect redelivery. Define result retention, backups and expiry independently of queue delivery. Avoid process-local queues and in-memory-only results for jobs that must survive scale-down. Scale workers by backlog and oldest-job age, independently of backend replicas. +### Connect a report job to Reflex state + +The handler below awaits an application service outside the state lock, then applies its result only if the signed-in session and selected request are still current. `report_app.auth.AuthState` is your own authentication state, not a Reflex API: its backend-only `_owner_id` comes from verified authentication, and `_session_revision` changes on every login/logout, including signing back in as the same owner. Your logout handler also clears the report UI. + +Supply asynchronous `report_app.reports` adapters backed by your database and broker: + +- `accept_report(owner, request_key, report)` validates access and inputs, atomically commits the job and outbox, and returns only after commit. Enforce `UNIQUE(owner, request_key)`: the same key and inputs return the existing job; the same key with different inputs is rejected. +- `read_report(owner, job_id, request_key)` reads from the **primary**, filtering by owner and the selected job ID, or by owner and request key when no ID exists. Only when both are empty does it recover that owner's latest job for this report type. Return `None` for no match, without falling back to an older job for a selected key. Both adapters return a record with `id`, `status` and `request_key`; service failures raise your `ReportServiceError`. + +```python +from uuid import uuid4 + +import reflex as rx +from report_app.auth import AuthState +from report_app.reports import ReportServiceError, accept_report, read_report + + +class ReportState(rx.State): + job_id: str = "" + status: str = "" + error: str = "" + _request_key: str = "" + _generation: int = 0 + + @rx.event + def new_report(self): + self._request_key = uuid4().hex + self._generation += 1 + self.job_id = self.status = self.error = "" + + @rx.event(background=True) + async def sync_report(self, submit: bool = False): + async with self: + auth = await self.get_state(AuthState) + identity = (auth._owner_id, auth._session_revision) + self._generation += 1 + generation = self._generation + if not identity[0]: + self.job_id = self.status = "" + return + if submit and not self._request_key: + self._request_key = uuid4().hex + request_key, job_id = self._request_key, self.job_id + + job, error = None, "" + try: + if submit: + job = await accept_report(identity[0], request_key, "monthly-sales") + else: + job = await read_report(identity[0], job_id, request_key) + except ReportServiceError: + error = "Request not confirmed. Retry or refresh before starting anew." + + async with self: + auth = await self.get_state(AuthState) + if identity != (auth._owner_id, auth._session_revision): + return + if generation != self._generation: + return + self.error = error + if not error: + self.job_id = job.id if job else "" + self.status = job.status if job else "" + if job: + self._request_key = job.request_key + + +def report_panel(): + return rx.vstack( + rx.button("New report", on_click=ReportState.new_report), + rx.button("Submit / retry", on_click=ReportState.sync_report(True)), + rx.button("Refresh / recover", on_click=ReportState.sync_report(False)), + rx.text(ReportState.job_id, ": ", ReportState.status), + rx.text(ReportState.error), + ) +``` + +Retries keep the same request key; **New report** deliberately creates a new logical request and key. After authentication, run `ReportState.sync_report` on page load to recover a job in a fresh session. If a session is lost before acceptance reaches the UI, recover from durable records before deliberately submitting a new request; a key held only in lost session state cannot deduplicate a new key. For multiple outstanding jobs, provide an authorized list/selection view instead of a latest-job lookup. + +These adapters and the worker must implement the lifecycle above; a background event alone does not make the job durable. + ## Keep regional state and durable data separate Keep frequent session-state access close to the backends: each region uses its own Redis session store, and active sessions are routed consistently to that region. Business records, preferences and jobs belong in stores with explicit write ownership, consistency, replication and recovery policies. @@ -145,23 +226,19 @@ For example, cache an expensive, repeatable lookup when bounded staleness is acc `REFLEX_REDIS_URL` configures Reflex state. Application caching uses your own client/configuration, with bounded TTLs, reused clients and limits on cache-miss database work. Keep cache clients outside serialized session state. No cache is required by this architecture. -### Sentinel is an optional integration, not a Reflex setting +### Use a failover endpoint -**Reflex's built-in Redis configuration does not support connecting directly through Sentinel.** In the reviewed source, the factory creates ordinary Redis clients from `redis://`, `rediss://` or `unix://` URLs. It does not discover a primary through Sentinel or construct a Redis Cluster client. Do not point `REFLEX_REDIS_URL` at a Sentinel control port or invent a Sentinel URL scheme. Redis Cluster is not supported by this session-state client path. - -The diagram's optional Sentinel panel illustrates a deployment you must implement and validate separately. It needs a Sentinel-aware proxy/endpoint or custom integration that connects application traffic to the current Redis primary. A provider's stable ordinary Redis endpoint may handle failover internally; that does not make Reflex Sentinel-aware. Test primary switching, connection recovery, state locks and persistence with your exact versions before relying on either approach. - -Sentinel monitors non-clustered Redis and promotes replicas; it does not shard data. The illustration uses three Sentinel instances in independent failure domains and an example failure-detection quorum of two. Failover also requires majority authorization. Redis replication is asynchronous, so failover can lose recent acknowledged writes. Durable business records and jobs need their own recovery guarantees. See [Redis Sentinel requirements](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/). +Configure `REFLEX_REDIS_URL` with an ordinary Redis URL for a provider-managed endpoint that handles failover. ## Keep builds and state versions compatible Use `uv` or another locking package manager to freeze dependencies across builds. Replicas sharing a state store must run compatible application code and state schemas. - Compatible framework upgrades can preserve saved state; plan for session reset on downgrade/rollback and read the release notes for migration instructions. -- Changing stored field names/types makes saved state incompatible. Changing defaults or method bodies does not change the current saved-state schema. -- Rebuild and deploy the frontend alongside changes to fields, computed-variable definitions or handler names. Frontend compatibility checks and saved-state compatibility checks are distinct. +- Changing stored base-variable names/types makes saved state incompatible. Changing defaults or method bodies does not change the saved-state schema hash. +- Rebuild and deploy the frontend alongside changes to fields, frontend computed-variable names/types or handler names. The frontend compatibility hash includes frontend computed variables and handler names; the saved-state schema hash does not. -**0.9 → 0.10 migration:** Reflex 0.9 and 0.10 cannot share a state store. 0.10 can load compatible 0.9 state; 0.9 discards 0.10 state. Do not roll mixed versions against the same Redis store or disk state directory. Upgrade all instances together, or use separate stores and accept session reset at cutover. Use a fresh dedicated state store for rollback; keep durable records outside session state. See [self-hosting compatibility guidance](/docs/hosting/self-hosting/). +See [self-hosting compatibility guidance](/docs/hosting/self-hosting/) for release-specific migration constraints. ## Use telemetry to choose the next change @@ -179,12 +256,9 @@ Use service, region, deployment version and database role to locate the slow bou | Instrumentor release | Declared dependency floors | | --- | --- | -| `reflex-otel==0.1.0` | Python >=3.10; `reflex-base >=0.9.11`. | -| `reflex-otel==0.2.0` and the reviewed source | Python >=3.11; `reflex-base >=0.10.0`. Do not install 0.2.0 against 0.9. | - -Both releases declare `opentelemetry-api >=1.30.0,<2.0` and instrumentation/ASGI instrumentation >=0.49b0,<1.0. These are dependency constraints, not certification of arbitrary SDK/exporter combinations. Lock the exact framework, instrumentor, SDK and exporter; verify backend and optional browser export together. +| `reflex-otel==0.2.0` | Python >=3.11; `reflex-base >=0.10.0`. | -Source checks use [Reflex d02ba540](https://github.com/reflex-dev/reflex/tree/d02ba5409579e24e67401f9dd6eb2fc4d3ef5e1b), the [0.1.0 manifest](https://github.com/reflex-dev/reflex/blob/3d47357ac158c51a0c131ec48c41232830bd9031/packages/reflex-otel/pyproject.toml), and the [0.2.0 manifest](https://github.com/reflex-dev/reflex/blob/40ffab8124ba257053c395309c87f1c818b4e0a2/packages/reflex-otel/pyproject.toml). Recheck another release at its exact tag/commit, and record resolved Python, Reflex, reflex-base, reflex-otel, Redis client/server, broker and result-store versions with test results. +This release declares `opentelemetry-api >=1.30.0,<2.0` and instrumentation/ASGI instrumentation >=0.49b0,<1.0. These are dependency constraints, not certification of arbitrary SDK/exporter combinations. Lock the exact framework, instrumentor, SDK and exporter; verify backend and optional browser export together. ## Validate recovery under load @@ -196,7 +270,7 @@ Here are example validation tasks for the components you choose to deploy. Run r - Remove the backend that accepted a report; reconnect elsewhere and retrieve the job/result. - Restart a worker mid-job; verify redelivery/retry behavior and absence of duplicate side effects. - Repeat a submission and interrupt queue publication; verify idempotency and recovery of pending outbox entries. -- Fail over the tested Redis endpoint or validated Sentinel integration; exercise locks, active sessions and reconnects. An unvalidated integration is not a supported failover setup. +- Fail over the provider-managed Redis endpoint; exercise locks, active sessions and reconnects, and check which writes survive under its persistence policy. - Move a user between regions; verify session reset, authorization and recovery of durable jobs. - Delay or disable a read replica; verify primary routing for fresh reads and bounded fallback. From 401bb91efc7b80637dedc3ad22e269339c2d3e9d Mon Sep 17 00:00:00 2001 From: Kastier1 <40179067+Kastier1@users.noreply.github.com.> Date: Mon, 19 Oct 2026 12:25:33 -0700 Subject: [PATCH 4/5] ENG-12238: reset report selection on authentication changes --- docs/app/tests/test_scaling_job_example.py | 107 +++++++++++++++++++-- docs/hosting/scaling.md | 13 ++- 2 files changed, 106 insertions(+), 14 deletions(-) diff --git a/docs/app/tests/test_scaling_job_example.py b/docs/app/tests/test_scaling_job_example.py index d0b082ec001..ef9d37e8825 100644 --- a/docs/app/tests/test_scaling_job_example.py +++ b/docs/app/tests/test_scaling_job_example.py @@ -4,6 +4,7 @@ import re import sys from contextlib import asynccontextmanager +from functools import wraps from pathlib import Path from types import ModuleType, SimpleNamespace from unittest.mock import AsyncMock @@ -21,6 +22,17 @@ GUIDE = Path(__file__).resolve().parents[2] / "hosting/scaling.md" +def run_async(test): + """Run an async case with core pytest, preserving fixture/parameter discovery.""" + + @wraps(test) + def run(*args, **kwargs): + """Execute one case in its own event loop.""" + return asyncio.run(test(*args, **kwargs)) + + return run + + class ReportServiceError(Exception): """The adapter could not confirm the requested operation.""" @@ -138,7 +150,7 @@ async def session(example, owner="alice"): await manager.close() -@pytest.mark.asyncio +@run_async @pytest.mark.parametrize("lost_response", [False, True]) async def test_retries_reuse_the_request_key_and_committed_job(example, lost_response): """A repeated submission, including lost acknowledgement, keeps one job.""" @@ -153,12 +165,12 @@ async def test_retries_reuse_the_request_key_and_committed_job(example, lost_res assert state.job_id == example.service.outbox[0] assert state.status == "queued" and not state.error async with state: - example.module.ReportState.new_report.fn(state) + await example.module.ReportState.new_report.fn(state) assert state._request_key != request_key assert not state.job_id and not state.status -@pytest.mark.asyncio +@run_async async def test_failed_accept_does_not_claim_an_accepted_job(example): """A failed acceptance leaves a retryable key without a job ID/status.""" example.service.fail_accept = True @@ -169,7 +181,7 @@ async def test_failed_accept_does_not_claim_an_accepted_job(example): assert not example.service.jobs and not example.service.outbox -@pytest.mark.asyncio +@run_async async def test_status_read_uses_owner_and_selected_job_id(example): """Possessing another owner's job ID does not return its status.""" async with session(example) as alice: @@ -183,6 +195,7 @@ async def test_status_read_uses_owner_and_selected_job_id(example): alice._request_key, ) async with session(example, "bob") as bob: + await example.module.ReportState.sync_report.fn(bob) async with bob: bob.job_id = job_id await example.module.ReportState.sync_report.fn(bob) @@ -190,7 +203,7 @@ async def test_status_read_uses_owner_and_selected_job_id(example): assert not bob.job_id and not bob.status -@pytest.mark.asyncio +@run_async @pytest.mark.parametrize("operation", ["submit", "read"]) @pytest.mark.parametrize("change", ["logout", "same_owner_relogin", "selection"]) async def test_stale_service_response_does_not_update_the_ui( @@ -207,7 +220,7 @@ async def test_stale_service_response_does_not_update_the_ui( await asyncio.wait_for(example.service.started.wait(), 2) async with state: if change == "selection": - example.module.ReportState.new_report.fn(state) + await example.module.ReportState.new_report.fn(state) else: auth = await state.get_state(example.auth) auth._session_revision += 1 @@ -218,7 +231,7 @@ async def test_stale_service_response_does_not_update_the_ui( assert not state.job_id and not state.status and not state.error -@pytest.mark.asyncio +@run_async async def test_fresh_state_recovers_job_and_retry_key(example): """A new session uses an authorized durable lookup, not old UI state.""" async with session(example) as original: @@ -233,7 +246,81 @@ async def test_fresh_state_recovers_job_and_retry_key(example): assert len(example.service.jobs) == len(example.service.outbox) == 1 -@pytest.mark.asyncio +@run_async +@pytest.mark.parametrize("next_owner", ["alice", "bob"]) +@pytest.mark.parametrize("clear_ui", [False, True]) +async def test_authentication_change_recovers_without_previous_session_selectors( + example, next_owner, clear_ui +): + """An account switch or relogin cannot reuse the old report selection.""" + async with session(example) as state: + await example.module.ReportState.sync_report.fn(state, True) + async with session(example, next_owner) as other: + await example.module.ReportState.sync_report.fn(other, True) + latest_id, latest_key = other.job_id, other._request_key + async with state: + auth = await state.get_state(example.auth) + auth._owner_id = next_owner + auth._session_revision += 2 # Logout and login between report events. + state.error = "Previous session error" + if clear_ui: + state.job_id = state.status = state.error = "" + example.service.pause = "read" + task = asyncio.create_task(example.module.ReportState.sync_report.fn(state)) + await asyncio.wait_for(example.service.started.wait(), 2) + assert not state.job_id and not state.status and not state.error + assert not state._request_key + example.service.release.set() + await asyncio.wait_for(task, 2) + assert example.service.calls[-1] == ("read", next_owner, "", "") + assert state.job_id == latest_id and state._request_key == latest_key + assert state.status == "queued" and not state.error + await example.module.ReportState.sync_report.fn(state, True) + assert len(example.service.jobs) == len(example.service.outbox) == 2 + + +@run_async +@pytest.mark.parametrize("account_change", [False, True]) +async def test_new_report_binds_identity_before_first_refresh(example, account_change): + """An explicit new selection survives initial or post-login recovery.""" + async with session(example, "bob") as previous: + await example.module.ReportState.sync_report.fn(previous, True) + older_id = previous.job_id + async with session(example, "alice" if account_change else "bob") as state: + if account_change: + await example.module.ReportState.sync_report.fn(state, True) + async with state: + if account_change: + auth = await state.get_state(example.auth) + auth._owner_id = "bob" + auth._session_revision += 2 + await example.module.ReportState.new_report.fn(state) + new_key = state._request_key + await example.module.ReportState.sync_report.fn(state) + assert example.service.calls[-1] == ("read", "bob", "", new_key) + assert state._request_key == new_key and not state.job_id + await example.module.ReportState.sync_report.fn(state, True) + assert state._request_key == new_key and state.job_id != older_id + + +@run_async +async def test_logout_clears_selection_without_calling_job_services(example): + """Signed-out refresh removes private selectors as well as visible fields.""" + async with session(example) as state: + await example.module.ReportState.sync_report.fn(state, True) + calls = len(example.service.calls) + async with state: + auth = await state.get_state(example.auth) + auth._owner_id = "" + auth._session_revision += 1 + state.error = "Previous session error" + await example.module.ReportState.sync_report.fn(state) + assert not state.job_id and not state.status and not state.error + assert not state._request_key + assert len(example.service.calls) == calls + + +@run_async @pytest.mark.parametrize("pending_submit", [False, True]) async def test_refresh_does_not_replace_a_new_request_with_an_older_job( example, pending_submit @@ -243,7 +330,7 @@ async def test_refresh_does_not_replace_a_new_request_with_an_older_job( await example.module.ReportState.sync_report.fn(state, True) old_job = state.job_id async with state: - example.module.ReportState.new_report.fn(state) + await example.module.ReportState.new_report.fn(state) new_key = state._request_key task = None if pending_submit: @@ -265,7 +352,7 @@ async def test_refresh_does_not_replace_a_new_request_with_an_older_job( assert len(example.service.jobs) == len(example.service.outbox) == 2 -@pytest.mark.asyncio +@run_async async def test_signed_out_state_does_not_call_job_services(example): """An absent trusted identity prevents both reads and submissions.""" async with session(example, "") as state: diff --git a/docs/hosting/scaling.md b/docs/hosting/scaling.md index 9005bd54a0f..ed26c1dedf8 100644 --- a/docs/hosting/scaling.md +++ b/docs/hosting/scaling.md @@ -122,7 +122,7 @@ A queue library alone does not guarantee durability. Select broker persistence a ### Connect a report job to Reflex state -The handler below awaits an application service outside the state lock, then applies its result only if the signed-in session and selected request are still current. `report_app.auth.AuthState` is your own authentication state, not a Reflex API: its backend-only `_owner_id` comes from verified authentication, and `_session_revision` changes on every login/logout, including signing back in as the same owner. Your logout handler also clears the report UI. +The handler below awaits an application service outside the state lock, then applies its result only if the signed-in session and selected request are still current. `report_app.auth.AuthState` is your own authentication state, not a Reflex API: its backend-only `_owner_id` comes from verified authentication, and `_session_revision` changes on every login/logout, including signing back in as the same owner. Report selection is bound to that identity and cleared when it changes. Your logout handler also clears the visible report UI immediately. Supply asynchronous `report_app.reports` adapters backed by your database and broker: @@ -142,10 +142,13 @@ class ReportState(rx.State): status: str = "" error: str = "" _request_key: str = "" + _identity: tuple[str, int] = ("", 0) _generation: int = 0 @rx.event - def new_report(self): + async def new_report(self): + auth = await self.get_state(AuthState) + self._identity = (auth._owner_id, auth._session_revision) self._request_key = uuid4().hex self._generation += 1 self.job_id = self.status = self.error = "" @@ -155,10 +158,12 @@ class ReportState(rx.State): async with self: auth = await self.get_state(AuthState) identity = (auth._owner_id, auth._session_revision) + if identity != self._identity or not identity[0]: + self._identity = identity + self.job_id = self.status = self.error = self._request_key = "" self._generation += 1 generation = self._generation if not identity[0]: - self.job_id = self.status = "" return if submit and not self._request_key: self._request_key = uuid4().hex @@ -197,7 +202,7 @@ def report_panel(): ) ``` -Retries keep the same request key; **New report** deliberately creates a new logical request and key. After authentication, run `ReportState.sync_report` on page load to recover a job in a fresh session. If a session is lost before acceptance reaches the UI, recover from durable records before deliberately submitting a new request; a key held only in lost session state cannot deduplicate a new key. For multiple outstanding jobs, provide an authorized list/selection view instead of a latest-job lookup. +Retries within the same authenticated session keep the request key; **New report** deliberately creates a new logical request and key. Run `ReportState.sync_report` on page load and after authentication changes to recover from durable records before deliberately submitting a new request. A key held only in lost session state cannot deduplicate a new key. For multiple outstanding jobs, provide an authorized list/selection view instead of a latest-job lookup. These adapters and the worker must implement the lifecycle above; a background event alone does not make the job durable. From 718849767258323166e9a2edc68d233e04504d60 Mon Sep 17 00:00:00 2001 From: Kastier1 <40179067+Kastier1@users.noreply.github.com.> Date: Mon, 19 Oct 2026 12:46:01 -0700 Subject: [PATCH 5/5] ENG-12238: require explicit report requests and isolate docs tests --- docs/app/tests/test_scaling_job_example.py | 110 ++++++++++++++++++--- docs/hosting/scaling.md | 8 +- 2 files changed, 103 insertions(+), 15 deletions(-) diff --git a/docs/app/tests/test_scaling_job_example.py b/docs/app/tests/test_scaling_job_example.py index ef9d37e8825..f2d59ddf4b6 100644 --- a/docs/app/tests/test_scaling_job_example.py +++ b/docs/app/tests/test_scaling_job_example.py @@ -16,6 +16,7 @@ from reflex.istate.manager.memory import StateManagerMemory from reflex.istate.manager.token import BaseStateToken from reflex.istate.proxy import StateProxy +from reflex.istate.shared import SharedState, SharedStateBaseInternal from reflex_base.event.context import EventContext from reflex_base.registry import RegistrationContext @@ -97,7 +98,10 @@ def example(monkeypatch): blocks = re.findall(r"^```python\n(.*?)^```", source, re.MULTILINE | re.DOTALL) matches = [block for block in blocks if "class ReportState(rx.State):" in block] assert len(matches) == 1 - with RegistrationContext(_config=rx.Config(app_name="scaling_example_test")): + with RegistrationContext.get().fork() as context: + context._set_config(rx.Config(app_name="scaling_example_test")) + if SharedStateBaseInternal.get_full_name() not in context.base_states: + RegistrationContext.register_base_state(SharedStateBaseInternal) auth = ModuleType("report_app.auth") monkeypatch.setitem(sys.modules, auth.__name__, auth) exec( @@ -150,13 +154,95 @@ async def session(example, owner="alice"): await manager.close() +async def start_report(example, state): + """Follow the explicit New report, then Submit UI flow.""" + async with state: + await example.module.ReportState.new_report.fn(state) + await example.module.ReportState.sync_report.fn(state, True) + + +@run_async +async def test_session_survives_shared_state_in_another_context(example): + """A prior shared-state demo must not break this isolated app's state locks.""" + with RegistrationContext.get().fork(): + + class PriorSharedState(SharedState): + """Model a shared-state demo evaluated in a separate app context.""" + + value: int = 0 + + async with session(example) as state: + await start_report(example, state) + assert state.job_id and state.status == "queued" + + +@run_async +@pytest.mark.parametrize("next_owner", ["alice", "bob"]) +@pytest.mark.parametrize("lost_response", [False, True]) +async def test_direct_submit_after_login_recovers_without_accepting_a_new_job( + example, next_owner, lost_response +): + """Submit after an authentication change recovers even a lost acceptance.""" + async with session(example) as state: + example.service.lose_response = lost_response + await start_report(example, state) + expected = example.service.jobs[("alice", state._request_key)] + if next_owner == "bob": + async with session(example, "bob") as other: + await start_report(example, other) + expected = example.service.jobs[("bob", other._request_key)] + example.service.lose_response = False + before = len(example.service.outbox) + async with state: + auth = await state.get_state(example.auth) + auth._owner_id = next_owner + auth._session_revision += 2 + state.job_id = state.status = state.error = "" + await example.module.ReportState.sync_report.fn(state, True) + assert example.service.calls[-1] == ("read", next_owner, "", "") + assert ( + state.job_id == expected.id and state._request_key == expected.request_key + ) + assert len(example.service.jobs) == len(example.service.outbox) == before + await example.module.ReportState.sync_report.fn(state, True) + assert len(example.service.jobs) == len(example.service.outbox) == before + + +@run_async +@pytest.mark.parametrize("existing_report", [False, True]) +async def test_fresh_state_submit_requires_explicit_new_or_recovers_existing( + example, existing_report +): + """Without a selected key, Submit never starts an unrequested new report.""" + expected = None + if existing_report: + async with session(example) as previous: + await start_report(example, previous) + expected = example.service.jobs[("alice", previous._request_key)] + before = len(example.service.outbox) + async with session(example) as state: + await example.module.ReportState.sync_report.fn(state, True) + assert example.service.calls[-1] == ("read", "alice", "", "") + assert len(example.service.jobs) == len(example.service.outbox) == before + if expected: + assert state.job_id == expected.id + assert state._request_key == expected.request_key and not state.error + else: + assert not state.job_id and not state._request_key + assert state.error == "Choose New report to start a report." + await example.module.ReportState.sync_report.fn(state, True) + assert len(example.service.jobs) == len(example.service.outbox) == before + await start_report(example, state) + assert len(example.service.jobs) == len(example.service.outbox) == before + 1 + + @run_async @pytest.mark.parametrize("lost_response", [False, True]) async def test_retries_reuse_the_request_key_and_committed_job(example, lost_response): """A repeated submission, including lost acknowledgement, keeps one job.""" async with session(example) as state: example.service.lose_response = lost_response - await example.module.ReportState.sync_report.fn(state, True) + await start_report(example, state) request_key = state._request_key example.service.lose_response = False await example.module.ReportState.sync_report.fn(state, True) @@ -175,7 +261,7 @@ async def test_failed_accept_does_not_claim_an_accepted_job(example): """A failed acceptance leaves a retryable key without a job ID/status.""" example.service.fail_accept = True async with session(example) as state: - await example.module.ReportState.sync_report.fn(state, True) + await start_report(example, state) assert state._request_key and state.error assert not state.job_id and not state.status assert not example.service.jobs and not example.service.outbox @@ -185,7 +271,7 @@ async def test_failed_accept_does_not_claim_an_accepted_job(example): async def test_status_read_uses_owner_and_selected_job_id(example): """Possessing another owner's job ID does not return its status.""" async with session(example) as alice: - await example.module.ReportState.sync_report.fn(alice, True) + await start_report(example, alice) job_id = alice.job_id await example.module.ReportState.sync_report.fn(alice) assert example.service.calls[-1] == ( @@ -211,6 +297,8 @@ async def test_stale_service_response_does_not_update_the_ui( ): """Responses cannot overwrite a changed authentication or selection.""" async with session(example) as state: + async with state: + await example.module.ReportState.new_report.fn(state) if operation == "read": await example.module.ReportState.sync_report.fn(state, True) example.service.pause = "accept" if operation == "submit" else "read" @@ -235,7 +323,7 @@ async def test_stale_service_response_does_not_update_the_ui( async def test_fresh_state_recovers_job_and_retry_key(example): """A new session uses an authorized durable lookup, not old UI state.""" async with session(example) as original: - await example.module.ReportState.sync_report.fn(original, True) + await start_report(example, original) job_id, request_key = original.job_id, original._request_key async with session(example) as fresh: assert not fresh.job_id and not fresh._request_key @@ -254,9 +342,9 @@ async def test_authentication_change_recovers_without_previous_session_selectors ): """An account switch or relogin cannot reuse the old report selection.""" async with session(example) as state: - await example.module.ReportState.sync_report.fn(state, True) + await start_report(example, state) async with session(example, next_owner) as other: - await example.module.ReportState.sync_report.fn(other, True) + await start_report(example, other) latest_id, latest_key = other.job_id, other._request_key async with state: auth = await state.get_state(example.auth) @@ -284,11 +372,11 @@ async def test_authentication_change_recovers_without_previous_session_selectors async def test_new_report_binds_identity_before_first_refresh(example, account_change): """An explicit new selection survives initial or post-login recovery.""" async with session(example, "bob") as previous: - await example.module.ReportState.sync_report.fn(previous, True) + await start_report(example, previous) older_id = previous.job_id async with session(example, "alice" if account_change else "bob") as state: if account_change: - await example.module.ReportState.sync_report.fn(state, True) + await start_report(example, state) async with state: if account_change: auth = await state.get_state(example.auth) @@ -307,7 +395,7 @@ async def test_new_report_binds_identity_before_first_refresh(example, account_c async def test_logout_clears_selection_without_calling_job_services(example): """Signed-out refresh removes private selectors as well as visible fields.""" async with session(example) as state: - await example.module.ReportState.sync_report.fn(state, True) + await start_report(example, state) calls = len(example.service.calls) async with state: auth = await state.get_state(example.auth) @@ -327,7 +415,7 @@ async def test_refresh_does_not_replace_a_new_request_with_an_older_job( ): """Refresh keeps a new request's key even before its acceptance returns.""" async with session(example) as state: - await example.module.ReportState.sync_report.fn(state, True) + await start_report(example, state) old_job = state.job_id async with state: await example.module.ReportState.new_report.fn(state) diff --git a/docs/hosting/scaling.md b/docs/hosting/scaling.md index ed26c1dedf8..1b8b8c50bab 100644 --- a/docs/hosting/scaling.md +++ b/docs/hosting/scaling.md @@ -165,13 +165,11 @@ class ReportState(rx.State): generation = self._generation if not identity[0]: return - if submit and not self._request_key: - self._request_key = uuid4().hex request_key, job_id = self._request_key, self.job_id job, error = None, "" try: - if submit: + if submit and request_key: job = await accept_report(identity[0], request_key, "monthly-sales") else: job = await read_report(identity[0], job_id, request_key) @@ -190,6 +188,8 @@ class ReportState(rx.State): self.status = job.status if job else "" if job: self._request_key = job.request_key + elif submit and not request_key: + self.error = "Choose New report to start a report." def report_panel(): @@ -202,7 +202,7 @@ def report_panel(): ) ``` -Retries within the same authenticated session keep the request key; **New report** deliberately creates a new logical request and key. Run `ReportState.sync_report` on page load and after authentication changes to recover from durable records before deliberately submitting a new request. A key held only in lost session state cannot deduplicate a new key. For multiple outstanding jobs, provide an authorized list/selection view instead of a latest-job lookup. +Choose **New report**, then **Submit / retry** to start a report. Only **New report** creates a new request key. Without a selected key, **Submit / retry** only recovers an existing job; if none exists, it prompts you to choose **New report**. Recovered jobs keep their original key for retries. Run `ReportState.sync_report` on page load and after authentication changes to recover from durable records. A key held only in lost session state cannot deduplicate a new key. For multiple outstanding jobs, provide an authorized list/selection view instead of a latest-job lookup. These adapters and the worker must implement the lifecycle above; a background event alone does not make the job durable.