Skip to content

fix: Use do_exchange for offline server reads to support HPA - #6846

Merged
ntkathole merged 1 commit into
feast-dev:masterfrom
ntkathole:fix/offline-server-do-exchange-hpa
Sep 21, 2026
Merged

ntkathole merged 1 commit into
feast-dev:masterfrom
ntkathole:fix/offline-server-do-exchange-hpa

Conversation

@ntkathole

@ntkathole ntkathole commented Sep 18, 2026

Copy link
Copy Markdown
Member

Problem

The remote offline server uses a two-phase Arrow Flight protocol for read operations:

  1. do_put — client sends command + entity data → server stores it in self.flights (an in-memory Python dict)
  2. get_flight_info + do_get — client retrieves results in separate gRPC calls → server looks up self.flights

With HPA (Horizontal Pod Autoscaler) or multiple replicas, these separate gRPC calls can be load-balanced to different pods. When do_put lands on Pod A but get_flight_info hits Pod B, the server raises KeyError("Flight not found.") because Pod B's self.flights dict has no record of the flight stored on Pod A.

Client              K8s Service (round-robin)       Pod A         Pod B
  │                          │                        │             │
  ├── do_put(data) ─────────►├───────────────────────►│             │
  │                          │              flights[key] = data     │
  │                          │                        │             │
  ├── get_flight_info() ────►├─────────────────────────────────────►│
  │                          │                        │   key not in│
  │                          │                        │   flights   │
  │   ◄── KeyError("Flight not found.") ◄───────────────────────────┤

Solution

Replace the two-phase read path with Arrow Flight's do_exchange RPC, which handles both the entity data upload and result download in a single bidirectional gRPC stream. This guarantees both phases are processed by the same server pod.

Client              K8s Service           Pod A
  │                      │                  │
  ├── do_exchange() ────►├─────────────────►│
  │   (single stream)    │                  ├─ receives entity data
  │                      │                  ├─ executes query
  │   ◄── results ◄──────┤◄─────────────────┤─ streams results back

@ntkathole
ntkathole requested a review from a team as a code owner September 18, 2026 06:49
The remote offline server used a two-phase Arrow Flight protocol for read
operations: do_put (stores entity data in an in-memory dict) followed by
get_flight_info + do_get (retrieves results). With HPA or multiple replicas,
these separate gRPC calls can be load-balanced to different pods, causing
'Flight not found' errors because the in-memory flights dict is per-pod.

Replace the two-phase read path with Arrow Flight's do_exchange RPC, which
handles both the upload and result download in a single bidirectional gRPC
stream. This guarantees both phases hit the same pod, making the offline
server compatible with horizontal scaling.

Signed-off-by: ntkathole <nikhilkathole2683@gmail.com>
@ntkathole
ntkathole force-pushed the fix/offline-server-do-exchange-hpa branch from 8a44a0f to 48ad47c Compare September 18, 2026 06:52
@codecov-commenter

codecov-commenter commented Sep 18, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 63.23529% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.48%. Comparing base (fa8f06b) to head (48ad47c).

Files with missing lines Patch % Lines
sdk/python/feast/offline_server.py 58.00% 14 Missing and 7 partials ⚠️
sdk/python/feast/infra/offline_stores/remote.py 77.77% 3 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6846      +/-   ##
==========================================
+ Coverage   47.43%   47.48%   +0.05%     
==========================================
  Files         422      422              
  Lines       52263    52321      +58     
  Branches     7582     7590       +8     
==========================================
+ Hits        24791    24846      +55     
+ Misses      25708    25702       -6     
- Partials     1764     1773       +9     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 48.81% <63.23%> (+0.05%) ⬆️
Files with missing lines Coverage Δ
sdk/python/feast/infra/offline_stores/remote.py 55.06% <77.77%> (+4.82%) ⬆️
sdk/python/feast/offline_server.py 35.75% <58.00%> (+6.35%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fa8f06b...48ad47c. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Marcus-Rosti Marcus-Rosti left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll try testing this, but overall the logic looks right for the flight server

@shuchu shuchu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ntkathole
ntkathole merged commit bae07fc into feast-dev:master Sep 21, 2026
34 of 35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants