Add upstream FeatureView lineage support for PushSource

[mao-liu]

Is your feature request related to a problem? Please describe.

In many real-world production ML architectures, derived features are computed outside of Feast by external stream or batch computation engines. These external pipelines read upstream features from existing Feast FeatureViews and push the transformed features into Feast via a PushSource backed by an offline batch store.

Currently, Feast's static registry lineage only tracks:

PushSource --> Target FeatureView

Because PushSource does not support declaring the upstream FeatureViews it was computed from, the relationship between the upstream feature views and the push source is lost in both the Feast UI Lineage graph and the registry metadata.

Describe the solution you'd like

We would like PushSource to support declaring upstream FeatureView dependencies via a source_views (or upstream_feature_views) parameter:

# 1. Upstream feature views
user_tx_fv = FeatureView(name="user_transaction_stats", ...)
user_credit_fv = FeatureView(name="user_credit_profile", ...)
# 2. PushSource declaring upstream views and offline batch source
risk_push_source = PushSource(
    name="risk_calc_pipeline",
    batch_source=FileSource(path="data/risk_offline.parquet"),
    source_views=[user_tx_fv, user_credit_fv],  # <-- Upstream dependencies
    description="External Spark job computing risk scores from transaction & credit features",
)
# 3. Target feature view receiving the pushed features
user_risk_fv = FeatureView(
    name="user_risk_target_fv",
    entities=[user_entity],
    ttl=timedelta(days=30),
    source=risk_push_source,
    schema=[Field(name="risk_score", dtype=Float32)],
)

With this metadata, Feast can render the complete directed graph in the Feast UI Lineage view:

[ user_transaction_stats ] (FeatureView) ──┐
                                           ├──→ [ risk_calc_pipeline ] (PushSource) ──→ [ user_risk_target_fv ] (FeatureView)
[ user_credit_profile ]    (FeatureView) ──┘

Proposed Implementation Details

1. Protobuf (protos/feast/core/DataSource.proto): Add upstream_feature_views to PushOptions:

message PushOptions {
  reserved 1;
  repeated string upstream_feature_views = 2;
}

2. Python SDK (sdk/python/feast/data_source.py): Update PushSource constructor, to_proto(), and from_proto() to accept and serialize source_views: Optional[List[Union[BaseFeatureView, str]]].
3. Registry Lineage (sdk/python/feast/lineage/registry_lineage.py): Update RegistryLineageGenerator._parse_direct_relationships() to emit EntityRelation(source=FeatureView, target=DataSource) for each upstream feature view defined on a PushSource.
4. UI Lineage Parser (ui/src/parsers/parseEntityRelationships.ts): Extract upstreamFeatureViews from dataSources to draw the corresponding edges in React Flow.
5. OpenLineage Emitter (sdk/python/feast/openlineage/mappers.py): Ensure emit_apply() creates the corresponding input dataset connections for OpenLineage producers.

Describe alternatives you've considered

1. Using OnDemandFeatureView (ODFV): ODFV handles on-the-fly transformations at request/serving time, but is not intended for pre-computed, stored, push-ingested features requiring offline batch storage.
2. Adding metadata in tags: Storing tags={"depends_on": "user_tx_fv,user_credit_fv"} records metadata, but is non-standard and does not render visual lineage relationships in the Feast UI graph.
3. Emitting manual OpenLineage events via CI/CD: Works if an OpenLineage backend is configured, but does not solve native Feast static registry lineage or out-of-the-box UI visualization.

Additional context
I am happy to contribute this change and submit a PR covering the proto update, Python SDK, registry lineage generator, and UI relationship parser if the community agrees with this direction!

[ntkathole]

@mao-liu Thanks for raising it, it's a valid feature request and would be a good contribution to accept. I can help with reviews.

[mao-liu]

Thanks @ntkathole - letting you know that I have began working on this and testing on my fork.

There are a couple of pre-existing issues on master that I've identified along the way during testing:

1. Feast registry lineage doesn't currently render stream sources (affects Push/Kafka/Kinesis sources)

• only batch sources are currently rendered
• rendering nodes and edges for stream sources is easy enough to add, and I will include it on my PR

2. Feast serve_lineage UI currently doesn't correctly display relationships between data sources and feature views

• Currently, feast apply emits a single openlineage job event with all sources and features in a single event
• OpenLineageProcessor then computes edges as the cartesian product of all sources and views in the same job - leading to all data sources in a feature repo connected to all feature views
• Is this the intended behaviour? Would you like me to open a separate issue for the community to track this behaviour?

We don't have any requirements to use OpenLineage... but wanted to raise that its lineage view might not be very visually helpful in its current state

[ntkathole]

@mao-liu Thank you for the findings, yes, those seems valid bugs. Feel free to tackle and raise issues.