Aggregated APIService's /openapi/v2 gets corrupted (ArbitraryKeys/SchemaProps/SwaggerSchemaProps leak as literal schema properties) when the APIService's kube-openapi is newer than the parent apiserver's

[andrewegel]

FWD: Analysis generated by Claude Sonnet 5, then edited / confirmed by a human (me)

What happened:

After upgrading metrics-server from v0.8.1 to v0.9.0 on an EKS 1.35 cluster, the cluster's merged /openapi/v2 document became corrupt. The apiserver logs (CloudWatch, kube-apiserver-* stream) show:

E0901 02:01:25.001 handler.go:160] Error in OpenAPI handler:
  ...resource.Quantity_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
  ...meta.v1.APIGroupList_v2 / APIGroup_v2 / APIResourceList...

and equivalent errors for meta.v1.APIGroupList_v2, meta.v1.APIGroup_v2, meta.v1.APIResourceList_v2 - To reproduce:

$ curl -sk -o /dev/null -w "protobuf -> HTTP %{http_code}\n" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf" \
    https://<cluster-endpoint>/openapi/v2
protobuf -> HTTP 503

$ curl -sk -o /dev/null -w "json     -> HTTP %{http_code}\n" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Accept: application/json" \
    https://<cluster-endpoint>/openapi/v2
json     -> HTTP 200

The 503 effectively DDOS's the KubeAPI server for everyone else using this KubeAPI server, leading to:

Error from server (ServiceUnavailable): the server is currently unable to handle the request

From other Clients.

metrics-server v0.9.0 bumped k8s.io/apiserver from v0.33.7 → v0.36.2, which transitively bumped k8s.io/kube-openapi from v0.0.0-20250318190949-c8a335a9a2ff (v0.8.1) to v0.0.0-20260520065146-aa012df4f4af (v0.9.0).

Within that range, commit add4291a9eec163878911f65bbb7534d08b05d31 ("Migrate callers to post-split go-json-experiment API", merged in #604) changed Schema.MarshalJSONTo in pkg/validation/spec/schema.go:

 func (s Schema) MarshalJSONTo(enc *jsontext.Encoder) error {
      type ArbitraryKeys map[string]interface{}
      var x struct {
-             ArbitraryKeys
+             ArbitraryKeys      ArbitraryKeys              `json:",inline"`
              SchemaProps        schemaPropsOmitZero        `json:",inline"`
              SwaggerSchemaProps swaggerSchemaPropsOmitZero `json:",inline"`
              Schema             string                     `json:"$schema,omitempty"`

Before this change, ArbitraryKeys was an anonymous/embedded field, implicitly flattened by the pre-split experimental JSON v2 API (MarshalNextJSON). After the change, it's a named field that depends on the new jsontext/post-split API correctly honoring the explicit json:",inline" tag to flatten it (along with SchemaProps and SwaggerSchemaProps) into the parent object.

The field names ArbitraryKeys, SchemaProps, SwaggerSchemaProps are leaking into the served schema as literal JSON properties instead of being inlined.

What you expected to happen:

Upgrading an aggregated APIService's implementation (metrics-server) to a newer line of its Kubernetes library dependencies shouldn't corrupt OpenAPI aggregation on a parent apiserver that is still within the normal supported version window.

Anything else we need to know?:

Not yet experimented: whether the malformed document is produced by metrics-server itself when serving its own spec, or by the parent apiserver's aggregator when merging it; and whether this requires a specific Go toolchain/build flag (e.g. GOEXPERIMENT=jsonv2) to route through MarshalJSONTo at all, versus the MarshalJSON path.

Environment:

• Parent apiserver: EKS v1.35.6-eks-bca9cf6

• Aggregated APIService: metrics-server, image 0.9.0 (previously 0.8.1, working)

• Metrics Server manifest

spoiler for Metrics Server manifest:

• Kubelet config:

spoiler for Kubelet config:

• Metrics server logs:

spoiler for Metrics Server logs:

• Status of Metrics API:

spolier for Status of Metrics API:

kubectl describe apiservice v1beta1.metrics.k8s.io

/kind bug

[kubernetes-prow]

This issue is currently awaiting triage.

If metrics-server contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[anandshivam44]

We got the same issue in our company last night

[andrewegel]

The EKS APIServer logs in full context if this helps:

E0902 18:40:18.933197      13 handler.go:160] Error in OpenAPI handler: [1,5865899] $root.definitions.io.k8s.apimachinery.pkg.api.resource.Quantity_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5871946] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.APIGroupList_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5873385] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.APIGroup_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5880034] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.APIResourceList_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5881805] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.APIResource_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5896459] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.Duration has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5898439] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.FieldsV1_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5899904] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.GroupVersionForDiscovery_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5905196] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5910399] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.ManagedFieldsEntry_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5921503] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5932702] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.OwnerReference_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5935850] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.ServerAddressByClientCIDR_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5937122] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.ShardInfo_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5942986] $root.definitions.io.k8s.apimachinery.pkg.apis.meta.v1.Time_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5951075] $root.definitions.io.k8s.apimachinery.pkg.version.Info_v2 has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5997227] $root.definitions.io.k8s.metrics.pkg.apis.metrics.v1beta1.ContainerMetrics has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,5997943] $root.definitions.io.k8s.metrics.pkg.apis.metrics.v1beta1.NodeMetrics has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,6000101] $root.definitions.io.k8s.metrics.pkg.apis.metrics.v1beta1.NodeMetricsList has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,6001774] $root.definitions.io.k8s.metrics.pkg.apis.metrics.v1beta1.PodMetrics has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps
[1,6003992] $root.definitions.io.k8s.metrics.pkg.apis.metrics.v1beta1.PodMetricsList has invalid properties: ArbitraryKeys, SchemaProps, SwaggerSchemaProps

[andrewegel]

FWD: Claude Analysis Sonnet 5 high - edited by me:

Update: reproduced directly against metrics-server:

Port-forwarded straight to the metrics-server pod's secure port (10250) and hit its own /openapi/v2 directly with a valid bearer token, bypassing the EKS-managed apiserver's aggregator:

curl -sk -H "Authorization: Bearer $TOKEN" \
  -H "Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf" \
  https://<metrics-server-pod-ip>:10250/openapi/v2

Same 503, same corruption signature, from metrics-server's pod logs — no aggregator involved. Whatever's wrong, metrics-server is generating it itself before the parent apiserver ever sees it.

We attempted to work around this by forcing UseOptimizedJSONMarshaling/UseOptimizedJSONUnmarshaling to false with a patch:

diff --git a/vendor/k8s.io/kube-openapi/pkg/internal/flags.go b/vendor/k8s.io/kube-openapi/pkg/internal/flags.go
--- a/vendor/k8s.io/kube-openapi/pkg/internal/flags.go
+++ b/vendor/k8s.io/kube-openapi/pkg/internal/flags.go
@@ -17,9 +17,11 @@
 package internal

 // Used by tests to selectively disable experimental JSON unmarshaler
-var UseOptimizedJSONUnmarshaling bool = true
+var UseOptimizedJSONUnmarshaling bool = false
 var UseOptimizedJSONUnmarshalingV3 bool = true

 // Used by tests to selectively disable experimental JSON marshaler
-var UseOptimizedJSONMarshaling bool = true
+var UseOptimizedJSONMarshaling bool = false
 var UseOptimizedJSONMarshalingV3 bool = true

But this didn't work. Since the corruption spans multiple pkg/validation/spec types the bug isn't in kube-openapi's per-type MarshalJSON/MarshalJSONTo dispatch. More likely candidate: something in k8s.io/apiserver's own OpenAPI v2 spec-building/aggregation code; Haven't bisected k8s.io/apiserver versions yet to narrow down which bump in that range introduces it.

Will update here if we get to it

[andrewegel]

We found that pinning the 0.9.0 tag back to these versions:

go mod edit \
        -require=k8s.io/api@v0.35.3 \
        -require=k8s.io/apimachinery@v0.35.3 \
        -require=k8s.io/apiserver@v0.35.3 \
        -require=k8s.io/client-go@v0.35.3 \
        -require=k8s.io/component-base@v0.35.3 \
        -require=k8s.io/kms@v0.35.3 \
        -require=k8s.io/metrics@v0.35.3 \
        -require=k8s.io/kube-openapi@v0.0.0-20250910181357-589584f1c912
go mod tidy
# Build

With zero modifications to metrics-server's code resolved our issue.

[aloknikhil]

We are seeing this too. Ran into it when using Pulumi which is a GRPC k8s client and fails to access the schema API.

With the new metrics server image

kubectl proxy --port=8001 &
sleep 5

for a in 'application/json' 'application/com.github.proto-openapi.spec.v2@v1.0+protobuf'; do
  printf '%-60s ' "$a"
  curl -sS -o /dev/null -w '%{http_code}  %{size_download}B\n' -H "Accept: $a" \
    http://127.0.0.1:8001/openapi/v2
done

application/json                                             200  9830091B
application/com.github.proto-openapi.spec.v2@v1.0+protobuf   503  0B

With the previous metrics server image

for a in 'application/json' 'application/com.github.proto-openapi.spec.v2@v1.0+protobuf'; do
  printf '%-60s ' "$a"
  curl -sS -o /dev/null -w '%{http_code}  %{size_download}B\n' -H "Accept: $a" \
    http://127.0.0.1:8001/openapi/v2
done

application/json                                             200  10230821B
application/com.github.proto-openapi.spec.v2@v1.0+protobuf   200  8892971B