Provider :system ClusterRole silently under-permissioned: core persists a truncated ProviderRevision.status.objectRefs and never re-derives it

[sourcehawk]

Type: Bug
Area: package manager (core) + rbac-manager

Summary

On one cluster, ProviderRevision.status.objectRefs for an active provider revision recorded 13 of 48 ManagedResourceDefinitions. rbac-manager derives the provider :system ClusterRoles from these lists. As a result, every active provider role on the cluster contained 302 of 337 managed-resource kinds. The roles granted no rules for s3.aws.upbound.io.

Thus the API server denied (403) all requests from the provider ServiceAccount to the API group of its managed resources. No S3 managed resource reconciled for at least 12.5 hours (see Timeline).

During this time, the ProviderRevision reported RuntimeHealthy=True and RevisionHealthy=True. The provider pod was Running and Ready with 0 restarts. rbac-manager logged Skipped no-op RBAC ClusterRole apply. It derived the role correctly from the truncated list and found no difference to apply. The only signal was the reflector 403 errors in the provider log.

Routine status reconciles did not repair the list. A core restart also did not: core was replaced mid-incident at T+8 h 34 m, and the list stayed at 13 for four more hours. The repair came 72 s after a second core restart. At that time, a no-op annotation was already present on the ProviderRevision (see Evidence 4).

Environment

Crossplane	v2.3.4 core + rbac-manager (upgraded from v2.2.0 12 h before onset, see Timeline)
Kubernetes	EKS 1.33 (client-go v0.33.0 in provider)
Providers	Upbound AWS family v2.3.0: provider-family-aws, provider-aws-s3, provider-aws-iam, provider-aws-ec2, provider-aws-elbv2, provider-aws-route53
MRDs / CRDs	333 / 248
MRD state	legacy *.aws.upbound.io Active; .m variants Inactive
MRAP	activates 6 legacy s3.aws.upbound.io kinds; iam in both variants; ec2/elbv2/route53 .m only

A second cluster has the identical Crossplane version and the identical provider package digest. The revision name hash is the image digest prefix, thus the package contents are byte-identical. This second cluster is not affected. We used it as the control.

Timeline

All times are UTC. T0 = 2026-08-05 01:38Z is the first observable evidence of the truncation. The rest of this report refers to the offsets in the second column.

Absolute	Offset	Event
2026-07-29 08:48	T−17 h	The provider provisions S3 resources for a tenant normally. The recorded list was complete at this point
2026-08-04 13:47	T−12 h	Crossplane core + rbac-manager upgraded v2.2.0 → v2.3.4 (GitOps-driven, new ReplicaSets for both)
2026-08-04 23:34	T−2 h	Core pod rescheduled onto the new ReplicaSet (node roll, not a deploy change)
2026-08-05 01:38	T0	rbac-manager rewrites the :system role of the storage provider → 302 kinds, s3.aws.upbound.io gone. First evidence that the recorded list is short.
2026-08-05 ~04:20	T+2 h 40 m	First composite blocked. This was the earliest tenant that needed a new S3 managed resource after T0
2026-08-05 07:02	T+5 h 24 m	A tenant resume creates fresh managed resources. They hang with blank conditions
2026-08-05 09:55	T+8 h 17 m	Incident triage begins (~2.5 h after the first tenant alert)
2026-08-05 10:12	T+8 h 34 m	Core pod replaced (…-775fb65547-qp79r → …-775fb65547-224p9, same ReplicaSet, thus a pod deletion, not a template change). Core re-reconciles revisions at startup: rbac-manager rewrites the iam :system role a minute later, still at 302 kinds. objectRefs remains 13.
2026-08-05 11:32	T+9 h 54 m	The :system role of the Inactive revision of the same provider is deleted → rbac-manager regenerates it complete (the own objectRefs of that revision is intact at 48)
2026-08-05 12:10	T+10 h 32 m	Provider pod replaced → Deployment status write → core transitions RuntimeHealthy. Core writes the revision status here and keeps objectRefs at 13.
2026-08-05 12:16	T+10 h 38 m	The :system role of the active revision is recreated → identical 302
2026-08-05 13:30	T+11 h 52 m	Role deleted manually → regenerated → identical 302 again
2026-08-05 14:03	T+12 h 25 m	rbac-manager restarted with --debug → logs Skipped no-op RBAC ClusterRole apply, zero writes
2026-08-05 14:10	T+12 h 32 m	Annotation written on the ProviderRevision → no effect for 4.5 min
2026-08-05 14:14:51	T+12 h 36 m	rollout restart deploy/crossplane (core pod …-766657454d-qd6nm). The annotation from T+12 h 32 m is still present
2026-08-05 14:16:03	T+12 h 37 m	Core writes status: objectRefs 13 → 48. rbac-manager applies the :system role for all six member revisions in 2 s.

Total exposure: 12 h 37 m from the first evidence to the repair. Approximately 10 h of this time was tenant-visible.

Evidence

1. The truncated status, and the arithmetic it produces

# affected cluster
$ kubectl get providerrevision <storage-revision> -o json | jq '.status.objectRefs | length'
13     # all ManagedResourceDefinition, all *.s3.aws.m.upbound.io, zero *.s3.aws.upbound.io

# control cluster, same revision name / digest
48     # 24 legacy + 24 .m

rbac-manager builds the :system role of each member provider from the family-wide union of the objectRefs of the active revisions:

active revision	objectRefs
iam	46
storage	13 (expected: 48)
family	5
ec2	202
elbv2	14
route53	22
total	302 (expected: 337)

This total is exactly the kind count that we observed in the generated roles. The 35 missing kinds are the 11 alphabetically last s3.aws.m.upbound.io kinds (bucketpolicies, bucketpublicaccessblocks, bucketreplicationconfigurations, bucketrequestpaymentconfigurations, buckets, bucketserversideencryptionconfigurations, bucketversionings, bucketwebsiteconfigurations, directorybuckets, objectcopies, objects) plus all 24 s3.aws.upbound.io kinds. s3.aws.upbound.io was also absent from the multi-group */finalizers rule. That rule listed the other ten groups, which included the legacy variant of every other family.

The union is family-wide. Thus all 7 active provider roles on the cluster were short in the same way: the family, iam, ec2, elbv2, and route53 roles too, not only the storage role.

2. rbac-manager operates correctly

With --debug, a freshly started rbac-manager logged:

DEBUG crossplane Skipped no-op RBAC ClusterRole apply
  {"controller": "rbac-roles/providerrevision.pkg.crossplane.io",
   "role-name": "crossplane:provider:<storage-revision>:system"}

rbac-manager computes the desired rules from the recorded list, compares them with the applied rules, and finds them equal. Two rbac-manager restarts produced zero writes. The resourceVersion of the affected roles did not change. The role was also deleted and regenerated three times (once by an unknown actor in the cluster, twice manually). Each time it came back byte-identical with 302 kinds.

For contrast: the inactive revision of the same provider has its own complete 48-entry objectRefs. The same rbac-manager binary deleted and regenerated its role minutes earlier. That role came back complete, with 24 kinds in each of the two s3 groups.

3. Routine status reconciles do not re-derive objectRefs

At T+10 h 32 m, the provider pod was replaced. kube-controller-manager wrote the status of the provider Deployment. Core observed that write and transitioned the RuntimeHealthy condition of the revision in the same second. That is, core wrote the revision status at that moment and kept objectRefs at 13. Two more role regenerations followed (T+10 h 38 m and T+11 h 52 m). Both were identical.

metadata.managedFields keeps only the latest write per (manager, operation, subresource). Thus the condition lastTransitionTime is the only record that remains of that earlier write.

4. What did and did not repair it

T+8h34m   core pod replaced (deletion; same ReplicaSet)  -> objectRefs STILL 13, for 4 more hours
T+12h32m  kubectl-annotate writes an annotation on the ProviderRevision -> no effect on its own (4.5 min)
T+12h36m  kubectl -n crossplane rollout restart deploy/crossplane  (annotation still present)
T+12h37m  manager=crossplane, subresource=status write -> objectRefs 13 -> 48   (+72 s from restart)
   +2 s   rbac-manager logs "Applied RBAC ClusterRole" for the :system role of ALL SIX
          member revisions (the family-wide union changed for all of them)

Thus a fresh core process alone is not sufficient. The restart at T+8 h 34 m shows this. That restart did trigger revision reconciles: rbac-manager rewrote a sibling role a minute later. The annotation alone is also not sufficient. Only the combination repaired the list. An annotation-only edit does not increase metadata.generation, thus it cannot force a spec-change reconcile. We cannot explain why its presence at startup makes a difference.

We also cannot fully isolate the annotation as the differential. Between the two core restarts, the cluster also saw a provider pod replacement (T+10 h 32 m), two deletions of the generated ClusterRole (T+10 h 38 m and T+11 h 52 m), and an rbac-manager restart (T+12 h 25 m). Each of these events is a possible contributor to the state that made the second startup re-derive the list.

5. Downstream effect

E reflector.go:200] "Failed to watch" err="failed to list *v1beta1.Bucket:
buckets.s3.aws.upbound.io is forbidden: User \"system:serviceaccount:<ns>:<sa>\"
cannot list resource \"buckets\" in API group \"s3.aws.upbound.io\" at the cluster scope"

This error repeated for every S3 kind, approximately 900 KB of log from one pod. The log also showed cannot update resource "bucketpolicies/status". Thus already-provisioned resources no longer received status updates but kept their latched Ready=True conditions. They look healthy but are unmanaged. Drift correction, lifecycle changes, and deletions are all blocked.

Composite resources that depend on the group hung with blank conditions on their leaf managed resources. In our case, 7 tenant-facing composites hung. They alerted between 5 minutes and 5 hours 35 minutes after their respective creations.

What we could not determine

• What truncated objectRefs. The provider provisioned S3 resources normally at T−17 h, thus the list was complete then. The earliest direct evidence of the truncation is the role rewrite at T0. That window contains the v2.2.0 → v2.3.4 upgrade (T−12 h) and a core pod reschedule (T−2 h). But we cannot attribute the truncation to either event, because the control cluster took the same upgrade and is healthy. The MRDs themselves were untouched (managedFields unchanged since March). All 48 s3 MRDs existed on the cluster during the outage, correctly owned by the active revision.
• Why 13, and why an alphabetical .m-only prefix. 13 of 48 is not an obvious page or size boundary. The emitted role was approximately 19 KB, much smaller than the ClusterRole and etcd limits.
• Why the second core restart re-derived the list and the first did not. A restart with no other change was effectively tested at T+8 h 34 m. It did not repair the list. The annotation was the only deliberate difference at the second restart. But three other cluster-state changes intervened (see Evidence 4). Thus we cannot attribute the repair to a single action.

Ruled out

• A provider crash or OOM (0 restarts)
• A wrong active revision
• The ClusterRoleBinding subject (correct)
• The ServiceAccount name (correct)
• The CRD ownerReferences (correct, active revision)
• MRD or MRAP drift (unchanged for months, all 48 MRDs present)
• GitOps pruning (the generated roles are not managed in git)
• rbac-manager caches or state that a restart clears (two restarts, zero writes)
• Rate limits
• A version-wide regression in 2.3.4 (the control cluster on 2.3.4 regenerates all 337 kinds correctly)
• The package contents (identical digest)
• A cap on the total kind count (the control cluster enumerates all 337)
• Object counts (the affected cluster has fewer CRDs than the control)

Impact

The failure is a silent and total loss of reconciliation for one API group. No Crossplane status field shows it: RuntimeHealthy=True, RevisionHealthy=True, no event, no log line above debug level. The failure survived a restart of the writing controller (core), two restarts of the reading controller (rbac-manager), two deletions of the generated role, and a provider pod replacement. In our case, it surfaced approximately 2.5 hours after onset as application-level alerts that looked unrelated. The root-cause analysis took most of one day.

Suggested changes

1. Re-derive objectRefs on each ProviderRevision reconcile, and set a condition when the recorded list disagrees with the MRDs and CRDs that the revision owns. A re-derivation at startup is not enough: one core restart mid-incident kept the truncated list in place.
2. Give rbac-manager an MRD/CRD EventSource. The rbac-roles/providerrevision controller watches only ProviderRevision and ClusterRole. Thus it cannot detect or repair a stale upstream snapshot. It will regenerate an under-permissioned role forever.
3. Log the number of granted kinds per generated role at info level. Then a regression of this shape is visible in the rbac-manager output, not only in provider reflector errors.

Workaround

Annotate the affected ProviderRevision. Then restart crossplane core. Then make sure that objectRefs returns to the expected count. In our case, a core restart alone was not sufficient.

For immediate mitigation, create a standalone ClusterRole and binding that grant the dropped group to the provider ServiceAccount. This role survives regeneration. If the role has no rbac.crossplane.io/system label and no ownerReferences, rbac-manager does not adopt it.

[sourcehawk]

Potentially a duplicate or related to #7536

Feel free to close as duplicate, although we did not encounter any crashloop behavior here

[IvanJosipovic]

We're seeing this issue occur almost daily on 4 clusters. Likely exacerbated due to Karpenter reconciling Nodes nightly which causes Crossplane Pod to restart. I posted details in #7536.

[vcatlassian]

I found one concrete producer-side path that can leave ProviderRevision.status.objectRefs incomplete. CachedClient.Get writes fetched package content to the cache asynchronously and ignores the Store result. If that write fails (for example, because the cache volume is full), FsPackageCache.Store leaves a truncated gzip at the final path. A later reconcile can parse that partial entry and persist only the recovered subset in status.objectRefs.

I opened crossplane/crossplane-runtime#1112 to wait for cache writes, drain early writer failures so registry parsing can finish, remove failed writes, and refetch corrupt entries for pull-enabled policies. This fixes the failure we reproduced, but may not explain every source of incomplete objectRefs described here.