Repository navigation
feat(auth,gax): support certificate-bound tokens and mTLS rotation for agent identities - #14668
Open
macastelaz wants to merge 14 commits into
Open
macastelaz wants to merge 14 commits into
macastelaz wants to merge 14 commits into
Conversation
Co-authored-by: Kirill Logachev <kirl@google.com> Co-authored-by: Min Zhu <zhumin@google.com> Co-authored-by: Lawrence Qiu <lawrenceqiu@google.com> Co-authored-by: yangyzs <171981480+yangyzs@users.noreply.github.com> Co-authored-by: cloud-java-bot <122572305+cloud-java-bot@users.noreply.github.com> Co-authored-by: Keshav Dandeva <keshavdandeva@google.com> Co-authored-by: Mattie Fu <mattiefu@google.com> Co-authored-by: Will Howes <whowes@google.com> Co-authored-by: Jin Seop Kim <jinseop@google.com> Co-authored-by: Jiang Zhu <jiangzzhu@google.com> Co-authored-by: Knut Olav Løite <koloite@gmail.com> Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: rahul2393 <irahul@google.com> Co-authored-by: Sakthivel Subramanian <179120858+sakthivelmanii@users.noreply.github.com> Co-authored-by: Gaole Meng <gaole@google.com>
This PR introduces a feature which enables the auth library to acquire bound access-tokens and bound id-tokens in Agentic Environments. 1) We detect certs in default paths and check if they match the SPIFFE format for agents. 2) If 1. is a yes then we call the MDS endpoint in a POST request with the certificate in the body. Note this PR was based on #13169 --- ## Manual Testing & End-to-End Verification We verified this feature end-to-end across both a **Live Cloud Run Agent Identity environment** (testing against the live Google Metadata Server, Security Token Service, and Vertex AI with the **Java Agent Development Kit (`com.google.adk:google-adk:1.9.0`)**) and a **10-Scenario Local Mock MDS Simulation Harness** (testing exact HTTP request payloads, true cryptographic certificate/key rotation, combined bundle private-key stripping, well-known directory discovery, non-agent SPIFFE fallback, environment variable precedence, non-atomic rotation retries, and asynchronous container startup polling). ### 1. Live Cloud Run Agent Identity Verification (`<PROJECT_ID>`, `us-central1`) We deployed a containerized Java test application built against this branch (`google-auth-library-oauth2-http:1.50.0-SNAPSHOT` @ `27776f6d62a`) + Java ADK (`com.google.adk:google-adk:1.9.0`) to Cloud Run with Agent Identity enabled (`--functional-type=agent --identity-type=agent-identity`). #### Execution A: Default Bound Token Acquisition (`GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN` unset / default `true`) - **Cloud Run Job Executions**: `agent-bound-token-java-test-cjrgp` (latest run with Step 4D) & `agent-bound-token-java-test-dd9gm` - **Workload Certificate Discovery**: - Detected `/var/run/secrets/workload-spiffe-credentials/credentials.json` - Extracted Leaf Certificate SAN URI: `spiffe://agents.global.org-<ORG_NUMBER>.system.id.goog/resources/run/projects/<PROJECT_NUMBER>/locations/us-central1/jobs/agent-bound-token-java-test` - Leaf Certificate SHA-256 (`x5t#S256`): `5Ggf2ofOIkHkvBAwfku6eKQVI5pYzW50whzv-VwSAM8` - **Bound Access Token (`ComputeEngineCredentials`)**: - Successfully acquired bound OAuth2 Access Token via `POST` to Metadata Server (`ya29.d.c0AZ4bNp...`). - **Bound ID Token (`IdTokenCredentials`) & Cryptographic Binding Verification (RFC 8705 § 3.1)**: - Acquired bound ID Token for target audience `https://example-target-service.run.app`. - Decoded JWT payload inline (`textPayload`) and verified live Google STS embedded the `cnf` (Confirmation) claim containing the SHA-256 thumbprint (`x5t#S256`) of the workload's leaf X.509 certificate: ```json "cnf": { "x5t#S256": "5Ggf2ofOIkHkvBAwfku6eKQVI5pYzW50whzv-VwSAM8" } ``` - Verified an **exact byte-for-byte match** (`[PASS] JWT x5t#S256 thumbprint EXACTLY matches local leaf certificate SHA-256!`). - **Java ADK (`com.google.adk:google-adk:1.9.0`) + `google-genai` & mTLS Proof-of-Possession Verification (Steps 4A, 4B, 4C, 4D)**: - **Step 4A (ADK `HttpClientFactory` Non-mTLS Transport)**: Inspected ADK's shared `OkHttpClient` (`sun.security.ssl.SSLSocketFactoryImpl`, no client cert). Calling Google APIs over this non-mTLS channel with the bound token is rejected at the auth layer with **`HTTP 401 UNAUTHENTICATED`**. - **Step 4B (Live ADK `LlmAgent` + `Gemini` Turn on Vertex AI)**: Invoking `InMemoryRunner.runAsync(...)` with `Gemini` (`gemini-2.5-flash`) fails on turn 1 with `com.google.genai.errors.ClientException: 401 . Request had invalid authentication credentials`, confirming the known limitation where `google-genai` sends bound tokens over a non-mTLS channel. - **Step 4C (True mTLS Contrast Test — Matching Workload Certificate)**: Presenting the **exact same bound access token** over a true mTLS `OkHttpClient` (configured with `/var/run/secrets/workload-spiffe-credentials/certificates.pem` + `private_key.pem`, `x5t#S256 = 5Ggf2ofOIkHkvBAwfku6eKQVI5pYzW50whzv-VwSAM8`) to `https://cloudresourcemanager.mtls.googleapis.com/v1/projects/<PROJECT_ID>` **passes authentication** (`HTTP 403 PERMISSION_DENIED` IAM check instead of `401 UNAUTHENTICATED`), proving Google API Frontend verified the token binding against the TLS client certificate handshake. - **Step 4D (Sender-Constraining / Proof-of-Possession Negative Test — Mismatched Client Certificate)**: Presenting the **exact same bound access token** to `https://cloudresourcemanager.mtls.googleapis.com/v1/projects/<PROJECT_ID>` over an mTLS `OkHttpClient` configured with a **different** X.509 client certificate (`x5t#S256 = -57ZjYVm89oczfdO02Hf3Sz-FaYQIVH1DD3c9zaBIKA`) is **rejected at the auth layer with `HTTP 401 UNAUTHENTICATED`**, confirming that Google API Frontend enforces cryptographic thumbprint matching (`cnf.x5t#S256 == SHA256(TLS client cert)`). #### Execution B: Opt-Out Unbound Token Acquisition (`GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false`) - **Cloud Run Job Execution**: `agent-bound-token-java-test-t7b98` - Updated job environment variable `--set-env-vars="GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false,GOOGLE_CLOUD_LOCATION=global,GOOGLE_GENAI_USE_VERTEXAI=true"`. - Verified `ComputeEngineCredentials` and `IdTokenCredentials` fell back to standard `HTTP GET` requests against MDS and issued standard unbound tokens (decoded JWT payload confirmed absence of the `cnf` claim). - **Java ADK Live Agent Turn (`Step 4B`)**: With `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false`, the unbound token works over `google-genai`'s non-mTLS channel and the live ADK `LlmAgent` + `Gemini` turn **SUCCEEDS**: `[ADK Event] author=bound-token-verify-agent, content=Hello! Yes, as an agent designed to test bound token behavior with the Java ADK, I am expected to receive and process bound tokens.` --- ### 2. Local End-to-End Simulation & Wire Verification (10 Scenarios) To verify internal wire-level, discovery, rotation, environment-variable, and error-handling behavior between the client and MDS, we executed our local simulation suite (`LocalSimulationRunner.java`) spinning up a local Mock MDS (`HttpServer`) across **10 end-to-end scenarios**: 1. **Bound Access Token (`ComputeEngineCredentials` + `GOOGLE_API_CERTIFICATE_CONFIG`)**: Verified `HTTP POST` to `/computeMetadata/v1/instance/service-accounts/default/token?scopes=https://www.googleapis.com/auth/cloud-platform`, verified JSON body `{"certificate_chain": "-----BEGIN CERTIFICATE-----\n..."}` (serialized as a single PEM string), and verified no extra fields are included in the JSON payload. 2. **Bound ID Token (`IdTokenCredentials` + `GOOGLE_API_CERTIFICATE_CONFIG`)**: Verified `HTTP POST` to `/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://target.run.app` (with `audience` passed as a URL query parameter) and JSON body `{"certificate_chain": "-----BEGIN CERTIFICATE-----\n..."}`. 3. **True Live Cryptographic Certificate & Key Rotation on Disk (`Gen-1` $\rightarrow$ `Gen-2`)**: Generated a new X.509 SPIFFE certificate and matching 2048-bit RSA key pair on disk, updated file `mtime`, and verified `AgentIdentityUtils.getAgentIdentityCertInfo()` invalidated its cache, verified the new key pair, and transmitted the rotated `Gen-2` certificate chain (`!req3.body.equals(req1.body)`). 4. **Opt-Out Override (`GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false`)**: Verified setting `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` switches requests back to `HTTP GET` with an empty request body. 5. **Well-Known Directory Combined Bundle (`credentialbundle.pem`) + Private Key Stripping**: Unset `GOOGLE_API_CERTIFICATE_CONFIG`, wrote a combined `credentialbundle.pem` containing **both** `-----BEGIN CERTIFICATE-----` and `-----BEGIN PRIVATE KEY-----` in the well-known directory, and verified that `AgentIdentityUtils` resolved `certPath == keyPath`, verified the key pair, and **stripped the `PRIVATE KEY` block** from the transmitted `POST` payload. 6. **Well-Known Directory Separate Files (`certificates.pem` + `private_key.pem`)**: Unset `GOOGLE_API_CERTIFICATE_CONFIG`, removed `credentialbundle.pem`, placed separate `certificates.pem` and `private_key.pem` in the well-known directory, and verified bound `HTTP POST` acquisition. 7. **Non-Agent SPIFFE Trust Domain Fallback (`shouldRequestBoundToken == false`)**: Configured a valid X.509 certificate with a standard GKE Workload Identity SAN (`spiffe://my-standard-gke-project.svc.id.goog/ns/default/sa/my-ksa`); verified `AgentIdentityUtils` did not throw, cached `shouldRequestBoundToken = false`, and fell back to standard `HTTP GET`. 8. **Environment Variable Precedence & `GOOGLE_API_USE_CLIENT_CERTIFICATE` Matrix**: - **8a**: Legacy `GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=false` (with `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN` unset) $\rightarrow$ falls back to `HTTP GET`. - **8b**: Modern `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=true` overrides legacy `GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=false` $\rightarrow$ sends bound `HTTP POST`. - **8c**: `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` with valid agent certs on disk $\rightarrow$ disables bound token (`HTTP GET`) without startup polling. - **8d**: `GOOGLE_API_USE_CLIENT_CERTIFICATE=true` with missing cert files $\rightarrow$ fails closed with `IOException` after retries instead of silently downgrading to an unbound token. 9. **Non-Atomic Rotation / Transient Key-Pair Mismatch Recovery (`CERT_KEY_MATCH_RETRIES`)**: Wrote a new `Gen-3` non-prod SPIFFE certificate (`spiffe://agents-nonprod.global.org-54321.system.id.goog/...`) first while delaying the matching `private_key.pem` update on a background thread; verified `loadAndVerifyCredentials()` retried cleanly via `CERT_KEY_MATCH_RETRIES` and transmitted `Gen-3`. 10. **Asynchronous Container Startup Credential Delivery Polling (`TOTAL_POLL_CYCLES`)**: Started with an empty well-known directory on initial startup (`GOOGLE_API_USE_CLIENT_CERTIFICATE=true`), delivered `certificates.pem` + `private_key.pem` asynchronously from a background thread after ~180ms, and verified the initial `refreshAccessToken()` polled until the files arrived and succeeded with a bound `HTTP POST`. --- ### 3. Reproducible Test Artifacts & Execution Logs (`gpaste` - Internal Corp Access Only) | Artifact / Log | Description | Link | | :--- | :--- | :--- | | **Combined Verification Summary & Logs** | Complete report & console outputs from Live Cloud Run (Bound `cjrgp` & Opt-Out `t7b98`) + Java ADK + Local 10-Scenario Simulation | https://paste.googleplex.com/4671448266964992 | | **`CloudRunAgentVerifyApp.java`** | Live Cloud Run + Java ADK verification app (validates SPIFFE cert, ADC access token, ID token `cnf.x5t#S256` match, and ADK Steps 4A/4B/4C/4D) | https://paste.googleplex.com/5431546950057984 | | **`LocalSimulationRunner.java`** | Self-contained local E2E test harness with Mock MDS (`HttpServer`) testing all 10 scenarios | https://paste.googleplex.com/4541159570014208 | | **`Dockerfile`** | Multi-stage Docker build resolving Java ADK 1.9.0 + `google-genai` and overlaying our local PR JAR | https://paste.googleplex.com/6569302979903488 | | **`deploy_cloud_run_job.sh`** | Turnkey script to build via Cloud Build, deploy Cloud Run Job with `--identity-type=agent-identity`, and execute | https://paste.googleplex.com/4908078735163392 | | **`build_and_run_local.sh`** | Turnkey script to compile and run the local simulation | https://paste.googleplex.com/6449866457350144 | | **Raw Cloud Logging — Bound Run (`cjrgp`)** | Plain-text Cloud Run logs from `gcloud logging read` for default Bound Token execution (with inline decoded JWT & ADK Steps 4A/4B/4C/4D) | https://paste.googleplex.com/5404054864396288 | | **Raw Cloud Logging — Opt-Out Run (`t7b98`)** | Plain-text Cloud Run logs from `gcloud logging read` for `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` execution (showing live ADK Gemini response) | https://paste.googleplex.com/5990735250325504 | #### Quick Reproduction Steps To replicate the live Cloud Run test in any GCP project with Agent Identity enabled: ```bash # 1. Place CloudRunAgentVerifyApp.java, LocalSimulationRunner.java, Dockerfile, and deploy_cloud_run_job.sh # in a directory alongside your local google-cloud-java checkout. chmod +x deploy_cloud_run_job.sh build_and_run_local.sh # 2. Run the local 10-scenario simulation suite: ./build_and_run_local.sh # 3. Deploy and execute the Cloud Run Job with Agent Identity enabled (Default Bound Token mode): ./deploy_cloud_run_job.sh <PROJECT_ID> us-central1 # 4. To test opt-out behavior (unblocks ADK non-mTLS Gemini calls): gcloud alpha run jobs update agent-bound-token-java-test \ --project=<PROJECT_ID> --region=us-central1 \ --set-env-vars="GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false,GOOGLE_CLOUD_LOCATION=global,GOOGLE_GENAI_USE_VERTEXAI=true" gcloud alpha run jobs execute agent-bound-token-java-test \ --project=<PROJECT_ID> --region=us-central1 --wait ``` --------- Co-authored-by: Pranav Iyer <pjiyer@google.com>
…ons (#13995) ## Description This PR adds transparent retries for mTLS workload certificate rotation to the gRPC and HTTP/JSON transports. When a request fails with `UNAUTHENTICATED` and the workload certificate on disk has changed, the transport is rebuilt with the new certificate and the request is retried once, without interrupting in-flight RPCs or streams. ### 🚀 Core Features & Architectural Updates • Rotation detection: `CertificateRotationTracker` compares the SHA-256 fingerprint of the workload certificate on disk with the certificate the transport was built with. The check only runs after an auth failure (never on the request path), and positive results are cached for at most 1 second so a burst of failures doesn't trigger a burst of file reads. Empty or mid-write files are ignored. • gRPC: `ChannelPool` replaces all of its channels when a rotation is detected. Calls already in flight finish on their old channels, which are shut down once idle. During a rotation refresh, a channel that can't be recreated is dropped rather than kept, so no traffic keeps using the old certificate; a statically sized pool is refilled in the background. • HTTP/JSON: `RefreshingHttpJsonChannel` swaps the channel's `HttpTransport` for one built with the new certificate. Calls already created keep using the transport they started with, so in-flight requests aren't interrupted. • Retries: `AttemptCallable` (unary) and `ServerStreamingAttemptCallable` (server streaming) refresh the transport after an `UNAUTHENTICATED` failure. If the transport moved to a new certificate during or after the attempt, the failure is flagged for a single immediate retry on the refreshed channel; its configured `isRetryable()` value is unchanged. • `ApiResultRetryAlgorithm` gives that retry once, immediately, without using a regular attempt. • If the free retry also fails after another rotation, the call stops instead of falling back to backoff retries. • For server streams, the free retry becomes available again once the stream has made progress, and the stream resumes through the existing resumption strategy. • Client and bidi streams refresh the transport on `UNAUTHENTICATED` but don't retry; the next stream uses the new certificate. • Plumbing: `TransportChannel` gains `shouldRefresh()`, `refresh()` and `getGeneration()`, which default to no-ops. `ApiCallContext.getTransportChannel()` gives retrying callables access to the channel, and `GrpcCallContext` / `HttpJsonCallContext` carry it through `merge()` and `withChannel()`. ### 🔒 System Hardening & Bug Fixes • Outstanding RPC leak (`ChannelPool.ReleasingClientCall`): if a call was cancelled before `start()`, `start()` threw without releasing the channel entry, leaving the channel with a permanently outstanding RPC count so it could never be cleaned up after a refresh. The entry is now released on that path and on cancellation before start. • HTTP/JSON refresh vs. shutdown: `shutdown()` / `shutdownNow()` are serialized with `refresh()` so a transport swap can't race with teardown. ###⚠️ Behavioral & Security Boundary Changes * mTLS is enabled automatically when a workload certificate config is present (`MtlsUtils`, auth library): * When `GOOGLE_API_CERTIFICATE_CONFIG`, or the default gcloud `certificate_config.json`, contains a `workload` certificate, client certificates are used even if `GOOGLE_API_USE_CLIENT_CERTIFICATE` is unset. * `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` still turns mTLS off. * mTLS misconfiguration now fails closed (`MtlsUtils`): * Previously, an invalid explicit config was swallowed and the client silently connected without mTLS. * Now client creation throws `IllegalStateException` in these cases: * the explicit `GOOGLE_API_CERTIFICATE_CONFIG` file is missing, unreadable or malformed; * the default config file exists but is unreadable or malformed; * a configured certificate or key file can't be read. * When mTLS is required but the transport can't load the client certificate, gRPC and HTTP/JSON channel creation fails with an `IOException` instead of falling back to a non-mTLS connection. ### 🧪 Testing #### Automated Testing • Added and updated comprehensive unit-tests reflecting the thread-safety fixes inside ChannelPoolTest.java and RefreshingHttpJsonChannelTest.java. • Corrected edge case test configurations to leverage realistic mocked X.509 certificates to properly exercise deep WorkloadCertificateUtils. getCertificateFingerprint() filesystem caching mechanisms. #### Manual Testing End-to-end tests with real generated GAPIC clients against local mTLS servers, run on this PR's head. Each scenario runs in its own JVM and checks the exact sequence of client certificates and outcomes the server saw. * **Setup** * Clients: `KeyManagementServiceClient` over gRPC and HTTP/JSON, and `GrpcBigQueryReadStub` for server-streaming `ReadRows` with offset-based resumption. * Servers: local gRPC and HTTPS servers that require a client certificate and return `UNAUTHENTICATED` / 401 for "revoked" certificates (by CN). * Rotation: the workload certificate under `GOOGLE_API_CERTIFICATE_CONFIG` is replaced by atomic rename, key first, then cert. * **gRPC unary:** * basic rotation: one rejected attempt, then a free retry on the new cert, and no extra attempts afterwards; * the server rejects every cert: one free retry, then stop; * a second rotation during the free retry: stops after 2 attempts, with no backoff retry; * UNAVAILABLE, then rotation; * corrupt cert on disk: no crash, the call fails, and the client recovers once a valid cert is written; * 20-call bursts across 2 rotations less than 1s apart: all succeed with exactly 1 refresh each; * a 3-channel pool: no stale channel after refresh; * 8 threads across 4 rotations, then `close()` with calls in flight: 70,624/70,624 calls succeeded, no hangs. * **gRPC server streaming:** * rotation before the stream starts; * rotation mid-stream: resumes at the right offset with no gaps or duplicates; * rotation retry re-armed after the stream makes progress, following an earlier UNAVAILABLE retry; * bounded: a second rotation without progress stops the stream, with no third attempt. * **HTTP/JSON:** basic rotation; reject-all; double rotation; a slow call started before rotation completes on the old cert while new calls use the new cert; corrupt cert, then restore; 8-thread stress with 4 rotations, then `close()` (49,561/49,561 calls succeeded, no hangs). The socket factory was verified for both Conscrypt and plain JDK TLS. * **No mTLS** (`GOOGLE_API_USE_CLIENT_CERTIFICATE=false`, or no certificate config): a single attempt with no client cert and no refresh, over both gRPC and HTTP/JSON, even when the cert files on disk change. * **Results:** * JDK 26 (JDK TLS): all 22 scenarios pass. * JDK 21 (Conscrypt): all gRPC, streaming and no-mTLS scenarios pass. * HTTP/JSON with mTLS and Conscrypt on TLS 1.3 fails with an existing `Unknown authType: GENERIC` handshake error, which #14556 fixes. With #14556 applied on top of this PR, all 22 scenarios pass on JDK 21 with Conscrypt active. * Harness source: https://paste.googleplex.com/6120902048219136 * Logs: https://paste.googleplex.com/5467233782988800 (this PR on JDK 26 and JDK 21), https://paste.googleplex.com/5563956459077632 (this PR + #14556 on JDK 21)
…igquery pom (#14587) ## Why Every PR into `agentic-identities-bound-token` is currently failing CI (for example #14559 and #14557, about 35 jobs each) with: ``` [ERROR] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: com.google.cloud:google-cloud-storage:jar -> duplicate declaration of version (?) [ERROR] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: com.google.cloud:google-cloud-datacatalog:jar -> duplicate declaration of version 1.102.0-SNAPSHOT [ERROR] The build could not read 1 project ``` `java-bigquery/google-cloud-bigquery/pom.xml` declares `google-cloud-storage` and `google-cloud-datacatalog` twice in `<dependencies>`. This has been the case for a while, but Maven 3.9 only logged a `[WARNING]`. The GitHub `ubuntu-24.04` runner image `20261004` upgraded Maven from 3.9.16 to 3.10.0, which treats it as an error, so any job that loads the reactor now fails immediately. ## Change Remove the second declaration of each dependency (11 lines, one file): - `google-cloud-storage`: drops the later test-scoped entry. This is the same change as #14586 on `main`. - `google-cloud-datacatalog`: drops the second, identical test-scoped entry. `main` no longer has this dependency in this pom, so #14586 doesn't need it. ## Verification - `mvn help:effective-pom` on the module before and after: the effective `<dependencies>` section is identical. `google-cloud-storage` stays test-scoped because the parent's `dependencyManagement` sets `<scope>test</scope>`, and the two datacatalog entries were identical. - Each artifact now appears once in `<dependencies>`; the pom parses as valid XML.
…ext (#14556) > [!IMPORTANT] > **This must merge before the `agentic-identities-bound-token` feature branch is merged to `main`.** Without it, HTTP/JSON clients fail by default wherever mTLS is enabled automatically (see below). ## Problem When mTLS is active, `InstantiatingHttpJsonChannelProvider.configureMtls()` builds a Conscrypt `SSLContext` but initializes it with the JDK (SunJSSE) PKIX `TrustManagerFactory`. On TLS 1.3, Conscrypt passes authType `"GENERIC"` to the trust manager. SunJSSE's end-entity checks reject that authType for server certificates that are CA-issued and carry a KeyUsage extension, which includes Google front ends. So every mTLS HTTP/JSON handshake fails with: ``` javax.net.ssl.SSLHandshakeException: Unknown authType: GENERIC Caused by: java.security.cert.CertificateException: Unknown authType: GENERIC ``` The bug is already on `main`, but there it only triggers when `GOOGLE_API_USE_CLIENT_CERTIFICATE=true` is set explicitly. On this feature branch, #13995 enables mTLS automatically whenever a workload certificate config is present, so on Cloud Run (agent identity) **every HTTP/JSON client fails by default**. gRPC isn't affected. The non-mTLS path isn't affected either, because google-http-client already pairs Conscrypt with a provider-matched trust manager there. A second provider mismatch shows up on JDK 26: the mTLS `SSLContext` also used the JDK's default (`SunX509`) `KeyManagerFactory`. Since JDK 26 ([JDK-8359956](https://bugs.openjdk.org/browse/JDK-8359956)), that key manager applies algorithm constraints to the client certificate. Called from a Conscrypt handshake, it rejects valid certificates (e.g. `SHA256withRSA`), so the client sends an empty certificate chain and the server aborts with `certificate_required`. ## Fix Use Conscrypt's own PKIX `TrustManagerFactory` (`TrustManagerFactory.getInstance("PKIX", conscryptProvider)`) for the mTLS `SSLContext`. I called the JDK API directly rather than `SslUtils.getPkixTrustManagerFactory(Provider)`, which only exists in google-http-client 2.2.0+. Likewise, use Conscrypt's own `KeyManagerFactory` (`KeyManagerFactory.getInstance("PKIX", conscryptProvider)`), so the trust manager and key manager both come from the same provider as the `SSLContext`. Conscrypt's trust manager loads the same default trust store as the JDK. Verified on JDK 21: 174 anchors in both, the same set, and both honor `-Djavax.net.ssl.trustStore` overrides identically. ## Testing - New `InstantiatingHttpJsonChannelProviderTls13Test`: a local JDK TLS 1.3 server presents a CA-issued leaf with KeyUsage and requires a client certificate. The test asserts that the transport completes the request and presents its client certificate. - Without the fix: fails with `Unknown authType: GENERIC`. - With the fix: passes. - Skips when Conscrypt native or TLS 1.3 is unavailable. - Full `gax-httpjson` suite on the current `agentic-identities-bound-token` (with #13995 merged): 194/194 pass on JDK 8, 11, 17, 21, 25 and 26. On JDK 26, the Tls13 test fails without the `KeyManagerFactory` change: the client sends an empty certificate chain and times out. - End-to-end with #13995's certificate-rotation support, on JDK 21 with Conscrypt active: 22 scenarios with real KMS and BigQuery Storage GAPIC clients against local mTLS servers, where the certificate rotates on disk. - With this fix: all 22 pass. That includes HTTP/JSON rotation, in-flight calls during refresh, and stress with `close()`. The refreshed transport still uses Conscrypt's socket factory. - Without this fix: every HTTP/JSON mTLS scenario fails with `Unknown authType: GENERIC`. Limiting the test server to TLS 1.2 makes the error go away, which confirms that the TLS 1.3 authType is the trigger. - Logs: https://paste.googleplex.com/5563956459077632 (with fix), https://paste.googleplex.com/5467233782988800 (without fix, JDK 21 set) - Live on Cloud Run (agent identity), combined #13873 + #13995 build, default env: | | gRPC | HTTP/JSON | |---|---|---| | Before | ✅ | ❌ `Unknown authType: GENERIC` | | After | ✅ | ✅ authenticated over mTLS with a bound token |
…te-bound ID tokens (#14559) > [!IMPORTANT] > **This must merge before the `agentic-identities-bound-token` feature branch is merged to `main`.** Docs PR #14557 documents this option. ## Why When an agent identity certificate is available, `ComputeEngineCredentials` requests certificate-bound ID tokens by default. A bound ID token is only accepted by targets that authenticate the caller over mTLS with the same certificate. Targets reached over standard HTTPS, such as a Cloud Run service's `*.run.app` URL or a custom domain, reject it with `401 Unauthorized`. Per the [ID token binding 1-pager](https://docs.google.com/document/d/1yL_xdDimlUlUyid6HDtqZUIMQjX9VznwzXn1SBABjZc/edit?pli=1&resourcekey=0-GxSXGdfH5Ni_nSI9f7xkQg&tab=t.0#heading=h.94fnt3dk2gx) (Sep 30 decision), ID tokens stay bound by default, and callers get a per-target `bind_id_token` setting to turn binding off where needed. The process-wide opt-out (`GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false`) is unchanged. ## What changes - New `IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN`. Callers pass it per target, the same way as `FORMAT_FULL` and `LICENSES_TRUE`: ```java IdTokenCredentials.newBuilder() .setIdTokenProvider((IdTokenProvider) credentials) .setTargetAudience("https://my-service-12345.us-central1.run.app") .setOptions(Arrays.asList(IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN)) .build(); ``` - `ComputeEngineCredentials.idTokenWithAudience()`: with the option, the certificate lookup is skipped and the identity endpoint is called with a plain GET. Without it, behavior is unchanged (bound whenever a certificate is available and binding isn't turned off by the env var). - Other credential types (`ServiceAccountCredentials`, `ImpersonatedCredentials`, `UserCredentials`) already ignore options they don't use, so the same code works outside agent identity environments. - Access tokens are unaffected. ### API shape `Option` is a set of on/off flags rather than a value, so this adds only the "off" flag (`DISABLE_BOUND_ID_TOKEN`). Leaving it unset means "auto", which currently means bound. This matches the 1-pager's tri-state (unset = auto) without adding an "enable" flag that would do nothing today; one can be added later without breaking callers if "auto" starts deciding per target. Python exposes the same control as `bind_id_token: Optional[bool]` (googleapis/google-cloud-python#18559). ## Tests ### Unit tests New `ComputeEngineCredentialsTest` cases, all with an agent identity certificate present and binding enabled: - `idTokenWithAudience_disableBoundIdToken_requestsUnboundToken`: GET, no request body, no forced `format=full`. - `idTokenWithAudience_disableBoundIdTokenWithFormatFull_requestsUnboundFullToken`: still GET; `format=full` is kept when the caller asks for it. - `idTokenWithAudience_disableBoundIdToken_skipsCertificateLookup`: with a certificate config pointing to a missing file, the default request fails, but the opted-out request succeeds, which shows the certificate is never read. - `idTokenCredentials_withDisableBoundIdTokenOption_requestsUnboundToken`: the option set on `IdTokenCredentials` reaches `ComputeEngineCredentials`. The existing bound-ID-token tests are unchanged and still pass. Full `oauth2_http` suite: 1064 tests, 0 failures. `fmt` is clean. ### Live test on GKE Run on a GKE cluster with agent identity enabled, using a test-only local merge of this PR with #14595 (GKE support) and #14556 (already on the base branch). A single process fetches ID tokens through `IdTokenCredentials` from `ComputeEngineCredentials` (ADC) and sends them to an agent-to-agent receiver pod. The receiver accepts only bound tokens over mTLS (whose `cnf` must match the client certificate), and only unbound tokens over plain HTTP. | Check | Defaults | `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` | |---|---|---| | ID token with `DISABLE_BOUND_ID_TOKEN` | **unbound** (no `cnf`); receiver: plain HTTP 200, mTLS 401 | unbound | | ID token for another target without the option, same credentials, fetched afterwards | **still bound** (`cnf` = certificate leaf); receiver: mTLS 200 | unbound | | Access token after the ID-token opt-out | **still bound** (401 on a plain, non-mTLS endpoint) | unbound | The option unbinds only the ID token it's set on. Other targets and access tokens are unaffected, and it's harmless when binding is already off. **Internal links (Googlers only):** [results](https://paste.googleplex.com/5926225961418752) · [harness](https://paste.googleplex.com/5713819712749568) · cluster [setup](https://paste.googleplex.com/5635555912712192) and [manifests](https://paste.googleplex.com/6258309762514944) (shared with #14595)
…14557) > [!IMPORTANT] > **Merge after #14559**, which adds `IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN` referenced here. Both must merge before the `agentic-identities-bound-token` feature branch is merged to `main`. ## What this adds Docs only; no code changes. - **`ComputeEngineCredentials` class Javadoc**: adds a brief summary noting that when a workload certificate for an agent identity is available, `ComputeEngineCredentials` requests certificate-bound access tokens and ID tokens by default, with pointers to `IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN` and the [Google Auth Library guide](https://cloud.google.com/java/getting-started/getting-started-with-google-auth-library). - The full "Certificate-bound tokens for agent identities" user guide section (covering environment variables, per-target ID token opt-out, and the ADK for Java limitation) is being added to the DevSite `getting-started-with-google-auth-library` page (`cl/996749889`) alongside the Cloud Run documentation (`cl/990396526`). ## Release notes `google-auth-library-java/CHANGELOG.md` is generated, so release notes come from commit messages. Suggested text for the `BEGIN_COMMIT_OVERRIDE` block of the feature-branch → `main` merge PR: ``` feat(auth): request certificate-bound tokens for agent identities by default. Set GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false to opt out. feat(auth): add IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN to request unbound ID tokens for specific targets. docs(auth): known limitation: ADK for Java doesn't support mTLS yet; agents that use it must set GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false. ``` ## Testing - `google-java-format` and `javadoc:javadoc` pass.
…kens and mTLS (#14595) ## Why GKE delivers agent identity credentials (Pod Certificates) as a single combined PEM file, holding the certificate chain and a PKCS#8 private key, at: ``` /var/run/secrets/workload-spiffe-credentials/x509.credential-bundle.private-key.pem ``` The feature branch already handles Cloud Run, where a certificate config points into the well-known directory. On GKE there is no certificate config, so neither the transport nor the token side found the certificate. This is the Java counterpart of googleapis/google-cloud-python#18559. Background: go/gke-security-identity/features/agent_identity_diy.md and go/sdk-mds-bound-token. ## What changes When **no certificate configuration exists** (`GOOGLE_API_CERTIFICATE_CONFIG` is unset and there is no `~/.config/gcloud/certificate_config.json`), a readable, non-empty bundle now: - **Transport:** turns on mTLS automatically. - `MtlsUtils.getWorkloadCertPath` / `useMtlsClientCertificate` return the bundle path. - `DefaultMtlsProviderFactory.create()` builds its keystore from the bundle via a new package-private `X509Provider.createForTransport()`. - Every GAX transport (`EndpointContext`, gRPC, HTTP/JSON) goes through that factory. - **Tokens:** `AgentIdentityUtils` resolves the same bundle, so `ComputeEngineCredentials` requests tokens bound to the certificate the transport presents. - With `GOOGLE_API_USE_CLIENT_CERTIFICATE=true`, the bundle takes precedence over the Cloud Run well-known files for the same reason. **Kept unchanged:** - `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` and `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` still opt out. - An explicit or default certificate config (including ECP) always beats the bundle, on both sides. - **The bundle is never polled for.** GKE writes it before the container starts, and polling for a missing file would delay every workload that isn't an agent. - **Workload identity federation X.509 is unchanged.** The public `X509Provider` constructors don't fall back to the bundle. Only the transport factory does. - If the bundle disappears during the read (rotation), the transport reports `CertificateSourceUnavailableException` rather than a generic `IOException`. If it's briefly missing after a credential was cached, the token side falls back to the cached credential, but only while the bundle is still allowed (no certificate configuration has appeared). **New API:** a public static `MtlsUtils.hasCertificateConfiguration(...)` on an `@InternalApi` class. This is additive only. ## Differences from Python (intentional) - Python also checks `CLOUDSDK_CONTEXT_AWARE_CERTIFICATE_CONFIG_FILE_PATH`. Java has no equivalent env var. - Python temporarily keeps default **ID tokens** unbound on Cloud Run (when `GOOGLE_API_CERTIFICATE_CONFIG` is set). That gate is temporary and Python-only, so it isn't copied here. On GKE, both libraries bind ID tokens by default. #14559 adds the per-target `DISABLE_BOUND_ID_TOKEN` opt-out. ## Related - #14556 (HTTP/JSON mTLS fix, merged) enables mTLS over HTTP/JSON transports. - #14559 (`DISABLE_BOUND_ID_TOKEN`) is the per-target ID-token opt-out. ## Tests ### Unit tests - `MtlsUtilsTest`: - bundle present, absent, a directory, or empty; - `USE_CLIENT_CERTIFICATE=false`; - a default config without a workload section (ECP) blocks the bundle; - an explicit config takes precedence; - `hasCertificateConfiguration` / `getGkeCredentialBundlePath`. - `X509ProviderTest`: - transport provider with cert-first and key-first bundles; - an explicit config, or an override path, takes precedence; - unavailable when there is no bundle; - the public constructor ignores the bundle (WIF). - `AgentIdentityUtilsTest`: - implicit discovery with zero sleeps; - the cached credential stays bound; - opt-outs, a default gcloud config, and a non-agent cert all leave the token unbound; - an empty or absent bundle returns null without polling; - explicit true finds the bundle, never polls for it, and prefers it over the Cloud Run files; - fallback to the cache after the bundle is deleted, and no fallback once a certificate config has appeared (implicit and explicit paths). Full `oauth2_http` suite: 1089 tests, 0 failures. `fmt` is clean. ### Live test on GKE - **Cluster:** a GKE cluster with agent identity enabled. - **Pods:** the test pods get the agent identity credential bundle at `/var/run/secrets/workload-spiffe-credentials/x509.credential-bundle.private-key.pem`. - **Image:** built from this PR's jars merged with #14556, which is already on the base branch and is required for HTTP/JSON mTLS. It uses ADC with default settings and no client options. - **Details:** cluster setup, manifests and full logs are in the internal links below. **Checks:** - **GCS over raw HTTP:** read through `storage.mtls.googleapis.com` with the transport certificate. The same token on plain `storage.googleapis.com` gets 401, which proves the token is bound. - **GAX Resource Manager client:** `getProject` over gRPC and HTTP/JSON. - **ID tokens:** sent to an agent-to-agent receiver pod. It accepts only bound tokens over mTLS whose `cnf` matches the client certificate, and only unbound tokens over plain HTTP. | Scenario | Token | GAX resolved endpoint | GAX gRPC / HTTP-JSON | GCS mTLS / plain | ID token mTLS / plain | |---|---|---|---|---|---| | Defaults | Bound; `cnf` = bundle leaf = transport cert | `cloudresourcemanager.mtls.googleapis.com` | ✅ / ✅ | 200 / **401** | **200** / 401 | | `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` | Unbound | mTLS endpoint | ✅ / ✅ | 200 / 200 | 401 / **200** | | `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` | Unbound | plain endpoint | ✅ / ✅ | 200 / 200 | 401 / **200** | | `GOOGLE_API_USE_CLIENT_CERTIFICATE=true` | Bound (bundle) | mTLS endpoint | ✅ / ✅ | 200 / **401** | **200** / 401 | | Pod without agent identity | Unbound, no mTLS | plain endpoint | authenticated / authenticated | — / authenticated | n/a | **Timing:** the first token fetch took about 0.5–1s in every scenario. A pod without the bundle has no polling delay. **Internal links (Googlers only):** [setup](https://paste.googleplex.com/5635555912712192) · [manifests](https://paste.googleplex.com/6258309762514944) · [harness](https://paste.googleplex.com/4882197073494016) · [build/run](https://paste.googleplex.com/5942061774274560) · [results](https://paste.googleplex.com/5072605959290880) **Out of scope for this PR (tracked separately):** - **Leaf-only keystore:** `X509Provider`'s keystore holds only the leaf certificate, through google-http-client `SecurityUtils.createMtlsKeyStore`. Google APIs over `*.mtls.googleapis.com` are unaffected. Agent-to-agent mTLS against a peer that trusts only the SPIFFE root fails unless the caller presents the full chain.
…14626) ## Why After the `main` sync (`03af2000542`), the `dependencies` check fails on every PR targeting this branch, for example #14559: ``` [ERROR] Used undeclared dependencies found: [ERROR] com.google.code.findbugs:jsr305:jar:3.0.2:compile ... gax-grpc / gax-httpjson: Dependency problems found ``` `main` moved GAX off `jsr305`: it uses JSpecify for nullness, and `gax-grpc`/`gax-httpjson` no longer declare `jsr305`. Two `javax.annotation` uses that came in with #13995 (certificate rotation retries) are still on this branch. They compile, because `jsr305` still arrives transitively, but `dependency:analyze` correctly flags them. ## What changes These are annotation-only edits with no behavior change: - **`gax-grpc/.../ChannelPool.java`:** the `@GuardedBy("entryWriteLock")` annotation (from `javax.annotation.concurrent`) becomes a `// Guarded by entryWriteLock.` comment. I didn't switch to Error Prone's `@GuardedBy`, because `gax-grpc` doesn't depend on Error Prone annotations, so that would just trade one undeclared dependency for another. On `main`, `gax-grpc` uses no `@GuardedBy`. - **`gax-httpjson/.../RefreshingHttpJsonChannelTest.java`:** `javax.annotation.Nullable` becomes `org.jspecify.annotations.Nullable`, as in the rest of GAX. These are the only `javax.annotation` imports in `gax-java`, `google-auth-library-java` and `java-core` that are on this branch but not on `main`. ## Tests `dependency:analyze` was run the same way as `.kokoro/dependencies.sh` (`mvn dependency:analyze -Pquick-build -DfailOnWarning=true -Dmdep.analyze.skip=false`) on `gax-grpc` and `gax-httpjson`: - without this change: `Used undeclared dependencies found: com.google.code.findbugs:jsr305`, BUILD FAILURE (the same as CI); - with this change: BUILD SUCCESS. Unit tests: `gax-grpc` 252 and `gax-httpjson` 234 tests, 0 failures. `fmt` is clean.
## Why
`X509Provider` and `SecureConnectProvider` build their mTLS key store
with google-http-client's `SecurityUtils.createMtlsKeyStore`. That
method keeps only the **first** `CERTIFICATE` in the PEM input, so Java
presents only the leaf certificate in the TLS handshake. This is
unchanged in the latest google-http-client release (2.2.0).
- **Google APIs over `*.mtls.googleapis.com` are unaffected**, because
Google's front ends already know the intermediates.
- **Peers that trust only a root CA can't verify the leaf without the
intermediate**, so they reject the handshake.
- A common case is agent-to-agent mTLS, where the receiving agent trusts
the SPIFFE trust bundle.
- Agent identity certificates on GKE (and Cloud Run) are issued by an
intermediate CA, so a Java agent that reuses the SDK's mTLS key store
can't call such a peer.
- Python sends the whole chain (`load_cert_chain`).
This bug predates the agent identity work; the agent identity testing
caught it.
## What changes
**New package-private
`MtlsKeyStoreUtils.createMtlsKeyStore(InputStream)`.** It replaces the
upstream method.
- **The change:** the key entry's chain holds **every distinct
certificate**, in input order. The first certificate is still the leaf.
- **Same as upstream:** key store type (JKS), alias, empty password,
accepted key format (PKCS#8 `PRIVATE KEY`), which private key is chosen
when there are several, and error messages.
- **Also different from upstream:** the input is decoded as UTF-8
instead of the platform default charset, and it's read to the end.
Upstream stopped once it had a certificate and a key.
**Used by:**
- `X509Provider.getKeyStore()`, on both the certificate-configuration
path (Cloud Run, WIF X.509) and the GKE credential-bundle path;
- `SecureConnectProvider.getKeyStore()` (ECP), which drains the bytes
already buffered on the helper's stdout after the process exits so a
background child process holding stdout open cannot block waiting for
EOF.
**Inputs that worked before still work.** Because more of the input is
now parsed, once a certificate and a private key have been found (the
point where upstream stopped reading):
- a malformed later PEM section ends the input instead of failing;
- an extra certificate that can't be parsed is left out of the chain
(and logged at `FINE`).
**Unchanged:**
- exception wrapping in the providers;
- the token side: the `cnf` binding is still the leaf thumbprint, read
from the certificate file;
- the public API (no clirr impact).
**Known limitations (unchanged from before):**
- GAX's deprecated `com.google.api.gax.rpc.mtls.MtlsProvider` (ECP-only)
still uses the upstream method. GAX's transports don't use it; they use
`com.google.auth.mtls.MtlsProvider` through
`DefaultMtlsProviderFactory`.
- There's no check that the private key matches the first certificate
(same as upstream).
## Tests
### Unit tests
- `MtlsKeyStoreUtilsTest` (new), using a generated test chain (root →
intermediate → leaf, EC P-256, long validity) in
`testresources/mtls/test_chain_{cert,key}.pem`:
- upstream `SecurityUtils.createMtlsKeyStore` keeps only the leaf for
the same input. This pins down the bug, and the test fails if upstream
ever fixes it;
- leaf + intermediate (and 3-certificate chains) in order, with the
certificates first or the key first, and duplicate certificates
deduplicated;
- the private key matches the leaf (sign/verify);
- a single certificate gives the same alias, type, certificate and key
as `SecurityUtils.createMtlsKeyStore`;
- non-certificate sections are ignored;
- with several private keys, the same key is chosen as upstream: the
last one before the first certificate, or the first one after it. Each
of these tests runs upstream on the same input and compares the encoded
key bytes;
- a malformed trailing section after a certificate and key is ignored,
while a malformed section after only a certificate or only a key throws
`IllegalArgumentException`, and an `IOException` mid-stream propagates;
- an unparsable or empty extra certificate is left out;
- an unparsable leaf, a missing certificate, or a missing key throws,
with the same messages as before.
- `X509ProviderTest`: a GKE bundle with an intermediate (certificates
first, and key first as GKE writes it), and a certificate config whose
`cert_path` holds a chain, all give leaf + intermediate.
- `SecureConnectProviderTest`: helper output with a chain gives leaf +
intermediate, and only buffered stdout bytes are read after the process
exits.
The provider tests fail without the fix (`expected: <2> but was: <1>`).
Full `oauth2_http` suite passes. `fmt` is clean.
### Live tests
| Environment | Check | Before | After |
|---|---|---|---|
| GKE agent identity | Agent-to-agent mTLS using the SDK's
`DefaultMtlsProviderFactory` key store, against a receiver that trusts
only the SPIFFE root | `Broken pipe` (handshake rejected) | **200** |
| GKE agent identity | GCS over `storage.mtls.googleapis.com`, GAX
clients over gRPC and HTTP/JSON, bound tokens | pass | pass (unchanged)
|
| Workload identity federation X.509 (STS mTLS) | Token exchange with a
leaf-only `cert_path` | success, leaf sent | success, leaf sent |
| Workload identity federation X.509 (STS mTLS) | Token exchange with a
leaf + intermediate `cert_path` | success, **leaf only** sent | success,
**leaf + intermediate** sent |
The certificates sent were confirmed from the TLS handshake debug log.
The live tests ran on the first commit (`b2876a4d3a9`). Subsequent
commits only refine how unusual or malformed inputs are handled, plus
tests, so well-formed GKE and WIF files behave the same.
**Internal links (Googlers only):** [live test
results](https://paste.googleplex.com/5884065727774720) · GKE
[setup](https://paste.googleplex.com/5635555912712192),
[manifests](https://paste.googleplex.com/6258309762514944) and
[harness](https://paste.googleplex.com/4882197073494016) (shared with
#14595)
…g with Python parity (#14658) ## Why An audit comparing the Java (`agentic-identities-bound-token`) and Python (`googleapis/google-cloud-python#18559`) agent identity and mTLS implementations surfaced several edge-case behavioral differences: 1. **GAX silent fallback on mTLS initialization error:** When `CertificateBasedAccess.useMtlsClientCertificate()` is `true` and `DefaultMtlsProviderFactory.create()` throws an `IOException` (for example, `CertificateSourceUnavailableException` when a certificate file disappears mid-read or a configured certificate path is invalid/corrupt), `EndpointContext`, `InstantiatingGrpcChannelProvider`, and `InstantiatingHttpJsonChannelProvider` caught the `IOException` and silently degraded to `mtlsProvider = null` (sending requests over an unauthenticated channel or regular endpoint without client certificates). 2. **Startup polling when `GOOGLE_API_CERTIFICATE_CONFIG` is unset:** In `AgentIdentityUtils.getWellKnownCertificatePathWithRetry()`, when `GOOGLE_API_CERTIFICATE_CONFIG` was unset and `GOOGLE_API_USE_CLIENT_CERTIFICATE=true` was set on a non-agent GCE VM where no certificates exist, Java polled the well-known Cloud Run paths for ~3 seconds and threw an `IOException` on timeout instead of falling back to unbound tokens in 0ms. 3. **Symlinked parent directories and empty (0-byte) files during startup polling:** - `AgentIdentityUtils.isPathInWellKnownDir(String)` used `Files.exists(path)` before calling `toRealPath()`, which skipped symlink resolution when the leaf certificate file did not yet exist on disk (e.g., on systems where `/var/run` is a symlink to `/run` and `GOOGLE_API_CERTIFICATE_CONFIG` uses `/run/secrets/workload-spiffe-credentials/...`). - `AgentIdentityCacheUtils.checkExistsOrAccessDenied(Path)` only checked file existence via `BasicFileAttributes` without verifying `attrs.isRegularFile() && attrs.size() > 0`, which could cause startup polling to return immediately on a newly created 0-byte file before certificate contents were written. 4. **`GOOGLE_API_USE_CLIENT_CERTIFICATE` non-boolean values:** `MtlsUtils.useMtlsClientCertificate()` and `MtlsUtils.getWorkloadCertPath()` only checked for `"true"` and `"false"`, allowing arbitrary non-boolean values (e.g., `"invalid"` or `"1"`) to fall through to auto-discovery instead of treating any non-`"true"` value as disabled. ## What changes - **GAX (`CertificateBasedAccess`, `EndpointContext`, `InstantiatingGrpcChannelProvider`, `InstantiatingHttpJsonChannelProvider`):** - Propagate `IOException` from `DefaultMtlsProviderFactory.create()` (routed via package-private `CertificateBasedAccess.createDefaultMtlsProvider()`) when `certificateBasedAccess.useMtlsClientCertificate()` is `true`, failing closed instead of degrading to `mtlsProvider = null`. - When `useMtlsClientCertificate()` is `false`, `IOException` during default `MtlsProvider` creation continues to be logged at `FINE` and ignored. - **`AgentIdentityUtils` & `AgentIdentityCacheUtils`:** - Remove the retry loop and `IOException` throw from `getWellKnownCertificatePathWithRetry()` when `GOOGLE_API_CERTIFICATE_CONFIG` is unset; check well-known paths once (`GKE_CREDENTIAL_BUNDLE_PATH`, `WELL_KNOWN_CREDENTIAL_BUNDLE_PATH`, `WELL_KNOWN_CERT_PATH` + `WELL_KNOWN_KEY_PATH`) and immediately return `certsPresent=false` if none are present. - Startup polling remains exclusively gated on `GOOGLE_API_CERTIFICATE_CONFIG` pointing inside `/var/run/secrets/workload-spiffe-credentials/`. - Add `resolveRealPathAllowingMissing(Path)` to resolve symlinks on the nearest existing ancestor directory and re-append remaining relative path components in `isPathInWellKnownDir(String)`. - Require `attrs.isRegularFile() && attrs.size() > 0` in `AgentIdentityCacheUtils.checkExistsOrAccessDenied(Path)` so polling waits until certificate files are non-empty regular files (`> 0` bytes). - **`MtlsUtils`:** - Treat any non-empty trimmed value of `GOOGLE_API_USE_CLIENT_CERTIFICATE` other than `"true"` (case-insensitive) as disabled (`false` / `null`) in both `useMtlsClientCertificate()` and `getWorkloadCertPath()`. ## Tests - **`EndpointContextTest`, `InstantiatingGrpcChannelProviderTest`, `InstantiatingHttpJsonChannelProviderTest`:** - Added tests verifying that `IOException` from `createDefaultMtlsProvider()` is propagated when `useMtlsClientCertificate()` is `true` and ignored when `useMtlsClientCertificate()` is `false`. - **`AgentIdentityUtilsTest`:** - Updated unset `GOOGLE_API_CERTIFICATE_CONFIG` tests to verify 0-sleep non-polling behavior and immediate `null` return when no well-known certificates exist, even when `GOOGLE_API_USE_CLIENT_CERTIFICATE=true`. - Added tests for 0-byte -> populated file polling in `getConfigFilePathWithRetry` and `getWorkloadCertPathFromConfigWithRetry`. - Added test for `isPathInWellKnownDir` with symlinked parent directory and non-existent leaf file. - **`MtlsUtilsTest`:** - Added tests verifying non-boolean values of `GOOGLE_API_USE_CLIENT_CERTIFICATE` (`"invalid"`) return `false` in `useMtlsClientCertificate()` and `null` in `getWorkloadCertPath()` even when workload certificates exist on disk.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces support for dynamic mTLS certificate rotation and token binding (Agent Identity) across both gRPC and HTTP/JSON transports. It adds utilities for discovering, caching, and validating certificates, and updates credentials and channel pools to dynamically refresh underlying transports when a certificate rotation is detected on disk. The reviewer feedback identifies two key issues: first, using "processOut.available()" in "SecureConnectProvider" to read stdout is flaky and can truncate certificate data, so it should read until EOF instead; second, calling "transportChannel.refresh()" synchronously inside "BidiStreamingCallable.onError" can block the transport's delivery thread due to blocking disk I/O, so it should be offloaded to a background thread.
Comment on lines
+157
to
+166
| for (int available = processOut.available(); | ||
| available > 0; | ||
| available = processOut.available()) { | ||
| byte[] buffer = new byte[available]; | ||
| int read = processOut.read(buffer); | ||
| if (read <= 0) { | ||
| break; | ||
| } | ||
| stdout.write(buffer, 0, read); | ||
| } |
Contributor
There was a problem hiding this comment.
Using processOut.available() to read the process's stdout is extremely flaky and can easily result in truncated or empty certificate data because available() can return 0 even when there is data to be read (especially for process streams, and especially on different OS/JVMs). Since the process has already exited, it is much safer and standard to read the stream until EOF (i.e., read() returns -1).
byte[] buffer = new byte[4096];
int read;
while ((read = processOut.read(buffer)) != -1) {
stdout.write(buffer, 0, read);
}
Contributor
Author
There was a problem hiding this comment.
Keeping processOut.available() here intentionally. Reading until EOF (read() != -1) after waitFor() hangs getKeyStore() indefinitely if the credential helper spawns a background child process that inherits stdout, which we explicitly addressed and tested in #14621 (SecureConnectProviderTest). Because waitFor() has already returned, the exited helper's output is already buffered in the pipe (or drained into a ByteArrayInputStream by the JDK process reaper).
Comment on lines
+264
to
+276
| public void onError(Throwable t) { | ||
| if (t instanceof UnauthenticatedException) { | ||
| TransportChannel transportChannel = mergedContext.getTransportChannel(); | ||
| if (transportChannel != null && transportChannel.shouldRefresh()) { | ||
| try { | ||
| transportChannel.refresh(); | ||
| } catch (Exception e) { | ||
| LOG.log( | ||
| Level.WARNING, | ||
| "Failed to refresh transport channel after authentication error", | ||
| e); | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
Calling transportChannel.refresh() synchronously inside onError can block the transport's delivery thread (e.g., gRPC inbound thread or HTTP response thread) because refresh() performs blocking I/O (such as reading certificate files from disk and initializing SSLContext). Consider executing the refresh asynchronously or offloading it to a background thread.
Contributor
Author
There was a problem hiding this comment.
Keeping transportChannel.refresh() synchronous here intentionally. This path only runs on an UNAUTHENTICATED error when shouldRefresh() confirms the certificate on disk actually rotated (which happens once every few hours or days). Unlike unary and server-streaming calls, GAX does not automatically retry bidi streams—the caller reconnects from responseObserver.onError(t). Offloading refresh() to a background thread would race with the caller's immediate reconnect and risk handing it the stale channel.
macastelaz
marked this pull request as ready for review
October 17, 2026 01:14
|
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merges the
agentic-identities-bound-tokenfeature branch intomain, adding end-to-end support for certificate-bound tokens and automatic mTLS with transparent certificate rotation for agent identities on Cloud Run and GKE.google-auth-library-java(oauth2_http)#13873,#14595,#14658): When an agent identity SPIFFE workload certificate is present (either viaGOOGLE_API_CERTIFICATE_CONFIGon Cloud Run or the GKE credential bundle at/var/run/secrets/workload-spiffe-credentials/x509.credential-bundle.private-key.pem),ComputeEngineCredentialsrequests certificate-bound access tokens and ID tokens viaPOSTto the Metadata Server with the leaf certificate chain.#13873,#14559,#14557):GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false(or legacyGOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES=false) disables token binding.IdTokenProvider.Option.DISABLE_BOUND_ID_TOKENrequests an unbound ID token for specific targets (such as plain HTTPS*.run.appURLs) while keeping other ID tokens and access tokens bound.#13995,#14595,#14658):MtlsUtils.useMtlsClientCertificate()enables mTLS automatically when a workload certificate config or GKE credential bundle is present, unlessGOOGLE_API_USE_CLIENT_CERTIFICATEis set to a non-"true"value.GOOGLE_API_CERTIFICATE_CONFIGfiles fail closed withIllegalStateException.#14621):X509ProviderandSecureConnectProvidernow populate the mTLS key store with the full certificate chain (leaf + intermediates) viaMtlsKeyStoreUtils.createMtlsKeyStorerather than only the leaf certificate.sdk-platform-java/gax-java(gax,gax-grpc,gax-httpjson)#13995):CertificateRotationTrackerchecks the SHA-256 fingerprint of the workload certificate on disk after anUNAUTHENTICATEDfailure.ChannelPool(gRPC) andRefreshingHttpJsonChannel(HTTP/JSON) rebuild transport channels with the rotated certificate without interrupting in-flight RPCs.AttemptCallable(unary) andServerStreamingAttemptCallable(server streaming) issue one immediate free retry on the refreshed channel when a certificate rotation is detected.#14658): WhenuseMtlsClientCertificate()istrue,EndpointContext,InstantiatingGrpcChannelProvider, andInstantiatingHttpJsonChannelProviderpropagateIOExceptions (while still falling back cleanly onCertificateSourceUnavailableExceptionwhen no certificate source exists on disk).#14556):InstantiatingHttpJsonChannelProvideruses Conscrypt's ownTrustManagerFactoryandKeyManagerFactoryfor the mTLSSSLContext, fixingUnknown authType: GENERICon TLS 1.3 and key-manager algorithm constraint mismatches on JDK 26+.Sub-PRs Included on
agentic-identities-bound-token#13873feat: Enable Bound Token for Agentic Identities#13995feat(gax): support transparent retries during mTLS certificate rotations#14556fix(gax-httpjson): use Conscrypt TrustManagerFactory for mTLS SSLContext#14559feat(auth): add DISABLE_BOUND_ID_TOKEN option to opt out of certificate-bound ID tokens#14557docs(auth): document certificate-bound tokens for agent identities#14595feat(auth): support GKE agent identity credential bundle for bound tokens and mTLS#14626fix(gax): drop jsr305 annotations left over after syncing with main#14621fix(auth): present the full certificate chain for mTLS#14658fix(auth,gax): align Java mTLS and agent identity certificate handling with Python parityTesting
google-auth-library-java(credentials,oauth2_http) andsdk-platform-java/gax-java(gax,gax-grpc,gax-httpjson) pass across JDK 8, 11, 17, 21, 25, and 26, including new unit tests for certificate discovery, bound token requests, startup polling, rotation retries, Conscrypt TLS 1.3 mTLS handshakes, and full certificate chains.45/45scenarios passed): Built all 9 changed artifacts (google-auth-library-credentials,google-auth-library-oauth2-http,api-common,gax,gax-grpc,gax-httpjson,google-cloud-core,google-cloud-core-http,google-cloud-core-grpc) together from the combined branch and ran 22 live GAPIC mTLS rotation scenarios, 14 local bound-token and GAX parity checks, 3 live Cloud Run Agent Identity job executions, and 6 live GKE Agent Identity pod scenarios.Full manual test matrix (45 scenarios)
1. Live GAPIC mTLS Rotation (
22/22Scenarios,125/125Checks)Executed real
KeyManagementServiceClient(gRPC and HTTP/JSON) andBigQueryReadClient(server-streamingReadRows) against live.mtls.googleapis.comendpoints (cloudkms.mtls.googleapis.com,bigquerystorage.mtls.googleapis.com) while rotating client certificates on disk.G1–G7)atomic-mv,inplace,symlink-swap,corrupt-recover,missing-recover,idle-gap,burst-25-rpcsKeyManagementServiceClientgRPC channel seamlessly picks up rotated certs across atomic rename, in-place write, Kubernetes-style symlink swap, corrupt/missing file recovery, idle gaps, and concurrent bursts.S1–S4)mid-stream,between-streams,reconnect-after-rotate,corrupt-mid-streamBigQueryReadClient.readRowsCallable()active streams continue uninterrupted during rotation; subsequent streams and reconnects use the new certificate.H1–H5)atomic-mv,inplace,corrupt-recover,missing-recover,burst-20-rpcsKeyManagementServiceClientHTTP/JSON transport reloads rotated certs and recovers from transient corrupt/missing files without dropping requests.N1a,N1b,C1)N1a,N1b,C1-conscrypt,C1-jdkSunJSSE) TLS providers.G1–S4assertions125/125individual assertions across 22 isolated JVMs.2. Local Bound-Token & GAX Parity (
14/14Checks)#13873/#14595cnf.x5t#S256matches leaf),GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false,GOOGLE_API_USE_CLIENT_CERTIFICATE=false, cert rotation updatesx5t#S256, andGOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICESbehavior.#14658GOOGLE_API_USE_CLIENT_CERTIFICATE=truewith no certs falls back immediately (43msGET) with no 30s polling delay.#14658GOOGLE_API_CERTIFICATE_CONFIGfails closed withIOExceptioninstead of silently downgrading.#14658#14658..datasymlinked parent directory resolves to the real directory for file watching.#14559IdTokenProvider.Option.DISABLE_BOUND_ID_TOKENreturns an unbound ID token without leaking state to subsequent default calls.#14621X509Providerpreserves the full 2-certificate chain (leaf + intermediate) in theKeyStore.fail-closed#14658GOOGLE_API_CERTIFICATE_CONFIGthrowsIllegalStateExceptionin bothInstantiatingGrpcChannelProviderandInstantiatingHttpJsonChannelProvider.zero-ms-fallback#14658GOOGLE_API_USE_CLIENT_CERTIFICATE=truewith no certs builds plain gRPC and HTTP/JSON channels immediately (0mspolling delay).3. Live Cloud Run Agent Identity (
3/3Scenarios)Tested on a Cloud Run Agent Identity job (
--functional-type=agent --identity-type=agent-identity).DefaultMtlsProviderFactorychain length =2(#14621). Access token acquired in414ms. ID tokencnf.x5t#S256matched leaf cert.DISABLE_BOUND_ID_TOKEN(#14559) returnedcnf: falseand next default call restoredcnf: true. GAXProjectsClient(gRPC & HTTP/JSON) resolvedcloudresourcemanager.mtls.googleapis.com:443and passed mTLS + bound-token auth (PERMISSION_DENIEDIAM check). Non-mTLS transport rejected bound token (401), true mTLS passed auth (403), and mismatched client cert was rejected (401).GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=falsecnfabsent). Non-mTLS endpoints passed auth (403IAM) and live Java ADKLlmAgent+Geminicall on Vertex AI succeeded (200 OK).GOOGLE_API_USE_CLIENT_CERTIFICATE=false450ms) and ID token (cnfabsent). GAXProjectsClient(gRPC & HTTP/JSON) resolved plaincloudresourcemanager.googleapis.com:443and passed auth (PERMISSION_DENIED). Live Java ADKLlmAgent+Geminicall on Vertex AI succeeded (200 OK).4. Live GKE Agent Identity (
6/6Scenarios)Tested on a GKE Agent Identity cluster against live GCP APIs (
cloudresourcemanager,storage) and an in-cluster SPIFFE mTLS receiver configured to trust only the SPIFFE root CA (requiring the full leaf + intermediate chain from#14621).s1-defaultinject-podcertificates=truebundle exists=true, first token fetch1430ms, SDK KeyStore chain length =2, mTLS receiver (SDK KeyStore) =200 OK, mTLS GCS passed auth (404), plain GCS rejected bound token (401), gRPC & HTTP/JSON resolved.mtls.googleapis.com:443and passed auth,DISABLE_BOUND_ID_TOKEN-> plain200/ mTLS401/ default after opt-out -> mTLS200/ access token still bound -> plain GCS401.s2-runtime-bound-falseinject-podcertificates=trueGOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=falsecnf=null), mTLS receiver rejected unbound token (401), plain receiver accepted (200), raw GCS and Storage SDK (HTTP & gRPC) passed auth (404), gRPC & HTTP/JSON resolved.mtls.googleapis.com:443and passed auth.s3-use-cert-falseinject-podcertificates=trueGOOGLE_API_USE_CLIENT_CERTIFICATE=falsecnf=null), gRPC & HTTP/JSON resolved plaincloudresourcemanager.googleapis.com:443and passed auth, plain receiver =200, plain GCS passed auth (404).s4-use-cert-trueinject-podcertificates=trueGOOGLE_API_USE_CLIENT_CERTIFICATE=trues1-default: SDK KeyStore mTLS receiver =200,.mtls.googleapis.com:443resolved and passed auth, allDISABLE_BOUND_ID_TOKENchecks passed.s5-no-annotationbundle exists=false, first token fetch646ms(no 30s hang), resolved plaincloudresourcemanager.googleapis.com:443, passed auth (PERMISSION_DENIED/403).s6-use-cert-true-no-annotationGOOGLE_API_USE_CLIENT_CERTIFICATE=true#14658on real GKE:bundle exists=false, first token fetch609ms(no 30s polling delay), resolved plaincloudresourcemanager.googleapis.com:443, passed auth (PERMISSION_DENIED/403).BEGIN_COMMIT_OVERRIDE
feat(auth): request certificate-bound tokens for agent identities by default. Set GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false to opt out.
feat(auth): add IdTokenProvider.Option.DISABLE_BOUND_ID_TOKEN to request unbound ID tokens for specific targets.
feat(auth): support GKE agent identity credential bundle for bound tokens and mTLS.
fix(auth): present the full certificate chain for mTLS.
docs(auth): known limitation: ADK for Java doesn't support mTLS yet; agents that use it must set GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false.
feat(gax): enable mTLS automatically when a workload certificate is present and support transparent retries during mTLS certificate rotations.
fix(gax-httpjson): use Conscrypt TrustManagerFactory and KeyManagerFactory for mTLS SSLContext.
END_COMMIT_OVERRIDE