Server/engine version coupling after #7374 (Kyuubi 1.12) (InternalSecurityAccessor IV change)

[oh0873]

Background

We run Kyuubi with internal security enabled (kyuubi.internal.security.enabled=true, formerly kyuubi.engine.security.enabled). In our deployment, the Kyuubi server and the Spark engine are upgraded on different cadences:

• Kyuubi server — deployed as a standalone service (we are moving to 1.12.0)
• Spark engine — kyuubi-spark-sql-engine_2.12 is baked into our Spark container images, which lag the server upgrade cycle

Problem

After upgrading the Kyuubi server to 1.12.0, sessions fail when the engine still runs an older kyuubi-spark-sql-engine jar (1.11.1 or earlier) from the Spark image.

We traced this to #7374, which changed the on-the-wire format of internal access tokens in InternalSecurityAccessor:

	Before (#7374)	After (#7374)
IV	Static all-zero IV, fixed at cipher init	Random IV per encryption
Token payload	hex(ciphertext)	hex(IV || ciphertext)

The server issues tokens via InternalSecurityAccessor.issueToken() when connecting to an engine (KyuubiSessionImpl). The engine validates them via EngineSecureAuthenticationProviderImpl using the same class from kyuubi-common. Because the encrypt/decrypt wire format changed, a 1.12 server cannot authenticate to an engine built from a pre-#7374 kyuubi-common jar, even when both sides share the same secret and cipher configuration.

We understand the security motivation for the IV fix. The operational impact is that this change effectively requires lockstep upgrades of server and engine jars whenever internal security is enabled — which is also required for REST API HA mode per internal secure access docs.

Impact on us

• We cannot roll out a Kyuubi server upgrade without simultaneously rebuilding and redeploying every Spark image that carries the engine jar.
• For large fleets, that coupling makes minor/patch server upgrades costly and slow.
• We did not find this requirement documented in upgrade or compatibility guidance.

Questions

1. What is the recommended rollout strategy for operators who decouple server and Spark image lifecycles? For example: upgrade all engine images first, then server — or the reverse?

2. Would the project consider a transitional compatibility approach, such as:

   • decrypt-time fallback (try IV-prefixed format, then legacy zero-IV format), or
   • a config flag to opt into legacy token format during a rolling migration window?

/cc @aajisaka @turboFei

[aajisaka]

@oh0873 Thank you for submitting the discussion. Sorry for breaking compatibility between versions.

To safely upgrade to Kyuubi 1.12.0+, adding a flag to allow the decrypt-time fallback makes sense to me. Let me try to implement.

[aajisaka]

Rethinking this, decrypt-time fallback itself does not fix the issue. This is because the old server can decrypt only the legacy format. We need a flag for new server to issue the legacy encryption format, so that the old server can decrypt it.

[aajisaka]

Opened #7762