|
| 1 | +--- |
| 2 | +authors: |
| 3 | + - "@jiripetrlik" |
| 4 | +state: draft |
| 5 | +links: |
| 6 | + - https://github.com/NVIDIA/OpenShell/issues/3028 |
| 7 | + - https://github.com/jiripetrlik/OpenShell/pull/1 |
| 8 | +--- |
| 9 | + |
| 10 | +# RFC 0015 - SDK Conformance Testing |
| 11 | + |
| 12 | +## Summary |
| 13 | + |
| 14 | +This RFC introduces a shared SDK conformance suite that verifies the supported |
| 15 | +OpenShell SDKs against a real gateway. The suite defines the observable |
| 16 | +behavior that an SDK must provide for selected gateway workflows, while each |
| 17 | +language keeps native test code and idioms. |
| 18 | + |
| 19 | +The first implementation is a Go adapter based on the exploratory work in the |
| 20 | +related pull request. The prototype currently uses a Podman-backed gateway; |
| 21 | +this RFC proposes a Docker-backed gateway as the portable default conformance |
| 22 | +environment. The first adapter covers sandbox lifecycle and execution, |
| 23 | +providers, and workspaces. Rust, Python, and TypeScript adapters follow |
| 24 | +incrementally. |
| 25 | + |
| 26 | +## Motivation |
| 27 | + |
| 28 | +The Go SDK has extensive unit coverage, but its client tests use an in-process |
| 29 | +transport and do not exercise a built `openshell-gateway`. That leaves a gap: |
| 30 | +a protocol, conversion, authentication, or asynchronous-lifecycle regression |
| 31 | +can pass SDK unit tests while breaking SDK consumers in a deployed system. |
| 32 | + |
| 33 | +Adding a Go-only E2E suite would improve that situation, but would leave the |
| 34 | +project with independently selected scenarios, assertions, harnesses, and CI |
| 35 | +requirements for every SDK. The result would be uneven compatibility coverage |
| 36 | +and no durable statement of which gateway behavior all official SDKs promise to |
| 37 | +support. |
| 38 | + |
| 39 | +OpenShell already has reusable CLI conformance scenarios in |
| 40 | +`openshell-conformance`, and language-specific E2E suites for Rust and Python. |
| 41 | +Those establish useful patterns, but CLI conformance cannot validate typed SDK |
| 42 | +interfaces, conversions, error classification, or SDK conveniences such as |
| 43 | +readiness waiting. A shared SDK contract is needed alongside them. |
| 44 | + |
| 45 | +This deserves an RFC because it defines a testing strategy and compatibility |
| 46 | +contract across SDKs, gateway behavior, E2E infrastructure, and CI. It also |
| 47 | +sets an extension boundary that future SDKs should follow. |
| 48 | + |
| 49 | +## Non-goals |
| 50 | + |
| 51 | +- Replacing unit tests, generated-protobuf checks, or each SDK's language-native |
| 52 | + integration tests. |
| 53 | +- Defining every public SDK method as conformant in the initial release. |
| 54 | +- Replacing the existing `openshell-conformance` CLI runner or changing its |
| 55 | + scenarios and plan format. |
| 56 | +- Requiring every scenario to run against every compute driver, identity mode, |
| 57 | + or operating system on every pull request. |
| 58 | +- Standardizing language APIs, error types, package layouts, or async models. |
| 59 | +- Providing cross-version compatibility testing against released gateway and SDK |
| 60 | + versions in the first implementation. |
| 61 | + |
| 62 | +## Proposal |
| 63 | + |
| 64 | +### A shared behavioral specification |
| 65 | + |
| 66 | +Add a versioned SDK conformance specification under `e2e/conformance/sdk/`. |
| 67 | +The specification defines scenarios in terms of externally observable gateway |
| 68 | +behavior: setup inputs, ordered operations, expected successful results, |
| 69 | +expected classified failures, and cleanup requirements. It does not encode |
| 70 | +language syntax or require one generic test runner to call every SDK. |
| 71 | + |
| 72 | +The initial specification version covers these workflows: |
| 73 | + |
| 74 | +| Area | Required behavior | |
| 75 | +| --- | --- | |
| 76 | +| Sandbox lifecycle | Create, get, list, wait for ready, delete, and observe eventual removal. | |
| 77 | +| Sandbox execution | Run a successful command, preserve sandbox filesystem state across executions, and surface a failed command's exit status and stderr. | |
| 78 | +| Providers | Create, get, list, update, attach, detach, and delete a provider; verify a sandbox receives placeholders rather than raw credential values. | |
| 79 | +| Workspaces | Create, get, list, delete, apply labels, scope sandbox visibility, and classify missing resources as not found. | |
| 80 | + |
| 81 | +A scenario may explicitly mark an operation as unsupported for an SDK only when |
| 82 | +the SDK's published API does not expose that capability. The adapter must report |
| 83 | +the omission in its results; it must not silently skip it. Adding an SDK feature |
| 84 | +that makes an omitted scenario available requires enabling that scenario in the |
| 85 | +same change. |
| 86 | + |
| 87 | +The specification is a set of versioned Markdown scenario documents. Each |
| 88 | +document gives a stable scenario identifier, preconditions, ordered operations, |
| 89 | +expected observable results, cleanup expectations, and permitted SDK-specific |
| 90 | +variations. Markdown is the normative contract because it keeps the behavior |
| 91 | +reviewable without introducing a parser or a general gateway workflow language. |
| 92 | + |
| 93 | +Adapters translate the Markdown requirements into native test code. A future |
| 94 | +machine-readable companion may reduce duplicated fixture data or produce |
| 95 | +coverage reports, but it must derive from the Markdown contract and is not |
| 96 | +required for the initial implementation. |
| 97 | + |
| 98 | +### Fixture ownership and isolation |
| 99 | + |
| 100 | +In this RFC, a fixture is predefined test data or setup that makes a scenario |
| 101 | +repeatable. Fixtures have three ownership levels: |
| 102 | + |
| 103 | +| Fixture type | Owner | Examples | Reuse rule | |
| 104 | +| --- | --- | --- | --- | |
| 105 | +| Conformance fixture | Shared scenario documents | A standard sandbox policy, an exec command, expected exit code, placeholder expectation, or expected `NotFound` result. | Shared by every SDK adapter that implements the scenario. | |
| 106 | +| Adapter fixture | One SDK adapter | Go `TestMain`, a configured Go client, context deadlines, helper functions, and language-native assertions. | Local to that SDK because its runtime and testing idioms differ. | |
| 107 | +| Live test resource | One test invocation | A sandbox, provider, workspace, credential value, or temporary file created while a test runs. | Never shared between parallel tests; each test uses a unique name and cleans it up. | |
| 108 | + |
| 109 | +The shared Markdown scenario is authoritative for conformance fixtures. It |
| 110 | +states the inputs and observations each SDK must use or verify, such as a |
| 111 | +failing command's exit code and stderr content. It does not prescribe how Go, |
| 112 | +Python, Rust, or TypeScript construct a client or express assertions. |
| 113 | + |
| 114 | +Adapter fixtures remove repeated language-specific setup without crossing SDK |
| 115 | +boundaries. They may share a gateway connection within one test process only |
| 116 | +when the adapter can do so safely. They must not cause tests to share mutable |
| 117 | +gateway objects or credentials. |
| 118 | + |
| 119 | +Live resources are deliberately isolated. Every test creates its own uniquely |
| 120 | +named sandbox, provider, and workspace as needed; it registers bounded cleanup |
| 121 | +before later operations can fail. A scenario may define a synthetic credential |
| 122 | +value for placeholder testing, but the adapter creates the provider that holds |
| 123 | +it for that invocation and must verify that the raw value is never returned to |
| 124 | +the sandbox. |
| 125 | + |
| 126 | +### Native SDK adapters |
| 127 | + |
| 128 | +Each supported SDK owns a native adapter and test entry point under `e2e/`. |
| 129 | +An adapter loads the shared scenarios, performs the corresponding SDK calls, |
| 130 | +and checks the standardized observations using that language's normal test |
| 131 | +framework. It may add language-specific assertions where they validate the SDK |
| 132 | +surface, such as Go context cancellation or TypeScript typing, without changing |
| 133 | +the shared contract. |
| 134 | + |
| 135 | +The adapters have a common responsibility: |
| 136 | + |
| 137 | +- create unique test resources and register cleanup before making later calls; |
| 138 | +- use bounded operation and cleanup deadlines; |
| 139 | +- tolerate documented asynchronous deletion by polling only the specified |
| 140 | + observable state; |
| 141 | +- emit enough resource identity and gateway diagnostics to make CI failures |
| 142 | + actionable; |
| 143 | +- run only when the harness explicitly supplies a reachable gateway. |
| 144 | + |
| 145 | +The Go adapter lives in `e2e/go/` as a separate Go module with a local |
| 146 | +`replace` directive to `sdk/go/`. Its files use the `e2e` build tag. The related |
| 147 | +draft PR demonstrates this shape and currently uses the Podman gateway harness, |
| 148 | +but it must be rebased, moved to the Docker default lane, and adjusted to the |
| 149 | +shared scenarios before it is accepted. |
| 150 | + |
| 151 | +```mermaid |
| 152 | +flowchart LR |
| 153 | + SPEC[Versioned SDK scenarios] --> GO[Go adapter] |
| 154 | + SPEC --> PY[Python adapter] |
| 155 | + SPEC --> TS[TypeScript adapter] |
| 156 | + SPEC --> RS[Rust SDK adapter] |
| 157 | + GO --> GW[Real gateway] |
| 158 | + PY --> GW |
| 159 | + TS --> GW |
| 160 | + RS --> GW |
| 161 | + CLI[CLI conformance runner] --> GW |
| 162 | +``` |
| 163 | + |
| 164 | +### Gateway harness and runtime coverage |
| 165 | + |
| 166 | +The default SDK conformance lane starts one Docker-backed gateway through the |
| 167 | +existing `e2e/with-docker-gateway.sh` harness. This gives all SDK adapters one |
| 168 | +repeatable local command and prevents the general `mise run e2e` task from |
| 169 | +implicitly requiring both Docker and Podman. |
| 170 | + |
| 171 | +Runtime-specific SDK scenarios are allowed when the expected result depends on |
| 172 | +a compute driver or deployment mode. They belong in explicitly named tasks and |
| 173 | +CI jobs, such as `e2e:go:podman`; they do not redefine the portable contract. |
| 174 | +The initial Go suite therefore runs in the Docker default lane. A Podman lane |
| 175 | +may be added later to validate its harness and driver behavior, but is not a |
| 176 | +prerequisite for the shared SDK contract. |
| 177 | + |
| 178 | +Every CI lane must capture gateway logs on failure. Per-language test timeouts |
| 179 | +must be shorter than the job timeout and should report the last observed state |
| 180 | +instead of relying on a global test-framework timeout. |
| 181 | + |
| 182 | +### Relationship to existing tests |
| 183 | + |
| 184 | +`openshell-conformance` remains the portable CLI conformance suite. It invokes |
| 185 | +the `openshell` binary and validates command-line behavior, so it is not an SDK |
| 186 | +adapter and does not consume SDK scenarios. |
| 187 | + |
| 188 | +SDK unit tests continue to validate conversion details, retry behavior, and |
| 189 | +language-specific ergonomics cheaply. Existing Rust and Python E2E tests remain |
| 190 | +valid. Their scenarios should be mapped to the shared specification over time; |
| 191 | +the project should not rewrite stable coverage merely to satisfy a new layout. |
| 192 | + |
| 193 | +The conformance specification is the source of truth for portable SDK behavior. |
| 194 | +Its scenario identifiers, required observations, and capability exceptions are |
| 195 | +reviewed as compatibility changes. |
| 196 | + |
| 197 | +## Implementation plan |
| 198 | + |
| 199 | +1. Add the version-one Markdown scenario documents and a short contributor |
| 200 | + guide under `e2e/conformance/sdk/`. Start with sandbox, exec, provider, and |
| 201 | + workspace workflows defined above. |
| 202 | +2. Rebase the Go prototype onto current `main`, move it to the Docker gateway |
| 203 | + harness, and implement it as the first adapter. Strengthen its assertions to |
| 204 | + check returned resource state, readiness state, provider read/list/update, |
| 205 | + failed exec behavior, and missing-sandbox errors. |
| 206 | +3. Add `mise run e2e:go` and a focused Go CI lane. Keep it out of the aggregate |
| 207 | + `e2e` task until the Docker default path is in place; then add it to the |
| 208 | + aggregate task. |
| 209 | +4. Add adapters for the Rust SDK, Python SDK, and TypeScript SDK. Migrate or |
| 210 | + map existing E2E coverage where it matches a scenario, retaining |
| 211 | + language-specific tests where it does not. |
| 212 | +5. Add runtime-specific lanes only for scenarios whose expected behavior varies |
| 213 | + by runtime. Document supported capability exceptions and their rationale. |
| 214 | +6. Update the relevant architecture and contributor documentation when the |
| 215 | + suite is accepted and implemented. |
| 216 | + |
| 217 | +## Risks |
| 218 | + |
| 219 | +- A broad DSL could become a second orchestration framework. Markdown keeps the |
| 220 | + contract descriptive, while adapters retain control flow in native code. |
| 221 | +- The contract could flatten meaningful language differences. Shared scenarios |
| 222 | + define gateway-visible outcomes, while each SDK retains its own type, error, |
| 223 | + and cancellation assertions. |
| 224 | +- Running every SDK against every runtime would make CI too slow and flaky. The |
| 225 | + default lane uses Docker and adds specialized lanes only for |
| 226 | + runtime-dependent behavior. |
| 227 | +- Asynchronous cleanup can leave leaked sandboxes or providers after failures. |
| 228 | + Adapters must register bounded cleanup early and failure diagnostics must |
| 229 | + identify resources for manual recovery. |
| 230 | +- A scenario may accidentally codify a gateway implementation detail. Reviews |
| 231 | + should specify observable behavior and avoid asserting storage internals, |
| 232 | + polling intervals, or transport implementation details. |
| 233 | + |
| 234 | +## Alternatives |
| 235 | + |
| 236 | +### Keep a Go-only E2E suite |
| 237 | + |
| 238 | +This delivers value quickly and the related PR is a useful starting point. |
| 239 | +However, it does not establish a shared SDK compatibility contract, leaving |
| 240 | +other bindings to independently choose what they test. It is retained as the |
| 241 | +first implementation phase, but not as the final design. |
| 242 | + |
| 243 | +### Put all E2E tests inside each SDK directory |
| 244 | + |
| 245 | +Keeping tests beside SDK code simplifies ownership, but makes a shared scenario |
| 246 | +catalog and common gateway-harness conventions harder to discover and reuse. |
| 247 | +The separate `e2e/` adapters make the real-gateway boundary explicit while SDK |
| 248 | +unit and integration tests remain near their implementation. |
| 249 | + |
| 250 | +### Extend the existing CLI conformance runner |
| 251 | + |
| 252 | +The runner already has reusable plans and lifecycle scenarios, but it executes |
| 253 | +the CLI. Adding language SDK invocation would couple it to multiple language |
| 254 | +runtimes and would still not validate SDK-native APIs. Shared scenario concepts |
| 255 | +can align, while the runners remain separate. |
| 256 | + |
| 257 | +### Use only generated-protobuf compatibility checks |
| 258 | + |
| 259 | +Schema checks detect wire changes, but cannot verify SDK conversions, readiness |
| 260 | +helpers, credential-placeholder safety, cleanup behavior, or error |
| 261 | +classification against a running gateway. |
| 262 | + |
| 263 | +## Prior art |
| 264 | + |
| 265 | +- `crates/openshell-conformance` provides a versioned plan and registered |
| 266 | + scenario model for CLI conformance. This RFC adopts the idea of explicit, |
| 267 | + reviewable scenarios while retaining SDK-native execution. |
| 268 | +- The existing `e2e/with-docker-gateway.sh` harness supplies a repeatable real |
| 269 | + gateway lifecycle for Rust and Python E2E tests. It is the portable default |
| 270 | + for the SDK suite. |
| 271 | +- The Go SDK's fake client and bufconn tests demonstrate the complementary |
| 272 | + unit-test layer. They remain appropriate for fast, isolated tests that do |
| 273 | + not need a real gateway. |
| 274 | +- The related Go E2E draft demonstrates parallel-safe resource naming, |
| 275 | + placeholder assertions, and eventual-deletion handling that the first adapter |
| 276 | + should preserve. |
| 277 | + |
| 278 | +## Open questions |
| 279 | + |
| 280 | +- When, if ever, should repeated fixture data or coverage reporting justify a |
| 281 | + machine-readable companion to the Markdown scenarios? |
| 282 | +- Which SDKs are in the initial supported-adapter set? This RFC assumes Go, |
| 283 | + Rust, Python, and TypeScript, subject to confirmation of release support. |
| 284 | +- Should independently released SDKs run the same scenarios against the |
| 285 | + latest gateway only, a release compatibility range, or both? |
| 286 | +- Which provider type can provide deterministic placeholder coverage without |
| 287 | + requiring a live external credential service in every CI lane? |
| 288 | +- Should scenario results be emitted as a machine-readable report for release |
| 289 | + qualification, or are native test reports sufficient initially? |
0 commit comments