Skip to content

Commit 1239f1f

Browse files
committed
docs(rfc): propose SDK conformance testing
Signed-off-by: Jiri Petrlik <jpetrlik@redhat.com>
1 parent 118b250 commit 1239f1f

1 file changed

Lines changed: 289 additions & 0 deletions

File tree

Lines changed: 289 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,289 @@
1+
---
2+
authors:
3+
- "@jiripetrlik"
4+
state: draft
5+
links:
6+
- https://github.com/NVIDIA/OpenShell/issues/3028
7+
- https://github.com/jiripetrlik/OpenShell/pull/1
8+
---
9+
10+
# RFC 0015 - SDK Conformance Testing
11+
12+
## Summary
13+
14+
This RFC introduces a shared SDK conformance suite that verifies the supported
15+
OpenShell SDKs against a real gateway. The suite defines the observable
16+
behavior that an SDK must provide for selected gateway workflows, while each
17+
language keeps native test code and idioms.
18+
19+
The first implementation is a Go adapter based on the exploratory work in the
20+
related pull request. The prototype currently uses a Podman-backed gateway;
21+
this RFC proposes a Docker-backed gateway as the portable default conformance
22+
environment. The first adapter covers sandbox lifecycle and execution,
23+
providers, and workspaces. Rust, Python, and TypeScript adapters follow
24+
incrementally.
25+
26+
## Motivation
27+
28+
The Go SDK has extensive unit coverage, but its client tests use an in-process
29+
transport and do not exercise a built `openshell-gateway`. That leaves a gap:
30+
a protocol, conversion, authentication, or asynchronous-lifecycle regression
31+
can pass SDK unit tests while breaking SDK consumers in a deployed system.
32+
33+
Adding a Go-only E2E suite would improve that situation, but would leave the
34+
project with independently selected scenarios, assertions, harnesses, and CI
35+
requirements for every SDK. The result would be uneven compatibility coverage
36+
and no durable statement of which gateway behavior all official SDKs promise to
37+
support.
38+
39+
OpenShell already has reusable CLI conformance scenarios in
40+
`openshell-conformance`, and language-specific E2E suites for Rust and Python.
41+
Those establish useful patterns, but CLI conformance cannot validate typed SDK
42+
interfaces, conversions, error classification, or SDK conveniences such as
43+
readiness waiting. A shared SDK contract is needed alongside them.
44+
45+
This deserves an RFC because it defines a testing strategy and compatibility
46+
contract across SDKs, gateway behavior, E2E infrastructure, and CI. It also
47+
sets an extension boundary that future SDKs should follow.
48+
49+
## Non-goals
50+
51+
- Replacing unit tests, generated-protobuf checks, or each SDK's language-native
52+
integration tests.
53+
- Defining every public SDK method as conformant in the initial release.
54+
- Replacing the existing `openshell-conformance` CLI runner or changing its
55+
scenarios and plan format.
56+
- Requiring every scenario to run against every compute driver, identity mode,
57+
or operating system on every pull request.
58+
- Standardizing language APIs, error types, package layouts, or async models.
59+
- Providing cross-version compatibility testing against released gateway and SDK
60+
versions in the first implementation.
61+
62+
## Proposal
63+
64+
### A shared behavioral specification
65+
66+
Add a versioned SDK conformance specification under `e2e/conformance/sdk/`.
67+
The specification defines scenarios in terms of externally observable gateway
68+
behavior: setup inputs, ordered operations, expected successful results,
69+
expected classified failures, and cleanup requirements. It does not encode
70+
language syntax or require one generic test runner to call every SDK.
71+
72+
The initial specification version covers these workflows:
73+
74+
| Area | Required behavior |
75+
| --- | --- |
76+
| Sandbox lifecycle | Create, get, list, wait for ready, delete, and observe eventual removal. |
77+
| Sandbox execution | Run a successful command, preserve sandbox filesystem state across executions, and surface a failed command's exit status and stderr. |
78+
| Providers | Create, get, list, update, attach, detach, and delete a provider; verify a sandbox receives placeholders rather than raw credential values. |
79+
| Workspaces | Create, get, list, delete, apply labels, scope sandbox visibility, and classify missing resources as not found. |
80+
81+
A scenario may explicitly mark an operation as unsupported for an SDK only when
82+
the SDK's published API does not expose that capability. The adapter must report
83+
the omission in its results; it must not silently skip it. Adding an SDK feature
84+
that makes an omitted scenario available requires enabling that scenario in the
85+
same change.
86+
87+
The specification is a set of versioned Markdown scenario documents. Each
88+
document gives a stable scenario identifier, preconditions, ordered operations,
89+
expected observable results, cleanup expectations, and permitted SDK-specific
90+
variations. Markdown is the normative contract because it keeps the behavior
91+
reviewable without introducing a parser or a general gateway workflow language.
92+
93+
Adapters translate the Markdown requirements into native test code. A future
94+
machine-readable companion may reduce duplicated fixture data or produce
95+
coverage reports, but it must derive from the Markdown contract and is not
96+
required for the initial implementation.
97+
98+
### Fixture ownership and isolation
99+
100+
In this RFC, a fixture is predefined test data or setup that makes a scenario
101+
repeatable. Fixtures have three ownership levels:
102+
103+
| Fixture type | Owner | Examples | Reuse rule |
104+
| --- | --- | --- | --- |
105+
| Conformance fixture | Shared scenario documents | A standard sandbox policy, an exec command, expected exit code, placeholder expectation, or expected `NotFound` result. | Shared by every SDK adapter that implements the scenario. |
106+
| Adapter fixture | One SDK adapter | Go `TestMain`, a configured Go client, context deadlines, helper functions, and language-native assertions. | Local to that SDK because its runtime and testing idioms differ. |
107+
| Live test resource | One test invocation | A sandbox, provider, workspace, credential value, or temporary file created while a test runs. | Never shared between parallel tests; each test uses a unique name and cleans it up. |
108+
109+
The shared Markdown scenario is authoritative for conformance fixtures. It
110+
states the inputs and observations each SDK must use or verify, such as a
111+
failing command's exit code and stderr content. It does not prescribe how Go,
112+
Python, Rust, or TypeScript construct a client or express assertions.
113+
114+
Adapter fixtures remove repeated language-specific setup without crossing SDK
115+
boundaries. They may share a gateway connection within one test process only
116+
when the adapter can do so safely. They must not cause tests to share mutable
117+
gateway objects or credentials.
118+
119+
Live resources are deliberately isolated. Every test creates its own uniquely
120+
named sandbox, provider, and workspace as needed; it registers bounded cleanup
121+
before later operations can fail. A scenario may define a synthetic credential
122+
value for placeholder testing, but the adapter creates the provider that holds
123+
it for that invocation and must verify that the raw value is never returned to
124+
the sandbox.
125+
126+
### Native SDK adapters
127+
128+
Each supported SDK owns a native adapter and test entry point under `e2e/`.
129+
An adapter loads the shared scenarios, performs the corresponding SDK calls,
130+
and checks the standardized observations using that language's normal test
131+
framework. It may add language-specific assertions where they validate the SDK
132+
surface, such as Go context cancellation or TypeScript typing, without changing
133+
the shared contract.
134+
135+
The adapters have a common responsibility:
136+
137+
- create unique test resources and register cleanup before making later calls;
138+
- use bounded operation and cleanup deadlines;
139+
- tolerate documented asynchronous deletion by polling only the specified
140+
observable state;
141+
- emit enough resource identity and gateway diagnostics to make CI failures
142+
actionable;
143+
- run only when the harness explicitly supplies a reachable gateway.
144+
145+
The Go adapter lives in `e2e/go/` as a separate Go module with a local
146+
`replace` directive to `sdk/go/`. Its files use the `e2e` build tag. The related
147+
draft PR demonstrates this shape and currently uses the Podman gateway harness,
148+
but it must be rebased, moved to the Docker default lane, and adjusted to the
149+
shared scenarios before it is accepted.
150+
151+
```mermaid
152+
flowchart LR
153+
SPEC[Versioned SDK scenarios] --> GO[Go adapter]
154+
SPEC --> PY[Python adapter]
155+
SPEC --> TS[TypeScript adapter]
156+
SPEC --> RS[Rust SDK adapter]
157+
GO --> GW[Real gateway]
158+
PY --> GW
159+
TS --> GW
160+
RS --> GW
161+
CLI[CLI conformance runner] --> GW
162+
```
163+
164+
### Gateway harness and runtime coverage
165+
166+
The default SDK conformance lane starts one Docker-backed gateway through the
167+
existing `e2e/with-docker-gateway.sh` harness. This gives all SDK adapters one
168+
repeatable local command and prevents the general `mise run e2e` task from
169+
implicitly requiring both Docker and Podman.
170+
171+
Runtime-specific SDK scenarios are allowed when the expected result depends on
172+
a compute driver or deployment mode. They belong in explicitly named tasks and
173+
CI jobs, such as `e2e:go:podman`; they do not redefine the portable contract.
174+
The initial Go suite therefore runs in the Docker default lane. A Podman lane
175+
may be added later to validate its harness and driver behavior, but is not a
176+
prerequisite for the shared SDK contract.
177+
178+
Every CI lane must capture gateway logs on failure. Per-language test timeouts
179+
must be shorter than the job timeout and should report the last observed state
180+
instead of relying on a global test-framework timeout.
181+
182+
### Relationship to existing tests
183+
184+
`openshell-conformance` remains the portable CLI conformance suite. It invokes
185+
the `openshell` binary and validates command-line behavior, so it is not an SDK
186+
adapter and does not consume SDK scenarios.
187+
188+
SDK unit tests continue to validate conversion details, retry behavior, and
189+
language-specific ergonomics cheaply. Existing Rust and Python E2E tests remain
190+
valid. Their scenarios should be mapped to the shared specification over time;
191+
the project should not rewrite stable coverage merely to satisfy a new layout.
192+
193+
The conformance specification is the source of truth for portable SDK behavior.
194+
Its scenario identifiers, required observations, and capability exceptions are
195+
reviewed as compatibility changes.
196+
197+
## Implementation plan
198+
199+
1. Add the version-one Markdown scenario documents and a short contributor
200+
guide under `e2e/conformance/sdk/`. Start with sandbox, exec, provider, and
201+
workspace workflows defined above.
202+
2. Rebase the Go prototype onto current `main`, move it to the Docker gateway
203+
harness, and implement it as the first adapter. Strengthen its assertions to
204+
check returned resource state, readiness state, provider read/list/update,
205+
failed exec behavior, and missing-sandbox errors.
206+
3. Add `mise run e2e:go` and a focused Go CI lane. Keep it out of the aggregate
207+
`e2e` task until the Docker default path is in place; then add it to the
208+
aggregate task.
209+
4. Add adapters for the Rust SDK, Python SDK, and TypeScript SDK. Migrate or
210+
map existing E2E coverage where it matches a scenario, retaining
211+
language-specific tests where it does not.
212+
5. Add runtime-specific lanes only for scenarios whose expected behavior varies
213+
by runtime. Document supported capability exceptions and their rationale.
214+
6. Update the relevant architecture and contributor documentation when the
215+
suite is accepted and implemented.
216+
217+
## Risks
218+
219+
- A broad DSL could become a second orchestration framework. Markdown keeps the
220+
contract descriptive, while adapters retain control flow in native code.
221+
- The contract could flatten meaningful language differences. Shared scenarios
222+
define gateway-visible outcomes, while each SDK retains its own type, error,
223+
and cancellation assertions.
224+
- Running every SDK against every runtime would make CI too slow and flaky. The
225+
default lane uses Docker and adds specialized lanes only for
226+
runtime-dependent behavior.
227+
- Asynchronous cleanup can leave leaked sandboxes or providers after failures.
228+
Adapters must register bounded cleanup early and failure diagnostics must
229+
identify resources for manual recovery.
230+
- A scenario may accidentally codify a gateway implementation detail. Reviews
231+
should specify observable behavior and avoid asserting storage internals,
232+
polling intervals, or transport implementation details.
233+
234+
## Alternatives
235+
236+
### Keep a Go-only E2E suite
237+
238+
This delivers value quickly and the related PR is a useful starting point.
239+
However, it does not establish a shared SDK compatibility contract, leaving
240+
other bindings to independently choose what they test. It is retained as the
241+
first implementation phase, but not as the final design.
242+
243+
### Put all E2E tests inside each SDK directory
244+
245+
Keeping tests beside SDK code simplifies ownership, but makes a shared scenario
246+
catalog and common gateway-harness conventions harder to discover and reuse.
247+
The separate `e2e/` adapters make the real-gateway boundary explicit while SDK
248+
unit and integration tests remain near their implementation.
249+
250+
### Extend the existing CLI conformance runner
251+
252+
The runner already has reusable plans and lifecycle scenarios, but it executes
253+
the CLI. Adding language SDK invocation would couple it to multiple language
254+
runtimes and would still not validate SDK-native APIs. Shared scenario concepts
255+
can align, while the runners remain separate.
256+
257+
### Use only generated-protobuf compatibility checks
258+
259+
Schema checks detect wire changes, but cannot verify SDK conversions, readiness
260+
helpers, credential-placeholder safety, cleanup behavior, or error
261+
classification against a running gateway.
262+
263+
## Prior art
264+
265+
- `crates/openshell-conformance` provides a versioned plan and registered
266+
scenario model for CLI conformance. This RFC adopts the idea of explicit,
267+
reviewable scenarios while retaining SDK-native execution.
268+
- The existing `e2e/with-docker-gateway.sh` harness supplies a repeatable real
269+
gateway lifecycle for Rust and Python E2E tests. It is the portable default
270+
for the SDK suite.
271+
- The Go SDK's fake client and bufconn tests demonstrate the complementary
272+
unit-test layer. They remain appropriate for fast, isolated tests that do
273+
not need a real gateway.
274+
- The related Go E2E draft demonstrates parallel-safe resource naming,
275+
placeholder assertions, and eventual-deletion handling that the first adapter
276+
should preserve.
277+
278+
## Open questions
279+
280+
- When, if ever, should repeated fixture data or coverage reporting justify a
281+
machine-readable companion to the Markdown scenarios?
282+
- Which SDKs are in the initial supported-adapter set? This RFC assumes Go,
283+
Rust, Python, and TypeScript, subject to confirmation of release support.
284+
- Should independently released SDKs run the same scenarios against the
285+
latest gateway only, a release compatibility range, or both?
286+
- Which provider type can provide deterministic placeholder coverage without
287+
requiring a live external credential service in every CI lane?
288+
- Should scenario results be emitted as a machine-readable report for release
289+
qualification, or are native test reports sufficient initially?

0 commit comments

Comments
 (0)