|
| 1 | +# Testing a deployment move between two nodes (zmount + rootfs) |
| 2 | + |
| 3 | +This is a dev runbook for the `zos.deployment.transfer` / `prepare` / `start` feature |
| 4 | +(branch `feat/deployment-transfer`). It moves a zmachine (mycelium network + zmount) |
| 5 | +from **node A** to **node B**, preserving zmount data *and* rootfs changes, via S3. |
| 6 | + |
| 7 | +## The flow at a glance |
| 8 | + |
| 9 | +``` |
| 10 | +node A S3 (MinIO) node B |
| 11 | + │ transfer (PUT) ──────────▶ zmount.raw │ |
| 12 | + │ transfer (PUT) ──────────▶ rootfs.tar │ |
| 13 | + │ │ prepare (GET) ◀────────┤ creates net+zmount+rootfs, pulls data |
| 14 | + │ └───────────────────────▶ │ (VM NOT booted yet) |
| 15 | + │ start ─────────┤ boots the VM on B |
| 16 | + ▼ cancel old contract ▼ VM running on B |
| 17 | +``` |
| 18 | + |
| 19 | +`transfer` pauses the source VM first, so the copy is consistent. |
| 20 | + |
| 21 | +## Prerequisites |
| 22 | + |
| 23 | +1. **Two zos-light nodes** A and B on the same farm/network, both reachable over RMB, |
| 24 | + both running a node image **built from this branch** (see "Build the node image"). |
| 25 | +2. **A twin + mnemonic** you control (the deployment owner), KYC-verified on the target |
| 26 | + chain env — `prepare` runs the same validation as `deploy` (twin match, signature, |
| 27 | + contract hash). Dev/QA net is fine. |
| 28 | +3. **A MinIO (or S3) bucket** reachable from both nodes, plus the `mc` client to mint |
| 29 | + presigned URLs. |
| 30 | +4. **A running source deployment on A**: a mycelium network + a zmount + a zmachine that |
| 31 | + mounts the zmount. Write a marker file into both the rootfs and the zmount so you can |
| 32 | + prove they survived the move. |
| 33 | + |
| 34 | +## Build the node image |
| 35 | + |
| 36 | +zoslight is already wired to this branch via a local replace (added during implementation): |
| 37 | + |
| 38 | +``` |
| 39 | +# in the zoslight repo |
| 40 | +grep 'zos_base =>' go.mod |
| 41 | +# replace github.com/threefoldtech/zos_base => /home/afouda/code/github/threefoldtech/zosbase |
| 42 | +go build ./... # sanity: compiles against the branch |
| 43 | +``` |
| 44 | + |
| 45 | +Build/flash the node image the way you normally do for zoslight (the replace makes the |
| 46 | +image include these RMB endpoints). For a throwaway test you can also run the affected |
| 47 | +daemons (api_gateway, provisiond, storaged) from source on a dev node. |
| 48 | + |
| 49 | +> Before pushing the branch, drop the local `replace` and pin a real `zos_base` |
| 50 | +> pseudo-version instead — the path replace only works on your machine. |
| 51 | +
|
| 52 | +## Mint presigned URLs (MinIO example) |
| 53 | + |
| 54 | +```bash |
| 55 | +mc alias set m http://MINIO:9000 ACCESS SECRET |
| 56 | +mc mb m/zos-move |
| 57 | + |
| 58 | +# uploads (used by node A on transfer) — PUT |
| 59 | +ZMOUNT_PUT=$(mc share upload --expire 12h m/zos-move/zmount.raw | awk '/share:/{print $2}') |
| 60 | +ROOTFS_PUT=$(mc share upload --expire 12h m/zos-move/rootfs.tar | awk '/share:/{print $2}') |
| 61 | +# downloads (used by node B on prepare) — GET |
| 62 | +ZMOUNT_GET=$(mc share download --expire 12h m/zos-move/zmount.raw | awk '/share:/{print $2}') |
| 63 | +ROOTFS_GET=$(mc share download --expire 12h m/zos-move/rootfs.tar | awk '/share:/{print $2}') |
| 64 | +``` |
| 65 | + |
| 66 | +The node streams a definite Content-Length, so plain presigned PUT/GET work (no chunked). |
| 67 | + |
| 68 | +## Driver program (uses the client SDK added on this branch) |
| 69 | + |
| 70 | +`client.NodeClient` gained `DeploymentTransfer`, `DeploymentPrepare`, `DeploymentStart`. |
| 71 | +Workload names below (`"data"`, `"vm"`) are the `Name` fields in *your* deployment. |
| 72 | + |
| 73 | +```go |
| 74 | +package main |
| 75 | + |
| 76 | +import ( |
| 77 | + "context" |
| 78 | + "time" |
| 79 | + |
| 80 | + "github.com/threefoldtech/zos_base/client" |
| 81 | + "github.com/threefoldtech/zos_base/pkg/gridtypes" |
| 82 | + "github.com/threefoldtech/zos_sdk_go/rmb-sdk-go" |
| 83 | +) |
| 84 | + |
| 85 | +func main() { |
| 86 | + cl, err := rmb.Default() // or a peer-backed rmb.Client for your env |
| 87 | + if err != nil { panic(err) } |
| 88 | + |
| 89 | + const ( |
| 90 | + twin = 1234 // your twin id |
| 91 | + nodeATwin = 1111 // node A twin id |
| 92 | + nodeBTwin = 2222 // node B twin id |
| 93 | + contractA = 100 // running contract on A |
| 94 | + ) |
| 95 | + |
| 96 | + nodeA := client.NewNodeClient(nodeATwin, cl) |
| 97 | + nodeB := client.NewNodeClient(nodeBTwin, cl) |
| 98 | + ctx := context.Background() |
| 99 | + |
| 100 | + // 0) grab the running deployment from A (you'll reuse it for prepare on B) |
| 101 | + dl, err := nodeA.DeploymentGet(ctx, contractA) |
| 102 | + if err != nil { panic(err) } |
| 103 | + |
| 104 | + // 1) A: pause + upload zmount + rootfs to S3 |
| 105 | + must(nodeA.DeploymentTransfer(ctx, contractA, []client.TransferItem{ |
| 106 | + {WorkloadName: "data", URL: ZMOUNT_PUT}, // zmount |
| 107 | + {WorkloadName: "vm", URL: ROOTFS_PUT}, // zmachine rootfs |
| 108 | + })) |
| 109 | + |
| 110 | + // 2) stage a contract on B with the SAME deployment hash (see note below), |
| 111 | + // set dl.ContractID = <new B contract>, dl.Version = 0. |
| 112 | + dl.ContractID = 200 |
| 113 | + dl.Version = 0 |
| 114 | + |
| 115 | + // 3) B: prepare (provision net+zmount, skip VM) and pull data back |
| 116 | + must(nodeB.DeploymentPrepare(ctx, dl, []client.TransferItem{ |
| 117 | + {WorkloadName: "data", URL: ZMOUNT_GET}, |
| 118 | + {WorkloadName: "vm", URL: ROOTFS_GET, Size: rootfsSize}, // rootfs volume size |
| 119 | + })) |
| 120 | + |
| 121 | + // 4) B: boot the VM |
| 122 | + time.Sleep(2 * time.Second) |
| 123 | + must(nodeB.DeploymentStart(ctx, dl.ContractID)) |
| 124 | + |
| 125 | + // 5) A: cancel the old contract on chain to decommission the source |
| 126 | +} |
| 127 | + |
| 128 | +func must(err error) { if err != nil { panic(err) } } |
| 129 | +``` |
| 130 | + |
| 131 | +### The on-chain contract for B (the manual bit) |
| 132 | + |
| 133 | +`prepare` validates against a node contract for **B** whose `DeploymentHash` matches the |
| 134 | +deployment. In production the grid/tfchain orchestration creates it; for a manual test, |
| 135 | +create a node contract on B for the same hash (`dl.ChallengeHash()`) using your usual tool |
| 136 | +(tfchain client / `tfcmd` / grid client), then use that contract id as `dl.ContractID` |
| 137 | +in step 2. Same-contract migration (updating the existing contract's node id on chain) is |
| 138 | +a later refinement — for now use a fresh contract on B. |
| 139 | + |
| 140 | +## Verify |
| 141 | + |
| 142 | +- `nodeB.DeploymentGet(ctx, 200)` → zmount + network `StateOk`, zmachine `StateOk` after start. |
| 143 | +- On node B host: the zmount file exists at `/mnt/<pool>/vdisks/<twin>-200-data` and the |
| 144 | + rootfs volume at `/mnt/<pool>/rootfs:<twin>-200-vm/rw`. |
| 145 | +- Console into the VM (mycelium IP is derived from the deployment seed, so it's the same as |
| 146 | + on A) and confirm your **marker files** in both the rootfs and the mounted zmount. |
| 147 | + |
| 148 | +## Cleanup |
| 149 | + |
| 150 | +Cancel the source contract on A (normal decommission path); the node tears down the paused |
| 151 | +deployment. Delete the S3 objects. |
| 152 | + |
| 153 | +## Known limitations (this phase) |
| 154 | + |
| 155 | +- `transfer` pauses vCPUs but does not `fsync` the guest — tiny consistency risk; for a |
| 156 | + clean test, quiesce the app inside the VM before transferring. |
| 157 | +- The rootfs volume quota on B is set at `prepare` time from the `Size` you pass; make it |
| 158 | + match the VM's rootfs size. |
| 159 | +- Same-contract (in-place node reassignment) orchestration is not implemented — use a new |
| 160 | + contract on B as above. |
| 161 | +``` |
0 commit comments