Skip to content

Commit 4020303

Browse files
authored
Merge pull request #33 from RenaCode/feat/vm-only
Etap 2: jeden układ (maszyna Lima + SMB), usunięty układ z obrazem dysku
2 parents ed28473 + efccecb commit 4020303

100 files changed

Lines changed: 2568 additions & 11459 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 77 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -7,35 +7,35 @@
77
Native Time Machine, backed by Google Drive. No external disk, no NAS, nothing
88
plugged in at home.
99

10-
Time Machine writes to a disk image that macOS mounts like any other volume. The
11-
image's contents live on Google Drive, behind a local write buffer. Backups land
12-
in the buffer at SSD speed and drain to the cloud in the background, so a dropped
13-
connection stalls the upload instead of interrupting the backup.
10+
Time Machine backs up over SMB to a small Linux machine that runs on this Mac
11+
([Lima](https://lima-vm.io), Apple Virtualization). The machine keeps the
12+
backup's small files on its own disk and its bands on Google Drive, behind a
13+
local upload buffer: backups land in the buffer at disk speed and drain to the
14+
cloud in the background, so a dropped connection stalls the upload instead of
15+
interrupting the backup.
1416

1517
```
16-
Time Machine
17-
-> /Volumes/CloudMachine attached sparsebundle; Time Machine sees plain APFS
18-
-> ~/.cloudmachine/drive rclone mount over FUSE-T
19-
-> ~/.cloudmachine/cache 100 GB write buffer
20-
-> gdrive:CloudMachine/mac-studio Google Drive
18+
Time Machine --SMB--> machine cmsmb (Lima)
19+
Samba /srv/tm-local/<Computer>.sparsebundle image metadata, local
20+
bands/ -> punchfs -> rclone mount 120 GB upload buffer
21+
gdrive:CloudMachine/<folder>-tm Google Drive
2122
```
2223

23-
**No network filesystem sits in the write path.** That is the whole point.
24-
Time Machine over SMB to a NAS or a cloud share is the common approach and the
25-
fragile one — a sparsebundle written directly over a network link corrupts when
26-
the link drops. Here Time Machine talks to a locally attached image and never
27-
knows the cloud exists.
24+
Why a machine at all: macOS 27 no longer accepts a disk image as a new Time
25+
Machine destination (`tmutil setdestination` answers error 45), and refuses SMB
26+
to the Mac's own address. A share served from a machine on the Mac is what Time
27+
Machine still accepts. Why each layer inside it is there: [`vm/README.md`](vm/README.md).
2828

2929
---
3030

3131
## Requirements
3232

33-
- macOS 14 (Sonoma) or newer, and an administrator password for one step
34-
(pointing Time Machine at CloudMachine).
33+
- macOS 14 (Sonoma) or newer, Apple silicon, and an administrator password for
34+
one step (pointing Time Machine at the machine).
3535
- A Google account with room to spare.
36-
- [Homebrew](https://brew.sh).
37-
- Nothing else at runtime. CloudMachine installs its own `rclone` and its own
38-
copy of FUSE-T.
36+
- [Homebrew](https://brew.sh) and Lima from it (`brew install lima`).
37+
- About 200 GB of free disk: the machine's disk file grows with the upload
38+
buffer.
3939

4040
The menu-bar app, CLI output and notifications follow the system language:
4141
Polish when Polish is the first preferred language, English otherwise.
@@ -47,25 +47,9 @@ so they read the same whoever sends them to you.
4747
- **Not notarised.** Releases are signed with a self-signed certificate, not
4848
with an Apple Developer ID. The Homebrew cask clears the quarantine flag; a
4949
DMG downloaded by hand gets Gatekeeper's "unidentified developer" warning.
50-
51-
### macOS 27: Time Machine over SMB to a local machine (preview)
52-
53-
macOS 27 no longer accepts a disk image as a new Time Machine destination
54-
(`tmutil setdestination` answers error 45), so the layout above cannot be set
55-
up from scratch there. The replacement, being rolled out next to it: a small
56-
Linux machine on this Mac serves Time Machine an SMB share, and the backup's
57-
bands go from it to Google Drive (details in [`vm/README.md`](vm/README.md)).
58-
59-
- Requires [Lima](https://lima-vm.io) from Homebrew (`brew install lima`) and
60-
a connected Google Drive (`configure-remote`).
61-
- `cloudmachine-agent vm-setup` creates and starts the machine and prints
62-
a `sudo tmutil setdestination smb://...` command - the one step that needs
63-
an administrator, once.
64-
- When Time Machine has created its empty image, stop the backup and run
65-
`cloudmachine-agent vm-prepare-bundle` (32 MB bands, moved to Drive).
66-
- `cloudmachine-agent vm-status` shows the machine; `vm-setup --install-agents`
67-
adds the agents that start it at login and pause Time Machine while the
68-
upload buffer is full.
50+
- **Google's daily upload limit and a full Drive are not named in the app.**
51+
rclone's log that says so lives in the machine; what the app does show is
52+
the backlog growing and the guard pausing Time Machine.
6953

7054
---
7155

@@ -78,33 +62,43 @@ open -a CloudMachine
7862

7963
CloudMachine lives in the menu bar. Its window opens on a **Required Setup
8064
Steps** card listing what is left to do on this Mac, in order, each with a
81-
button — or, for the two steps an app cannot do, a command to copy:
65+
button — or, for the steps an app cannot do, a command to copy:
8266

83-
1. **Install rclone** and **Install FUSE-T.** CloudMachine downloads its own
84-
copies; nothing else is installed system-wide.
67+
1. **Install rclone.** CloudMachine downloads its own copy; it signs in to
68+
Google Drive and writes the configuration the machine gets a copy of.
8569
2. **Connect Google Drive** — a command to run in Terminal. It opens Google's
8670
sign-in in the browser and waits for your approval. Enter your own OAuth
8771
credentials first (see [below](#your-own-google-oauth-credentials)).
88-
3. **Grant Full Disk Access** — opens the right pane of System Settings.
72+
3. **Install Lima** — `brew install lima`, to copy.
73+
4. **Set up the machine** — creates and starts `cmsmb` (the first start
74+
downloads Ubuntu and installs packages, several minutes), copies rclone's
75+
configuration into it and sets its SMB password.
76+
5. **Grant Full Disk Access** — opens the right pane of System Settings.
8977
Without it CloudMachine cannot read when Time Machine last *finished* a
9078
backup, which is the one check that matters.
91-
4. **Install agents** — the launchd agents that keep the Drive mounted, attach
92-
the image and watch the backup.
93-
5. **Create image**, then **Attach image** — the backup image on Google Drive.
94-
The size is a ceiling, not an allocation: the image is sparse, and Drive
95-
only holds what has been written.
96-
6. **Point Time Machine at CloudMachine** — a `sudo tmutil setdestination`
97-
command to copy, because only an administrator can change it.
79+
6. **Install agents** — the launchd agents that start the machine at login,
80+
pause Time Machine while the upload buffer is full, and watch the backup.
81+
7. **Point Time Machine at the machine** — a `sudo tmutil setdestination
82+
smb://tm:…@<address>/CloudMachine` command. The card shows the password
83+
masked; **Copy** puts the real command on the clipboard.
84+
8. **Convert the image** — once Time Machine has created its first, empty
85+
image, stop that backup (`tmutil stopbackup`) and press the button
86+
(`cloudmachine-agent vm-prepare-bundle`): the image gets 32 MB bands, and
87+
the bands move to Google Drive.
9888

9989
When the card disappears, setup is done. Turn on automatic backups in System
10090
Settings → General → Time Machine, or run `sudo tmutil enable`.
10191

92+
The same from Terminal: `install-rclone`, `configure-remote`, `vm-setup
93+
--install-agents` (prints the `tmutil` command), then `vm-prepare-bundle`;
94+
`vm-status` shows everything at once.
95+
10296
### Several Macs, one Google account
10397

10498
Install CloudMachine on each Mac and go through the same steps; they can all
10599
use the same Google account and the same OAuth credentials. Each Mac backs up
106-
into its own folder, `gdrive:CloudMachine/<folder>`, holding
107-
`<folder>.sparsebundle`, so their backups never mix.
100+
into its own folder, `gdrive:CloudMachine/<folder>-tm`, so their backups never
101+
mix.
108102

109103
The folder name is chosen once, in the **Connect Google Drive** step: the card
110104
has a **Folder on Google Drive** field, filled in from the computer name, and
@@ -122,33 +116,38 @@ It works two ways:
122116

123117
- **Time Machine quota.** The card gives a `sudo tmutil setquota` command to
124118
copy; Time Machine then deletes this Mac's oldest backups to stay under it.
125-
The quota is 70% of the limit, because the image on Drive grows larger than
126-
the backup inside it (see `renacode-infra: docs/projekty/cloudmachine/design.md`).
119+
The quota is 70% of the limit, because the bands on Drive grow larger than
120+
the backup inside them (see `renacode-infra: docs/projekty/cloudmachine/design.md`).
127121
- **An alarm on the real usage.** The watchdog measures this Mac's folder on
128122
Drive every few hours and warns from 90% of the limit, and again above it.
129-
The card and `drive-status` show the usage.
123+
The card and `vm-status` show the usage.
130124

131125
### Upgrading and uninstalling
132126

133-
`brew upgrade --cask renacode/tap/cloudmachine` replaces the app without
134-
restarting the Google Drive mount. The cask in the tap can be a version
135-
behind the latest GitHub Release: it is updated by a pull request after each
136-
release (see `packaging/README.md`).
137-
When Homebrew reopens the app, it reloads the background agents so they run
138-
the new version; `cloudmachine-agent drive-status` shows `Agents: OK`. Changes
139-
to the mount itself wait for its next start. One such change is rclone's
140-
control interface moving to a private socket: until then the old mount takes
141-
commands on `127.0.0.1:5572` without a password, and the app window shows a red
142-
card, the menu bar an alert, and the watchdog a notification (once, then after
143-
1, 2, 4 and every 7 days). **Restart the mount** on that card - or
144-
`cloudmachine-agent restart-mount` - detaches the image, waits until everything
145-
has reached Google Drive, restarts the mount and attaches the image again; it
146-
refuses while Time Machine is backing up. Restarting the Mac after
147-
`prepare-shutdown` does the same. If a new version does not show up, run
148-
`brew update` first - Homebrew refreshes the tap only now and then. Neither `brew uninstall` nor
149-
`--zap` touches the launchd agents or the upload buffer in `~/.cloudmachine`,
150-
which may hold backups that have not reached Google Drive yet. Run
151-
`cloudmachine-agent prepare-shutdown` before uninstalling.
127+
`brew upgrade --cask renacode/tap/cloudmachine` replaces the app; the machine
128+
keeps running. The cask in the tap can be a version behind the latest GitHub
129+
Release: it is updated by a pull request after each release (see
130+
`packaging/README.md`). If a new version does not show up, run `brew update`
131+
first - Homebrew refreshes the tap only now and then. When Homebrew reopens
132+
the app, it reloads the background agents so they run the new version;
133+
`cloudmachine-agent vm-status` shows `Agents: OK`.
134+
135+
**Coming from the disk-image layout** (an rclone mount on the Mac and an image
136+
in `/Volumes/CloudMachine`): the app removes its launchd agents
137+
(`gdrive-buffer`, `gdrive-attach`, `buffer-guard`) on its first launch and
138+
lists in `~/Library/Logs/CloudMachine/cloudmachine.log` what that layout left
139+
behind - `~/.cloudmachine/cache`, `drive`, `fuse`, the old image in
140+
`gdrive:CloudMachine/<folder>` - without deleting any of it. The new layout
141+
starts a new backup in `<folder>-tm`; delete the old one when you no longer
142+
need to restore from it.
143+
144+
Before restarting the Mac, **Prepare for restart** in the menu (or
145+
`cloudmachine-agent prepare-shutdown`) stops the backup, gives the upload up
146+
to 30 minutes, and shuts the machine down cleanly; it starts again at the next
147+
login. Neither `brew uninstall` nor `--zap` touches the launchd agents or the
148+
machine, whose buffer may hold backups that have not reached Google Drive yet:
149+
run `prepare-shutdown`, then `limactl delete cmsmb` and remove
150+
`~/Library/LaunchAgents/com.renacode.cloudmachine.*.plist`.
152151

153152
### Your own Google OAuth credentials
154153

@@ -166,13 +165,13 @@ compete with every other rclone user.
166165
The connection uses scope `drive.file`, which grants access only to files this
167166
application itself created. Full `drive` scope would hand out read, write and
168167
**delete** over the entire Google account, which is far more than a folder of
169-
disk-image bands needs — especially with `--drive-use-trash=false`, where a
168+
backup bands needs — especially with `--drive-use-trash=false`, where a
170169
delete has no bin to recover from.
171170

172171
On a Mac that is already connected, keep the connection it has. A new sign-in
173172
with different credentials sees only what it created itself, so the existing
174173
backup would vanish from view; `configure-remote` refuses to replace a working
175-
connection unless told twice, and checks the new token against the image first
174+
connection unless told twice, and checks the new token against the backup first
176175
(see `renacode-infra: docs/projekty/cloudmachine/setup-cli.md`, "Replacing the connection").
177176

178177
---
@@ -184,17 +183,16 @@ repository (`renacode-infra: docs/projekty/cloudmachine/`):
184183

185184
- Running CloudMachine (`operations.md`) — checking from Terminal,
186185
knowing when backups stop, reading the logs, before rebooting.
187-
- Restoring (`restoring.md`) — getting files back, and checking the image.
186+
- Restoring (`restoring.md`) — getting files back.
188187
- How it works, and what it costs (`design.md`) — measured traffic and
189188
storage, and why each piece is the way it is.
190189
- Setting up from Terminal (`setup-cli.md`) — the same setup without the app.
191-
- Building from source (`building.md`) — local builds, tests, releases,
192-
measurement harnesses.
190+
- Building from source (`building.md`) — local builds, tests, releases.
193191

194192
---
195193

196194
## Licence
197195

198-
MIT. FUSE-T and rclone keep their own licences. One file is GPLv2:
196+
MIT. rclone, Lima and Samba keep their own licences. One file is GPLv2:
199197
`vm/punchfs/punchfs.c`, based on libfuse's passthrough example (see
200198
`vm/punchfs/README.md`).

‎launchd/com.renacode.cloudmachine.app.plist.template‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,8 @@
2121
would launch it over and over. The user closing the window should stay
2222
closed, not be an invitation to restart.
2323
24-
The backups themselves do NOT depend on this app - the gdrive-buffer,
25-
gdrive-attach and buffer-guard agents do them. This is only a viewer.
24+
The backups themselves do NOT depend on this app - the machine (started
25+
by vm-start) and the vm-guard agent do them. This is only a viewer.
2626
-->
2727
<key>StandardOutPath</key>
2828
<string>__CM_LOG_DIR__/launchd-app.out.log</string>

‎launchd/com.renacode.cloudmachine.backup-health.plist.template‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,9 @@
2323
Every 30 minutes, with an hourly cycle and a 3 h threshold - often enough
2424
that a broken cycle does not sit unnoticed for a day, and rarely enough to
2525
cost nothing. The watchdog reads the local Time Machine preferences file,
26-
NOT the Google Drive mount: a supervisor that hangs together with what it
27-
supervises goes silent exactly when it should speak.
26+
not the backup destination, and every question to the machine has a time
27+
limit: a supervisor that hangs together with what it supervises goes
28+
silent exactly when it should speak.
2829
-->
2930
<key>StartInterval</key>
3031
<integer>1800</integer>

‎launchd/com.renacode.cloudmachine.buffer-guard.plist.template‎

Lines changed: 0 additions & 28 deletions
This file was deleted.

‎launchd/com.renacode.cloudmachine.gdrive-attach.plist.template‎

Lines changed: 0 additions & 32 deletions
This file was deleted.

‎launchd/com.renacode.cloudmachine.gdrive-buffer.plist.template‎

Lines changed: 0 additions & 34 deletions
This file was deleted.

launchd/vm/com.renacode.cloudmachine.vm-guard.plist.template renamed to launchd/com.renacode.cloudmachine.vm-guard.plist.template

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@
66
<string>com.renacode.cloudmachine.vm-guard</string>
77
<!--
88
The machine's buffer guard (`vm-guard`): one check per run, so
9-
StartInterval rather than KeepAlive. Installed only by
10-
`cloudmachine-agent vm-setup --install-agents`, never by install-launchd.
9+
StartInterval rather than KeepAlive. Installed by install-launchd
10+
(also run by `vm-setup --install-agents`).
1111
-->
1212
<key>ProgramArguments</key>
1313
<array>

launchd/vm/com.renacode.cloudmachine.vm-start.plist.template renamed to launchd/com.renacode.cloudmachine.vm-start.plist.template

File renamed without changes.

0 commit comments

Comments
 (0)