Skip to content

Commit 4f81f4b

Browse files
authored
Merge pull request #32 from RenaCode/feat/vm-samba-time-machine
Etap 1: Time Machine przez SMB do maszyny Lima (obok starego układu)
2 parents 1f6278d + 4020303 commit 4f81f4b

117 files changed

Lines changed: 6481 additions & 11315 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/release.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ on:
4747
- "mac-app/Sources/**"
4848
- "mac-app/Resources/**"
4949
- "launchd/**"
50+
- "vm/**"
5051
- "config/**"
5152
- "packaging/homebrew/**"
5253
workflow_dispatch:

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ logs/
88
# rclone config (contains OAuth tokens) - never commit
99
rclone.conf
1010
*.conf
11+
# ...except the machine's templates (Samba, FUSE, systemd drop-ins) - no
12+
# secrets in them; rclone.conf stays ignored there too.
13+
!vm/**/*.conf
14+
vm/**/rclone.conf
1115

1216
# Generated launchd files (contain the user's local paths)
1317
launchd/*.plist

‎README.md‎

Lines changed: 79 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -7,35 +7,35 @@
77
Native Time Machine, backed by Google Drive. No external disk, no NAS, nothing
88
plugged in at home.
99

10-
Time Machine writes to a disk image that macOS mounts like any other volume. The
11-
image's contents live on Google Drive, behind a local write buffer. Backups land
12-
in the buffer at SSD speed and drain to the cloud in the background, so a dropped
13-
connection stalls the upload instead of interrupting the backup.
10+
Time Machine backs up over SMB to a small Linux machine that runs on this Mac
11+
([Lima](https://lima-vm.io), Apple Virtualization). The machine keeps the
12+
backup's small files on its own disk and its bands on Google Drive, behind a
13+
local upload buffer: backups land in the buffer at disk speed and drain to the
14+
cloud in the background, so a dropped connection stalls the upload instead of
15+
interrupting the backup.
1416

1517
```
16-
Time Machine
17-
-> /Volumes/CloudMachine attached sparsebundle; Time Machine sees plain APFS
18-
-> ~/.cloudmachine/drive rclone mount over FUSE-T
19-
-> ~/.cloudmachine/cache 100 GB write buffer
20-
-> gdrive:CloudMachine/mac-studio Google Drive
18+
Time Machine --SMB--> machine cmsmb (Lima)
19+
Samba /srv/tm-local/<Computer>.sparsebundle image metadata, local
20+
bands/ -> punchfs -> rclone mount 120 GB upload buffer
21+
gdrive:CloudMachine/<folder>-tm Google Drive
2122
```
2223

23-
**No network filesystem sits in the write path.** That is the whole point.
24-
Time Machine over SMB to a NAS or a cloud share is the common approach and the
25-
fragile one — a sparsebundle written directly over a network link corrupts when
26-
the link drops. Here Time Machine talks to a locally attached image and never
27-
knows the cloud exists.
24+
Why a machine at all: macOS 27 no longer accepts a disk image as a new Time
25+
Machine destination (`tmutil setdestination` answers error 45), and refuses SMB
26+
to the Mac's own address. A share served from a machine on the Mac is what Time
27+
Machine still accepts. Why each layer inside it is there: [`vm/README.md`](vm/README.md).
2828

2929
---
3030

3131
## Requirements
3232

33-
- macOS 14 (Sonoma) or newer, and an administrator password for one step
34-
(pointing Time Machine at CloudMachine).
33+
- macOS 14 (Sonoma) or newer, Apple silicon, and an administrator password for
34+
one step (pointing Time Machine at the machine).
3535
- A Google account with room to spare.
36-
- [Homebrew](https://brew.sh).
37-
- Nothing else at runtime. CloudMachine installs its own `rclone` and its own
38-
copy of FUSE-T.
36+
- [Homebrew](https://brew.sh) and Lima from it (`brew install lima`).
37+
- About 200 GB of free disk: the machine's disk file grows with the upload
38+
buffer.
3939

4040
The menu-bar app, CLI output and notifications follow the system language:
4141
Polish when Polish is the first preferred language, English otherwise.
@@ -47,6 +47,9 @@ so they read the same whoever sends them to you.
4747
- **Not notarised.** Releases are signed with a self-signed certificate, not
4848
with an Apple Developer ID. The Homebrew cask clears the quarantine flag; a
4949
DMG downloaded by hand gets Gatekeeper's "unidentified developer" warning.
50+
- **Google's daily upload limit and a full Drive are not named in the app.**
51+
rclone's log that says so lives in the machine; what the app does show is
52+
the backlog growing and the guard pausing Time Machine.
5053

5154
---
5255

@@ -59,33 +62,43 @@ open -a CloudMachine
5962

6063
CloudMachine lives in the menu bar. Its window opens on a **Required Setup
6164
Steps** card listing what is left to do on this Mac, in order, each with a
62-
button — or, for the two steps an app cannot do, a command to copy:
65+
button — or, for the steps an app cannot do, a command to copy:
6366

64-
1. **Install rclone** and **Install FUSE-T.** CloudMachine downloads its own
65-
copies; nothing else is installed system-wide.
67+
1. **Install rclone.** CloudMachine downloads its own copy; it signs in to
68+
Google Drive and writes the configuration the machine gets a copy of.
6669
2. **Connect Google Drive** — a command to run in Terminal. It opens Google's
6770
sign-in in the browser and waits for your approval. Enter your own OAuth
6871
credentials first (see [below](#your-own-google-oauth-credentials)).
69-
3. **Grant Full Disk Access** — opens the right pane of System Settings.
72+
3. **Install Lima** — `brew install lima`, to copy.
73+
4. **Set up the machine** — creates and starts `cmsmb` (the first start
74+
downloads Ubuntu and installs packages, several minutes), copies rclone's
75+
configuration into it and sets its SMB password.
76+
5. **Grant Full Disk Access** — opens the right pane of System Settings.
7077
Without it CloudMachine cannot read when Time Machine last *finished* a
7178
backup, which is the one check that matters.
72-
4. **Install agents** — the launchd agents that keep the Drive mounted, attach
73-
the image and watch the backup.
74-
5. **Create image**, then **Attach image** — the backup image on Google Drive.
75-
The size is a ceiling, not an allocation: the image is sparse, and Drive
76-
only holds what has been written.
77-
6. **Point Time Machine at CloudMachine** — a `sudo tmutil setdestination`
78-
command to copy, because only an administrator can change it.
79+
6. **Install agents** — the launchd agents that start the machine at login,
80+
pause Time Machine while the upload buffer is full, and watch the backup.
81+
7. **Point Time Machine at the machine** — a `sudo tmutil setdestination
82+
smb://tm:…@<address>/CloudMachine` command. The card shows the password
83+
masked; **Copy** puts the real command on the clipboard.
84+
8. **Convert the image** — once Time Machine has created its first, empty
85+
image, stop that backup (`tmutil stopbackup`) and press the button
86+
(`cloudmachine-agent vm-prepare-bundle`): the image gets 32 MB bands, and
87+
the bands move to Google Drive.
7988

8089
When the card disappears, setup is done. Turn on automatic backups in System
8190
Settings → General → Time Machine, or run `sudo tmutil enable`.
8291

92+
The same from Terminal: `install-rclone`, `configure-remote`, `vm-setup
93+
--install-agents` (prints the `tmutil` command), then `vm-prepare-bundle`;
94+
`vm-status` shows everything at once.
95+
8396
### Several Macs, one Google account
8497

8598
Install CloudMachine on each Mac and go through the same steps; they can all
8699
use the same Google account and the same OAuth credentials. Each Mac backs up
87-
into its own folder, `gdrive:CloudMachine/<folder>`, holding
88-
`<folder>.sparsebundle`, so their backups never mix.
100+
into its own folder, `gdrive:CloudMachine/<folder>-tm`, so their backups never
101+
mix.
89102

90103
The folder name is chosen once, in the **Connect Google Drive** step: the card
91104
has a **Folder on Google Drive** field, filled in from the computer name, and
@@ -103,33 +116,38 @@ It works two ways:
103116

104117
- **Time Machine quota.** The card gives a `sudo tmutil setquota` command to
105118
copy; Time Machine then deletes this Mac's oldest backups to stay under it.
106-
The quota is 70% of the limit, because the image on Drive grows larger than
107-
the backup inside it (see `renacode-infra: docs/projekty/cloudmachine/design.md`).
119+
The quota is 70% of the limit, because the bands on Drive grow larger than
120+
the backup inside them (see `renacode-infra: docs/projekty/cloudmachine/design.md`).
108121
- **An alarm on the real usage.** The watchdog measures this Mac's folder on
109122
Drive every few hours and warns from 90% of the limit, and again above it.
110-
The card and `drive-status` show the usage.
123+
The card and `vm-status` show the usage.
111124

112125
### Upgrading and uninstalling
113126

114-
`brew upgrade --cask renacode/tap/cloudmachine` replaces the app without
115-
restarting the Google Drive mount. The cask in the tap can be a version
116-
behind the latest GitHub Release: it is updated by a pull request after each
117-
release (see `packaging/README.md`).
118-
When Homebrew reopens the app, it reloads the background agents so they run
119-
the new version; `cloudmachine-agent drive-status` shows `Agents: OK`. Changes
120-
to the mount itself wait for its next start. One such change is rclone's
121-
control interface moving to a private socket: until then the old mount takes
122-
commands on `127.0.0.1:5572` without a password, and the app window shows a red
123-
card, the menu bar an alert, and the watchdog a notification (once, then after
124-
1, 2, 4 and every 7 days). **Restart the mount** on that card - or
125-
`cloudmachine-agent restart-mount` - detaches the image, waits until everything
126-
has reached Google Drive, restarts the mount and attaches the image again; it
127-
refuses while Time Machine is backing up. Restarting the Mac after
128-
`prepare-shutdown` does the same. If a new version does not show up, run
129-
`brew update` first - Homebrew refreshes the tap only now and then. Neither `brew uninstall` nor
130-
`--zap` touches the launchd agents or the upload buffer in `~/.cloudmachine`,
131-
which may hold backups that have not reached Google Drive yet. Run
132-
`cloudmachine-agent prepare-shutdown` before uninstalling.
127+
`brew upgrade --cask renacode/tap/cloudmachine` replaces the app; the machine
128+
keeps running. The cask in the tap can be a version behind the latest GitHub
129+
Release: it is updated by a pull request after each release (see
130+
`packaging/README.md`). If a new version does not show up, run `brew update`
131+
first - Homebrew refreshes the tap only now and then. When Homebrew reopens
132+
the app, it reloads the background agents so they run the new version;
133+
`cloudmachine-agent vm-status` shows `Agents: OK`.
134+
135+
**Coming from the disk-image layout** (an rclone mount on the Mac and an image
136+
in `/Volumes/CloudMachine`): the app removes its launchd agents
137+
(`gdrive-buffer`, `gdrive-attach`, `buffer-guard`) on its first launch and
138+
lists in `~/Library/Logs/CloudMachine/cloudmachine.log` what that layout left
139+
behind - `~/.cloudmachine/cache`, `drive`, `fuse`, the old image in
140+
`gdrive:CloudMachine/<folder>` - without deleting any of it. The new layout
141+
starts a new backup in `<folder>-tm`; delete the old one when you no longer
142+
need to restore from it.
143+
144+
Before restarting the Mac, **Prepare for restart** in the menu (or
145+
`cloudmachine-agent prepare-shutdown`) stops the backup, gives the upload up
146+
to 30 minutes, and shuts the machine down cleanly; it starts again at the next
147+
login. Neither `brew uninstall` nor `--zap` touches the launchd agents or the
148+
machine, whose buffer may hold backups that have not reached Google Drive yet:
149+
run `prepare-shutdown`, then `limactl delete cmsmb` and remove
150+
`~/Library/LaunchAgents/com.renacode.cloudmachine.*.plist`.
133151

134152
### Your own Google OAuth credentials
135153

@@ -147,13 +165,13 @@ compete with every other rclone user.
147165
The connection uses scope `drive.file`, which grants access only to files this
148166
application itself created. Full `drive` scope would hand out read, write and
149167
**delete** over the entire Google account, which is far more than a folder of
150-
disk-image bands needs — especially with `--drive-use-trash=false`, where a
168+
backup bands needs — especially with `--drive-use-trash=false`, where a
151169
delete has no bin to recover from.
152170

153171
On a Mac that is already connected, keep the connection it has. A new sign-in
154172
with different credentials sees only what it created itself, so the existing
155173
backup would vanish from view; `configure-remote` refuses to replace a working
156-
connection unless told twice, and checks the new token against the image first
174+
connection unless told twice, and checks the new token against the backup first
157175
(see `renacode-infra: docs/projekty/cloudmachine/setup-cli.md`, "Replacing the connection").
158176

159177
---
@@ -165,15 +183,16 @@ repository (`renacode-infra: docs/projekty/cloudmachine/`):
165183

166184
- Running CloudMachine (`operations.md`) — checking from Terminal,
167185
knowing when backups stop, reading the logs, before rebooting.
168-
- Restoring (`restoring.md`) — getting files back, and checking the image.
186+
- Restoring (`restoring.md`) — getting files back.
169187
- How it works, and what it costs (`design.md`) — measured traffic and
170188
storage, and why each piece is the way it is.
171189
- Setting up from Terminal (`setup-cli.md`) — the same setup without the app.
172-
- Building from source (`building.md`) — local builds, tests, releases,
173-
measurement harnesses.
190+
- Building from source (`building.md`) — local builds, tests, releases.
174191

175192
---
176193

177194
## Licence
178195

179-
MIT. FUSE-T and rclone keep their own licences.
196+
MIT. rclone, Lima and Samba keep their own licences. One file is GPLv2:
197+
`vm/punchfs/punchfs.c`, based on libfuse's passthrough example (see
198+
`vm/punchfs/README.md`).

‎launchd/com.renacode.cloudmachine.app.plist.template‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,8 @@
2121
would launch it over and over. The user closing the window should stay
2222
closed, not be an invitation to restart.
2323
24-
The backups themselves do NOT depend on this app - the gdrive-buffer,
25-
gdrive-attach and buffer-guard agents do them. This is only a viewer.
24+
The backups themselves do NOT depend on this app - the machine (started
25+
by vm-start) and the vm-guard agent do them. This is only a viewer.
2626
-->
2727
<key>StandardOutPath</key>
2828
<string>__CM_LOG_DIR__/launchd-app.out.log</string>

‎launchd/com.renacode.cloudmachine.backup-health.plist.template‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,9 @@
2323
Every 30 minutes, with an hourly cycle and a 3 h threshold - often enough
2424
that a broken cycle does not sit unnoticed for a day, and rarely enough to
2525
cost nothing. The watchdog reads the local Time Machine preferences file,
26-
NOT the Google Drive mount: a supervisor that hangs together with what it
27-
supervises goes silent exactly when it should speak.
26+
not the backup destination, and every question to the machine has a time
27+
limit: a supervisor that hangs together with what it supervises goes
28+
silent exactly when it should speak.
2829
-->
2930
<key>StartInterval</key>
3031
<integer>1800</integer>

‎launchd/com.renacode.cloudmachine.buffer-guard.plist.template‎

Lines changed: 0 additions & 28 deletions
This file was deleted.

‎launchd/com.renacode.cloudmachine.gdrive-attach.plist.template‎

Lines changed: 0 additions & 32 deletions
This file was deleted.

‎launchd/com.renacode.cloudmachine.gdrive-buffer.plist.template‎

Lines changed: 0 additions & 34 deletions
This file was deleted.
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
3+
<plist version="1.0">
4+
<dict>
5+
<key>Label</key>
6+
<string>com.renacode.cloudmachine.vm-guard</string>
7+
<!--
8+
The machine's buffer guard (`vm-guard`): one check per run, so
9+
StartInterval rather than KeepAlive. Installed by install-launchd
10+
(also run by `vm-setup --install-agents`).
11+
-->
12+
<key>ProgramArguments</key>
13+
<array>
14+
<string>__CM_AGENT_BIN__</string>
15+
<string>vm-guard</string>
16+
</array>
17+
<key>StartInterval</key>
18+
<integer>60</integer>
19+
<key>RunAtLoad</key>
20+
<true/>
21+
<key>EnvironmentVariables</key>
22+
<dict>
23+
<key>PATH</key>
24+
<string>/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
25+
</dict>
26+
<key>StandardOutPath</key>
27+
<string>__CM_LOG_DIR__/launchd-vm-guard.out.log</string>
28+
<key>StandardErrorPath</key>
29+
<string>__CM_LOG_DIR__/launchd-vm-guard.err.log</string>
30+
</dict>
31+
</plist>

0 commit comments

Comments
 (0)