Skip to content

Commit f4083dd

Browse files
authored
Close a computer's browsers before its memory runs out, and a Bot's idle tabs (#879)
1 parent 4766e42 commit f4083dd

34 files changed

Lines changed: 1818 additions & 39 deletions

‎.env.example‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -359,6 +359,20 @@ COMPUTER_TOKEN=
359359
# COMPUTER_BROWSER_BACKEND=managed
360360
# COMPUTER_BROWSER_MODE=headed
361361

362+
# When a computer closes a browser besides the cap (COMPUTER_MAX_BROWSERS) and the idle timeout
363+
# (COMPUTER_BROWSER_IDLE_MS). At or above this share of its memory limit, read as the kubelet reads it
364+
# (the working set, so the page cache does not count), it closes browsers nobody has used for a
365+
# minute, least recently used first, until it is ten points under. Memory is read before each launch
366+
# and once a minute, so a page that grows between two checks can still run it out of memory. A
367+
# container with no memory limit is measured against the machine's memory; a computer outside a
368+
# container closes nothing for memory. In the all-in-one image the share counts the database and the
369+
# API server too, on purpose: the out-of-memory kill counts them as well. 0 switches it off.
370+
# COMPUTER_MEMORY_HIGH_PERCENT=85
371+
# How long a Bot's background tab may sit untouched before it is closed, in ms: a tab the Bot left
372+
# behind, such as the page a sign-in popup opened from. The page the Bot is on, a person's own
373+
# browser, and any tab of a Bot whose wheel a person holds are never closed. 0 keeps them.
374+
# COMPUTER_TAB_IDLE_MS=600000
375+
362376
# Web Bot Auth: every request a Bot's browser makes is signed as coming from this deployment
363377
# (Ed25519, HTTP Message Signatures), and the public key is published at
364378
# <OPENBOT_PUBLIC_URL>/.well-known/http-message-signatures-directory. Off unless set to on, because

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,7 @@ jobs:
141141
bun test tests/sign-in.test.ts tests/session-cookies.test.ts tests/follows-popup.test.ts
142142
tests/demonstration-live.test.ts tests/webrtc-proxy.test.ts tests/web-bot-auth-browser.test.ts
143143
tests/handoff-page-restore.test.ts tests/enter-text.test.ts tests/person-browser.test.ts tests/person-browser-computer.test.ts
144-
tests/screencast-keys-browser.test.ts
144+
tests/screencast-keys-browser.test.ts tests/browser-memory.test.ts
145145
working-directory: agent-computer
146146
env:
147147
OPENBOT_COMPUTER_BROWSER: '1'

‎CHANGELOG.md‎

Lines changed: 50 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.
88

99
## Unreleased
1010

11-
**Before upgrading.** Fourteen things change for an existing deployment:
11+
**Before upgrading.** Fifteen things change for an existing deployment:
1212
- A hosted computer's browser is now headed by default. It uses a little more memory than headless
1313
did; set `COMPUTER_BROWSER_MODE=headless` (Helm: `computers.browserMode: headless`) to keep the old
1414
behaviour.
@@ -61,6 +61,55 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.
6161
to keep the old behaviour.
6262
- Interfaces a Bot drew before the upgrade (A2UI 0.9 surfaces) no longer draw when a conversation is
6363
reopened; the messages around them stay. New ones use A2UI 1.0 (below).
64+
- A computer now closes browsers nobody is using, least recently used first, when its memory
65+
reaches 85% of its limit, and closes a Bot's background tabs after ten minutes untouched (below).
66+
In the all-in-one image that share counts the database and the API server too.
67+
`COMPUTER_MEMORY_HIGH_PERCENT=0` and `COMPUTER_TAB_IDLE_MS=0` (Helm: `computers.memoryHighPercent`
68+
and `computers.tabIdleMs`) keep the old behaviour.
69+
70+
### A computer closes browsers nobody is using when its memory runs high
71+
72+
A computer runs one browser for each Bot that is browsing, and the only limits were a count and half
73+
an hour untouched. Neither looked at memory, so a few heavy pages could take a container past its
74+
memory limit well under the count, and the kernel then killed processes without regard to which Bot
75+
was busy. On Kubernetes it kills everything in the container at once, so every Bot's browser closed
76+
together and nothing could reach the computer until it came back. A computer now reads its own
77+
memory the way the kubelet does (the working set, which leaves out the file cache the kernel can
78+
drop), before it starts a browser and once a minute. At or above 85% of its limit it closes browsers
79+
one at a time, least recently used first, until it is under 75%, so the next launch does not cross
80+
the line again straight away. Launches that arrive together share one pass rather than closing a
81+
browser each. A Bot whose browser was closed starts again where it left off on its next step, still
82+
signed in, because its profile is on disk.
83+
84+
Only a browser nobody is using is closed. Never closed for memory: the browser being started, the
85+
most recently used one, any browser used in the last minute (a Bot between two steps asks for
86+
nothing while its model works out the next), a person's own browser, cloud or native, whose Recorder
87+
session would end, and the browser of a Bot whose wheel a person holds or has been asked to take.
88+
When every browser is in use, none is closed.
89+
90+
This is a check, not a guarantee. Memory is read before each launch and once a minute, so a page
91+
that grows between two checks can still run the container out of memory. A container with no memory
92+
limit is measured against the machine's memory, and a computer outside a container, such as OpenBot
93+
Desktop's, closes nothing for memory. In the all-in-one image the database and the API server run in
94+
the same container as the browsers, and the share counts them too. That is deliberate: the kernel's
95+
out-of-memory kill counts everything in the container and can take the database down, and closing a
96+
browser nobody is using first is how the database is protected. Each close is logged as
97+
`computer-browser-closed-for-memory` with the working set and the limit in bytes.
98+
`COMPUTER_MEMORY_HIGH_PERCENT` (Helm: `computers.memoryHighPercent`) sets the share where closing
99+
starts, and `0` switches it off.
100+
101+
The tabs a Bot leaves behind are closed too. A Bot always works in its newest page, so an older tab,
102+
such as the page a sign-in popup was opened from or a link a site opened in a new tab, is one it
103+
cannot go back to, and it held memory for as long as the browser stayed up. A background tab that
104+
nothing has used, navigated or brought to the front for ten minutes is now closed. The page the Bot
105+
is on is never closed, and neither is any tab of a person's own browser or of a Bot whose wheel a
106+
person holds. `COMPUTER_TAB_IDLE_MS` (Helm: `computers.tabIdleMs`) sets the time, and `0` keeps them.
107+
108+
Through Helm, `computers.browserIdleMs` and `computers.maxBrowsers` now reach the computer as
109+
written. An hour (`browserIdleMs: 3600000`) used to render as `3.6e+06`, which the computer ignored
110+
for its default of thirty minutes, and `browserIdleMs: 0` (keep browsers open) was dropped. All four
111+
of these settings now also reach the computers `computers.mode: sandbox` makes, which had none of
112+
them.
64113

65114
### Generated interfaces use A2UI 1.0
66115

‎agent-computer/src/browser-eviction.ts‎

Lines changed: 122 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/**
2-
* Which Bots' browsers to close, and when.
2+
* Which Bots' browsers to close, and when, and which of a Bot's tabs.
33
*
44
* Its own file, with no Playwright import, for the reason `authorisation.ts` and `bot-id.ts` are:
55
* `profiles.ts` imports Playwright at module scope, so anything living there needs a browser merely
@@ -62,21 +62,96 @@ export function chooseEvictions(
6262
const entries = [...running];
6363
if (entries.length <= max) return [];
6464
// A cap of zero keeps nothing, the newest included; that is the mistake reading the setting guards.
65-
const newest =
66-
max > 0
67-
? entries.reduce((latest, entry) =>
68-
entry[1].usedAt >= latest[1].usedAt ? entry : latest,
69-
)
70-
: undefined;
65+
const newest = max > 0 ? newestOf(entries) : undefined;
7166

67+
return inClosingOrder(entries, newest, goesLast)
68+
.slice(0, entries.length - max)
69+
.map(([botId]) => botId);
70+
}
71+
72+
/** The most recently used, which is the one a launch has just added. Ties go to the later entry. */
73+
function newestOf(
74+
entries: [string, Evictable][],
75+
): [string, Evictable] | undefined {
76+
if (entries.length === 0) return undefined;
77+
return entries.reduce((latest, entry) =>
78+
entry[1].usedAt >= latest[1].usedAt ? entry : latest,
79+
);
80+
}
81+
82+
/**
83+
* The order browsers are closed in when some have to go: every Bot's before a person's native window
84+
* (`keptOpen`), and least recently used first within each. `spare` is left out, which is the newest.
85+
*/
86+
function inClosingOrder(
87+
entries: [string, Evictable][],
88+
spare: [string, Evictable] | undefined,
89+
goesLast: (botId: string) => boolean,
90+
): [string, Evictable][] {
7291
return entries
73-
.filter((entry) => entry !== newest)
92+
.filter((entry) => entry !== spare)
7493
.sort(
7594
([aId, a], [bId, b]) =>
7695
Number(goesLast(aId)) - Number(goesLast(bId)) || a.usedAt - b.usedAt,
96+
);
97+
}
98+
99+
/**
100+
* How long after its last use a browser still counts as in use, and so is never closed for memory.
101+
*
102+
* A browser's use is recorded when something asks for its page: a Bot's step, a person's input, the
103+
* live screen. Between two steps a Bot asks for nothing while its model works out the next one, which
104+
* can take tens of seconds, and nothing here can tell that pause from a Bot that has finished.
105+
* Closing the browser then closes it in the middle of a run: the page the Bot was on is gone, and its
106+
* next step starts a cold browser, which is the very jump in memory the close was meant to head off.
107+
* A minute covers a model's turn between steps. It also covers a browser that has just launched, so
108+
* the next launch never closes the one before it, which is what would otherwise turn a full computer
109+
* into a loop of each Bot's launch closing the last Bot's browser.
110+
*/
111+
export const RECENTLY_USED_MS = 60_000;
112+
113+
/**
114+
* Which to close next because the computer's memory runs high (`container-memory.ts`).
115+
*
116+
* One at a time, because the caller reads memory again after each close and stops as soon as it is
117+
* enough: a count says in advance how many have to go, and bytes do not. Least recently used first,
118+
* the cap's order, so the two never disagree about who goes first.
119+
*
120+
* Never a browser in use, which here means four things:
121+
* - The one being launched. The caller runs this before a launch and passes only the browsers
122+
* already running, so the Bot that asked is not among them.
123+
* - The most recently used of those, however long ago that was. So at least one browser is always
124+
* left, because one browser that is on its own too big for the computer is the limit's problem,
125+
* and closing it would only start it again on its next step.
126+
* - Any used within `recentlyUsedMs` ({@link RECENTLY_USED_MS}), which is a Bot in the middle of a
127+
* run or a browser that has just started.
128+
* - Any `spared` names, which is a Bot whose wheel a person holds or has been asked to take: closing
129+
* it closes it under somebody's hands, or under a page the Bot is waiting for somebody to work.
130+
*
131+
* And never a person's own browser, cloud or native. Their Recorder runs in it, and closing it ends
132+
* what they were recording without a word to them. It still counts toward the cap, which closes it
133+
* last; memory pressure is not a reason to end somebody's session.
134+
*/
135+
export function chooseForMemory(
136+
running: Iterable<[string, Evictable]>,
137+
now: number,
138+
{
139+
recentlyUsedMs = RECENTLY_USED_MS,
140+
spared = () => false,
141+
}: {
142+
recentlyUsedMs?: number;
143+
spared?: (botId: string) => boolean;
144+
} = {},
145+
): string | undefined {
146+
const entries = [...running];
147+
const cutoff = now - recentlyUsedMs;
148+
return inClosingOrder(entries, newestOf(entries), () => false)
149+
.filter(
150+
([botId, entry]) =>
151+
entry.usedAt <= cutoff && !isPersonBrowserId(botId) && !spared(botId),
77152
)
78-
.slice(0, entries.length - max)
79-
.map(([botId]) => botId);
153+
.map(([botId]) => botId)
154+
.at(0);
80155
}
81156

82157
/**
@@ -104,3 +179,40 @@ export function chooseIdle(
104179
.filter(([botId, entry]) => entry.usedAt <= cutoff && !stays(botId))
105180
.map(([botId]) => botId);
106181
}
182+
183+
/** Enough of a tab for the choice. Keeps this file independent of what else is on a page. */
184+
export type IdleTab = {
185+
/** When the Bot or a person last acted on it, it navigated, or it was the page in front. */
186+
usedAt: number;
187+
/** The page the Bot is on, which is the one the live screen shows and its next action lands in. */
188+
front: boolean;
189+
/** A page that is never the Bot's to lose: an extension's own (the Recorder's) or DevTools. */
190+
kept?: boolean;
191+
};
192+
193+
/**
194+
* Which of a Bot's background tabs to close because nothing has touched them.
195+
*
196+
* A Bot is always on its newest page (`chooseLivePage`): a window a site opens is where its work
197+
* went, and it has no way back to an older tab except that newer ones close. So a tab left behind
198+
* (the page a sign-in popup was opened from, a link that opened in a new tab and was never closed)
199+
* holds a renderer and its memory for nothing, for as long as the browser stays up. Closed past a
200+
* timeout rather than at once, because an opener whose popup is still mid sign-in is waiting for
201+
* that popup to come back to it.
202+
*
203+
* Never the page in front, which is the one the Bot is on, so closing tabs never moves it: it stays
204+
* the newest open page, which is what the Bot follows. And never anything when which page is in
205+
* front is not exactly one, because then any tab could be the one the Bot is on. That also means the
206+
* last tab is never closed, since a single tab is the one in front. A timeout of zero or less
207+
* switches this off.
208+
*/
209+
export function chooseIdleTabs<T extends IdleTab>(
210+
tabs: readonly T[],
211+
idleTimeoutMs: number,
212+
now: number,
213+
): T[] {
214+
if (idleTimeoutMs <= 0) return [];
215+
if (tabs.filter((tab) => tab.front).length !== 1) return [];
216+
const cutoff = now - idleTimeoutMs;
217+
return tabs.filter((tab) => !tab.front && !tab.kept && tab.usedAt <= cutoff);
218+
}

0 commit comments

Comments
 (0)