Skip to content

Commit 59b00a6

Browse files
feat: remote skill download and auto-sync (#468)
* feat: add remote skill download and auto-sync Download default skills from remote catalog on first setup with bundled fallback when offline. Background sync every 45 minutes checks for new/updated skills without overwriting user-customized ones. Tracks installed defaults via content hashes in a local manifest file. * feat: make skills catalog URL configurable and add generation script Add SKILLS_CATALOG_URL env var (following CODEGEN_SERVICE_URL pattern) with fallback to the default constant. Add script to generate catalog.json from bundled defaults for static hosting. * feat: add R2 upload script and use cdn.browseros.com for catalog URL Add upload-skills-catalog.ts that generates and uploads catalog.json to Cloudflare R2 (same infra as existing build artifacts). Update default catalog URL to cdn.browseros.com/skills/v1/catalog.json. * test: add E2E tests for remote skill sync against live CDN * fix: address code review findings — security, validation, DRY - Add path traversal protection via safeSkillDir in writeSkillFile and readSkillContent (reuses existing validation from service.ts) - Add runtime type guards for catalog JSON and manifest JSON parsing - Fix seedFromRemote to return false on partial failure so bundled fallback kicks in - Add per-skill error handling in syncRemoteSkills so one bad skill doesn't crash the entire sync - Wire stopSkillSync into Application.stop() shutdown path - Extract version from frontmatter in seedFromBundled instead of hardcoding '1.0' - Consolidate duplicated logic: reuse installSkill/writeSkillFile/ contentHash/saveManifest from remote-sync.ts in seed.ts - Extract shared catalog generation into scripts/catalog-utils.ts * test: add flow tests for all four sync scenarios against live CDN * refactor: remove redundant scripts and inline catalog generation Drop generate-skills-catalog.ts, catalog-utils.ts, and e2e-remote-sync.test.ts (covered by flows.test.ts). Inline catalog generation into upload-skills-catalog.ts. * test: add full E2E server flow test against live CDN Tests all 7 steps of the real server lifecycle: fresh seed from CDN, no-op sync, user edit preservation, skill reinstall, custom skill protection, background timer firing, and second startup skip. * chore: remove e2e-server-flow test * fix: address Greptile review — entry validation, size limit, DRY, no-op saves - Validate individual skill entries in catalog (id, version, content must all be strings) not just the top-level shape - Add 1MB response size limit on catalog fetch to prevent resource exhaustion from compromised/misconfigured CDN - Skip manifest save when sync cycle had no changes (avoids unnecessary disk I/O every 45 minutes) - Share extractVersion via remote-sync.ts export, remove duplicate from seed.ts * fix: prevent bundled fallback from overwriting partial remote seeds When seedFromRemote partially fails, the bundled fallback now skips skills already in the manifest (installed by the partial remote seed). Also adds Content-Length early check before downloading the full catalog response body. * fix: run sync immediately on startup, not just on interval Previously the first sync fired 45 minutes after boot. Now startSkillSync runs one sync immediately so returning users get skill updates right away. * refactor: simplify sync — remote always wins, remove manifest Remote catalog is the source of truth. If a skill exists in the catalog, its version is compared against local frontmatter and overwritten when newer. No manifest file, no content hashes. User-created skills (IDs not in catalog) are never touched. * fix: skip bundled skills already installed by partial remote seed * chore: remove unreliable Content-Length check * chore: remove size limit checks, fetch timeout is sufficient
1 parent 44af9ae commit 59b00a6

11 files changed

Lines changed: 618 additions & 7 deletions

File tree

packages/browseros-agent/apps/server/src/env.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ export const INLINED_ENV = {
1919
CODEGEN_SERVICE_URL: process.env.CODEGEN_SERVICE_URL,
2020
POSTHOG_API_KEY: process.env.POSTHOG_API_KEY,
2121
BROWSEROS_CONFIG_URL: process.env.BROWSEROS_CONFIG_URL,
22+
SKILLS_CATALOG_URL: process.env.SKILLS_CATALOG_URL,
2223
} as const
2324

2425
export const REQUIRED_FOR_PRODUCTION = [

packages/browseros-agent/apps/server/src/main.ts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ import { fetchDailyRateLimit } from './lib/rate-limiter/fetch-config'
2828
import { RateLimiter } from './lib/rate-limiter/rate-limiter'
2929
import { Sentry } from './lib/sentry'
3030
import { seedSoulTemplate } from './lib/soul'
31+
import { startSkillSync, stopSkillSync } from './skills/remote-sync'
3132
import { seedDefaultSkills } from './skills/seed'
3233
import { registry } from './tools/registry'
3334
import { VERSION } from './version'
@@ -112,12 +113,14 @@ export class Application {
112113
)
113114

114115
this.logStartupSummary(controllerServerStarted)
116+
startSkillSync()
115117

116118
metrics.log('http_server.started', { version: VERSION })
117119
}
118120

119121
stop(reason?: string): void {
120122
logger.info('Shutting down server...', { reason })
123+
stopSkillSync()
121124

122125
// Immediate exit without graceful shutdown. Chromium may kill us on update/restart,
123126
// and we need to free the port instantly so the HTTP port doesn't keep switching.
Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
import { mkdir, readFile, writeFile } from 'node:fs/promises'
2+
import { join } from 'node:path'
3+
import { TIMEOUTS } from '@browseros/shared/constants/timeouts'
4+
import { EXTERNAL_URLS } from '@browseros/shared/constants/urls'
5+
import { INLINED_ENV } from '../env'
6+
import { getSkillsDir } from '../lib/browseros-dir'
7+
import { logger } from '../lib/logger'
8+
import { safeSkillDir } from './service'
9+
import type { RemoteSkillCatalog, RemoteSkillEntry } from './types'
10+
11+
let syncTimer: ReturnType<typeof setInterval> | null = null
12+
13+
export function extractVersion(content: string): string {
14+
const match = content.match(/^\s*version:\s*["']?([^"'\n]+)["']?/m)
15+
return match?.[1]?.trim() || '1.0'
16+
}
17+
18+
function isValidSkillEntry(entry: unknown): entry is RemoteSkillEntry {
19+
if (typeof entry !== 'object' || entry === null) return false
20+
const e = entry as Record<string, unknown>
21+
return (
22+
typeof e.id === 'string' &&
23+
typeof e.version === 'string' &&
24+
typeof e.content === 'string'
25+
)
26+
}
27+
28+
function isValidCatalog(data: unknown): data is RemoteSkillCatalog {
29+
if (typeof data !== 'object' || data === null) return false
30+
const d = data as Record<string, unknown>
31+
return (
32+
typeof d.version === 'number' &&
33+
Array.isArray(d.skills) &&
34+
d.skills.every(isValidSkillEntry)
35+
)
36+
}
37+
38+
function getCatalogUrl(): string {
39+
return INLINED_ENV.SKILLS_CATALOG_URL || EXTERNAL_URLS.SKILLS_CATALOG
40+
}
41+
42+
export async function fetchRemoteCatalog(): Promise<RemoteSkillCatalog | null> {
43+
try {
44+
const response = await fetch(getCatalogUrl(), {
45+
signal: AbortSignal.timeout(TIMEOUTS.SKILLS_FETCH),
46+
})
47+
if (!response.ok) {
48+
logger.warn('Failed to fetch remote skill catalog', {
49+
status: response.status,
50+
})
51+
return null
52+
}
53+
const data: unknown = await response.json()
54+
if (!isValidCatalog(data)) {
55+
logger.warn('Remote skill catalog has invalid format')
56+
return null
57+
}
58+
return data
59+
} catch (err) {
60+
logger.debug('Remote skill catalog unavailable', {
61+
error: err instanceof Error ? err.message : String(err),
62+
})
63+
return null
64+
}
65+
}
66+
67+
async function getLocalVersion(skillId: string): Promise<string | null> {
68+
try {
69+
const safeDir = safeSkillDir(skillId)
70+
const content = await readFile(join(safeDir, 'SKILL.md'), 'utf-8')
71+
return extractVersion(content)
72+
} catch {
73+
return null
74+
}
75+
}
76+
77+
export async function writeSkillFile(
78+
skillId: string,
79+
content: string,
80+
): Promise<void> {
81+
const safeDir = safeSkillDir(skillId)
82+
await mkdir(safeDir, { recursive: true })
83+
await writeFile(join(safeDir, 'SKILL.md'), content)
84+
}
85+
86+
export async function syncRemoteSkills(): Promise<{
87+
installed: number
88+
updated: number
89+
}> {
90+
const result = { installed: 0, updated: 0 }
91+
const catalog = await fetchRemoteCatalog()
92+
if (!catalog) return result
93+
94+
for (const remoteSkill of catalog.skills) {
95+
try {
96+
const localVersion = await getLocalVersion(remoteSkill.id)
97+
98+
if (!localVersion) {
99+
await writeSkillFile(remoteSkill.id, remoteSkill.content)
100+
result.installed++
101+
continue
102+
}
103+
104+
if (localVersion === remoteSkill.version) {
105+
continue
106+
}
107+
108+
await writeSkillFile(remoteSkill.id, remoteSkill.content)
109+
result.updated++
110+
} catch (err) {
111+
logger.warn('Failed to sync skill', {
112+
id: remoteSkill.id,
113+
error: err instanceof Error ? err.message : String(err),
114+
})
115+
}
116+
}
117+
118+
return result
119+
}
120+
121+
export async function seedFromRemote(): Promise<boolean> {
122+
const catalog = await fetchRemoteCatalog()
123+
if (!catalog || catalog.skills.length === 0) return false
124+
125+
let seeded = 0
126+
127+
for (const skill of catalog.skills) {
128+
try {
129+
await writeSkillFile(skill.id, skill.content)
130+
seeded++
131+
} catch (err) {
132+
logger.warn('Failed to seed remote skill', {
133+
id: skill.id,
134+
error: err instanceof Error ? err.message : String(err),
135+
})
136+
}
137+
}
138+
139+
if (seeded > 0) {
140+
logger.info(`Seeded ${seeded}/${catalog.skills.length} skills from remote catalog`)
141+
}
142+
143+
return seeded === catalog.skills.length
144+
}
145+
146+
async function runSync(): Promise<void> {
147+
try {
148+
const { installed, updated } = await syncRemoteSkills()
149+
if (installed > 0 || updated > 0) {
150+
logger.info('Remote skill sync completed', { installed, updated })
151+
}
152+
} catch (err) {
153+
logger.warn('Skill sync failed', {
154+
error: err instanceof Error ? err.message : String(err),
155+
})
156+
}
157+
}
158+
159+
export function startSkillSync(): void {
160+
if (syncTimer) return
161+
162+
runSync()
163+
164+
syncTimer = setInterval(runSync, TIMEOUTS.SKILLS_SYNC_INTERVAL)
165+
syncTimer.unref()
166+
}
167+
168+
export function stopSkillSync(): void {
169+
if (syncTimer) {
170+
clearInterval(syncTimer)
171+
syncTimer = null
172+
}
173+
}

packages/browseros-agent/apps/server/src/skills/seed.ts

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
1-
import { mkdir, readdir, writeFile } from 'node:fs/promises'
1+
import { readdir, stat } from 'node:fs/promises'
22
import { join } from 'node:path'
33
import { getSkillsDir } from '../lib/browseros-dir'
44
import { logger } from '../lib/logger'
55
import { DEFAULT_SKILLS } from './defaults'
6+
import { seedFromRemote, writeSkillFile } from './remote-sync'
67

78
async function hasExistingSkills(skillsDir: string): Promise<boolean> {
89
try {
@@ -13,16 +14,27 @@ async function hasExistingSkills(skillsDir: string): Promise<boolean> {
1314
}
1415
}
1516

17+
async function skillExists(skillsDir: string, id: string): Promise<boolean> {
18+
try {
19+
await stat(join(skillsDir, id, 'SKILL.md'))
20+
return true
21+
} catch {
22+
return false
23+
}
24+
}
25+
1626
export async function seedDefaultSkills(): Promise<void> {
1727
const skillsDir = getSkillsDir()
1828
if (await hasExistingSkills(skillsDir)) return
1929

30+
const remoteSucceeded = await seedFromRemote()
31+
if (remoteSucceeded) return
32+
2033
let seeded = 0
2134
for (const skill of DEFAULT_SKILLS) {
35+
if (await skillExists(skillsDir, skill.id)) continue
2236
try {
23-
const targetDir = join(skillsDir, skill.id)
24-
await mkdir(targetDir, { recursive: true })
25-
await writeFile(join(targetDir, 'SKILL.md'), skill.content)
37+
await writeSkillFile(skill.id, skill.content)
2638
seeded++
2739
} catch (err) {
2840
logger.warn('Failed to seed skill', {
@@ -33,6 +45,6 @@ export async function seedDefaultSkills(): Promise<void> {
3345
}
3446

3547
if (seeded > 0) {
36-
logger.info(`Seeded ${seeded} default skills`)
48+
logger.info(`Seeded ${seeded} default skills (bundled)`)
3749
}
3850
}

packages/browseros-agent/apps/server/src/skills/service.ts

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,7 @@ export function slugify(name: string): string {
1919
.replace(/^-|-$/g, '')
2020
}
2121

22-
// Prevents path traversal — ensures resolved path stays inside skills directory
23-
function safeSkillDir(id: string): string {
22+
export function safeSkillDir(id: string): string {
2423
const skillsDir = getSkillsDir()
2524
const resolved = resolve(skillsDir, id)
2625
if (!resolved.startsWith(`${skillsDir}${sep}`)) {

packages/browseros-agent/apps/server/src/skills/types.ts

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,3 +38,15 @@ export type CreateSkillInput = {
3838
export type UpdateSkillInput = Partial<CreateSkillInput> & {
3939
enabled?: boolean
4040
}
41+
42+
export type RemoteSkillEntry = {
43+
id: string
44+
version: string
45+
content: string
46+
}
47+
48+
export type RemoteSkillCatalog = {
49+
version: number
50+
skills: RemoteSkillEntry[]
51+
}
52+
Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
/**
2+
* E2E flow tests against live CDN.
3+
*/
4+
5+
import { afterAll, beforeAll, describe, it, mock } from 'bun:test'
6+
import assert from 'node:assert'
7+
import { mkdir, readdir, readFile, rm, writeFile } from 'node:fs/promises'
8+
import { tmpdir } from 'node:os'
9+
import { join } from 'node:path'
10+
11+
let testDir: string
12+
13+
mock.module('../../src/lib/browseros-dir', () => ({
14+
getSkillsDir: () => testDir,
15+
}))
16+
17+
mock.module('../../src/env', () => ({
18+
INLINED_ENV: {
19+
SKILLS_CATALOG_URL: 'https://cdn.browseros.com/skills/v1/catalog.json',
20+
},
21+
}))
22+
23+
const { seedFromRemote, syncRemoteSkills } =
24+
await import('../../src/skills/remote-sync')
25+
26+
async function listSkills(): Promise<string[]> {
27+
const entries = await readdir(testDir)
28+
return entries.filter((e) => !e.startsWith('.')).sort()
29+
}
30+
31+
beforeAll(async () => {
32+
testDir = join(tmpdir(), `flow-test-${Date.now()}`)
33+
await mkdir(testDir, { recursive: true })
34+
})
35+
36+
afterAll(async () => {
37+
await rm(testDir, { recursive: true, force: true })
38+
})
39+
40+
describe('Flow tests against live CDN', () => {
41+
it('seeds all skills from CDN on fresh install', async () => {
42+
const result = await seedFromRemote()
43+
assert.strictEqual(result, true)
44+
const skills = await listSkills()
45+
assert.strictEqual(skills.length, 12)
46+
})
47+
48+
it('sync does nothing when already up to date', async () => {
49+
const result = await syncRemoteSkills()
50+
assert.strictEqual(result.installed, 0)
51+
assert.strictEqual(result.updated, 0)
52+
})
53+
54+
it('remote overwrites local edits when version differs', async () => {
55+
const skillPath = join(testDir, 'summarize-page', 'SKILL.md')
56+
const original = await readFile(skillPath, 'utf-8')
57+
58+
// User edits the file AND we fake a version mismatch
59+
const edited = original.replace(/version: "1.0"/, 'version: "0.9"') + '\n## My Notes\n'
60+
await writeFile(skillPath, edited)
61+
62+
const result = await syncRemoteSkills()
63+
assert.strictEqual(result.updated >= 1, true)
64+
65+
const afterSync = await readFile(skillPath, 'utf-8')
66+
assert.ok(!afterSync.includes('My Notes'))
67+
})
68+
69+
it('installs skill deleted locally', async () => {
70+
await rm(join(testDir, 'save-page'), { recursive: true })
71+
72+
const result = await syncRemoteSkills()
73+
assert.strictEqual(result.installed, 1)
74+
75+
const content = await readFile(join(testDir, 'save-page', 'SKILL.md'), 'utf-8')
76+
assert.ok(content.includes('name: save-page'))
77+
})
78+
79+
it('user-created skill is never touched', async () => {
80+
const customDir = join(testDir, 'my-workflow')
81+
await mkdir(customDir, { recursive: true })
82+
const custom = '---\nname: my-workflow\ndescription: custom\n---\n# Mine\n'
83+
await writeFile(join(customDir, 'SKILL.md'), custom)
84+
85+
await syncRemoteSkills()
86+
87+
const afterSync = await readFile(join(customDir, 'SKILL.md'), 'utf-8')
88+
assert.strictEqual(afterSync, custom)
89+
})
90+
})

0 commit comments

Comments
 (0)