Skip to content

Commit 3cfcd25

Browse files
committed
docs: teach the skill to cut and animate to the beat from an analysed track
1 parent 847a5f4 commit 3cfcd25

4 files changed

Lines changed: 88 additions & 2 deletions

File tree

‎skills/shotstack/SKILL.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ If a check fails (no network, no npm, a sandbox), skip it and carry on.
2424

2525
**Open-ended first render? Ask before composing.** If the ask doesn't state what the video is (subject, format or destination) — e.g. "render my first video", "make me a video", "set up Shotstack" — read [`references/onboarding.md`](references/onboarding.md) and ask its discovery questions, then WAIT for answers. Do not write an Edit first. An open-ended ask is not a specification, and a stray JSON file in the project doesn't make the user experienced. Compose without asking only when the user has stated what the video is, or explicitly says to just show anything.
2626

27-
Commands for the Shotstack video rendering API. `render` submits an Edit JSON and returns a render ID; `status` polls a render until done. `ingest` uploads your own local files (or fetches remote URLs) and hosts them so they can be referenced from an Edit. `validate` lints an Edit — run it before every render. `studio` opens an edit in the browser editor for a human to preview, tweak, and render. `template list|get|create|update|delete` reads and writes templates saved to the account: `list` shows saved templates (id + name); `get <id>` fetches a saved template's Edit JSON (e.g. one the user designed in Studio) so you can render or remix it; `create <file> --name <name>` saves an edit as a new template; `update <id> <file>` overwrites one; `delete <id>` removes one. `models` lists the generation models for prompt-driven assets and whether this key can use each one; `models <id>` prints the options a model accepts. `generate <model> "<prompt>"` generates one asset on its own and prints its URL.
27+
Commands for the Shotstack video rendering API. `render` submits an Edit JSON and returns a render ID; `status` polls a render until done. `ingest` uploads your own local files (or fetches remote URLs) and hosts them so they can be referenced from an Edit. `validate` lints an Edit — run it before every render. `studio` opens an edit in the browser editor for a human to preview, tweak, and render. `template list|get|create|update|delete` reads and writes templates saved to the account: `list` shows saved templates (id + name); `get <id>` fetches a saved template's Edit JSON (e.g. one the user designed in Studio) so you can render or remix it; `create <file> --name <name>` saves an edit as a new template; `update <id> <file>` overwrites one; `delete <id>` removes one. `models` lists the generation models for prompt-driven assets and whether this key can use each one; `models <id>` prints the options a model accepts. `generate <model> "<prompt>"` generates one asset on its own and prints its URL. `analyze <url>` measures a music track's tempo, beats, bars, drops and sections, free and without a key.
2828

2929
**Default loop — no API key required:** compose → `validate` (offline lint) → `studio` (browser preview). Only the API-backed commands (`render`, `status`, `ingest`, `models`, `generate`) need a key. Reach for `render` when you specifically need a final MP4 exported in the cloud; reach for `studio` for everything else — it is the cheapest way to see an edit and the right default for any creative a human will review.
3030

@@ -117,6 +117,8 @@ shotstack generate seedance-2.0-text-to-video "Waves break over black sand at du
117117

118118
**Generation costs credits on stage as well as v1.** Before generating, run the same command with `--quote`, tell the user the model and the credits it prints (it says "up to" when the final charge can be lower), and wait for a yes. Identical requests share one job and its cached result: repeating a request while it runs, or after it finishes, returns the same job without charging again. That is also how to pick up a job started without `--watch`. Sending a failed generation again retries it, and that is charged.
119119

120+
**Cutting or animating to music?** Measure the track with `shotstack analyze <url> --output json` and follow [`references/beat-sync.md`](references/beat-sync.md): write every cut and keyframe in bars from the map, never from the BPM you asked for.
121+
120122
**Read [`references/generation.md`](references/generation.md) before writing a prompt.** It covers what each model responds to, when to put text in a `rich-text` layer instead of the image, and how to size and chain generations inside an edit.
121123

122124
## Default workflow: preview in Studio (no key, no credits)
@@ -199,6 +201,7 @@ This skill ships sub-references for the gnarly bits:
199201
- [`references/html5-snippets.md`](references/html5-snippets.md) — copy-paste motion-graphic clips: kinetic headline, value reveal, shine sweep, pulsing CTA, film grain
200202
- [`references/fonts.md`](references/fonts.md) — built-in fonts, Google Fonts URL pattern, custom-font workflow
201203
- [`references/generation.md`](references/generation.md) — per-model prompting craft for AI-generated video, images, speech and music, and what a timeline adds to a prompt. Read before writing any `prompt`.
204+
- [`references/beat-sync.md`](references/beat-sync.md) — cutting and animating to music: get the track first, analyse it, write every cue in bars, drive `html5` from the map, check frames at bar lines
202205
- [`references/asset-library.md`](references/asset-library.md) — placeholder videos, images, music
203206
- [`references/troubleshooting.md`](references/troubleshooting.md) — common errors and fixes
204207

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Beat sync
2+
3+
Cut and animate to a track by measuring it, not by trusting the tempo you asked for. A generated track keeps a stated BPM but puts its drops and sections where it likes, and a tempo found by ear or by a general-purpose library is often half a beat off the kick. A cut that lands a quarter-beat late reads as a mistake.
4+
5+
## 1. Get the track before you compose
6+
7+
Analysis needs a file. A `prompt` on the timeline's `audio` asset only generates at render time, so there is nothing to measure while you write the edit.
8+
9+
- **The user's track:** use its hosted URL.
10+
- **A library track:** use its URL as is.
11+
- **A generated track:** generate it on its own first, then use the URL it returns as a plain `audio` `src`. Still state the BPM in the prompt; measure the result anyway.
12+
13+
<!-- surface:cli -->
14+
Host a local file with `shotstack ingest upload <file> --watch`. Generate a track with `shotstack generate <music-model> "<prompt>" --length <seconds> --watch`, after quoting it with `--quote`.
15+
<!-- /surface -->
16+
17+
## 2. Analyse it
18+
19+
Analysis is free, takes 15–30 s and reads the first 10 minutes of the track.
20+
21+
<!-- surface:cli,api -->
22+
It needs no API key. The `src` must be a direct, public link to an audio or video file: share pages and redirects are refused.
23+
<!-- /surface -->
24+
<!-- surface:cli -->
25+
```sh
26+
shotstack analyze https://shotstack-assets.s3.amazonaws.com/music/unminus/lit.mp3 --output json > map.json
27+
```
28+
<!-- /surface -->
29+
30+
## 3. Read the map
31+
32+
| Field | Meaning |
33+
|---|---|
34+
| `steady` | `true`: the tempo holds, and `bpm`, `beat`, `bar`, `meter` and `origin` describe a grid. `false`: use `beats[]` and `downbeats[]` instead. |
35+
| `bpm`, `beat`, `bar` | Tempo, and the beat and bar lengths in seconds. |
36+
| `meter` | Beats per bar. |
37+
| `origin` | Time of the first downbeat. Bar `n` starts at `origin + n × bar`. |
38+
| `bars.count`, `bars.energy[]` | Number of bars, and loudness per bar from 0 to 1. |
39+
| `drops[]` | `{ bar, time }` where the low end comes in hard. |
40+
| `sections[]` | `{ bar, time, change }`, where `change` is `rise` or `fall`. |
41+
| `downbeat.confidence` | How sure the bar phase is, 0 to 1. |
42+
| `precision` | How far to trust each part: `grid` is `frame` or `beat`, `sections` is `bar`, `downbeat` is `estimate`. |
43+
| `beats[]`, `downbeats[]` | Beat and bar times, only when `steady` is `false`. |
44+
45+
## 4. Write every cue in bars
46+
47+
- **Cuts on bar lines.** A shot holds a whole number of bars. Compute each `start` and `length` from the map and keep three decimals; never round to whole seconds.
48+
- **Let energy set the pace.** Hold two or four bars where `bars.energy` is low and cut every bar where it is high.
49+
- **Spend the big moment on a drop.** Put the reveal, the logo or the hardest cut at a `drops[].time`.
50+
- **Change scene at sections.** A `rise` opens a new idea; a `fall` is where to breathe.
51+
- **Accents on beats.** A pulse, flash or kick-on-the-beat sits at `origin + k × beat`.
52+
- **No grid?** When `steady` is `false`, cut on `downbeats[]` and accent on `beats[]`.
53+
- **Downbeat off by one?** The bar phase is an estimate. If `downbeat.confidence` is low, look at the first drop: if it lands mid-bar, shift `origin` by a beat.
54+
55+
## 5. Drive `html5` motion from the map
56+
57+
An `html5` clip's time is local to the clip. Put the map's numbers and the clip's own `start` in the page and place every tween from them, on a timeline the harness seeks (not `paused`):
58+
59+
```js
60+
const ORIGIN = 0.062, BAR = 1.9998, BEAT = BAR / 4, CLIP_START = 8.062; // from the map and this clip
61+
const at = (bar, beat = 0) => ORIGIN + bar * BAR + beat * BEAT - CLIP_START;
62+
const tl = gsap.timeline();
63+
tl.from(".title", { yPercent: 110, duration: BEAT, ease: "power3.out" }, at(4))
64+
.to(".title", { scale: 1.06, duration: BEAT / 2, yoyo: true, repeat: 1 }, at(5))
65+
.to(".title", { opacity: 0, duration: BEAT }, at(7, 3));
66+
```
67+
68+
Built-in motion works the same way: a clip whose `start` is a bar line makes its `transition` land on the beat.
69+
70+
## 6. Keep the music a plain audio clip
71+
72+
Put the track on its own track as an `audio` asset at `start: 0` with `length: "end"`; a fade (`effect: "fadeOut"`) is fine. Anything that moves the audio moves the grid with it. If you trim the track or start it later, add that offset to every time: timeline time = map time − `trim` + the audio clip's `start`.
73+
74+
## 7. Check frames at bar lines
75+
76+
Before calling it done, look at a few cuts and the first drop at their bar times: the new shot should be on screen at the bar line, not a frame or two after it.
77+
78+
<!-- surface:cli -->
79+
Preview with `shotstack studio`, or render on stage and pull a frame at a bar time with `ffmpeg -ss <time> -i out.mp4 -frames:v 1 bar.png`.
80+
<!-- /surface -->

‎skills/shotstack/references/generation.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,9 @@ Speech length is not knowable in advance. Set the clip's `length` to `"auto"` an
6464

6565
**Eleven Music.** A prompt answers genre, mood, instrumentation, tempo and production era whether or not you intend it to. Whatever you leave out, the model chooses.
6666

67-
**Production vocabulary does real work** — "dusty", "vinyl crackle", "sparse", "ethereal". Give a BPM when timing matters; the model holds a stated tempo precisely. Naming a key helps, less reliably.
67+
**Production vocabulary does real work** — "dusty", "vinyl crackle", "sparse", "ethereal". Naming a key helps, less reliably.
68+
69+
**State a BPM, then measure.** The model holds a stated tempo closely, but it places drops and sections where it likes, whatever the prompt says. Before timing cuts or motion to a generated track, analyse it: see [`beat-sync.md`](beat-sync.md).
6870

6971
**Naming a musician or band is refused**, as is quoting copyrighted lyrics. The error includes a suggested alternative prompt. Describe the era and production style instead: "1990s trip-hop with detuned Rhodes and a heavy swung break".
7072

‎skills/shotstack/shared/agent-core.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,7 @@ Timing, easing and rhythm are yours to design. Built-in motion takes fixed optio
146146
- **Combining on an existing asset:** keyframes move the whole clip while a `rich-text` preset animates the text inside it, so animate each property one way per clip. Keyframe `scale` multiplies `fit`, so a large scale on a low-resolution source shows its pixels.
147147
- **rich-text `animation`** is entrance-only (exits use the clip `transition.out`) with engine-fixed easing. `style` works only on `typewriter`/`shift`; `direction` is required for `slideIn`/`ascend`/`shift`/`movingLetters`.
148148
- **`html5`** gives full control: any seekable GSAP, anime.js, Lottie or CSS animation, any easing, per-element timing.
149+
- **To music:** measure the track before timing anything to it, then write cuts and motion in bars. Method in [`references/beat-sync.md`](../references/beat-sync.md).
149150
- **Brand once:** palette and font as top-level `merge[]` fields (`{{ink}}`, `{{accent}}`, `{{font}}`) resolve inside `html5` `html`/`css`/`js` too, so one edit re-skins the whole video.
150151

151152
## Positioning & coordinates

0 commit comments

Comments
 (0)