Skip to content

Commit 45a2f1a

Browse files
ColinGPT9claude
andauthored
Basketball: a rebuilt 9:16 crop around the play, clip starts and ends on the play, and a Crop choice (#151)
The crop is planned for the whole clip before a frame is rendered (sports/basketball/crop/): - The ball first, from a basketball-trained detector (E-BARD's YOLOv8n, CC BY 4.0), linked into tracks per camera shot with the camera's own movement taken out. - The play's phases: possession, a pass (the crop reaches the receiver before the catch), the release, the flight, the rim and the result. - The rim, net and backboard in frame from the ball nearing the rim to the result; a short shot keeps the shooter and the basket together from the release. - It pans and never zooms; the TV scoreboard is left out. - Close-ups are centred on the face of the one person the camera is on. - A team's own network, whose camera shows the players closer, is told court from crowd as well as a national broadcast. Clips start on the play's own possession and end when the play is over: before the next possession, a replay (the network's replay logo included) or another quarter, with the announcer's sentence about the play finished. Free throws at a stopped clock are never called a basket, and Best moments keeps plays and reactions only. The Sports row's Crop (basketball): by default the picture sits between the apps' top bar and their caption, username and music line, with a blurred band above and below (gaming/layouts.json's "all" zone), or Full screen. A clip keeps its choice on re-render, and the Generate list and Ask Clips Kitty know it. The Quarter control and the Highlights post style are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent c1a1143 commit 45a2f1a

86 files changed

Lines changed: 13356 additions & 2933 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 45 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -124,9 +124,8 @@ were often broken in a way that only showed up on somebody else's machine.
124124
option off. Works best with clear audio and voices that differ: two people who sound
125125
alike talking over each other outdoors are often left one colour. In a video where
126126
the streamer says less than what they are watching, the colours can be the other way
127-
round. Not for the Highlights post style, which has its own colour, or for translated
128-
subtitles. Runs on this PC (two small voice models, about 20 seconds once per video
129-
and a couple per clip). (#126)
127+
round. Not for translated subtitles. Runs on this PC (two small voice models, about 20
128+
seconds once per video and a couple per clip). (#126)
130129

131130
**Where it gets a caption wrong, fix it by hand.** In the clip editor's Captions tab,
132131
**Fix speakers** marks the other speaker's words. Click a word to switch who says it
@@ -149,26 +148,36 @@ were often broken in a way that only showed up on somebody else's machine.
149148
than a first-quarter one. The broadcast's cutaways to the crowd, the bench and
150149
courtside are found too: merged into the play's clip, or clipped on their own with
151150
**Fan reactions**, **Celebrity reactions**, **Crowd reactions** or **Bench reactions**.
152-
A person is named only when the broadcast captions them. A **Quarter** menu picks part
153-
of the game, the 9:16 crop follows the ball, the play around it and the rim, and a game
154-
filmed 9:16 keeps its picture. It runs on the same Whisper and AI model as everything
155-
else. Measured on three NBA broadcasts: the score bug is read piece by piece (two-row
156-
bugs, logos beside bare scores), court and crowd shots are told apart by how big the
157-
people in them are, and the crop no longer snaps when the camera pans. See
151+
A person is named only when the broadcast captions them. The whole game is clipped, and
152+
each clip's vertical crop is planned around its play: the ball and whoever has it, the
153+
receiver of a pass before the catch, and the shot followed to the basket with the rim,
154+
the net and the backboard in the picture. The crop is one picture that pans with the
155+
play and never zooms in or out, like a House of Highlights clip, the TV scoreboard left
156+
out. It sits clear of the apps' text, between TikTok's, Reels' and Shorts' top bar and
157+
the caption at the bottom with a blurred band above and below it, so neither hides the
158+
basket or the floor (the buttons down the right still sit over its edge, and the crop
159+
keeps the rim left of them); **Crop** in the Sports row makes it full screen instead.
160+
When the shooter and the basket don't both fit, it stays on
161+
the ball's side and pans with it to the rim, and when the ball can't be seen it follows
162+
the player who last had it, or the play. A close-up stays on the person the camera is
163+
on, not a blurred shoulder in front. A clip starts
164+
on the court, not on a close-up, the crowd or an earlier play left over from a condensed
165+
broadcast, and ends when the play is over: before the broadcast cuts to the next
166+
possession, a replay (the network's replay logo included) or another quarter, or before
167+
the camera follows the other team up the court, with everything the announcer says about
168+
the play before then and no glimpse of the next shot as it ends. A trip to the free-throw
169+
line stays in one clip, every attempt of it. A clip can come out a little under your
170+
shortest length rather than show the next play. **Best moments** keeps plays and
171+
reactions, not the commentators talking between them, and two free throws are never
172+
called a basket. A whole game joined from two
173+
broadcasts, each with its score in its own place, is read in both, and a team's own
174+
network, whose camera shows the players closer, is told court from crowd as well as a
175+
national broadcast. A game filmed 9:16 keeps its picture. It runs on the same Whisper and AI
176+
model as everything else. Measured on an NBA Finals broadcast: the ball stayed in the
177+
crop in 95% of the frames checked (46% with the first crop), and the rim and the
178+
backboard in all of them as the ball reached the rim (47% and 32%). See
158179
[docs/SPORTS.md](docs/SPORTS.md#basketball).
159180

160-
- **A Highlights post style.** Pick **Post style → Highlights** with the caption
161-
settings (it stays there with captions off) and Shorts come out looking like the big
162-
sports highlight pages' reels. The clip is framed full screen as usual. Over it goes
163-
their stacked title card: a big yellow headline on a black box and a smaller line on a
164-
yellow box under it, with the emoji in colour. Captions turn yellow and ALL CAPS in
165-
the middle of the frame. Titles, card lines and hashtags are written in that voice
166-
too, and names only appear when they are said in the clip or in the video title. The
167-
card can sit in the lower third or at the top (a clip with a hook title keeps it in
168-
the lower third), and you can change its words, or switch a clip's style, in the clip
169-
editor. 16:9 videos keep the standard look. Add your own handle or logo with a
170-
watermark. The standard look stays the default.
171-
172181
### Changed
173182

174183
- **The shape options say which shape they are about.** **Longform (16:9 clips)** makes
@@ -214,21 +223,21 @@ were often broken in a way that only showed up on somebody else's machine.
214223
other switches already did. (#138)
215224

216225
- **Processing a video again keeps your edits.** **Process again** made every clip it
217-
found again from the start: cuts, mutes, a layout, captions you fixed, colours,
218-
branding and title-card words you had applied in the editor stayed in the editor but
219-
were missing from the clip's file, and a channel that posts automatically could post
220-
that file. A clip you edited is now made again with your edits, on this computer or a
221-
render PC, and goes into a match's story reels with them. Clips you never edited, and
222-
new ones, are made with the settings you chose, as before, even when you set up a
223-
Gaming / Reaction split before processing. A clip you edited keeps its own look. In a
224-
Gaming / Reaction job it keeps its own split, and a split that was found automatically
225-
or remembered for the creator takes a webcam you set up or remembered for this run. A
226-
split you set up before an earlier run or drew for that clip stays as it is, even with
227-
Gaming / Reaction off (not in a match): change it in the clip editor. A split you turned
228-
off for a clip stays off. In a match, your cuts, layout, caption fixes and colour
229-
adjustments are kept, and the caption style, colours, branding and title card are the
230-
job's; **Re-render** a clip to make it with its own. 16:9 Longform clips too. A clip
231-
with music you added renders on this computer even when a render PC is chosen.
226+
found again from the start: cuts, mutes, a layout, captions you fixed, colours and
227+
branding you had applied in the editor stayed in the editor but were missing from the
228+
clip's file, and a channel that posts automatically could post that file. A clip you
229+
edited is now made again with your edits, on this computer or a render PC, and goes
230+
into a match's story reels with them. Clips you never edited, and new ones, are made
231+
with the settings you chose, as before, even when you set up a Gaming / Reaction split
232+
before processing. A clip you edited keeps its own look. In a Gaming / Reaction job it
233+
keeps its own split, and a split that was found automatically or remembered for the
234+
creator takes a webcam you set up or remembered for this run. A split you set up
235+
before an earlier run or drew for that clip stays as it is, even with Gaming /
236+
Reaction off (not in a match): change it in the clip editor. A split you turned off
237+
for a clip stays off. In a match, your cuts, layout, caption fixes and colour
238+
adjustments are kept, and the caption style, colours and branding are the job's;
239+
**Re-render** a clip to make it with its own. 16:9 Longform clips too. A clip with
240+
music you added renders on this computer even when a render PC is chosen.
232241
- **A model download that failed no longer says it finished.** When Ollama could not
233242
fetch a model (no connection, a name that does not exist, a full disk), the Models
234243
page, setup and the box above all showed the download as done, with nothing installed.

‎NOTICE‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,16 @@ significant ones, with their licences:
7676
class names (config/
7777
audioset_labels.txt) are CC BY 4.0,
7878
Google
79+
E-BARD detection models CC BY 4.0 the basketball, the hoop and the
80+
(Gabriele Giudici) players, for basketball's 9:16 framing
81+
(sports/basketball/crop/sources.py);
82+
the YOLOv8n weights (huggingface.co/
83+
GabrieleGiudici/E-BARD-detection-
84+
models, revision 3f4789c), unmodified,
85+
fetched by scripts/
86+
fetch_basketball_model.py. Trained
87+
with ultralytics, so its AGPL-3.0
88+
terms apply as to the YOLOv8 weights
7989
pyannote segmentation-3.0 MIT who is talking when, for the second
8090
speaker's caption colour (analysis/
8191
voice_turns.py); (c) 2022 CNRS, the

‎analysis/metadata.py‎

Lines changed: 3 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -30,20 +30,6 @@
3030
"required": ["title", "description", "hashtags"],
3131
"additionalProperties": False,
3232
}
33-
# The highlights post style (video/post_style.py) also asks for the two
34-
# lines of its title card.
35-
_CARD_FIELDS = {"headline": {"type": "string"}, "subline": {"type": "string"}}
36-
BATCH_SCHEMA_HIGHLIGHTS = {
37-
"type": "object",
38-
"properties": {"items": {"type": "array", "items": {
39-
"type": "object",
40-
"properties": {"index": {"type": "integer"}, **_METADATA_FIELDS, **_CARD_FIELDS},
41-
"required": ["index", "title", "description", "hashtags", "headline", "subline"],
42-
"additionalProperties": False,
43-
}}},
44-
"required": ["items"],
45-
"additionalProperties": False,
46-
}
4733
BATCH_SCHEMA = {
4834
"type": "object",
4935
"properties": {"items": {"type": "array", "items": {
@@ -62,10 +48,6 @@ class ClipMetadata:
6248
title: str
6349
description: str
6450
hashtags: list[str] = field(default_factory=list)
65-
# The highlights post style's title card (video/post_style.py). Empty
66-
# for every other style; never stored as metadata, only as render options.
67-
headline: str = ""
68-
subline: str = ""
6951

7052

7153
def generate_metadata(
@@ -101,11 +83,6 @@ def generate_metadata(
10183

10284

10385
BATCH_PROMPT_PATH = Path(__file__).resolve().parent.parent / "config" / "prompts" / "metadata_batch.txt"
104-
# Per post style (video/post_style.py): the words written to match the look.
105-
# A style not listed here writes the usual Shorts metadata.
106-
STYLE_PROMPT_PATHS = {
107-
"highlights": BATCH_PROMPT_PATH.with_name("metadata_batch_highlights.txt"),
108-
}
10986

11087

11188
def generate_metadata_batch(
@@ -116,20 +93,16 @@ def generate_metadata_batch(
11693
batch_size: int = 8,
11794
creator_context: str = "",
11895
rules: str = "",
119-
style: str = "default",
12096
) -> list[ClipMetadata]:
12197
"""Metadata for ALL clips in a few LLM calls instead of one per clip —
12298
on a long stream this cuts dozens of model calls from the analysis time.
12399
Any clip the model skips or mangles falls back to hook-based metadata.
124100
creator_context (optional): learned facts about the creator — series
125101
names, running jokes, collaborators — for more accurate titles/hashtags.
126102
rules (optional): a sport's own rules for its clips' titles (basketball:
127-
which player to name); "" for every other job, whose prompt is unchanged.
128-
style: the clip's post style; "highlights" writes highlight-page captions."""
103+
which player to name); "" for every other job, whose prompt is unchanged."""
129104
results: list[ClipMetadata] = [_fallback(c, video_title) for c in candidates]
130-
template = STYLE_PROMPT_PATHS.get(style, BATCH_PROMPT_PATH).read_text(encoding="utf-8")
131-
card = style == "highlights"
132-
schema = BATCH_SCHEMA_HIGHLIGHTS if card else BATCH_SCHEMA
105+
template = BATCH_PROMPT_PATH.read_text(encoding="utf-8")
133106
if rules:
134107
template = template.replace("{clips}", "RULES FOR THESE CLIPS:\n" + rules + "\n\n{clips}")
135108
if creator_context:
@@ -151,7 +124,7 @@ def generate_metadata_batch(
151124
.replace("{clips}", "\n\n".join(blocks))
152125
)
153126
try:
154-
data = _parse(generate_json(llm, prompt, schema))
127+
data = _parse(generate_json(llm, prompt, BATCH_SCHEMA))
155128
except Exception:
156129
data = None
157130
if not data or not isinstance(data.get("items"), list):
@@ -168,8 +141,6 @@ def generate_metadata_batch(
168141
title=_clean_title(item.get("title", "")) or fallback.title,
169142
description=str(item.get("description", "")).strip() or fallback.description,
170143
hashtags=_clean_hashtags(item.get("hashtags", [])) or fallback.hashtags,
171-
headline=_clean_card_line(item.get("headline", ""), 40) if card else "",
172-
subline=_clean_card_line(item.get("subline", ""), 48) if card else "",
173144
)
174145
for c, m in zip(candidates, results):
175146
_earned(c, m)
@@ -334,19 +305,6 @@ def _clean_title(title: str) -> str:
334305
return title[:MAX_TITLE_LEN].strip()
335306

336307

337-
def _clean_card_line(text, limit: int) -> str:
338-
"""One line of the highlights title card: one line, no quotes, no
339-
hashtags, and short. An overlong one is cut at a word; the card shrinks
340-
and wraps what is left (video/post_style.py)."""
341-
text = re.sub(r"\s+", " ", re.sub(r"[<>\"]", "", str(text or ""))).strip()
342-
# A hashtag has a letter in it: "#1 PICK" and "#23" are a draft rank and a
343-
# jersey number, and stay (the card's own filter in video/post_style.py).
344-
text = re.sub(r"(^|\s)#(?=\w*[^\W\d_])\w+", " ", text).strip()
345-
if len(text) > limit:
346-
text = text[:limit].rsplit(" ", 1)[0]
347-
return text
348-
349-
350308
def _clean_hashtags(tags) -> list[str]:
351309
"""One hashtag per entry, at most five.
352310

‎clips-studio.spec‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -224,6 +224,13 @@ _asd = ROOT / "models" / "pretrain_TalkSet.model"
224224
if _asd.exists():
225225
datas += [(str(_asd), ".")]
226226

227+
# The basketball detector (E-BARD's YOLOv8n, 6 MB), fetched by
228+
# scripts/fetch_basketball_model.py, for sports/basketball/crop. Without it a
229+
# basketball clip is framed with the COCO model and the rim finder.
230+
_bball = ROOT / "models" / "BODD_yolov8n_0001.pt"
231+
if _bball.exists():
232+
datas += [(str(_bball), ".")]
233+
227234
# Game-sound weights (PANNs MobileNetV1, 24 MB), fetched by
228235
# scripts/fetch_panns.py. Without them a gaming stream is scored on chat and
229236
# the streamer's voice only.

‎config/prompts/metadata_batch_highlights.txt‎

Lines changed: 0 additions & 19 deletions
This file was deleted.

0 commit comments

Comments
 (0)