Sending more speaker samples into the diarization pipeline than actual speakers causes pipeline to emit more speakers than are actually present

[agileramblings]

Update: Sorry about the churn on this issue. As I investigate and spend more time (and tokens on it), I'm refining what I should have actually posted. The unexpected behaviour is still true. Just the details provided by the agent are changing. I'll try to do better.

I'm building an app that does real-time transcription and diarization of a meeting. It uses the Speaches v0.9.0-rc.3 in a docker container running on the local host.

I'm sending known speakers (identified in previous sessions) into the diarization for follow-up sessions so that I don't have to re-associate them every time. New Speakers would be expected to show up as SPEAKER_NN and need to be identified (by my app).

My app currently sends all known speakers to diarization and hopes that Speaches would use those embedding to identify clusters, but wouldn't use those embeddings with the implication that they must all be in the meeting.

Here is the report from my agent.

Diarization: known speakers are not optional hints (unmatched speakers get force-labeled); no speaker-count controls

Summary

POST /v1/audio/diarization treats every known_speaker_reference as guaranteed present: any diarized speaker is relabeled with the closest supplied name, regardless of how poor the match is. There is no confidence threshold and no way to bound the speaker count, so passing a "known voices" catalog larger than the set of people actually in the recording produces mislabeled speakers rather than leaving the extras as SPEAKER_NN.

▎ Note: an earlier version of this issue described the reference count forcing min_speakers (via the old onnx-diarization pipeline). That was fixed by the switch to pyannote (commit 023ad86) — clustering is now independent of reference count. The gaps below remain on main.

Environment

- Endpoint: POST /v1/audio/diarization
- Model: pyannote/speaker-diarization-community-1 (pyannote-audio >= 4.0.3)

Actual behavior

Matching in _map_to_known_speakers (src/speaches/routers/diarization.py) picks the nearest known speaker by cosine similarity with no minimum threshold (best_sim = -2.0), so every diarized speaker is always assigned some supplied name:

best_name = diarized_spk
best_sim = -2.0
for known_name, known_emb in known_embeddings.items():
    ...
    sim = float(np.dot(diarized_emb, known_emb) / denom)
    if sim > best_sim:
        best_sim = sim
        best_name = known_name
mapping[diarized_spk] = best_name

Consequences:
- A real speaker who has no reference is still relabeled as whichever supplied name is least dissimilar, instead of remaining SPEAKER_NN.
- The endpoint exposes no num_speakers / min_speakers / max_speakers, so callers cannot bound or fix the detected speaker count even though pyannote community-1 supports all three as pipeline call kwargs.

Steps to reproduce

1. Take a recording with K real speakers (e.g. K = 3).
2. Prepare M > K reference clips (16 kHz mono WAV as data:audio/wav;base64,...), including the K real speakers plus voices not present.
3. POST multipart/form-data with file, response_format=json, and repeated known_speaker_names[] / known_speaker_references[] pairs.
4. Observe that speakers with no matching reference are still assigned one of the supplied names (mislabeled), rather than kept as SPEAKER_NN.

Expected behavior

Known speakers should be optional hints: a reference is applied only when its similarity clears a confidence threshold; otherwise the diarized speaker keeps its SPEAKER_NN label. Callers should optionally be able to constrain the speaker count.

Suggested fix

1. Add a known_speaker_threshold form param (default cosine-similarity match, so unmatched speakers staySPEAKER_NN.
2. Add num_speakers / min_speakers / max_speakers forannote pipeline call.

Separate minor quirk

Reference fields must be sent with a [] suffix (knownker_references[]) because the router binds them viaForm(alias=...). The un-suffixed spelling shown in openapi.json is silently ignored (HTTP 200, references dropped), and mismatched name/reference list lengths raise a 500 via zip(...,  both spellings and returning a 422 on lengthmismatch. (Can be split into its own issue.)

Is this something that could be adjusted? I'm adding pre-participant curation to my app, but it would be cool that my app could just diarize with known speakers and find whomever (and only whomever) was actually in the meeting.

I'm hoping to submit a PR for this code, but leaving this here until it is addressed.

[agileramblings]

Draft PR submitted with fixes.