validate model translation output before accepting a batch

[spgsroot]

Problem

A structurally valid model response is not necessarily a complete translation. Model-backed providers can return the expected JSON and segment count while leaving part of a translated text value in the source language.

Koharu currently accepts such a response as successful. In a batch workflow, the invalid segment may survive until manual review or export.

Reproduction

Tested through Koharu's OpenAI-compatible translator path with:

• model: HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF
• source: English (en-US)
• target: Russian (ru-RU)
• input:

If I'll make him cum first like last time...
Shlurbs Shilurp

With an explicit source language and Koharu's built-in prompt, the translator returned:

[
  "Если я’ll make him cum first like last time...",
  "Шлюрб Шилурп"
]

A stricter system prompt containing both NEVER output Latin letters and Every text value MUST be in Russian Cyrillic still returned:

{
  "translations": [
    {
      "id": 0,
      "text": "Если я заставлю him come first like last time..."
    }
  ]
}

Both responses satisfy the JSON and segment-count contracts but violate the requested translation contract. An explicit source language and stronger prompt instructions reduce ambiguity but do not guarantee complete translation.

Proposed behavior

Allow optional validation rules for translated text fields. For example, a Cyrillic-only English-to-Russian workflow can reject:

[A-Za-z]

For model-backed providers:

1. Generate the complete response.
2. Validate every translated text field.
3. Preserve every valid translation.
4. Retry only the invalid source segments, with the invalid translated fields and violated rules in corrective feedback.
5. Merge corrected segments back into their original positions.
6. Validate the merged result and retry the still-invalid subset once more.
7. If a segment remains invalid after two targeted retries, return an explicit validation error instead of accepting known-bad output.

This avoids regenerating an otherwise valid batch because one segment failed validation. It also gives the model a smaller correction task while retaining bounded, fail-closed behavior.

For providers that cannot consume corrective prompts, return the validation error without retrying.

Validation should be opt-in. Workflows that legitimately retain Latin characters, brand names, URLs, or identifiers can leave the rule disabled or configure a narrower expression.

Practical retry result

Generic feedback that only named the forbidden rule did not correct this model. With deterministic generation it still produced:

{
  "translations": [
    {
      "id": 0,
      "text": "Если я заставлю его конquer first like last time..."
    }
  ]
}

When corrective feedback included the actual invalid translated field, the same model corrected the segment. The targeted-retry implementation was then exercised through Translator::translate on a three-segment batch:

[
  "Доброе утро.",
  "Если я заставлю его кончить первым, как в прошлый раз...",
  "Шлёп, шлёп"
]

All fields pass [A-Za-z], and already-valid translations remain in their original positions.

The implementation experiment is in #1007, latest update: spgsroot/koharu@8234ade2.

Quality and availability impact

This does not claim that a regex can judge general translation quality. It enforces a user-selected, objectively testable postcondition.

Without validation, Koharu silently commits a known-invalid partial translation. A fail-open fallback after retries would restore that same failure mode and make the validator unreliable. The proposed behavior instead reduces batch failures by isolating correction to invalid segments, while still returning an explicit error if the configured invariant cannot be satisfied.

At most three provider calls are made: one complete batch and two retries containing only the remaining invalid segments. Providers without corrective-prompt support do not make redundant requests.

Proposed acceptance criteria

• Validation is optional and disabled by default.
• Rules are compiled when configuration or the translation processor is created, not per segment.
• Only translated text fields are checked.
• Valid translations are preserved across retries.
• Only invalid source segments are regenerated.
• Corrective feedback identifies the invalid fields and matched rules.
• Model-backed providers perform at most two targeted retries.
• A still-invalid result returns an explicit validation error.
• Providers without corrective-prompt support do not retry.
• Existing JSON and segment-count validation remains unchanged.

Product scope questions

A previous implementation was submitted as #1007 and closed because this product behavior had not been agreed with maintainers first. I am opening this issue before requesting review again.

The previous PR also included explicit source-language selection and a full system-prompt override. Those controls can be considered separately from deterministic output validation:

1. Would maintainers accept optional output validation with targeted, fail-closed retries?
2. Should Koharu expose custom regex rules, language-specific presets, or both?
3. Should explicit source-language selection be part of the same change or a separate proposal?
4. Should a custom prompt replace the complete system prompt, or should Koharu always retain its structural output invariants?

[spgsroot]

Updated the prototype after reproducing the pipeline failure shown above. The translator now preserves valid batch results and retries only the remaining invalid source segments, with two bounded targeted retries. The reproduced three-segment batch completed successfully through the real Translator::translate path. Implementation: spgsroot/koharu@8234ade2; focused translator tests: 57 passed.