Skip to content

Make Accuracy Checker annotation conversion deterministic - #4053

Merged
sstrehlk merged 2 commits into
openvinotoolkit:masterfrom
pwolnows:fix_classification_adapter
Sep 3, 2026
Merged

sstrehlk merged 2 commits into
openvinotoolkit:masterfrom
pwolnows:fix_classification_adapter

Conversation

@pwolnows

@pwolnows pwolnows commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Description

This PR improves Accuracy Checker validation reliability in two areas:

  • selects classification outputs explicitly when a model exposes multiple outputs;
  • makes filesystem-derived annotation ordering deterministic before annotation or subset selection.

Changes

Classification adapter

  • Prefer an explicitly configured output name when resolving classification model outputs.
  • Preserve the existing fallback behavior for models without an explicit output selection.

Deterministic annotation conversion

  • Sort image inputs in ImageProcessingConverter, including recursive inputs.
  • Sort generated reference images in ParametricImageProcessing before creating Denoise annotations.
  • Sort LibriSpeech transcript files before creating annotations.
  • Sort CamVid-32 label files before its val_subset_ratio logic selects the validation subset.
  • Warn when a parametric image-processing input has no generated reference pair.

Sorting removes dependence on filesystem directory-entry order. Consequently, the same dataset contents produce the same annotation order and the same fixed-seed subset after a dataset restore or annotation-cache regeneration.

Affected Validation Workloads

The deterministic ordering change covers the currently observed unstable validation subsets:

  • Denoise
  • Colorization
  • QuartzNet 15x5 English
  • Tiramisu FC-DenseNet-103 / CamVid-32

Validation

  • Ran python3 -m compileall for the modified annotation converter modules.
  • Ran git diff --check.
  • Verified all filesystem enumeration paths touched by the four affected converters are explicitly sorted.
  • Checked editor diagnostics for modified converter modules: no errors.

Follow-up

Regenerate validation references for the affected workloads using the deterministic annotation order. Additional annotation converters identified by the broader audit will be addressed separately.

@pwolnows pwolnows changed the title Fix classification adapter explicit output selection Make Accuracy Checker annotation conversion deterministic Sep 3, 2026
@sstrehlk
sstrehlk merged commit 2f37066 into openvinotoolkit:master Sep 3, 2026
6 of 14 checks passed
@pwolnows
pwolnows deleted the fix_classification_adapter branch September 3, 2026 07:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants