Skip to content

Fix NaFlex non-base patch sizes and fail fast on unsupported towers - #1195

Open
chenghuichen wants to merge 1 commit into
mlfoundations:mainfrom
chenghuichen:fix-patch
Open

Fix NaFlex non-base patch sizes and fail fast on unsupported towers#1195
chenghuichen wants to merge 1 commit into
mlfoundations:mainfrom
chenghuichen:fix-patch

Conversation

@chenghuichen

@chenghuichen chenghuichen commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Wire OpenCLIP's NaFlex data path to timm's existing patch interpolation support, so --naflex-patch-sizes works with non-base patch sizes.

  • Enable timm's parameter-free patch interpolator for compatible NaFlex image towers.
  • Pass the model's base patch size and interpolation capability to the data config.
  • Flatten base-size patches only; preserve [N, Ph, Pw, C] for non-base train and eval patches.
  • Fail early when a tower without interpolation support is used with a non-base patch size.

Problem

OpenCLIP exposes --naflex-patch-sizes, but did not fully connect non-base patches to timm's interpolator. Multi-size
training could reach a model with interpolation disabled, while a single non-base size and eval patches were flattened
before the model could recover their spatial dimensions. These paths failed with:

RuntimeError: mat1 and mat2 shapes cannot be multiplied

Fix

Patch flattening is now based on the model's base patch size rather than the number of configured choices:

base patch      -> [N, P*P*C]
non-base patch  -> [N, Ph, Pw, C]

This rule is applied consistently to training and evaluation. Compatible towers automatically enable timm's existing
interpolator; unsupported towers fail during configuration instead of later in matrix multiplication. The interpolator
adds no checkpoint parameters or buffers.

Tests

Added coverage for 16/32 train and eval paths, forward/backward, checkpoint round trips, automatic enablement, and
unsupported-tower fail-fast behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant