Fix NaFlex non-base patch sizes and fail fast on unsupported towers - #1195
Open
chenghuichen wants to merge 1 commit into
Open
Fix NaFlex non-base patch sizes and fail fast on unsupported towers#1195chenghuichen wants to merge 1 commit into
chenghuichen wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wire OpenCLIP's NaFlex data path to timm's existing patch interpolation support, so
--naflex-patch-sizesworks with non-base patch sizes.[N, Ph, Pw, C]for non-base train and eval patches.Problem
OpenCLIP exposes
--naflex-patch-sizes, but did not fully connect non-base patches to timm's interpolator. Multi-sizetraining could reach a model with interpolation disabled, while a single non-base size and eval patches were flattened
before the model could recover their spatial dimensions. These paths failed with:
Fix
Patch flattening is now based on the model's base patch size rather than the number of configured choices:
This rule is applied consistently to training and evaluation. Compatible towers automatically enable timm's existing
interpolator; unsupported towers fail during configuration instead of later in matrix multiplication. The interpolator
adds no checkpoint parameters or buffers.
Tests
Added coverage for 16/32 train and eval paths, forward/backward, checkpoint round trips, automatic enablement, and
unsupported-tower fail-fast behavior.