ImageFilter.BoxBlur(float("nan")) causes a heap-buffer-overflow

[marinelay]

I've been fuzzing Python C extension modules for a small research project, and this one came up.
It reproduces with the binary wheel from a plain pip install Pillow.

Versions

Pillow 12.3.0 binary wheel and Pillow main commit (807d689a83738027b6f6e0f219a6a6dd30e01c08), CPython 3.12.3, Ubuntu 24.04 x86_64, glibc 2.39.

Reproducer

from PIL import Image, ImageFilter

Image.new("RGB", (1, 1)).filter(ImageFilter.BoxBlur(float("nan")))

Running it produces:

$ python reproducer.py
Segmentation fault (core dumped)

An ASan build reports invalid NaN-to-integer conversion followed by a heap-buffer-overflow write.

heap-buffer-overflow WRITE of size 1
    #0 ImagingLineBoxBlur32 src/libImaging/BoxBlur.c:73
    #1 ImagingHorizontalBoxBlur src/libImaging/BoxBlur.c:212
    #2 ImagingBoxBlur src/libImaging/BoxBlur.c:273

I would expect a non-finite radius to be rejected rather than converted into overflowing native edge arithmetic.