Extracted Frames Are Horizontally Oriented, Unable to Retrieve Rotation Metadata (displaymatrix: rotation of -90.00 degrees) in JavaCV

[TenShenSun]

Problem Description:
I am extracting frames from a video using JavaCV (version 1.5.7, which internally depends on OpenCV 4.5.5). However, I encountered the following issues:

The extracted frames are horizontally oriented, even though the video should be played vertically.
I am unable to retrieve the rotation metadata (displaymatrix: rotation of -90.00 degrees), which is printed in the FFmpeg logs but not accessible via JavaCV.
Behavior differs from OpenCV-Python:
In OpenCV-Python (versions below 4.11.0), the grabbed frames are automatically rotated correctly.
In JavaCV, the grabbed frames remain unrotated (horizontal).
JavaCV Dependencies Used:

<!-- JavaCV dependency -->
<dependency>
    <groupId>org.bytedeco</groupId>
    <artifactId>javacv</artifactId>
    <version>1.5.7</version>
</dependency>
<!-- JavaCV platform dependency to include platform-specific binaries -->
<dependency>
    <groupId>org.bytedeco</groupId>
    <artifactId>javacv-platform</artifactId>
    <version>1.5.7</version>
</dependency>

Code for Frame Extraction:

public static void extractFramesFromByteArray(byte[] videoBytes, String outputDir) {
    Java2DFrameConverter frameConverter = new Java2DFrameConverter();
    try (ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(videoBytes);
         FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(byteArrayInputStream);
    ) {
        frameGrabber.start(); // Start the grabber
        int frameNumber = 0;

        Frame frame;
        while ((frame = frameGrabber.grabImage()) != null) {
            // Convert Frame to BufferedImage
            BufferedImage bufferedImage = frameConverter.getBufferedImage(frame);

            if (bufferedImage != null) {
                // Save the frame as an image file
                String imageName = String.format("frame_w%s_h%s_%s.png",
                        bufferedImage.getWidth(), bufferedImage.getHeight(), frameNumber);
                File outputFile = new File(outputDir, imageName);
                ImageIO.write(bufferedImage, "png", outputFile);
                frameNumber++;
            }
        }
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}

FFmpeg Logs Printed by JavaCV (av_dump_format in FFmpegFrameGrabber):
During debugging, I noticed that JavaCV internally logs the following metadata when calling:
av_dump_format(oc, 0, filename, 0);

FFmpeg Log Output:

Metadata:
    major_brand     : iso5
    minor_version   : 1
    compatible_brands: isomiso5hlsf
    creation_time   : 2025-03-17T07:40:53.000000Z
  Duration: 00:00:07.77, start: 0.000000, bitrate: 1029 kb/s
  Stream #0:0[0x1](und): Video: h264 (Baseline) (avc1 / 0x31637661), yuvj420p(pc, progressive), 640x480, 1025 kb/s, 30.11 fps, 29.92 tbr, 600 tbn (default)
    Metadata:
      creation_time   : 2025-03-17T07:40:53.000000Z
      handler_name    : Core Media Video
      vendor_id       : [0][0][0][0]
    Side data:
       displaymatrix: rotation of -90.00 degrees**

Issues:
Unable to retrieve the displaymatrix rotation metadata
I tried using frameGrabber.getVideoMetadata("rotate"), but it returns null.
This suggests that JavaCV does not expose displaymatrix directly, unlike ffprobe, which correctly retrieves the rotation metadata.
Extracted frames in JavaCV are horizontal, but in OpenCV-Python (versions < 4.11.0), they are automatically corrected
OpenCV-Python applies rotation automatically, whereas JavaCV does not.
How can I make JavaCV behave the same way?
Questions:

1. How can I retrieve displaymatrix: rotation of -90.00 degrees in JavaCV?
2. Why does OpenCV-Python automatically apply rotation, but JavaCV does not?
3. What is the best way to correctly rotate extracted frames in JavaCV, given that the rotation metadata is not directly accessible?

[saudet]

Duplicate of #1361 (comment)

[TenShenSun]

thks!!!

[TenShenSun]

getDisplayRotation() available in JavaCV 1.5.8 but missing in 1.5.11 – Will it be introduced?

I upgraded JavaCV from version 1.5.8 to 1.5.11, and I noticed that the method getDisplayRotation() is no longer available in 1.5.11.

In 1.5.8, I was able to retrieve the video rotation using:

double rotation = grabber.getDisplayRotation();

However, after upgrading to 1.5.11, this method no longer exists, and I couldn't find any alternative in the documentation or release notes.

My Questions:
Will this method be introduced in a future release?
Is there an alternative way to get the video rotation in JavaCV 1.5.11?
If I want to use both the latest features of 1.5.11 and getDisplayRotation(), what would be the best approach?
Dependencies Used:

<dependency>
    <groupId>org.bytedeco</groupId>
    <artifactId>javacv</artifactId>
    <version>1.5.8</version>
</dependency>

<dependency>
    <groupId>org.bytedeco</groupId>
    <artifactId>javacv-platform</artifactId>
    <version>1.5.8</version>
</dependency>

Any insights or recommendations would be greatly appreciated! @saudet

[saudet]

It's still there:
https://bytedeco.org/javacv/apidocs/org/bytedeco/javacv/FFmpegFrameGrabber.html#getDisplayRotation--

[TenShenSun]

Oh, I re-tested it, and it actually works! It seems there might have been some issues with my previous testing process. Thank you so much for your patience, help, and detailed explanations!