Trying to build an overlay one video or another

[saket-rev]

I want tried to create a video in which one video overlay another video the main video in whole screen and overlay in right bottom

Here is the code -

import java.io.*;
import lombok.extern.slf4j.Slf4j;
import org.bytedeco.javacv.*;
import org.bytedeco.ffmpeg.global.avutil;
import org.bytedeco.ffmpeg.avutil.AVRational.*;

@Slf4j
public class VideoGen {
    public static void main(String[] args) {
        try {
            // Enable FFmpeg logging
            FFmpegLogCallback.set();
            avutil.av_log_set_level(avutil.AV_LOG_DEBUG);  // Set FFmpeg log level to DEBUG

            // Set paths for the user video and agent video (local file paths)
            String userVideoPath = "/Users/saket.prasad/Downloads/user_video.mp4";
            String agentVideoPath = "/Users/saket.prasad/Downloads/agent_video.mp4";
            String outputDir = "/Users/saket.prasad/Downloads";

            // Generate a timestamp for the file name
            String timeStamp = String.valueOf(System.currentTimeMillis());
            timeStamp = timeStamp.substring(timeStamp.length() / 2);

            String customerAttemptId = "1234";

            // Set the output file name based on customer attempt ID and timestamp
            String outputFileName = "output_video_" + customerAttemptId + "_" + timeStamp + ".mp4";
            String outputPath = outputDir + File.separator + outputFileName;
            log.info("Local output video filename for attemptId - " + customerAttemptId + " : " + outputPath);

            // Ensure the output directory exists
            File outputDirFile = new File(outputDir);
            if (!outputDirFile.exists()) {
                outputDirFile.mkdirs();  // Creates the directory if it doesn't exist
            }

            File outputVideo = new File(outputPath);
            log.info("Output video file created for attemptId - " + customerAttemptId + " : " + outputVideo.getPath());

            try (InputStream userInputStream = new FileInputStream(userVideoPath);
                 InputStream agentInputStream = new FileInputStream(agentVideoPath)) {

                log.info("FileInputStream created for user video and agent video for attemptId - " + customerAttemptId);

                // Initialize the frame grabber for both videos
                try (FFmpegFrameGrabber userGrabber = new FFmpegFrameGrabber(userInputStream);
                     FFmpegFrameGrabber agentGrabber = new FFmpegFrameGrabber(agentInputStream)) {

                    userGrabber.start();
                    agentGrabber.start();


                    if (userGrabber.getVideoCodec() == 0 || agentGrabber.getVideoCodec() == 0) {
                        throw new Exception("One of the video codecs is invalid or not found.");
                    }

                    // Check if the time base is properly set (not null)
                    if (userGrabber.getFormatContext().streams(0).time_base() == null) {
                        log.error("Base video time base is null.");
                    }


                    // Ensure grabbers started successfully
                    if (userGrabber.getLengthInFrames() < 0 || agentGrabber.getLengthInFrames() < 0) {
                        log.error("Video grabbers failed to initialize correctly.");
                        return;
                    }

                    // Initialize the frame recorder to write to the output video
                    try (FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(outputVideo, userGrabber.getImageWidth(), userGrabber.getImageHeight())) {
                        recorder.setVideoCodec(userGrabber.getVideoCodec()); // Keep original video codec
                        recorder.setAudioChannels(userGrabber.getAudioChannels());  // Keep original audio channels
                        recorder.setAudioCodec(userGrabber.getAudioCodec());  // Copy audio
                        recorder.setFrameRate(userGrabber.getFrameRate());  // Set the frame rate
                        recorder.setVideoBitrate(2000000);  // Optionally set a fixed bitrate for more control over file size
                        recorder.start();

                        // Set the overlay filter (agent video on top of user video)
                        String overlayFilter = "[1:v]scale=" + userGrabber.getImageWidth() + ":" + userGrabber.getImageHeight() + "[scaled];[0:v][scaled]overlay=W-w-10:H-h-10";  // Match user video dimensions
                        try (FFmpegFrameFilter frameFilter = new FFmpegFrameFilter(overlayFilter, userGrabber.getImageWidth(), userGrabber.getImageHeight())) {
                            frameFilter.setPixelFormat(avutil.AV_PIX_FMT_YUV420P); // Ensure pixel format is set
                            frameFilter.start();

                            log.info("Frame filter started for " + customerAttemptId);

                            Frame userFrame, agentFrame;
                            while ((userFrame = userGrabber.grab()) != null && (agentFrame = agentGrabber.grab()) != null) {
                                // Push frames into the filter
                                frameFilter.push(userFrame, 0); // User video frame
                                frameFilter.push(agentFrame, 1); // Agent video frame

                                // Pull the filtered frame (overlayed video)
                                Frame filteredFrame = frameFilter.pull();

                                // Record the filtered frame (overlayed video)
                                recorder.record(filteredFrame);
                            }

                            log.info("Overlay added to the video successfully for attemptId : " + customerAttemptId);
                        }
                    }
                }
            } catch (Exception e) {
                log.error("Exception adding video overlay for attemptId: " + customerAttemptId, e);
            }

            // Return the output file path as confirmation
            log.info("Processed video saved locally at: " + outputPath);

        } catch (Exception e) {
            e.printStackTrace();
            log.error("An error occurred during video processing: ", e);
        }
    }
}`
```


But getting this error -

org.bytedeco.javacv.FFmpegFrameFilter$Exception: avfilter_graph_config() error -22 (For more details, make sure FFmpegLogCallback.set() has been called.)
	at org.bytedeco.javacv.FFmpegFrameFilter.startVideoUnsafe(FFmpegFrameFilter.java:411)
	at org.bytedeco.javacv.FFmpegFrameFilter.startUnsafe(FFmpegFrameFilter.java:308)
	at org.bytedeco.javacv.FFmpegFrameFilter.start(FFmpegFrameFilter.java:283)
	at com.whizdm.videokyc.core.document.VideoGen.main(VideoGen.java:84)
	Suppressed: java.lang.NullPointerException: Cannot invoke "org.bytedeco.ffmpeg.avutil.AVRational.releaseReference()" because "this.time_base" is null
		at org.bytedeco.javacv.FFmpegFrameFilter.releaseUnsafe(FFmpegFrameFilter.java:170)
		at org.bytedeco.javacv.FFmpegFrameFilter.release(FFmpegFrameFilter.java:151)
		at org.bytedeco.javacv.FFmpegFrameFilter.stop(FFmpegFrameFilter.java:519)
		at org.bytedeco.javacv.FrameFilter.close(FrameFilter.java:142)
		at com.whizdm.videokyc.core.document.VideoGen.main(VideoGen.java:82)

[saudet]

We can easily accomplish this with the ffmpeg program:
http://bytedeco.org/javacpp-presets/ffmpeg/apidocs/org/bytedeco/ffmpeg/ffmpeg.html