Pushing video frame data to custom stream server

[mrtndat]

Hi,
I am using your library to push frame data to RTMP server and it works fine.
Now, I need to push that frame to 3rd library which does not expose a rtmp url. It means, I can not use startStream() API.
I want to use RootEncoder to capture camera frame, apply filter, then use that 3rd SDK to push this frame to its stream server.

What I tried.

1. Accessed the raw camera image using addImageListener. then convert the buffer to NV21 format and pass it to 3rd lib. It is working well, but it is missing the filters, because at this phase, the filter renderer is not applied yet.
2. Tried to intercept the VideoEncoder.sendBuffer(byteBuffer, bufferInfo) to send this byteBuffer to 3rd lib, but it seems this byteBuffer's format is not recognized. I tried I420, ARGB, NV21 but the 3rd did not recognize it.
3. Tried to expose the texture id of the Screen Renderer and pass this texture id and eglContext to the 3rd lib. However, I just got crash saying that the eglContext is not set, when using EGL14.eglGetCurrentContext() as eglContext, it results in a green frame.

What I intend to do.

1. Create a custom Filter to access the texture id and pass this id to 3rd but im not sure if it works because this Filter phase is supposed to do rendering not pushing.

My question is: Is there a callback that notifies a processed frame data, or provide a way to push the processed frame data to external consumer.

Thanks

[pedroSG94]

Hello,

In that case I recommend you create your own RtmpStream class. Copy this class:
https://github.com/pedroSG94/RootEncoder/blob/master/library/src/main/java/com/pedro/library/rtmp/RtmpStream.kt
Now, you can remove RtmpClient and use your own library instead of RtmpClient

After that, you can use the class exactly the same way you are currently using RootEncoder library.
This apply to all classes in the library. Just open the source code and replace the client class.

[mrtndat]

Hi @pedroSG94
You mean this method, right.
override fun getVideoDataImp(videoBuffer: ByteBuffer, info: MediaCodec.BufferInfo) {
rtmpClient.sendVideo(videoBuffer, info)
}

Instead of using rtmp client, I will use the 3rd lib to send the video buffer?
May i know what this buffer format is? is it NV21 or I420?

[pedroSG94]

Yes, that method.
The buffer is already encoded. H264, H265 or AV1 depend of your codec selected (h264 by default)

[mrtndat]

Thanks, but I have tried it, the 3rd lib only support sending frame in NV21, I420 or ARGB format.

[pedroSG94]

Hello,

In that case you can decode the frame. Copy this class (it is not the same in the current release, I did a modification to support this case):
https://github.com/pedroSG94/RootEncoder/blob/master/extra-sources/src/main/java/com/pedro/extrasources/BufferDecoder.kt
Now, you can decode frames using that class and the result should be nv21. Code example:

    val decoder = BufferDecoder(VideoCodec.H264)
    //use the same values than in prepareVideo, width, height, fps and rotation
    decoder.prepare(640, 480, 30, 0)
    //null because we want receive a buffer when decode instead of render a surface.
    //the return indicate the raw color received when decode. check MediaCodecInfo.CodecCapabilities values
    val resultColor = decoder.start(null)
    //call it when needed. null if fail or if you set a surface in the start method
    val rawBuffer = decoder.decode(data)
    //stop it after finish
    decoder.stop()

[mrtndat]

Thanks, but I can not access the BufferDecoder class from that link (it returns 404).

[pedroSG94]

I moved the class my bad:
https://github.com/pedroSG94/RootEncoder/blob/master/encoder/src/main/java/com/pedro/encoder/input/decoder/BufferDecoder.kt

[mrtndat]

I will try it and let you know.
Thanks

[mrtndat]

hi @pedroSG94
Thanks for your support. I have not tried your new class yet but I managed to get it work just to use the WIT#3 (What I tried) that I mentioned in the first post.
I create a new OnTextureFrameAvailable listener and invoke the method in the draw() method of the GlStreamInterface.

    // render VideoEncoder (stream and record)
    if (surfaceManagerEncoder.isReady && mainRender.isReady() && !limitFps) {
      val w = if (muteVideo) 0 else encoderWidth
      val h = if (muteVideo) 0 else encoderHeight
      surfaceManagerEncoder.makeCurrent()
      mainRender.drawScreenEncoder(w, h, orientation, streamOrientation,
        isStreamVerticalFlip, isStreamHorizontalFlip)
      if (surfaceManager.eglContext != EGL14.EGL_NO_CONTEXT) {
          onTexFrameAvailable?.invoke(
              surfaceManager.eglContext,
              mainRender.screenRender.texId,
              w,
              h,
              streamOrientation
          )
      }
      surfaceManagerEncoder.swapBuffer()
    }

The previous issue happens at the 3rd lib. I fixed it and it works now.
Thanks

[mrtndat]

Do you think the above change is safe @pedroSG94 ?

[pedroSG94]

Hello,

If it is working for you I see no problem.
In performance maybe is the best solution, because you skip video encoding and decoding.

[mrtndat]

thanks