Some questions about merging with latest jsmpeg version

[roelvz]

Hello,

Do you think it's feasible to merge the latest version of jsmpeg into jsmpeg-vnc? I would like to do this because the latest jsmpeg version includes an audio decoder. I first tried with video only, but it doesn't work out of the box. The decoder thinks the received steam is garbage. I guess this is because the c++ part does not send a valid MPEG1-stream (because of a custom header each frame).

I tried to remove the custom header, but this does not seem to be working. Do you think it should be that simple? Or is there anything else I could be missing?

A better solution would be to take the custom header into account in the newest version of jsmpeg. Would this be hard? Could you give some directions on where to start?

For the audio part in the c++ code, I think this is pretty straightforward, but any tips are welcome.

Thank you

[phoboslab]

For jsmpeg-vnc to work with the current jsmpeg version, it needs to send out an MPEGTS Stream. FFMpeg has a muxer to create this stream. So, yeah, it's certainly possible.

I want to come back to this project eventually, but I'm buried with "real" work at the moment. Don't hold your breath :/

[roelvz]

Just for info, I wrote c++ code (started from jsmpeg-vnc for the idea but it turned out to me completely new code) that works together with the latest version of jsmpeg and also includes audio.

[stefan-scott]

@roelvz Is your version (with sound) published anywhere? I would be interested to check it out!

[dragonballa]

Just for info, I wrote c++ code (started from jsmpeg-vnc for the idea but it turned out to me completely new code) that works together with the latest version of jsmpeg-vnc and also includes audio.

Hi Roel....by any chance you could share this?

[roelvz]

I cannot make the code available, but here are some tips that may be helpful:

• Instead of only encoding the video (and audio), after the encoding you'll also have to use a muxer to output an MPEGTS stream.
• Encoding in ffmpeg is done using libavcodec and is already done in jsmpeg-vnc. Encoding audio is similar. You'll need to use AV_CODEC_ID_MPEG1VIDEO for video and AV_CODEC_ID_MP2 for audio with 2 channels. This is what I did back then for the audio:

    _audioCodec = avcodec_find_encoder(AV_CODEC_ID_MP2);
    _audioEncoder = avcodec_alloc_context3(_audioCodec);
    _audioEncoder->sample_rate = outputSampleRate;
    _audioEncoder->sample_fmt = AV_SAMPLE_FMT_S16;
    _audioEncoder->channel_layout = av_get_default_channel_layout(2);
    _audioEncoder->channels = av_get_channel_layout_nb_channels(_audioEncoder->channel_layout);
    _audioEncoder->bit_rate = audioBitrate;
    _audioEncoder->time_base = AVRational{1, _audioEncoder->sample_rate};
    auto result = avcodec_open2(_audioEncoder, _audioCodec, nullptr);

• Once you have the encoded video and audio, you need to mux them. Ffmpeg uses libavformat for this. Useful functions for initialisation are avformat_alloc_context, av_guess_format, avformat_new_stream, avcodec_parameters_from_context and avformat_write_header. For getting the data onto the WebSocket you can use avio_alloc_context.
• The actual encoding happens with avcodec_send_frame and avcodec_receive_packet for both video and audio.
• The actual muxing happens with av_write_frame for both video and audio.
• You may have to resample the audio first. Check out swr_alloc (init) and swr_convert (real time). This happens before encoding off course (comparable to the scaling for video, which is already present in jsmpeg-vnc).

Note: I'm not using this code anymore and it was all build to work with ffmpeg 4.0.2. Some of these functions are probably deprecated or deleted in recent ffmpeg versions.