bytedeco/javacv

Synchronization between multiple streamed audio RTP and low latency.

Open

#1,091 建立於 2018年11月20日

在 GitHub 查看
 (9 留言) (0 反應) (0 負責人)Java (1,583 fork)batch import
enhancementhelp wantedquestion

倉庫指標

Star
 (6,985 star)
PR 合併指標
 (30 天內沒有已合併 PR)

描述

Hello, everyone I'm using JavaCV for almost 5 months I was able to implement a basic application that receives multiple RTSP streams from Android phones as shown in this image. screenshot from 2018-11-20 18-06-49

My implementation creates a thread for each device and instantiate a FFmpegFrameGrabber object with the RTSP address and port to the device stream. The delay is noticeable and varies between 0.8 ~ 1.2 seconds approximately.

So I have two questions. It is possible to achieve real-time stream or reduce the delay to 0.2 seconds? I can achieve latency (between 0.2 to 0.3 seconds) when I run this command:

ffplay -fflags nobuffer -flags low_delay -framedrop \
 -strict experimental -rtsp_transport tcp rtsp://<host>:<port>

For example the AudioPlayer has this code:

public AudioPlayer(String source, GrabberListener grabberListener)
{
    if (grabberListener == null) return;
    if (source.isEmpty()) return;
    Thread playThread = new Thread(() -> {
        try {
            FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(source);
            grabber.setOption("rtsp_transport", "tcp");
            grabber.start();

            playing = true;
            LogHelper.e(TAG, "audioPlaying(on)");

            if (grabber.getSampleRate() > 0 && grabber.getAudioChannels() > 0) {
                AudioFormat audioFormat = new AudioFormat(grabber.getSampleRate(), 16, grabber.getAudioChannels(), true, true);

                DataLine.Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
                soundLine = (SourceDataLine) AudioSystem.getLine(info);
                soundLine.open(audioFormat);
                soundLine.start();

                gainControl = (FloatControl) soundLine.getControl(FloatControl.Type.MASTER_GAIN);
                grabberListener.onGainControl(gainControl);
            }

            ExecutorService executor = Executors.newSingleThreadExecutor();
            while (running.get()) {
                Frame frame = grabber.grab();
                if (frame == null) break;

                if (frame.samples != null) {
                    ShortBuffer channelSamplesFloatBuffer = (ShortBuffer) frame.samples[0];
                    channelSamplesFloatBuffer.rewind();

                    ByteBuffer outBuffer = ByteBuffer.allocate(channelSamplesFloatBuffer.capacity() * 2);
                    float[] samples = new float[channelSamplesFloatBuffer.capacity()];

                    float lastPeak = 0f;

                    for (int i = 0; i < channelSamplesFloatBuffer.capacity(); i++) {
                        short val = channelSamplesFloatBuffer.get(i);
                        outBuffer.putShort(val);
                    }

                    // Convert bytes to samples here
                    for (int i = 0, s = 0; i < channelSamplesFloatBuffer.capacity(); ) {
                        int sample = 0;

                        sample |= channelSamplesFloatBuffer.get(i++) & 0xFF; // (reverse these two lines
                        sample |= channelSamplesFloatBuffer.get(i++) << 8;   //  if the format is big endian)

                        // Normalize to range of +/-1.0f
                        samples[s++] = sample / 32768f;
                    }

                    float rms = 0f;
                    float peak = 0f;
                    for (float sample : samples) {
                        float abs = Math.abs(sample);
                        if (abs > peak) {
                            peak = abs;
                        }

                        rms += sample * sample;
                    }

                    rms = (float) Math.sqrt(rms / samples.length);

                    if (lastPeak > peak) {
                        peak = lastPeak * 0.875f;
                    }

                    lastPeak = peak;

                    grabberListener.onAudioSpectrum(rms, peak);

                    if (soundLine == null) return;
                    try {
                        executor.submit(() -> {
                            soundLine.write(outBuffer.array(), 0, outBuffer.capacity());
                            outBuffer.clear();
                        }).get();
                    } catch (InterruptedException interruptedException) {
                        Thread.currentThread().interrupt();
                    }
                }
            }

            executor.shutdownNow();
            executor.awaitTermination(1, SECONDS);

            playing = false;
            LogHelper.e(TAG, "audioPlaying(off)");

            if (soundLine != null) {
                soundLine.stop();
            }

            grabber.stop();
            grabber.release();
            grabberListener.onStop();
        } catch (Exception e) {
            LogHelper.e(TAG, e);
            playing.set(false);
            grabberListener.onError(new RuntimeException("Could not open input stream.", e));
        }
    });
    playThread.start();
}

And my second question is It is possible to synchronize these RTP streams? probably using the NTP timestamp? However, the NTP timestamp is in the Sender Report RTCP Packet can I have access to that packet from the FFmpegFrameGrabber??

貢獻者指南