bytedeco/javacv

Synchronization between multiple streamed audio RTP and low latency.

Open

#1.091 aberto em 20 de nov. de 2018

Ver no GitHub
 (9 comments) (0 reactions) (0 assignees)Java (1.583 forks)batch import
enhancementhelp wantedquestion

Métricas do repositório

Stars
 (6.985 stars)
Métricas de merge de PR
 (Nenhuma PRs mesclada em 30d)

Description

Hello, everyone I'm using JavaCV for almost 5 months I was able to implement a basic application that receives multiple RTSP streams from Android phones as shown in this image. screenshot from 2018-11-20 18-06-49

My implementation creates a thread for each device and instantiate a FFmpegFrameGrabber object with the RTSP address and port to the device stream. The delay is noticeable and varies between 0.8 ~ 1.2 seconds approximately.

So I have two questions. It is possible to achieve real-time stream or reduce the delay to 0.2 seconds? I can achieve latency (between 0.2 to 0.3 seconds) when I run this command:

ffplay -fflags nobuffer -flags low_delay -framedrop \
 -strict experimental -rtsp_transport tcp rtsp://<host>:<port>

For example the AudioPlayer has this code:

public AudioPlayer(String source, GrabberListener grabberListener)
{
    if (grabberListener == null) return;
    if (source.isEmpty()) return;
    Thread playThread = new Thread(() -> {
        try {
            FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(source);
            grabber.setOption("rtsp_transport", "tcp");
            grabber.start();

            playing = true;
            LogHelper.e(TAG, "audioPlaying(on)");

            if (grabber.getSampleRate() > 0 && grabber.getAudioChannels() > 0) {
                AudioFormat audioFormat = new AudioFormat(grabber.getSampleRate(), 16, grabber.getAudioChannels(), true, true);

                DataLine.Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
                soundLine = (SourceDataLine) AudioSystem.getLine(info);
                soundLine.open(audioFormat);
                soundLine.start();

                gainControl = (FloatControl) soundLine.getControl(FloatControl.Type.MASTER_GAIN);
                grabberListener.onGainControl(gainControl);
            }

            ExecutorService executor = Executors.newSingleThreadExecutor();
            while (running.get()) {
                Frame frame = grabber.grab();
                if (frame == null) break;

                if (frame.samples != null) {
                    ShortBuffer channelSamplesFloatBuffer = (ShortBuffer) frame.samples[0];
                    channelSamplesFloatBuffer.rewind();

                    ByteBuffer outBuffer = ByteBuffer.allocate(channelSamplesFloatBuffer.capacity() * 2);
                    float[] samples = new float[channelSamplesFloatBuffer.capacity()];

                    float lastPeak = 0f;

                    for (int i = 0; i < channelSamplesFloatBuffer.capacity(); i++) {
                        short val = channelSamplesFloatBuffer.get(i);
                        outBuffer.putShort(val);
                    }

                    // Convert bytes to samples here
                    for (int i = 0, s = 0; i < channelSamplesFloatBuffer.capacity(); ) {
                        int sample = 0;

                        sample |= channelSamplesFloatBuffer.get(i++) & 0xFF; // (reverse these two lines
                        sample |= channelSamplesFloatBuffer.get(i++) << 8;   //  if the format is big endian)

                        // Normalize to range of +/-1.0f
                        samples[s++] = sample / 32768f;
                    }

                    float rms = 0f;
                    float peak = 0f;
                    for (float sample : samples) {
                        float abs = Math.abs(sample);
                        if (abs > peak) {
                            peak = abs;
                        }

                        rms += sample * sample;
                    }

                    rms = (float) Math.sqrt(rms / samples.length);

                    if (lastPeak > peak) {
                        peak = lastPeak * 0.875f;
                    }

                    lastPeak = peak;

                    grabberListener.onAudioSpectrum(rms, peak);

                    if (soundLine == null) return;
                    try {
                        executor.submit(() -> {
                            soundLine.write(outBuffer.array(), 0, outBuffer.capacity());
                            outBuffer.clear();
                        }).get();
                    } catch (InterruptedException interruptedException) {
                        Thread.currentThread().interrupt();
                    }
                }
            }

            executor.shutdownNow();
            executor.awaitTermination(1, SECONDS);

            playing = false;
            LogHelper.e(TAG, "audioPlaying(off)");

            if (soundLine != null) {
                soundLine.stop();
            }

            grabber.stop();
            grabber.release();
            grabberListener.onStop();
        } catch (Exception e) {
            LogHelper.e(TAG, e);
            playing.set(false);
            grabberListener.onError(new RuntimeException("Could not open input stream.", e));
        }
    });
    playThread.start();
}

And my second question is It is possible to synchronize these RTP streams? probably using the NTP timestamp? However, the NTP timestamp is in the Sender Report RTCP Packet can I have access to that packet from the FFmpegFrameGrabber??

Guia do colaborador