bytedeco/javacv

extract frame from video in android device use JavaCV, but every time the model's inference results are not same

Open

#1,938 opened on Nov 15, 2022

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Java (1,583 forks)batch import
help wantedquestion

Repository metrics

Stars
 (6,985 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

Hello, I am using the JavaCV to extract frame from video in android device and as input to my machine learning model, and I used the FFmpegFrameFilter to set the fps of video to get same number of frame every time. but every time I run the model, the model's inference results are not same, do you know the reason? these my code below :

       FFmpegLogCallback.set();
      grabber.start();
      FFmpegFrameFilter frameFilter = new FFmpegFrameFilter("fps=fps=5", "anull",
              grabber.getImageWidth(), grabber.getImageHeight(), grabber.getAudioChannels());
      frameFilter.setSampleFormat(grabber.getSampleFormat());
      frameFilter.setSampleRate(grabber.getSampleRate());
      frameFilter.setPixelFormat(grabber.getPixelFormat());
      frameFilter.setFrameRate(grabber.getFrameRate());
      frameFilter.start();
      Frame capturedFrame;
      Frame pullFrame;
      while (true) {
        try {
          capturedFrame = grabber.grab();
          if (capturedFrame == null) {
            Log.w("!!! end cvQueryFrame..video:{}", frontcamVideoSourcePath);
            break;
          }
          if (capturedFrame.image != null) {
            frameFilter.push(capturedFrame);
          }
          if ((pullFrame = frameFilter.pull()) != null) {
            if (pullFrame.image != null) {
              Bitmap bitmap = convertToBitmap.convert(pullFrame);
              setImage(bitmap, frontcamRunner.getSurface(), true);
              ResultProto objectResult = frontcamRunner.run();
              resultsObject.add(objectResult));

When I do the snapshot test, it shows every time the inference results are not same like:

Changed content at line 922:
expecting:
  ["scores: 0.9639015",
   "scores: 0.4762144",
   "scores: 0.29251003",
   "boxes: 359.04742",
   "boxes: 108.379776",
   "boxes: 366.3406",
   "boxes: 112.13722",
   "boxes: 359.66046",
   "boxes: 108.81177",
   "boxes: 366.08936",
   "boxes: 116.47825",
   "boxes: 358.7235",
   "boxes: 110.984955",
   "boxes: 366.55127",
   "boxes: 114.66323"]
but was:
  ["scores: 0.9536576",
   "scores: 0.47547817",
   "scores: 0.25218058",
   "boxes: 358.9765",
   "boxes: 108.3856",
   "boxes: 366.24115",
   "boxes: 112.15494",
   "boxes: 359.57233",
   "boxes: 108.810745",
   "boxes: 366.00732",
   "boxes: 116.48662",
   "boxes: 358.6304",
   "boxes: 110.98422",
   "boxes: 366.46204",
   "boxes: 114.66691"]

Changed content at line 941:
expecting:
  ["boxes: 358.75476",
   "boxes: 108.227554",
   "boxes: 365.989",
   "boxes: 111.96691"]
but was:
  ["boxes: 358.73907",
   "boxes: 108.22945",
   "boxes: 365.9668",
   "boxes: 111.97165"]

Changed content at line 949:
expecting:
  ["boxes: 358.5876",
   "boxes: 108.080986",
   "boxes: 365.78546",
   "boxes: 111.79181"]
but was:
  ["boxes: 358.58386",
   "boxes: 108.081505",
   "boxes: 365.78018",
   "boxes: 111.79303"]

Changed content at line 957:
expecting:
  ["boxes: 358.52716",
   "boxes: 107.90219",
   "boxes: 365.63968",
   "boxes: 111.619354"]
but was:
  ["boxes: 358.5263",
   "boxes: 107.902336",
   "boxes: 365.63843",
   "boxes: 111.619675"]

Changed content at line 965:
expecting:
  ["boxes: 358.5914",
   "boxes: 107.647606",
   "boxes: 365.6294",
   "boxes: 111.37422"]
but was:
  ["boxes: 358.5912",
   "boxes: 107.64763",
   "boxes: 365.62915",
   "boxes: 111.37428"]

Changed content at line 977:
expecting:
  ["boxes: 358.92163"]
but was:
  ["boxes: 358.92157"]

Changed content at line 979:
expecting:
  ["boxes: 365.94983",
   "boxes: 111.76167"]
but was:
  ["boxes: 365.94977",
   "boxes: 111.761696"]

Changed content at line 985:
expecting:
  ["boxes: 359.38873"]
but was:
  ["boxes: 359.3887"]

Changed content at line 987:
expecting:
  ["boxes: 365.98245",
   "boxes: 115.30535"]
but was:
  ["boxes: 365.98242",
   "boxes: 115.305374"]

Changed content at line 995:
expecting:
  ["boxes: 366.0174"]
but was:
  ["boxes: 366.01736"]

Contributor guide