Weird timing issue on recordings made with VideoInputFrameGrabber

Right now I have a very weird problem, but you guys might now what's happening so I hope you have an idea.

In the recordings I make with the opencv / javacv libaries I add two text lines in every frame: the current date / time and the time of the recording. I'm recording at 25 fps, so the first 25 frames get 0:00 on the screen, the second 25 frames 0:01, etc.

Somehow video players like Windows Media Player and VLC do not agree with the timing of the videos I produce and I wanted to see where my timing goes wrong so I added (for debugging only) the ms and frame number of every frame. It turns out logically I'm not doing it wrong, but both video players are off, they 'turn to the next second' a couple of frames later than they should (see attachments).

But of course this cannot be the actual case, probably I'm doing something which isn't according to the specs or file type? Maybe the first second of a mp4 recording should actually have a few frames more than 25 to get things like GOP or CRF initialized??

I did find somewhere that using timestamps (https://github.com/bytedeco/javacv/blob/master/samples/WebcamAndMicrophoneCapture.java) might help, but it doesn't :-(

So, do you have any idea why players like WMP and VLC think that frame 28 of a recording at 25 fps still belongs to the first second / frame 53 still belongs to 0:01 and not 0:02??