Refresh rate syncing under linux/X

Billy Biggs <vektor@dumbterm.net>

Video sequences are produced for a specific playback framerate. At rates of 24fps and higher, frame display durations approach monitor refresh rates. For fluid motion, each frame in a video sequence must be displayed for the same number of screen refreshes.

This is often impossible. Consider playback of a 24fps movie on a 60hz display. The optimal strategy is to display frames in a 2-3-2-3 refresh pattern, which appears as objectionable judder at such a low refresh rate (identical to film-to-ntsc conversions, the 'credits staggering up the screen' effect). Similarly, a deinterlaced NTSC stream at 59.94fps jumps badly on a monitor run at 72hz or 75hz.

A great article by Dave Marsh on these issues is here: http://www.microsoft.com/whdc/archive/TempRate.mspx.

The XF86VidMode extension gives us some ability to switch the monitor refresh to provide a better match for the video signal. The difficulty is then trying to keep playback in sync. Some possible techniques are:

1. Adapt the monitor's clock dynamically. Dave Marsh suggests that applications could monitor the clock drift and adapt the display's refresh rate.
2. Sync the video to the output refresh. For DVD watching this option is very reasonable, especially if you own a projector that can do both 60hz for TV and 72hz for film.
3. Create temporally interpolated frames. Motion-vector interpolation is expensive, but becoming more viable.

Each technique above requires accurate information about the display's refresh frequency, and timestamps on when the retrace occurs. It's unclear how to best provide a clean API to this in X. Unfortunately, achieving useable accuracy will likely require kernel drivers, maybe as part of DRI? This is where discussion is required.

I previously started this thread on Xpert regarding XFree86 and refresh sync information. For reference I include the link here.

Effects of 10ms scheduling on video refresh.

Linux and other UNIX variants often use a kernel timer resolution (HZ value) of about 10ms. From the nanosleep(2) manpage under linux:

       Therefore, nanosleep pauses always for at least the speci­
       fied time, however it can take up to  10  ms  longer  than
       specified  until  the  process becomes runnable again. For
       the same reason, the value returned in case of a delivered
       signal  in *rem is usually rounded to the next larger mul­
       tiple of 1/HZ s.

This can cause negative effects on applications which must perform actions at very specific times. One application which is believed to require high resolution timing is video frame blitting. The study below attempts to model the effects of a 10ms timer resolution on video frame blitting in the ideal case, that is, where the application always wakes up on a multiple of 10ms.

Effects of uneven blit patterns on video

Jumpy frame display patterns become most annoying during long camera pans, scrolling credits or text, or other instances where something is smoothly flowing across the screen. The image tends to jump or display odd motion sequences. Since we observe the speed of objects by how quickly they move across the screen, an unfortunate blit pattern causes motion in the video sequence to appear as if it is speeding up and slowing down rapidly, or shuddering.

Even worse, even with the best possible frame pattern for the display refresh, the effects can be visible. Many north american TV viewers complain about the jagged motion created by the 3:2 pulldown technique for converting 24fps movies to 29.97fps video sequences, where the blit pattern is 3:2:3:2, especially during credit rolls at the end of movies. It is important to choose a refresh rate which matches the output display framerate whenever possible. For this reason, a refresh rate of 72hz is popular with hardware deinterlacers and projectors for watching movies.

The numbers below and their error

The numbers below represent the calculated results of patterns produced given various techniques for deciding when to blit a frame. In all of the cases below, the algorithm does not take into account when the refresh occurs in deciding when to blit, it simply blits at a regular interval, starting 5ms after a base refresh. For results which depend on HZ, we assume HZ of 10ms and that our renderer thread wakes up on exactly aligned 10ms intervals and can blit immediately.

The 'exact' numbers represent the pattern produced by blitting at exactly the correct time for when the frame should be displayed, disregarding the refresh rate. The 'nearest' results represent blitting on the 10ms interval closest to when the frame should be blit (round up or down whichever is smallest). The 'next' results represent blitting where we always round up, and the 'prev' results represent blitting where we always round down.

Source code

The source code used to generate these numbers is showpattern.c which didn't take too long to put together.

Results

The table below shows the refresh per frame pattern for each of the sets of framerate and refresh rate which we tested. We show results for the following input:

1. 24fps, recordings from film source, or NTSC video with 3:2 pulldown inverted.
2. 25fps, recordings from progressive 525/50 sources, such as films which were converted to PAL.
3. 2997fps, recording from progressive 625/59.94 sources, or single-field deinterlaced video material.
4. 50fps, recordings from full framerate 525/50 sources.

I didn't test 59.94fps because I forgot to. I'll maybe do that tomorrow, but maybe not. The links below show the detailed results page showing when the frames were blit, I recommend looking at least at one of them to see what it's like.

Discussion

I learned is that the nearest technique performed very close to optimal, except in some cases where it was very bad (namely 24fps and 29.97fps at 85hz). Cases such as 24fps at 72hz showed quite clearly that next and prev are poor techniques.

Some tests should be done to see how often user-level programs actually get scheduled in on average. I suspect that SCHED_FIFO or similar access will be required to ensure that the nearest results can be met under the stress of DVD decoding.

The poor performance of all 10ms-rounded techniques on 29.97 at 75hz and 29.97fps at 85hz indicate to me that higher scheduling resolution (such as obtained using /dev/rtc) is required for playback of video at maximum quality.

Bias

I am, of course, advocating for higher resolution timing. The reasonably performance of 'nearest' surprised me, especially for the 24fps at 72hz problem.