How to scrub a video on scroll (play video with scroll)

Scroll progress in, currentTime out

Pick a stretch of the page for the film to run over. As the reader scrolls through it, work out how far along they are as a number from 0 to 1, multiply it by the video’s duration, and set the video’s currentTime to that. The video never plays: it stays paused, and each scroll moves it to a new frame. Scroll back and the frames run backwards, because the progress goes down.

It takes a few lines of script, not CSS. Scroll-driven animations (animation-timeline) animate CSS properties, and currentTime is not one.

Pin the video and follow the scroll by hand

The usual layout holds the video still while it scrubs: a stage with position: sticky and one window’s height, inside a track as tall as the scroll you want plus one window. A 400vh track keeps the stage stuck, and the film running, for 300vh of scroll.

<section class="film-track" id="film-track">
  <div class="film-stage">
    <video id="film" muted playsinline preload="auto"
      aria-label="Headphones turning, then taken apart">
      <source src="film.webm" type="video/webm">
      <source src="film.mp4" type="video/mp4">
    </video>
  </div>
</section>

/* The track holds the scroll room: the stage stays stuck for the
   track's height less one window, and the film runs that long. */
.film-track {
  position: relative;
  height: 400vh;
}

.film-stage {
  position: sticky;
  top: 0;
  height: 100vh;
  height: 100svh; /* never taller than a phone's smallest window */
}

.film-stage video {
  display: block;
  width: 100%;
  height: 100%;
  object-fit: contain;
}

The script measures the track, not the video. While the stage is stuck the video moves down the page with the scroll, so its own position says nothing about how far the reader is; the track’s does. Progress is 0 when the track’s top reaches the top of the window and 1 when its bottom reaches the bottom.

const track = document.getElementById('film-track');
const video = document.getElementById('film');
const reduced = matchMedia('(prefers-reduced-motion: reduce)').matches;

let lastTime = NaN; // the time last written to the video
let queued = false; // a frame is already booked

// 0 when the track's top reaches the top of the window, 1 when its
// bottom reaches the bottom: exactly the stretch the stage is stuck.
function progress() {
  const box = track.getBoundingClientRect();
  const room = box.height - innerHeight;
  return Math.min(1, Math.max(0, -box.top / room));
}

function update() {
  queued = false;
  const time = progress() * video.duration;
  // Every write to currentTime starts a seek, even to the time the
  // video is already at. Above and below the track the progress is
  // held at 0 or 1, so every scroll there asks for the same time.
  if (time === lastTime) return;
  lastTime = time;
  video.currentTime = time;
}

// Read the layout and seek in an animation frame, one per frame.
function onScroll() {
  if (queued) return;
  queued = true;
  requestAnimationFrame(update);
}

function start() {
  if (reduced) {
    video.currentTime = video.duration; // the last frame, held still
    return;
  }
  update();
  addEventListener('scroll', onScroll, { passive: true });
  addEventListener('resize', onScroll);
}

// The duration is NaN until the metadata is in.
if (video.readyState >= 1) start();
else video.addEventListener('loadedmetadata', start, { once: true });

• A passive listener and an animation frame. { passive: true } tells the browser the listener never cancels the scroll, so scrolling does not wait for it. The layout read and the seek happen in requestAnimationFrame, at most once a frame however many scroll and resize events arrive in between.
• Write only on a change. Setting currentTime starts a seek every time, even to the value it already has, and a seek asks the decoder for a frame. Above and below the track the progress is held at 0 or 1, so every scroll there would ask for the same frame again. Comparing with the last time written costs nothing and skips them.
• Wait for the metadata. Until loadedmetadata, duration is NaN and there is nothing to seek. The readyState check covers a video whose metadata arrived before the script ran.

Encode a keyframe every few frames

This is the step that decides whether the scrub is smooth, and it is not in the code. A video file stores a few whole pictures, keyframes, and between them only what changed. To show any other frame, the decoder starts at the keyframe before it and works forward. Playing, that costs nothing extra. Scrubbing, every scroll is a seek, and each one pays for the frames since the last keyframe.

Encoders space keyframes for playback, far apart: left at their defaults, both H.264 and VP9 put 2 keyframes in this site’s 5.4-second, 162-frame film. Measured on the scroll video scrub generator, the slowest 5% of seeks on those files took 24–116 ms across the three browser engines, when a frame at 60 fps lasts 16.7 ms. Re-encoded with a keyframe every 3 frames (54 of them), the slowest 5% took 18 ms or less everywhere, as Fig. 1 shows. That is the stutter, and the fix.

# H.264 in MP4: a keyframe every 3 frames (-g 3), no audio track,
# and the index moved to the front of the file (+faststart)
ffmpeg -i input.mov -an -c:v libx264 -preset slow -crf 20 -g 3 \
  -pix_fmt yuv420p -movflags +faststart film.mp4

# VP9 in WebM, with the same keyframe interval
ffmpeg -i input.mov -an -c:v libvpx-vp9 -b:v 0 -crf 32 -g 3 -row-mt 1 film.webm

# Count the keyframes in what you made
ffmpeg -i film.mp4 -vf showinfo -f null - 2>&1 | grep -c "iskey:1"

-g is the most frames from one keyframe to the next. libx264 also takes it in its own spelling, -x264-params keyint=3; on the site’s film the two wrote byte-identical files. The last line counts what you got: with -g 3, at least a third of the frames, since the encoder may add one at a scene cut. -an drops the audio track, which a scrubbed film never plays.

Shorter intervals cost bytes. The H.264 film was 487 KB at the default, 954 KB with -g 3 and 2073 KB with every frame a keyframe (-g 1), whose median seek was never more than 2 ms from -g 3’s, in either codec or any engine. A keyframe every 15 frames still kept the slowest 5% at 17 ms or under. Anything from 1 to 15 frames is a fair range; footage differs, so the generator times seeks on your own file.

Load the file, and serve it with Range requests

A seek can only land on data the browser has. preload="auto" asks for the whole file, so once it has arrived every frame is local; preload="metadata" fetches only the duration and size, and each seek then waits on the network for its part of the file. preload is a hint the browser may scale back, to save a reader’s data for one, so expect the first scroll on a slow connection to wait.

• +faststart for MP4. It moves the file’s index (the moov box) in front of the frames. Without it the index comes last, and the browser has to go to the end of the file for it before it knows the duration.
• Range requests. Browsers fetch and seek a video by byte ranges. Serve it from a server that answers them, with 206 Partial Content and Accept-Ranges: bytes: against one that does not, Chromium did not seek at all in our tests, and Apple documents byte ranges as required for media on iOS.
• A poster. poster shows until the first frame is ready. Make it the frame the scrub starts on, so nothing jumps; the generator draws that frame from the video and saves it.
• Two sources. WebM first, then MP4 with H.264, which every major browser plays. A browser takes the first <source> it can play.

muted and playsinline, for iPhone

The scrub itself never calls play(); it only writes currentTime. The two attributes are for anything else that might start the video. Without playsinline, an iPhone plays video in its fullscreen player, and muted keeps a play() allowed by autoplay rules. Leave out autoplay, loop and controls: autoplay would run the film on its own clock, and controls would let a tap do the same.

Reduced motion: show the last frame

A film that moves with the scroll is motion the reader did not start. Under prefers-reduced-motion: reduce, the script above seeks once to the end and never listens to the scroll, so the reader gets the last frame, held still. Make that frame one that stands on its own, and if chapters of copy follow the film, show them all. The aria-label on the video says what it shows; anything the reader should read belongs in real text beside it, not painted into the film. What every other kind of scroll effect should do is in reduced motion for scroll animations.

The same with scrollVideo

svg-scroll-draw’s scrollVideo is the script above with the edges filed off, in its own 3.3 KB entry. For the pinned film, point triggerElement at the track and run the window from top top to bottom bottom:

import { scrollVideo } from 'svg-scroll-draw/video';

const film = scrollVideo('#film', {
  // The stage is sticky, so the video's own box moves with the scroll.
  // Measure the track that holds the scroll room instead.
  triggerElement: document.getElementById('film-track'),
  trigger: { start: 'top top', end: 'bottom bottom' },
});

// film.destroy() takes the listeners off and lets the video go.

It pauses the video, sets preload when the video has none, waits for the metadata, and writes currentTime = from + (to − from) × progress, only when that time changes. It measures the trigger window once and again 150 ms after a resize or a layout change, rather than on every scroll, and it runs frames only while the scroll is inside the window. Under reduced motion it seeks once to to and does not follow the scroll; turned on during the visit, it goes to that frame and stops there.

The scroll video scrub generator sets the pin, window, range and easing on a video from your own disk and writes the HTML, CSS and JavaScript or a React component. The scroll video example is this listing with chapters of copy that follow the film, and Aural is a whole product page built around it.

When an image sequence is the better choice

A well-encoded video scrubs smoothly and stays one compressed file. An image sequence, numbered stills drawn on a <canvas>, has no keyframes at all: each frame is its own picture, so any scroll position shows its frame at once, in every browser, with no codec or Range requests in the way. The cost is memory, since every loaded frame is kept decoded at width × height × 4 bytes: a hundred 1920×1080 frames come to 829 MB.

For a short, high-value turn, a product spinning in a few dozen frames, use frames; for long footage, use a video encoded as above. How to build the frames version is in the scroll image sequence post, and the scroll image sequence generator turns a video into frames in your browser.