Client-side video crop + audio mux on iOS Safari with HEVC — any viable approach?

12 hours ago 1
ARTICLE AD BOX

I'm building a web app where users upload a video, the app crops the first 60 seconds client-side, extracts audio for server-side AI processing, then muxes the returned audio back into the cropped video. This must work on mobile including iOS.

The problem

iPhones record HEVC (H.265) by default. Every client-side approach I've tried fails on iOS Safari for large HEVC files:

WebCodecs: VideoDecoder.isConfigSupported({ codec: "hev1..." }) → false
ffmpeg.wasm: fetchFile() copies entire file to JS heap → OOM (tab crash)
captureStream: video.captureStream() not supported on iOS Safari
canvas: must decode HEVC frames → ~8MB × 5 frame buffer → OOM
File.slice() works for byte-level cropping with no decode, but I still need to mux a new audio track in afterward — which brings the same memory constraints back.

What works

- iOS + H.264: WebCodecs pipeline works fine (decode → re-encode at 720p via VideoEncoder → mp4box.js mux)
- Android Chrome: WebCodecs or ffmpeg.wasm both work

What I've considered

1. Extract audio-only via mp4box.js + AudioDecoder (AAC is always supported even on iOS HEVC) — solves the crop step, not the mux step
2. Server-side mux — works but adds latency and bandwidth
3. mp4box.js container-level audio track replacement — still requires loading moov + mdat into memory

Question

Is there any iOS Safari 16.4+ web API that allows replacing the audio track in an MP4/MOV container without decoding video frames? Or any memory-safe approach to mux new audio into a large HEVC file client-side?

Or is server-side mux the only realistic option here?
Read Entire Article