decodeImageFromPixels per frame for shader pipeline in Flutter causes frame drops; better way to feed fragment shader?

3 weeks ago 21

ARTICLE AD BOX

I have 2 rendering paths:

Pixel mode: draw indexed pixels via CustomPainter Filter mode (CRT/LCD): convert frame buffer to BGRA bytes and call ui.decodeImageFromPixels, then pass ui.Image to a fragment shader

Filter mode drops frames badly on Web and Android emulator.

Simplified code:

class FilteredFrameViewState extends State<FilteredFrameView> { ui.Image? _image; FrameData? _pendingFrame; bool _decoding = false; void onFrame(FrameData frame) { _pendingFrame = frame; if (!_decoding) { _decodeNext(); } } void _decodeNext() { final frame = _pendingFrame; if (frame == null) return; _pendingFrame = null; _decoding = true; final bytes = Uint8List.view(frame.bgraBuffer.buffer); ui.decodeImageFromPixels( bytes, 128, 128, ui.PixelFormat.bgra8888, (image) { if (!mounted) return; setState(() { final old = _image; _image = image; old?.dispose(); _decoding = false; }); if (_pendingFrame != null) { _decodeNext(); } }, ); } @override Widget build(BuildContext context) { if (_image == null) return const SizedBox.shrink(); return CustomPaint( painter: ShaderPainter(image: _image!, program: shaderProgram), child: const SizedBox.expand(), ); } }

Is decodeImageFromPixels fundamentally too expensive for per-frame usage in Flutter? And What is the recommended way to stream frequently changing pixel data into a fragment shader without re-decoding each frame? Also, is there a way to keep data on GPU between frames in pure Flutter (no platform channel/FBO plugin)?

The target is 60 FPS on Web + Android and Dynamic low-resolution framebuffer with post-process shader.