Neuausrichtung von 360°-Videos in Flutter: Sphärische Projektion, Quaternion-Keyframes und GPU-Export
Wie wir rohes, equirectangular 360°-Material in ein flüssiges, gerahmtes, flaches Video verwandelt haben – vollständig innerhalb einer Flutter-App.
Das Problem, vor dem Sie niemand warnt
Ein normales Video zeigt Ihnen eine offensichtliche Sache: den Frame, auf den die Kamera gerichtet war. Ein 360°-Video hat so etwas nicht. Die Kamera hat alles aufgenommen – eine vollständige Kugel von Pixeln, gespeichert als ein flaches equirectangular Rechteck (ein 2:1-Bild, bei dem die X axis die Longitude und die Y axis die Latitude ist). Bevor Sie dieses Material auf Instagram, YouTube oder einem Telefonbildschirm zeigen können, muss jemand eine Frage beantworten, die die Kamera absichtlich unbeantwortet ließ:
Wohin soll der Betrachter wann schauen?
„Reframing“ ist der Akt, eine virtuelle Kamera im Laufe der Zeit durch diese Sphäre zu bewegen – Panning, Tilting, Zooming – und ein normales flaches 16:9- (oder 9:16- oder 2.35:1-)Video zu rendern. Dies ist die Funktion, um die GoPro Player, Insta360 Studio und Adobe Premiere’s GoPro VR Reframe plugin herum aufgebaut sind.
Wir haben es vollständig in Flutter entwickelt. Dieser Beitrag ist ein Überblick über die wirklich schwierigen Teile: die Projektionsmathematik, die gimbal-lock-freie Interpolation, die ±180°-Seam und eine GPU-Export-Pipeline, die echte Android-Decoder überleben muss.
Der end-to-end flow:
360 video aufnehmen/importieren
→ einen leichtgewichtigen Preview-Proxy transkodieren
→ Benutzer tippt / zeichnet / erkennt ein Subject automatisch
→ object tracking erzeugt einen Pfad
→ Keyframes generieren + glätten
→ SLERP-Interpolation der Camera Orientation pro Frame (live preview)
→ nativer GPU Export zu einem flachen Video
→ FFmpeg Post-Pass (moov atom + A/V sync fix)
Architektur auf einen Blick
Wir haben das System in drei sauberen Layers gehalten, damit die Mathematik isoliert von Flutter testbar ist:
| Schicht | Lebt in | Verantwortlichkeit |
|---|---|---|
| Models | lib/data/models/reframe/ | SphericalCoordinates/Quaternion, CameraOrientation/MotionConstraints, ReframeKeyframe, ReframeProject |
| Services | lib/services/reframe/ | SphericalProjection (ray casting), CoordinateConverter, KeyframeInterpolator/Generator/Smoother, ReframeExportService |
| Presentation | lib/presentation/.../reframe/ | Riverpod ReframeNotifier, der native OpenGL Spherical360Player, bbox-drawing overlay, keyframe timeline |
Die golden Rule: all the trigonometry is pure Dart with no Flutter imports. Das bedeutet, jede Projection- und Interpolation-Function ist unit-testable ohne einen widget tree, und dieselbe Math drives both the live preview und the Export.
1. Das Herzstück: equirectangular ⟷ spherical
Alles beginnt mit einem Mapping. Ein equirectangular Frame ist einfach ein Rectangle, bei dem:
- X (0 → width) die longitude (yaw) von −180° bis +180° abdeckt
- Y (0 → height) die latitude (pitch) von +90° (top/zenith) bis −90° (bottom/nadir) abdeckt
SphericalCoordinates ist bewusst tiny — nur yaw und pitch. Roll, FOV und zoom gehören zur camera, nicht zur direction. Hier ist die Conversion, auf der das ganze Feature rests on:
factory SphericalCoordinates.fromEquirectangular({
required double x, required double y,
required double videoWidth, required double videoHeight,
}) {
final normalizedX = x / videoWidth;
final normalizedY = y / videoHeight;
// X bildet auf Yaw ab: 0->-180°, 0.5->0°, 1.0->180°
final yaw = (normalizedX - 0.5) * 360.0;
// Y bildet auf Pitch ab: 0->90° (oben), 0.5->0° (Horizont), 1.0->-90° (unten)
final pitch = (0.5 - normalizedY) * 180.0;
return SphericalCoordinates(
yaw: yaw.clamp(-180.0, 180.0),
pitch: pitch.clamp(-90.0, 90.0),
);
}
The inverse ist the mirror image (normalizedX = yaw/360 + 0.5), und we use it constantly to draw a tracked bounding box back onto the player.
„Tap on what you see“ — perspective ray casting
Der User doesn't interact with the equirectangular rectangle. They interact with a rendered perspective view. So when they tap a point on the player, we need to cast a ray through a virtual pinhole camera and find which direction on the sphere they hit. That's SphericalProjection.screenToSpherical:
final ndcX = (2.0 * screenPoint.dx / playerSize.width) - 1.0;
final ndcY = 1.0 - (2.0 * screenPoint.dy / playerSize.height);
final aspectRatio = playerSize.width / playerSize.height;
final tanHalfFov = math.tan(cameraFov * math
.pi / 360.0); // halber Winkel
double rayX = ndcX * aspectRatio * tanHalfFov;
double rayY = ndcY * tanHalfFov;
double rayZ = 1.0;
// ...normalisieren, den Strahl um Pitch (um X) dann Yaw (um Y) drehen...
final sphericalYaw = math.atan2(finalRayX, finalRayZ) * 180.0 / math.pi;
final sphericalPitch = math.asin(finalRayY.clamp(-1.0, 1.0)) * 180.0 / math.pi;
Das .clamp(-1.0, 1.0) vor asin looks paranoid, but floating-point drift will hand you 1.0000001 und asin will return NaN — and a single NaN poisons every downstream frame. Defensive clamping around inverse trig ist a recurring theme in this codebase.
2. Smooth camera motion: quaternions, not Euler angles
The naïve approach ist to store yaw/pitch at each keyframe and linearly interpolate the numbers. It looks terrible. Interpolating Euler angles gives you gimbal lock, non-uniform angular velocity, und ugly snaps near the poles.
Wir interpolieren orientation as a quaternion using SLERP (spherical linear interpolation). Each direction converts to a quaternion via a ZYX Euler convention:
Quaternion toQuaternion() {
final yawRad = yaw * math.pi / 180.0;
final pitchRad = pitch * math.pi / 180.0;
final cy = math.cos(yawRad * 0.5); final sy = math.sin(yawRad * 0.5);
final cp = math.cos(pitchRad * 0.5); final sp = math.sin(pitchRad * 0.5);
return Quaternion(w: cy * cp, x: cy * sp, y: sy * cp, z: -sy * sp);
}
And the SLERP itself has the two production-grade safety valves every implementation needs:
static Quaternion slerp(Quaternion a, Quaternion b, double t) {
a = a.normalize(); b = b.normalize();
var dotProduct = a.dot(b);
if (dotProduct < 0.0) { b = -b; dotProduct = -dotProduct; } // kürzeste Hemisphäre wählen
if (dotProduct > 0.9995) { // fast parallel -> auf LERP zurückgreifen
return Quaternion(w: a.w + t*(b
.w-a.w), x: a
.x + t*(b.x-a.x),
y: a.y + t*(b.y-a.y), z: a.z + t*(b
.z-a.z)).normalize();
}
final theta0 = math.acos(dotProduct);
final theta = theta0 * t;
final sinTheta = math.sin(theta);
final sinTheta0 = math.sin(theta0);
final s0 = math.cos(theta) - dotProduct * sinTheta / sinTheta0;
final s1 = sinTheta / sinTheta0;
return Quaternion(w: s0*a.w + s1*b.w, /* ...*/).normalize();
}
Two non-negotiable details:
- Kürzeste Hemisphere — q und −q represent the same orientation. If the dot product is negative, you negate one input or your camera takes the long way around the sphere.
- Near-parallel LERP fallback — when two keyframes are almost identical, sinTheta0 → 0 und you divide by nearly zero. Falling back to plain LERP above dot > 0.9995 avoids the blow-up.
A subtle but important design call: we SLERP the direction, but LERP the FOV/zoom and angle-LERP the roll. Spherical interpolation ist the right tool for direction; a zoom ist just a scalar and should move linearly:
final qInterp = Quaternion.slerp(from.direction.toQuaternion(), to.direction.toQuaternion(), t);
final interpDirection = SphericalCoordinates.fromQuaternion(qInterp).normalize();
final interpFov = _lerp(from.fov, to.fov, t);
final interpRoll = _lerpAngle(from.roll, to.roll, t); // wickelt über ±180
final interpZoom = _lerp(from.zoom, to.zoom, t);
💡 When converting back from a quaternion to yaw/pitch, we use atan2 rather than asin "for numerical stability (avoids asin domain issues)." After a SLERP, the quaternion can be very slightly denormalized, and atan2 degrades gracefully where asin throws NaN.
Easing: handgemacht, auf dem eingehenden Ãœbergang
Each keyframe owns the easing curve for the transition into it. The curves are written out by hand so they're identical on every platform:
case KeyframeEasing.linear: return t;
case KeyframeEasing.easeInOut: return t * t * (3 - 2 * t); // smoothstep S-Kurve
case KeyframeEasing.easeIn: return t * t; // quadratisch
case KeyframeEasing.easeOut: return 1 - (1 - t) * (1 - t);
3. The ±180° seam: the bug that haunts every 360 feature
Hier ist die Trap. Yaw lives on a circle: +179° and −179° are 2° apart, not 358°. The instant your subject walks across the back of the sphere, naïve math snaps the camera halfway around the world. This single issue shows up at four different layers of the system, and each needs its own fix.
Layer 1 — primitives. Two tiny helpers that the entire codebase leans on:
static double normalizeYaw(double yaw) {
while (yaw > 180) { yaw -= 360; }
while (yaw < -180) { yaw += 360; }
return yaw;
}
// „KRITISCH für seam-safe Tracking ... vermeidet 180°-Sprünge“
static double shortestYawDelta(double from, double to) {
double delta = to - from;
if (delta > 180) delta -= 360;
if (delta < -180) delta += 360;
return delta;
}
Layer 2 — keyframe generation accumulates a continuous (unbounded) yaw so SLERP always interpolates the short way. Instead of clamping every sample to ±180°, we add up the shortest deltas and let the running value drift past 180° on purpose:
final normalizedContinuousYaw = SphericalCoordinates.normalizeYaw(continuousYaw);
final yawDelta = SphericalCoordinates.shortestYawDelta(normalizedContinuousYaw
, rawCoords.yaw);
continuousYaw = continuousYaw + yawDelta; // kann absichtlich ±180 überschreiten
coords = SphericalCoordinates(yaw: continuousYaw, pitch: rawCoords.pitch);
Layer 3 — bounding-box reconstruction. When the user draws a box that straddles the seam, the four corners report yaws like +170°, +175°, −175°, −170°. SphericalProjection detects this (any corner > 90° and any corner < −90°) and rebuilds a continuous-yaw "seam-safe" bbox by shifting the negatives by +360° before measuring its span.
Layer 4 — the live player smooths toward its target using shortestYawDelta so dragging across the seam never lurches.
If you remember one thing about building 360 tooling: the seam is not an edge case, it's a cross-cutting concern. Budget for it everywhere yaw appears.
4. Das Gefühl von Insta360/GoPro erzeugen
Smooth interpolation gets you a technically correct camera. It doesn't get you a good one. Three pieces of "feel engineering" close that gap.
Motion constraints — clamp velocity and acceleration. Raw tracking data ist jittery. We pass keyframes through MotionConstraints, which caps how fast and how abruptly the virtual camera can move (e.g. maxYawSpeed = 120°/s, maxPitchSpeed = 90°/s, maxAcceleration = 180°/s²), with cinematic(), responsive(), and production() presets. This ist the difference between "security camera tracking a fly" and "operator with a steady hand."
Adaptive FOV — zoom to fit the subject. We measure the subject's angular size in equirectangular space and pick an FOV so it fills a target fraction of the frame:
final angularWidth = (bboxWidth / videoWidth) * 360.0;
final angularHeight = (bboxHeight / videoHeight) * 180.0;
final objectAngularSize = math.max(angularWidth, angularHeight);
final idealFov = objectAngularSize / targetFillRatio; // targetFillRatio ≈ 0.4
return idealFov.clamp(minFov, maxFov); // 45°–120°
Motion lead + face→body blend. We push the framing ~15% of the bbox width ahead in the direction of travel, so the subject isn't pinned against the edge they're walking toward. And when face detection drops out, we don't snap to the body center — we lerp from the last-known face center to the body center over ~10 frames.
5. The two-video trick: preview proxy vs. full-res original
This ist the architectural decision that makes the whole thing usable on real phones.
A 360° GoPro/Insta360 file ist often 4096×2048 HEVC. Budget Android decoders choke trying to play that back smoothly, let alone while running object tracking on top. So we run tracking and live preview on a transcoded proxy (~1440×720, libx264 -preset veryfast -crf 28), and reserve the full-resolution original for the final export only.
// „zuverlässiger als MediaMetadataRetriever für 360°-Videos
// die oft ungewöhnliche Codecs (HEVC, hochauflösend) verwenden“
final maxSafeWidth = 2048, maxSafeHeight = 1080;
// 2:1 4K (4096×2048) ist unsafe -> transcodeForPlayback() auslösen
The catch you must handle: keyframe bounding boxes are computed in proxy coordinates and must be scaled up to original coordinates before export. Get this scaling wrong and your export frames a slightly different region than the user previewed. (We also use ffprobe, not MediaMetadataRetriever, for metadata and first-frame extraction — Android's built-in retriever ist flaky on the unusual codecs 360 cameras emit.)
6. GPU export: OpenGL + MediaCodec over a MethodChannel
Re-projecting every frame of a 4K sphere through FFmpeg's v360 filter on a phone CPU ist far too slow. (We do keep the v360 string around — it neatly documents the projection semantics, v360=e:flat:... meaning "equirectangular in, flat out" — but it's not the live path.)
The real export hands the keyframes to a native OpenGL ES + MediaCodec renderer over a MethodChannel:
final result = await _methodChannel.invokeMethod('exportVideoGPU', {
'videoPath': resolvedVideoPath,
'outputPath': outputPath,
'keyframes': keyframesList,
'initialOrientation': initialOrientationMap,
'width': settings.exportWidth,
'height': settings.exportHeight,
'fps': settings.outputFps.toInt(),
'includeAudio': settings.includeAudio,
if (isTrimmed) 'trimStartMs': trimStartMs,
if (isTrimmed) 'trimEndMs': trimEndMs,
});
We gate this behind a capability check (Android API 21+, a hardware H.264 encoder, OpenGL ES 3.0) and stream progress back over an EventChannel. A few hard-won gotchas:
- Der −90° Yaw-Offset. Unsere Dart convention puts yaw = 0 at the center of the equirectangular image (u = 0.5); the OpenGL renderer puts yaw = 0 at u = 0.75. So every keyframe yaw is remapped by −90° (and re-wrapped to ±180°) on the way across the channel — symmetrically applied to both the export channel and the live-player channel, so the preview and the export agree.
- Degenerate keyframe counts. Zero keyframes → synthesize two identical ones from the initial orientation (static export). One keyframe → duplicate it. The renderer always wants at least a start and an end.
- Trimming re-bases timestamps. A trimmed export shifts every keyframe by t − trimStartMs.
Der FFmpeg Post-Pass, den niemand erwartet
Man könnte meinen, GPU export sei die finish line. Ist es aber nicht — because Android's MediaMuxer writes the moov atom at the end of the file and can carry the source's audio PTS offset. The result: ExoPlayer (and Flutter's video_player) freezes the picture while the clock keeps running. So after the GPU pass, FFmpeg does a fast, lossless cleanup:
// Video kopieren (keine Neukodierung -> kein Qualitätsverlust), Audio neu kodieren zur PTS-Korrektur,
// moov atom für sofortige Wiedergabe an den Anfang verschieben.
-c:v copy -af "aresample=async=1:first_pts=0" -movflags +faststart
And a brutal little gotcha guarding it: we probe for an audio track first, because passing audio flags to a video that has no audio track makes FFmpeg hang forever on some devices. The whole post-pass ist wrapped in a 60-second timeout with cancel-on-timeout.
7. Die Live-Vorschau ist ebenfalls ein native player
The in-editor preview isn't a Flutter CustomPaint — it's a native OpenGL surface embedded via AndroidView (with a viewType like 'your_app/spherical360player'), with its own per-view method/event channel. The Dart Spherical360PlayerController mirrors yaw/pitch/roll/fov and applies the same −90° offset before sending.
Die angenehmen Details des Bediengefühls:
- Frame-rate-independent smoothing — factor = 1 - exp(-speed * dt) so the easing looks the same at 60fps and 90fps, with the seam-safe shortest-path yaw baked in.
- Momentum/inertia after a fling, with friction-based decay — disabled during playback, because during playback the camera must obey the keyframes, not the user's last flick.
- One GestureDetector whose onScaleUpdate does pan and pinch-zoom together; double-tap toggles play/pause. FOV clamps to 30°–120°, pitch to ±90°, at the controller boundary.
One tiny but telling UX fix: before switching from "tracking" to "generating keyframes," we insert a 120 ms delay so Flutter actually paints "100%." Without it, the final progress frame and the status change land in the same microtask batch and the user never sees completion.
Zustandsmaschine, kurz zusammengefasst
All of this ist coordinated by a Riverpod ReframeNotifier driving a status enum:
idle → loadingVideo → transcoding
→ (detecting | selectingObject | drawingBboxOnPlayer | autoDetectingOnPlayer)
→ tracking → generatingKeyframes → editing → exporting → completed | error
Multiple entry points feed the same tracker: tap-to-detect (YOLO overlay), manual bbox, or the Insta360-style draw-directly-on-the-player gesture. They all converge on KeyframeGenerator.production().generate(...) → KeyframeSmoother.smooth(...), and from there the interpolator drives every previewed and exported frame.
Erkenntnisse
- Die Mathematik rein halten. No Flutter imports in the projection/interpolation layer. It's the only reason this was testable.
- Quaternion SLERP for direction; LERP for scalars. Don't interpolate Euler angles, and don't SLERP your zoom.
- The ±180° seam ist a cross-cutting concern, not an edge case. It reappears in primitives, generation, projection, and the player.
- Konventionen einmal symmetrisch abstimmen. The −90° yaw offset between Dart and OpenGL is applied at both channel boundaries so preview == export.
- Proxy for interaction, original for output. Just remember to scale your coordinates between the two.
- "Rendering abgeschlossen" ≠"plays correctly." Budget for an FFmpeg post-pass to fix moov atom placement and audio PTS, and probe before you add audio flags.
- Feel ist a feature. Motion constraints, adaptive FOV, and motion-lead are what separate "technically reframed" from "looks like a pro shot it."
Das Ergebnis: a fully on-device 360°-reframe editor — tap a subject, let it track, tweak the keyframes, and export a clean flat video — built in Flutter with a thin native GPU layer doing only the work Dart can't do fast enough.
