#Why Observability Tools Are Lying to You About Multi-Modal Data
#May 26, 2026
Your dashboards are green. Your footage is unusable.
This is the central problem with applying conventional observability to multi-modal pipelines. Infrastructure observability tools were designed to measure latency, error rates, and throughput. When a packet drops or a GPU stalls, these metrics catch it. For APIs, they're a reasonable proxy for whether the system is working.
A multi-modal pipeline isn't an API call with a return value. It's a stream of frames, tracks, and events.
A capture can succeed by every infrastructure metric — sub-30ms frame times, zero dropped packets, 100% uptime — and still be a failure. The tracker lost the subject in Q3. Two-thirds of the generated frames have geometry clipping. The moderation model quietly stopped firing on a class it used to catch. Your dashboard showed nothing.
#What these tools measure vs. what matters
Infrastructure observability answers: "Is the system running?" It measures whether your components are responding and whether requests are completing.
Multi-modal observability needs to answer: "Was the capture any good?" These are categorically different questions.
A pipeline "worked" by infrastructure metrics if frames were transmitted, a model returned a response, and the session closed cleanly. A pipeline worked for your users if the footage is usable, the events are correctly detected, and nothing quietly broke. There is no infrastructure metric that captures the distance between those two outcomes.
#The deeper problem
The issue isn't that infrastructure observability is bad at what it does. It's that log-and-metric approaches apply the wrong model to multi-modal data.
Most teams instrument multi-modal pipelines the same way they'd instrument a REST endpoint: log the inputs, log the outputs, measure the latency. If the model returned a tensor and the session didn't error, the system is "healthy." This misses everything.
Video, audio, and telemetry carry information that logs don't. A tracker that returned coordinates but drifted five meters off the subject. A generated frame that has perfect pixel counts and completely broken geometry. A robot that reported "task complete" while the object was still on the floor. The metadata says one thing; the media says another. Log-based tooling only sees the metadata.
#What you actually need to index
Multi-modal observability needs to operate at the semantic level, across five dimensions:
- Objects — entities on screen and their identity across time.
- Motion — trajectory, velocity, contact, collision. What is moving and how.
- Events — what actually happened between frame N and frame N+k.
- Context — environment, lighting, camera state, sensor health.
- Intent — what the operator, subject, or model was trying to do.
These aren't features on top of observability. They are the observability layer for multi-modal data.
When a pipeline fails, the signal is almost never in the logs alone. It's in the motion of the tracked subject in second 47, the object dropout between frames 1200 and 1230, the context shift where the lighting changed and the model stopped generalizing. These are indexable, queryable, and detectable at scale — but only if your tooling knows to look for them.
#The consequence of getting this wrong
Teams running multi-modal pipelines at scale without semantic observability are in a strange position: they know something is wrong — downstream users are unhappy, reshoot rates are up, event counts don't match reality — but they can't find it. They review clips manually. They build crude tag searches. They look at frame counts as a proxy for footage quality.
This is expensive, slow, and doesn't scale. Every new capture is another needle in a growing haystack.
The observability gap is the reason ops teams spend weeks debugging what should take hours. The infrastructure is fine. The system is lying.