Blog

What is sensor fusion? Types, uses and examples

Sensor fusion explained: how camera, LiDAR and radar data combine, fusion levels and types, and why it matters for autonomous systems.

By Appen · Expert insight
Updated October 2026

In March 2018, an Uber test vehicle in Tempe, Arizona, first detected a pedestrian 5.6 seconds before impact. Its system classified her first as a vehicle, then as an unknown object and a bicycle. Each time the classification changed, it dropped her tracking history, so it never correctly predicted her path (NTSB, 2019). Sensor fusion is the process of combining data from multiple sensors, such as cameras, light detection and ranging (LiDAR) and radar, into a single, more reliable picture of the surroundings. Done well, it lets one sensor cover another's blind spots. This article explains what sensor fusion is, why it matters, how the main fusion types and algorithms work, and how to prepare multi-sensor training data.

What is sensor fusion?

Sensor fusion is the process of combining data from two or more sensors so the merged result is more accurate, complete or dependable than any single sensor could produce. In autonomous vehicles (AVs), robots and smartphones, it merges camera images, LiDAR point clouds, radar returns and inertial measurements into one consistent model of the surroundings.

"The intuition behind sensor fusion is that the more information we can collect, the better understanding we have of the world," says Kuo-Chin Lien, Appen’s Head of Computer Vision when this article was first published. That information only helps if it's labelled consistently, which is why production pipelines keep human-in-the-loop review for the cases where sensors disagree.

Why sensor fusion matters for autonomous systems

Every sensor fails somewhere. Fusion exists because the failures rarely overlap.

  • Lighting: Cameras lose detail in glare, tunnels and darkness. LiDAR and radar emit their own signal, so darkness doesn't affect them, though intense sunlight can still degrade some LiDARs (Yeong et al., 2021).
  • Weather: Heavy rain, fog and snow scatter LiDAR pulses. Radar works through them but has low resolution (Yeong et al., 2021).
  • Classification instability: One sensor stream can flip between labels from frame to frame, as in the Tempe crash (NTSB, 2019). Agreement across sensors stabilizes tracks.
  • Calibration drift: Vibration and temperature shift sensor mounts over time, so fused data quietly goes out of alignment.

Fusion is also how safety standards get met. ISO 26262 covers functional safety in road vehicles: hazards caused by malfunctioning electrical and electronic systems (ISO, 2018). ISO 21448, known as safety of the intended functionality (SOTIF), covers hazards that come from functional insufficiencies, such as sensor performance limits, when nothing has actually failed (ISO, 2022). Road testing alone can't prove reliability: one estimate puts it at more than 11 billion miles to show, with 95% confidence and 80% power, that an AV's fatality rate is 20% lower than human drivers' (Kalra & Paddock, 2016).

Beyond AVs, sensor fusion supports:

  • Advanced driver-assistance systems (ADAS): Radar plus camera for adaptive cruise control and automatic emergency braking.
  • Navigation: An inertial measurement unit (IMU) plus a global navigation satellite system (GNSS) keeps position accurate through tunnels and urban canyons.
  • Smartphones and wearables: The accelerometer, gyroscope and magnetometer combine to estimate orientation.
  • Internet of Things (IoT) and robotics: Merged readings from cameras, depth and environmental sensors drive building automation and warehouse robots.

How sensor fusion works

Sensor fusion technology: sensor types compared

Dimension Camera LiDAR Radar Ultrasonic IMU
What it measures Color, texture, signs 3D geometry (point cloud) Range and relative speed Distance at close range Acceleration and rotation
Effective range Medium (depends on lighting) Medium to long Long Short (a few meters) None (measures the vehicle itself)
Depth accuracy Moderate (inferred) High High Good at short range Drifts over time without an outside reference
Works in bad weather Poor Moderate Strong Moderate Not affected
Works in low light Poor Strong Strong Strong Not affected
Typical role Classification Shape and localization Speed, all-weather backup Parking, close obstacles Motion estimation between other sensor readings

Camera, LiDAR and radar ratings follow the sensor comparison in Yeong et al. (2021), Table 7. Ultrasonic ratings follow Vargas et al. (2021), Table 1, and Matos et al. (2024). IMU ratings follow Matos et al. (2024), Table 1.

The three categories of sensor fusion

Durrant-Whyte (1988) grouped fusion by how the sensors relate to each other:

Dimension Complementary Competitive (redundant) Cooperative
How the sensors relate Each sees a different part of the scene Several measure the same thing Combined to work out something none could alone
Goal Completeness Accuracy and fault tolerance New information (such as depth)
Example Four cameras giving 360° coverage Two radars on the same zone Stereo cameras estimating depth
If one sensor fails Coverage gap Graceful degradation Derived output is lost

The three levels of sensor fusion

Dimension Low level (early, data) Mid level (feature) High level (late, decision)
What gets fused Raw pixels and points Extracted features Each sensor's detections
Information kept Highest High Lowest
Compute cost Highest Medium Lowest
Calibration sensitivity Very high High Low
Tolerance of a failed sensor Low Medium High

Recent architectures such as BEVFusion merge camera and LiDAR features in a shared bird's-eye view (BEV), a mid-level (feature) approach that set a new state of the art on the nuScenes benchmark when it was published (Liu et al., 2023).

Sensor fusion algorithms: the Kalman filter and others

The Kalman filter repeats two steps: it predicts the next state from a motion model, then updates that prediction with each new measurement, giving more weight to the less noisy sensor (Kalman, 1960).

Dimension Kalman filter Extended Kalman filter (EKF) Unscented Kalman filter (UKF) Particle filter Deep learning fusion
Assumes Linear, Gaussian noise Nearly linear Nonlinear, Gaussian Any distribution Patterns learned from data
Compute cost Very low Low Medium High High (needs a GPU)
Handles nonlinearity No Approximately Well Very well Very well
Typical use Tracking, IMU/GNSS Vehicle localization Attitude estimation Robot localization Camera–LiDAR perception
Source Kalman, 1960 Thrun et al., 2005 Julier & Uhlmann, 1997 Thrun et al., 2005 Feng et al., 2021

Worked example: fusing two distance readings. LiDAR measures a pedestrian at 24.8 m (standard deviation 0.1 m, variance 0.01). Radar measures 25.3 m (standard deviation 0.5 m, variance 0.25). The Kalman update weights each reading by the inverse of its variance:

  • Fused distance = (24.8/0.01 + 25.3/0.25) ÷ (1/0.01 + 1/0.25) = 2,581.2 ÷ 104 = 24.82 m
  • Fused variance = 1/104 = 0.0096 (standard deviation about 0.098 m)

The result sits close to the more precise LiDAR reading and is slightly more certain than either sensor alone.

How to prepare sensor fusion training data

Lien names temporal synchronization and spatial calibration as the "two essential steps to organize data". In his words: "You can only construct the temporal causality and spatial geometry correctly with accurate synchronization and calibration."

  1. Synchronize timestamps: Align every sensor to one clock with a hardware trigger or the Precision Time Protocol (PTP). nuScenes, for example, triggers each camera exposure as the LiDAR sweeps across the center of that camera's view (Caesar et al., 2020). Small offsets add up: at 20 m/s, a 10 ms offset moves an object 20 cm.
  2. Calibrate spatially: Estimate each sensor's internal (intrinsic) and mounting (extrinsic) parameters. A mean camera reprojection error under 1 pixel is a common acceptance rule (MathWorks, n.d.). Then spot-check that LiDAR points projected onto the images land on object edges.
  3. Design the annotation layout: Show the synchronized camera views next to the point cloud, add static visual hints such as projected cuboids, and render only the information each task needs. Lien's team found this helps annotators deliver results faster and with less error.
  4. Split the job into stages: Label 3D cuboids on keyframes first, then assign tracking IDs across frames, then add attributes. Each stage builds on the last, which makes the next annotator's job easier.
  5. Set quality gates: Score annotators' cuboids against a gold standard using 3D intersection over union (IoU). For reference, the KITTI 3D object benchmark scores detections at IoU ≥ 0.7 for cars and ≥ 0.5 for pedestrians and cyclists (Geiger et al., 2012). Those are benchmark scoring thresholds, not an annotation standard, so set your own gates at or above them.
  6. Send disagreements to expert review: Frames where sensors disagree or objects are heavily occluded go to senior annotators, and what they decide feeds back into the guidelines.

Worked example: a quality-gate check. Three annotators box a partly hidden pedestrian 25 m away. Their IoU scores against the gold cuboid are 0.81, 0.74 and 0.58. All three clear KITTI's 0.5 pedestrian threshold, but the third is well below the other two, so the frame is flagged and goes to step 6 for review.

How Appen supports sensor fusion annotation

Appen's 3D sensor fusion and LiDAR annotation work applies the steps above through these controls:

  • Multi-sensor labeling: Coordinated annotation across LiDAR, radar and camera feeds for the same scene, with consistent object identities across modalities and object tracking across sequential frames. Quality is checked through multiple independent review rounds, geometric consistency checks, statistical quality sampling, inter-annotator agreement standards and gold-standard task validation.
  • Test questions: Gold test questions qualify contributors before a task (quiz mode) and keep checking them during it (work mode). If a contributor falls below the accuracy threshold, the job manager is notified and can remove them and their work from the job.
  • AI Data Platform (ADAP): Tooling for 3D point cloud and 4D annotation, with configurable workflows and multi-stage reviews, so a job can be split into the stages Lien describes. The ADAP LiDAR tool can project 3D cuboids into 2D boxes on the camera images and interpolate cuboids between keyframes (Appen, n.d.).
  • Trained annotators: LiDAR annotators are trained on the sensor characteristics and scene geometry of each client's data, so labels stay consistent across point densities and weather conditions. Appen's contributor network covers 1M+ people across 170+ countries.

For more detail, see Appen's 3D sensor fusion and LiDAR annotation services, the explainer on how LiDAR works, and the overview of physical AI training data.

Frequently asked questions

Is sensor fusion a form of AI?

Not always. Classic fusion with Kalman or particle filters is statistical estimation and involves no learning. Modern AV stacks, however, increasingly use neural networks that learn how to fuse camera, LiDAR and radar features, often in a shared bird's-eye view. These models need large, precisely synchronized, labelled multi-sensor datasets, so annotation quality directly limits how well learned fusion performs.

What is the difference between early and late fusion?

Early fusion merges raw sensor data before detection. It keeps the most information but needs precise calibration and heavy compute. Late fusion runs a separate detector for each sensor and merges the results. It's simpler and keeps working if one sensor fails, but it throws away cross-sensor cues, such as a faint camera edge that a LiDAR return could have confirmed. Many production systems use mid-level fusion as a compromise.

What software tools are used for sensor fusion?

Common choices include the Robot Operating System (ROS) robot_localization package, which provides ready-made EKF and UKF nodes, and the MATLAB Sensor Fusion and Tracking Toolbox for prototyping. OpenCV handles camera calibration. Teams benchmark against public datasets such as nuScenes, recorded with six cameras, one LiDAR and five radars (Caesar et al., 2020), and KITTI (Geiger et al., 2012).

How does sensor fusion work in a smartphone?

Phones combine an accelerometer, gyroscope and magnetometer to estimate orientation. The gyroscope is accurate over short periods but drifts. The accelerometer gives a gravity reference and the magnetometer gives a compass heading, but both are noisy. A complementary or Kalman filter uses the gyroscope for fast changes and the other two to correct drift, which powers screen rotation, augmented reality, step counting and map heading.

Why is sensor fusion annotation harder than 2D image labelling?

Each object needs a 3D cuboid with position, size and heading, and that cuboid must line up across every camera view and every frame. LiDAR returns get sparse at distance: a pedestrian 50 m away may return only a handful of points. Leftover calibration offsets also have to be judged visually, which is why staged workflows and gold-standard IoU checks matter more here than in 2D tasks.

References

Tags:
Sensor Fusion
Talk to our team