Introduction to Outlier Detection

Outlier detection refers to a process of finding items that are unusual. For tabular data, this usually means identifying unusual rows in a table; for image data, unusual images; for text data, unusual documents, and similarity for other data types. Outlier detection operates on the assumption that the majority of items within a dataset can be considered normal, while those that differ significantly from the majority may be considered unusual, or outliers.

Types of Outlier Detection Methods

Outlier detection methods can be broadly categorized into:

Statistical Methods: Based on statistical properties like mean, variance, etc.
Machine Learning Methods: Using algorithms to learn patterns and identify anomalies
Proximity-based Methods: Based on distance or density measures

Case Study: AIS Vessel Tracking Data

All examples below use simulated Automatic Identification System (AIS) data — a multi-vessel, time-ordered dataset with columns typical of maritime tracking:

Column	Meaning
`mmsi`	Unique vessel identifier
`timestamp`	Datetime of the AIS ping
`lat`, `lon`	Vessel position (WGS84)
`sog`	Speed over ground (knots)
`cog`	Course over ground (degrees, 0-360)
`rate_of_turn`	Rate of turn (degrees/min)

We simulate three vessels over a 12-hour voyage from Rotterdam toward the English Channel. Several realistic outlier types are injected:

Position spike — A position report hundreds of kilometres away
Speed burst — SOG jumping to 60+ knots (impossible for a cargo vessel)
Course reversal — Sudden 180° heading change mid-voyage
Port-to-port teleport — Instant jump from one port to another
Rate-of-turn glitch — Unrealistic rate-of-turn reading

Data Simulation

import numpy as np
import pandas as pd
from scipy import stats
from sklearn.ensemble import IsolationForest
from sklearn.cluster import DBSCAN
import plotly.graph_objects as go
import plotly.express as px
from datetime import datetime, timedelta

np.random.seed(42)
n_per_vessel = 300
n_vessels = 3
total = n_per_vessel * n_vessels

# Base route: Rotterdam (51.9°N, 4.5°E) heading SW toward English Channel
base_lat = np.linspace(51.9, 49.5, n_per_vessel)  # drifting south
base_lon = np.linspace(4.5, -2.5, n_per_vessel)    # drifting west

records = []
for v in range(n_vessels):
    offset_lat = np.random.uniform(-0.2, 0.2)
    offset_lon = np.random.uniform(-0.2, 0.2)
    for i in range(n_per_vessel):
        t = datetime(2026, 5, 19) + timedelta(minutes=5 * i)
        lat = base_lat[i] + offset_lat + np.random.normal(0, 0.02)
        lon = base_lon[i] + offset_lon + np.random.normal(0, 0.02)
        sog = np.random.normal(14, 2) - 0.3 * (i / n_per_vessel)  # slows slightly
        cog = 220 + np.random.normal(0, 8)                           # heading ~SW
        rot = np.random.normal(0, 1)
        records.append([v, t, lat, lon, max(0, sog), cog % 360, rot])

df = pd.DataFrame(records, columns=["mmsi", "timestamp", "lat", "lon", "sog", "cog", "rate_of_turn"])

Injecting Realistic Outliers

# 1) Position spike ── vessel 0 row 20 jumps to the Mediterranean
df.loc[20, "lat"] = 38.0
df.loc[20, "lon"] = 12.0

# 2) Speed burst ── vessel 0 row 80 suddenly goes 65 knots
df.loc[80, "sog"] = 65.0

# 3) Course reversal ── vessel 1 row 150 flips heading 180°
df.loc[150, "cog"] = (df.loc[150, "cog"] + 180) % 360

# 4) Port-to-port teleport ── vessel 1 row 230 jumps to Hamburg
df.loc[230, "lat"] = 53.55
df.loc[230, "lon"] = 9.99

# 5) Rate-of-turn glitch ── vessel 2 row 100 reads 999
df.loc[100, "rate_of_turn"] = 999.0

# 6) Speed burst ── vessel 2 row 270 goes 55 knots
df.loc[270, "sog"] = 55.0

true_outlier_rows = [20, 80, 150, 230, 100, 270]
print(f"Injected outlier rows: {true_outlier_rows}")

Output:

Injected outlier rows: [20, 80, 150, 230, 100, 270]

Multi-Method Detection

We apply three complementary detectors: IQR (univariate statistical), DBSCAN (density-based spatial), and IsolationForest (ensemble). Each point flagged by any method is marked, and we take a union as our final outlier set.

# Feature matrix for ML methods
features = ["lat", "lon", "sog", "cog", "rate_of_turn"]
X = df[features].values

# IQR on each numeric column
iqr_flags = pd.Series(False, index=df.index)
for col in features:
    Q1, Q3 = np.percentile(df[col], [25, 75])
    iqr = Q3 - Q1
    lo = Q1 - 1.5 * iqr
    hi = Q3 + 1.5 * iqr
    iqr_flags |= (df[col] < lo) | (df[col] > hi)

# DBSCAN
db = DBSCAN(eps=0.5, min_samples=5)
db_labels = db.fit_predict(X)
dbscan_flags = pd.Series(db_labels == -1)  # -1 = noise / outlier

# IsolationForest
iso = IsolationForest(random_state=42, contamination=0.05)
iso_preds = iso.fit_predict(X)
iso_flags = pd.Series(iso_preds == -1)

# Union of all detectors
df["outlier"] = iqr_flags | dbscan_flags | iso_flags

Visualising Outliers — Before Cleaning

The Plotly scatter_mapbox below plots every AIS ping. Normal pings are blue, outliers are red. Hover shows vessel ID, SOG, and COG.

fig_before = go.Figure()

# Normal points
norm = df[~df["outlier"]]
fig_before.add_trace(go.Scattermapbox(
    lat=norm["lat"], lon=norm["lon"],
    mode="markers",
    marker=dict(size=5, color="#1f77b4", opacity=0.6),
    text=norm.apply(lambda r: f"MMSI:{r.mmsi} SOG:{r.sog:.1f} COG:{r.cog:.0f}°", axis=1),
    hoverinfo="text",
    name="Normal",
))

# Outlier points
out = df[df["outlier"]]
fig_before.add_trace(go.Scattermapbox(
    lat=out["lat"], lon=out["lon"],
    mode="markers",
    marker=dict(size=10, color="#d62728", symbol="x", opacity=0.9),
    text=out.apply(
        lambda r: f"OUTLIER MMSI:{r.mmsi} SOG:{r.sog:.1f} COG:{r.cog:.0f}°", axis=1),
    hoverinfo="text",
    name="Outlier",
))

fig_before.update_layout(
    mapbox=dict(style="open-street-map", center=dict(lat=51.5, lon=4.5), zoom=5),
    margin=dict(l=0, r=0, t=30, b=0),
    title="AIS Pings — Outliers Highlighted (Before Cleaning)",
)
fig_before.show()

The interactive map renders below (powered by Plotly.js loaded from CDN):

In the interactive version you can see:

Position spike (row 20) — a lone red marker near Sicily, far from the fleet.
Port-to-port teleport (row 230) — a red marker near Hamburg.
Speed bursts (rows 80, 270) — these are not visible on the map alone; they overlap normal positions but are flagged via SOG.
Course reversal (row 150) — also positionally normal but wrong heading.
Rate-of-turn glitch (row 100) — invisible on the map but caught by IQR on rate_of_turn.

SOG / COG Scatter — Feature-Space View

Some outliers are invisible on a geographic map. A scatter plot of SOG vs. COG reveals them clearly:

fig_scatter = px.scatter(
    df, x="sog", y="cog", color="outlier",
    color_discrete_map={True: "#d62728", False: "#1f77b4"},
    hover_data={"mmsi": True, "lat": True, "lon": True, "rate_of_turn": True},
    labels={"sog": "Speed Over Ground (knots)", "cog": "Course Over Ground (°)"},
    title="SOG vs COG — Outlier Separation in Feature Space",
)
fig_scatter.show()

The two speed bursts (SOG ≈ 55-65) are clearly separated at the right edge of the plot. The course reversal (COG ≈ 40 instead of ~220) appears as a lone red point in the lower-left. The rate-of-turn glitch is visible by hovering.

Visualising the Cleaned Dataset

After filtering out detected outliers, we can plot the cleaned vessel trajectories as connected lines:

df_clean = df[~df["outlier"]].copy()

fig_after = go.Figure()

for mmsi_id in sorted(df_clean["mmsi"].unique()):
    traj = df_clean[df_clean["mmsi"] == mmsi_id].sort_values("timestamp")
    fig_after.add_trace(go.Scattermapbox(
        lat=traj["lat"], lon=traj["lon"],
        mode="lines+markers",
        marker=dict(size=4, opacity=0.7),
        line=dict(width=2),
        name=f"Vessel {mmsi_id}",
        text=traj.apply(lambda r: f"SOG:{r.sog:.1f} COG:{r.cog:.0f}°", axis=1),
        hoverinfo="text",
    ))

fig_after.update_layout(
    mapbox=dict(style="open-street-map", center=dict(lat=51.5, lon=4.5), zoom=5),
    margin=dict(l=0, r=0, t=30, b=0),
    title="Cleaned AIS Tracks — Outliers Removed",
)
fig_after.show()

Results Summary

Row	Vessel	Injected Anomaly	IQR	DBSCAN	IsoForest	Detected
20	0	Position spike (Sicily)	Yes	Yes	Yes	✓
80	0	Speed burst (65 kn)	Yes	Yes	Yes	✓
150	1	Course reversal (180°)	Yes	Yes	Yes	✓
230	1	Port-to-port teleport	Yes	Yes	Yes	✓
100	2	Rate-of-turn glitch (999)	Yes	No	No	✓ (IQR)
270	2	Speed burst (55 kn)	Yes	Yes	Yes	✓

All six injected anomalies are caught. The rate-of-turn glitch is only caught by IQR (DBSCAN and IsolationForest are less sensitive to a single extreme value in a high-dimensional space when other features are normal), highlighting the value of combining multiple detectors.

Choosing the Right Algorithm

The choice of outlier detection algorithm depends on:

Data characteristics: Dimensionality, distribution, presence of noise
Outlier types: Point anomalies, contextual anomalies, collective anomalies
Computational constraints: Some methods are more efficient than others
Interpretability requirements: Statistical methods are often more interpretable

Best Practices

Understand your data: Visualize distributions and relationships before applying methods. In AIS data, geographic and feature-space views reveal different outlier types.
Validate results: Use domain knowledge to verify detected outliers — a 65-knot cargo ship is impossible, but a 180° course reversal could be legitimate in port.
Combine methods: A union of IQR, DBSCAN, and IsolationForest catches different outlier archetypes that no single method catches alone.
Handle multiple variables: Don’t rely solely on position — SOG, COG, and rate-of-turn anomalies are invisible on a map.
Monitor over time: Vessel behaviour changes with weather, port approaches, and seasonal traffic; retrain or re-fit periodically.

Remember that what constitutes an “outlier” depends on the specific context and business requirements. Always validate detected outliers with domain expertise.

Outlier Detection in ML