Skip to content

Preprocessing

API Description
filter Trajectory filtering.
compress Trajectory compression.
stay_locations Stops detection.
cluster Cluster stop locations.
latlng_to_h3 Rust-accelerated batch lat/lng → H3 cell conversion.
trajectory_to_od Build a long-format Origin-Destination matrix directly from a raw trajectory.

fastmob.preprocessing.filter(traj, max_speed_kmh=500.0, include_loops=False, speed_kmh=5.0, max_loop=6, ratio_max=0.25, *, datetime_col=None, lat_col=None, lng_col=None, uid_col=None, is_sorted=False, method='speed', **method_kwargs)

Filter trajectory noise by removing outlier points using a named algorithm.

Parameters:

Name Type Description Default
traj Any

Trajectory dataframe; any Narwhals-compatible eager backend.

 required
max_speed_kmh float

Parameters for the default method="speed" algorithm; unused by every other method. See the method="speed" description below.

 500.0
include_loops float

Parameters for the default method="speed" algorithm; unused by every other method. See the method="speed" description below.

 500.0
speed_kmh float

Parameters for the default method="speed" algorithm; unused by every other method. See the method="speed" description below.

 500.0
max_loop float

Parameters for the default method="speed" algorithm; unused by every other method. See the method="speed" description below.

 500.0
ratio_max float

Parameters for the default method="speed" algorithm; unused by every other method. See the method="speed" description below.

 500.0
datetime_col str | None

Explicit column name overrides; auto-detected when None.

 None
lat_col str | None

Explicit column name overrides; auto-detected when None.

 None
lng_col str | None

Explicit column name overrides; auto-detected when None.

 None
uid_col str | None

Explicit column name overrides; auto-detected when None.

 None
is_sorted

Whether the trajectory is already sorted by user and time, with nulls and NaNs removed. Setting this to True can speed up processing but may lead to incorrect results if the data is not properly preprocessed.

 False
method str

Name of the outlier-detection algorithm to run. One of "speed" (default, backward-compatible with every prior filter() call that never passed method), "hampel", "greedy", "smart_greedy", or "zheng".

 'speed'
**method_kwargs Any

Method-specific parameters for method != "speed", forwarded to the matching FILTER_METHODS[method] preparer.

  • speed: uses max_speed_kmh, include_loops, speed_kmh, max_loop, ratio_max above (unchanged from every prior release).
  • hampel: window_size (default 5) — centered rolling window length, in points, over the per-user consecutive-point speed (km/h) series; n_sigma (default 3.0) — outlier threshold as a multiple of 1.4826 * MAD(window).
  • greedy: max_speed_kmh (default 100.0) — maximum physically-consistent speed, in km/h, between each point and the immediately preceding point.
  • smart_greedy: max_speed_kmh (default 100.0) — same consistency threshold as greedy, but keeps the single longest run of mutually consistent points rather than testing fixed adjacent pairs.
  • zheng: max_speed_kmh (default 100.0) — consistency threshold as above; min_seg_size (default 1) — minimum consecutive-consistent-run length required to keep a segment (shorter runs are dropped in full).
 {}

Returns:

Type Description
DataFrame

Filtered trajectory in the same backend as input.

Examples:

>>> import pandas as pd
>>> import fastmob
>>> url = fastmob.data.BRIGHTKITE_SAMPLE
>>> df = pd.read_csv(
...     url,
...     sep="\t",
...     header=0,
...     nrows=5000,
...     names=["uid", "datetime", "lat", "lng", "location id"],
... )
>>> df["datetime"] = pd.to_datetime(df["datetime"], errors="coerce")
>>> df["location_id"] = df["location id"].astype("string")
>>> df = df.dropna(subset=["uid", "datetime", "lat", "lng"])[
...     ["uid", "datetime", "lat", "lng", "location_id"]
... ]
>>> print(df.head().to_string(index=False))
 uid                  datetime       lat         lng                              location_id
   0 2010-10-16 06:02:04+00:00 39.891383 -105.070814         7a0f88982aa015062b95e3b4843f9ca2
   0 2010-10-16 03:48:54+00:00 39.891077 -105.068532         dd7cd3d264c2d063832db506fba8bf79
   0 2010-10-14 18:25:51+00:00 39.750469 -104.999073 9848afcc62e500a01cf6fbf24b797732f8963683
   0 2010-10-14 00:21:47+00:00 39.752713 -104.996337         2ef143e12038c870038df53e0478cefc
   0 2010-10-13 23:31:51+00:00 39.752508 -104.996637         424eb3dd143292f9e013efa00486c907
>>> from fastmob.preprocessing import filter
>>> filtered = filter(df, max_speed_kmh=500.0)
>>> print(len(filtered))
4898
>>> print(filtered.head().to_string(index=False))
 uid                  datetime       lat         lng                      location_id
   0 2009-05-25 20:56:10+00:00 37.774929 -122.419415 ee81ef22a22411ddb5e97f082c799f59
   0 2009-05-25 21:35:28+00:00 37.600747 -122.382376 248b82709e6c11ddbf68003048c0801e
   0 2009-05-25 21:37:44+00:00 37.600747 -122.382376 248b82709e6c11ddbf68003048c0801e
   0 2009-05-25 21:42:47+00:00 37.600747 -122.382376 248b82709e6c11ddbf68003048c0801e
   0 2009-05-25 22:13:23+00:00 37.615223 -122.389979 be2f1e669cc111dd9a50003048c0801e
References

fastmob.preprocessing.compress(traj, spatial_radius_km=0.2, *, datetime_col=None, lat_col=None, lng_col=None, uid_col=None)

Compress trajectory by collapsing nearby points into single representative points.

All points within spatial_radius_km from an initial point are collapsed into one point with median lat/lng and the initial point's timestamp.

Parameters:

Name Type Description Default
traj Any

Trajectory dataframe; any Narwhals-compatible eager backend.

 required
spatial_radius_km float

Minimum distance (km) between consecutive output points.

 0.2
datetime_col str | None

Explicit column name overrides; auto-detected when None.

 None
lat_col str | None

Explicit column name overrides; auto-detected when None.

 None
lng_col str | None

Explicit column name overrides; auto-detected when None.

 None
uid_col str | None

Explicit column name overrides; auto-detected when None.

 None

Returns:

Type Description
DataFrame

Compressed trajectory in the same backend as input.

Examples:

>>> import pandas as pd
>>> import fastmob
>>> url = fastmob.data.BRIGHTKITE_SAMPLE
>>> df = pd.read_csv(
...     url,
...     sep="\t",
...     header=0,
...     nrows=5000,
...     names=["uid", "datetime", "lat", "lng", "location id"],
... )
>>> df["datetime"] = pd.to_datetime(df["datetime"], errors="coerce")
>>> df["location_id"] = df["location id"].astype("string")
>>> df = df.dropna(subset=["uid", "datetime", "lat", "lng"])[
...     ["uid", "datetime", "lat", "lng", "location_id"]
... ]
>>> print(df.head().to_string(index=False))
 uid                  datetime       lat         lng                              location_id
   0 2010-10-16 06:02:04+00:00 39.891383 -105.070814         7a0f88982aa015062b95e3b4843f9ca2
   0 2010-10-16 03:48:54+00:00 39.891077 -105.068532         dd7cd3d264c2d063832db506fba8bf79
   0 2010-10-14 18:25:51+00:00 39.750469 -104.999073 9848afcc62e500a01cf6fbf24b797732f8963683
   0 2010-10-14 00:21:47+00:00 39.752713 -104.996337         2ef143e12038c870038df53e0478cefc
   0 2010-10-13 23:31:51+00:00 39.752508 -104.996637         424eb3dd143292f9e013efa00486c907
>>> from fastmob.preprocessing import compress
>>> compressed = compress(df, spatial_radius_km=0.2)
>>> print(len(compressed))
3173
>>> print(compressed.head().to_string(index=False))
 uid                  datetime       lat         lng                      location_id
   0 2009-05-25 20:56:10+00:00 37.774929 -122.419415 ee81ef22a22411ddb5e97f082c799f59
   0 2009-05-25 21:35:28+00:00 37.600747 -122.382376 248b82709e6c11ddbf68003048c0801e
   0 2009-05-25 22:13:23+00:00 37.615223 -122.389979 be2f1e669cc111dd9a50003048c0801e
   0 2009-05-26 02:21:12+00:00 39.878664 -104.682105 e12721ce84e911dd8019003048c0801e
   0 2009-05-26 04:59:44+00:00 39.739154 -104.984703 ee8b1d0ea22411ddb074dbd65f1665cf
References

fastmob.preprocessing.stay_locations(traj, minutes_for_a_stop=20.0, spatial_radius_km=0.2, leaving_time=True, no_data_for_minutes=1000000000000.0, min_speed_kmh=None, include_last=True, *, datetime_col=None, lat_col=None, lng_col=None, uid_col=None)

Detect stay locations (stops) in trajectory data.

A stop is detected when the individual stays within spatial_radius_km for at least minutes_for_a_stop minutes. Stop coordinates are median lat/lng.

Parameters:

Name Type Description Default
traj Any

Trajectory dataframe; any Narwhals-compatible eager backend.

 required
minutes_for_a_stop float

Minimum duration (minutes) to qualify as a stop.

 20.0
spatial_radius_km float

Radius (km) within which points are grouped into a stop.

 0.2
leaving_time bool

If True, add a 'leaving_datetime' column with the departure time.

 True
no_data_for_minutes float

Gap threshold (minutes) above which data is treated as missing.

 1000000000000.0
min_speed_kmh float | None

If set, trim trailing high-speed points from the end of each stop.

 None
include_last bool

Whether to close and emit the still-open stay at the end of a user's track, i.e. one the user hadn't "stepped out of" yet when tracking ended. Matches trackintel's generate_staypoints(include_last=...), except trackintel defaults this to False.

 True
datetime_col str | None

Explicit column name overrides; auto-detected when None.

 None
lat_col str | None

Explicit column name overrides; auto-detected when None.

 None
lng_col str | None

Explicit column name overrides; auto-detected when None.

 None
uid_col str | None

Explicit column name overrides; auto-detected when None.

 None

Returns:

Type Description
DataFrame

Stop locations in the same backend as input. Schema: [uid_col, lat_col, lng_col, datetime_col, (leaving_datetime)]

Examples:

>>> import pandas as pd
>>> import fastmob
>>> url = fastmob.data.BRIGHTKITE_SAMPLE
>>> df = pd.read_csv(
...     url,
...     sep="\t",
...     header=0,
...     nrows=5000,
...     names=["uid", "datetime", "lat", "lng", "location id"],
... )
>>> df["datetime"] = pd.to_datetime(df["datetime"], errors="coerce")
>>> df["location_id"] = df["location id"].astype("string")
>>> df = df.dropna(subset=["uid", "datetime", "lat", "lng"])[
...     ["uid", "datetime", "lat", "lng", "location_id"]
... ]
>>> print(df.head().to_string(index=False))
 uid                  datetime       lat         lng                              location_id
   0 2010-10-16 06:02:04+00:00 39.891383 -105.070814         7a0f88982aa015062b95e3b4843f9ca2
   0 2010-10-16 03:48:54+00:00 39.891077 -105.068532         dd7cd3d264c2d063832db506fba8bf79
   0 2010-10-14 18:25:51+00:00 39.750469 -104.999073 9848afcc62e500a01cf6fbf24b797732f8963683
   0 2010-10-14 00:21:47+00:00 39.752713 -104.996337         2ef143e12038c870038df53e0478cefc
   0 2010-10-13 23:31:51+00:00 39.752508 -104.996637         424eb3dd143292f9e013efa00486c907
>>> from fastmob.preprocessing import stay_locations
>>> stops = stay_locations(df, spatial_radius_km=0.2, minutes_for_a_stop=20.0)
>>> print(len(stops))
3029
>>> print(stops.head().to_string(index=False))
      lat         lng            datetime  uid    leaving_datetime
37.774929 -122.419415 2009-05-25 20:56:10    0 2009-05-25 21:35:28
37.600747 -122.382376 2009-05-25 21:35:28    0 2009-05-25 22:13:23
37.615223 -122.389979 2009-05-25 22:13:23    0 2009-05-26 02:21:12
39.878664 -104.682105 2009-05-26 02:21:12    0 2009-05-26 04:59:44
39.739154 -104.984703 2009-05-26 04:59:44    0 2009-05-26 16:43:59
References

fastmob.preprocessing.cluster(traj, cluster_radius_km=0.1, min_samples=1, *, h3_resolution=None, datetime_col=None, lat_col=None, lng_col=None, uid_col=None, n_jobs=None)

Cluster stop locations with H3 cells and connected components.

Points are assigned to an H3 cell. Cells containing fewer than min_samples points are noise; active cells touching at an edge are joined into a cluster. cluster_radius_km selects a calibrated grid resolution for compatibility with the previous DBSCAN API. Pass h3_resolution (0--15) to control the grid explicitly; it takes precedence over cluster_radius_km. n_jobs is accepted for API compatibility and has no effect.

fastmob.preprocessing.latlng_to_h3(traj, resolution=9, *, lat_col=None, lng_col=None, output_col='h3_cell')

Convert each row's (lat, lng) to an H3 cell index, Rust-accelerated.

Rows with a non-finite or out-of-range coordinate (or a real Arrow null, for Polars/PyArrow input) get a null cell rather than failing the whole batch: NaN/inf coordinates in a NumPy-backed frame map to the reserved sentinel 2**64 - 1 (not a valid H3 index), while an Arrow-backed frame gets a genuine null slot in the output column.

Parameters:

Name Type Description Default
traj Any

Trajectory dataframe; any Narwhals-compatible eager backend.

 required
resolution int

H3 resolution, 0-15.

 9
lat_col str | None

Explicit column name overrides; auto-detected when None.

 None
lng_col str | None

Explicit column name overrides; auto-detected when None.

 None
output_col str

Name of the new column holding the H3 cell index.

 'h3_cell'

Returns:

Type Description
DataFrame

traj with output_col added, in the same backend as input.

Examples:

>>> import pandas as pd
>>> from fastmob.preprocessing import latlng_to_h3
>>> df = pd.DataFrame({"lat": [37.769377], "lng": [-122.388519]})
>>> result = latlng_to_h3(df, resolution=9)
>>> hex(result["h3_cell"].iloc[0])
'0x89283082e73ffff'

fastmob.preprocessing.trajectory_to_od(traj, resolution=9, *, uid_col=None, datetime_col=None, lat_col=None, lng_col=None, drop_self_loops=True)

Build a long-format Origin-Destination matrix directly from a raw trajectory.

Tessellates each point into an H3 cell (via :func:fastmob.preprocessing.latlng_to_h3), sorts each user chronologically, and pairs each point's cell with the next point's cell (per user) to derive one trip per consecutive fix. Delegates the final counting to :func:fastmob.measures.collective.od.od_matrix.

Parameters:

Name Type Description Default
traj Any

Trajectory dataframe; any Narwhals-compatible eager backend.

 required
resolution int

H3 resolution, 0-15.

 9
uid_col str | None

Explicit column name overrides; auto-detected when None. When uid_col cannot be found, the whole frame is treated as one user.

 None
datetime_col str | None

Explicit column name overrides; auto-detected when None. When uid_col cannot be found, the whole frame is treated as one user.

 None
lat_col str | None

Explicit column name overrides; auto-detected when None. When uid_col cannot be found, the whole frame is treated as one user.

 None
lng_col str | None

Explicit column name overrides; auto-detected when None. When uid_col cannot be found, the whole frame is treated as one user.

 None
drop_self_loops bool

When True (default), consecutive fixes that land in the same cell are not counted as a trip -- matches how "did the user actually move somewhere" trip derivation is normally defined. Set False to also count same-cell "stayed put" pairs (as MoveInside traffic).

 True

Returns:

Type Description
DataFrame

Long-format OD counts with columns ["origin", "destination", "count"], matching :func:od_matrix's contract. Callers that want the wide pivoted form used by some report code can .pivot(on="destination", index="origin", values="count") this themselves.

Examples:

>>> import pandas as pd
>>> from fastmob.preprocessing import trajectory_to_od
>>> traj = pd.DataFrame(
...     {
...         "uid": ["u1", "u1", "u1"],
...         "datetime": pd.to_datetime(["2020-01-01 00:00", "2020-01-01 01:00", "2020-01-01 02:00"]),
...         "lat": [37.7793, 37.6007, 37.6007],
...         "lng": [-122.4194, -122.3824, -122.3824],
...     }
... )
>>> result = trajectory_to_od(traj, resolution=7)
>>> len(result)
1