Skip to content

Open in Colab Launch Binder

Use your own columns and dataframe library

Question. My data do not use the column names the library expects, or they live in Polars or Arrow instead of pandas. How do I use them without converting?

You will get the same measures computed from pandas, Polars and PyArrow tables with non-standard column names, each returning the same kind of table it was given.

Prerequisites. Start with a trajectory. Nothing is downloaded.

Data. Three invented positions of one person in Rome (synthetic), stored with unusual column names (person, observed_at, latitude_deg, longitude_deg). The same table is built three times, as pandas, Polars and PyArrow.

%pip install -q 'fastmob>=0.3.4' pandas matplotlib polars pyarrow

The table, with non-standard names

Fastkit-Mobility looks for columns by name: the first match in each list below is used, and a missing user column means all rows belong to one person.

Role Accepted names, in order
time datetime, check-in_time, timestamp, time
latitude lat, latitude
longitude lng, lon, longitude, long
person (optional) uid, user, user_id

Our columns match none of these, so we will name them explicitly.

import fastmob
import pandas as pd
import polars as pl
import pyarrow as pa

times = ["2020-01-01 08:00:00", "2020-01-01 09:00:00", "2020-01-01 10:00:00"]
columns = {
    "person": ["alice"] * 3,
    "observed_at": pd.to_datetime(times),
    "latitude_deg": [41.8902, 41.9028, 41.9109],
    "longitude_deg": [12.4922, 12.4964, 12.4818],
}
as_pandas = pd.DataFrame(columns)
as_polars = pl.DataFrame(columns)
as_arrow = pa.table(columns)
as_pandas
person observed_at latitude_deg longitude_deg
0 alice 2020-01-01 08:00:00 41.8902 12.4922
1 alice 2020-01-01 09:00:00 41.9028 12.4964
2 alice 2020-01-01 10:00:00 41.9109 12.4818

Without telling the library the names

Calling a measure directly fails with a message that lists the names it looked for: a missing column is never guessed.

try:
    fastmob.jump_lengths(as_pandas)
except ValueError as error:
    print(type(error).__name__ + ":", str(error)[:300])

ValueError: Could not find required column(s): 'datetime' (looked for: ['datetime', 'check-in_time', 'timestamp', 'time']); 'latitude' (looked for: ['lat', 'latitude']); 'longitude' (looked for: ['lng', 'lon', 'longitude', 'long']). Available columns: ['person', 'observed_at', 'latitude_deg', 'longitude_deg'].

The error names every column it was looking for and the columns it found, so the fix is obvious. Nothing is guessed from the data.

Tell the library the names

Every measure takes datetime_col, lat_col, lng_col and uid_col. Pass them once in a dictionary to keep calls short.

names = {
    "datetime_col": "observed_at",
    "lat_col": "latitude_deg",
    "lng_col": "longitude_deg",
    "uid_col": "person",
}

jumps_pandas = fastmob.jump_lengths(as_pandas, **names)
radius_pandas = fastmob.radius_of_gyration(as_pandas, **names)

print(type(jumps_pandas).__module__ + "." + type(jumps_pandas).__name__)
jumps_pandas

pandas.DataFrame
person jump_lengths
0 alice [1.44354, 1.506992]

The result is a pandas table with one row per person and the list of jump lengths in kilometres: 1.44 km and 1.51 km, the same as in the first notebook.

Polars and PyArrow in, the same type out

The same call works on a Polars DataFrame and a PyArrow Table; the result comes back as the type you passed in, with no conversion on your side.

jumps_polars = fastmob.jump_lengths(as_polars, **names)
jumps_arrow = fastmob.jump_lengths(as_arrow, **names)

for label, result in (
    ("pandas", jumps_pandas),
    ("Polars", jumps_polars),
    ("PyArrow", jumps_arrow),
):
    print(f"{label:8s} in -> {type(result).__module__}.{type(result).__name__}")
jumps_polars

pandas   in -> pandas.DataFrame
Polars   in -> polars.dataframe.frame.DataFrame
PyArrow  in -> pyarrow.lib.Table
shape: (1, 2)
personjump_lengths
strlist[f64]
"alice"[1.44354, 1.506992]

Each input type comes back as itself: pandas to pandas, Polars to Polars, PyArrow to PyArrow. The library does not copy your data to another library to compute the result.

values = [round(float(x), 4) for x in jumps_pandas["jump_lengths"].iloc[0]]
print("pandas ", values)
print("Polars ", [round(x, 4) for x in jumps_polars["jump_lengths"].to_list()[0]])
print(
    "PyArrow", [round(x, 4) for x in jumps_arrow.column("jump_lengths").to_pylist()[0]]
)

pandas  [1.4435, 1.507]
Polars  [1.4435, 1.507]
PyArrow [1.4435, 1.507]

The three backends return the same values.

One person without a user column

When a table holds one person's trajectory, leave out uid_col: the whole table is one person, and the result has no user column.

single = as_pandas.drop(columns="person")
one_person = fastmob.radius_of_gyration(
    single, datetime_col="observed_at", lat_col="latitude_deg", lng_col="longitude_deg"
)
one_person
radius_of_gyration
0 1.074642

Without a person column the result is one row without a user column: the radius of gyration of the whole table, 1.07 km, equal to Alice's earlier value.

Interpretation and limits

  • The measures read the time column as it is. A column of text must be converted to a datetime first (pd.to_datetime, or str.to_datetime in Polars): the notebook did it when building the tables.
  • Time zones are not converted. If your data carry a time zone, convert it explicitly (see Process trajectories).
  • Distances are in kilometres whatever the backend.
  • Results that are tables come back in the input's type; results that are arrays or numbers (for example jump_lengths(..., merge=True)) come back as NumPy arrays or Python floats.

Next steps

jump_lengths ยท radius_of_gyration