Evaluation Measures

Return the common part of commuters (CPC) between observed and predicted flows.

CPC is 2 * sum(min(observed, predicted)) / (sum(observed) + sum(predicted)) over origin-destination pairs: 1 when the two flow tables are identical, 0 when they share no trips. Self-loops and missing endpoints are ignored.

Parameters:

Returns:

Compare the trip-distance distributions of two sets of trips (CPCD).

Distances are grouped in 2 km bins and the result is sum(min(count_observed, count_predicted)) / number of observed trips: 1 for identical distance distributions, 0 for none in common.

Parameters:

Returns:

Return the common part of links (CPL) between observed and predicted flows.

CPL is 2 * |links in both| / (|links observed| + |links predicted|): the share of origin-destination links that exist in both tables, whatever their weight. It is 1 when both have exactly the same links.

Parameters:

Returns:

Score one or several flow models against observed origin-destination flows.

Every model is scored on the same set of pairs, leaving out self-loops, and a pair that is missing from a table has flow 0 there. By default (pairs="catalogue") the pairs are all the pairs of the locations of observed, so the score of a model does not depend on which other models are scored with it; report the n_pairs column with the scores. With pairs="union" they are the pairs that have flow in the observed table or in any model, which is the only choice when observed has no locations, but then a model's score changes when another model is added. The Poisson deviance and pseudo R-squared treat the flows as counts, so compare models whose total flow is on the scale of the observed one (for example generated with the observed outflows).

Two cautions. The common part of links (CPL) compares which pairs have flow. Expected flows (out_format="flows") are positive on every pair, so their CPL is fixed by the number of observed pairs and says nothing about the model; use sampled (integer) flows for CPL. And a model that was fitted to the observed flows is scored in-sample: compare it with a parameter-free model, or score it on flows that were held out, before concluding that it is better.

Parameters:

Returns:

Raises:

Examples:

>>> import pandas as pd
>>> from fastmob import FlowDataFrame
>>> from fastmob.measures.evaluation import compare_flow_models
>>> table = pd.DataFrame({"origin": ["a", "a", "b"], "destination": ["b", "c", "c"], "flow": [10, 5, 7]})
>>> flows = FlowDataFrame(table, tile_id="tile_id")
>>> compare_flow_models(flows, flows, pairs="union").column("cpc").to_pylist()
[1.0]

Compare value_col between two dataframes, optionally grouped by group_col.

metric defaults to wasserstein_distance and accepts any (array, array) -> float callable, e.g. jensen_shannon_divergence. When group_col is given, the comparison runs once per label present in both dataframes (sorted for determinism) and overall is the mean across groups.

Return the information gain (KL divergence variant) of true over pred.

where \(N = \sum_i y_i\).

Parameters:

Returns:

Examples:

>>> from fastmob import information_gain
>>> observed = [10, 20, 30, 40]
>>> predicted = [12, 18, 33, 37]
>>> print(round(information_gain(observed, predicted), 3))
0.005

Return Jensen-Shannon divergence between two distributions.

Inputs are normalised internally (raw counts and probabilities both work) and must have the same length -- unlike wasserstein_distance, the two arrays are paired by index (one entry per category/bin), not independent samples.

Return the Kullback–Leibler divergence \(D_\mathrm{KL}(\mathrm{true}\,\|\,\mathrm{pred})\).

Inputs are normalized to probability distributions before evaluation, matching the historical scipy.stats.entropy(true, pred) behavior.

Parameters:

Returns:

Examples:

>>> from fastmob import kullback_leibler_divergence
>>> observed = [10, 20, 30, 40]
>>> predicted = [12, 18, 33, 37]
>>> print(round(kullback_leibler_divergence(observed, predicted), 3))
0.005

Return the maximum signed error max(true_i - pred_i).

Parameters:

Returns:

Examples:

>>> from fastmob import max_error
>>> observed = [10, 20, 30, 40]
>>> predicted = [12, 18, 33, 37]
>>> print(round(max_error(observed, predicted), 3))
3.0

Return the mean squared error between true and predicted values.

Parameters:

Returns:

Examples:

>>> from fastmob import mse
>>> observed = [10, 20, 30, 40]
>>> predicted = [12, 18, 33, 37]
>>> print(round(mse(observed, predicted), 3))
6.5

Return the normalized root mean squared error (RMSE / sum(true)).

Parameters:

Returns:

Examples:

>>> from fastmob import nrmse
>>> observed = [10, 20, 30, 40]
>>> predicted = [12, 18, 33, 37]
>>> print(round(nrmse(observed, predicted), 3))
0.025

Return the coefficient of determination R².

where

Parameters:

Returns:

Examples:

>>> from fastmob import r_squared
>>> observed = [10, 20, 30, 40]
>>> predicted = [12, 18, 33, 37]
>>> print(round(r_squared(observed, predicted), 3))
0.948

Return the root mean squared error between true and predicted values.

Parameters:

Returns:

Examples:

>>> from fastmob import rmse
>>> observed = [10, 20, 30, 40]
>>> predicted = [12, 18, 33, 37]
>>> print(round(rmse(observed, predicted), 3))
2.55

Compute the spatio-temporal Wasserstein distance between two distributions.

Each input is a DataFrame representing a spatial-temporal distribution with columns for a time bin (HH:MM string), a weight/volume value, and a WGS84 latitude/longitude centroid — the exact shape produced by fastmob.measures.collective.build_stvd.

Distances are computed as an entropy-regularised (Sinkhorn) Earth Mover's Distance over an explicit cost matrix combining the great-circle (Haversine) distance between centroids with a cyclical-time term: two points at times t1/t2 contribute 2 * r * sin(d_theta / 2) metres, where r = alpha * cyclical_period / (2*pi), theta = 2*pi*(t / cyclical_period), and d_theta is the shortest angular difference between the two times.

Parameters:

Returns:

Examples:

>>> import pandas as pd
>>> from fastmob.measures.evaluation import stvd_emd
>>> dist_a = pd.DataFrame(
...     {
...         "time_bin": ["08:00", "08:10"],
...         "mean_volume": [2.0, 1.0],
...         "center_lat": [0.0, 0.0],
...         "center_lng": [0.0, 0.001],
...     }
... )
>>> dist_b = pd.DataFrame(
...     {
...         "time_bin": ["08:00", "08:10"],
...         "mean_volume": [1.0, 2.0],
...         "center_lat": [0.0, 0.0],
...         "center_lng": [0.0, 0.002],
...     }
... )
>>> print(round(stvd_emd(dist_a, dist_b), 3))
74.156

Return the 1-D Wasserstein (earth mover's) distance between two empirical samples.

The distance is the average amount by which the values of one sample must be moved to turn it into the other, in the units of the values. Non-finite values are dropped; the result is nan if a sample is empty.

Parameters:

Returns: