Fitting

Fit a distribution to positive values by maximum likelihood, with confidence intervals.

The values are grouped into geometric bins on [xmin, xmax] and the family is fitted by maximising the multinomial log-likelihood of the bin counts. The confidence intervals are percentile intervals of a bootstrap of the bin counts, which treats the values as independent: when they come from a limited number of people pass groups so that whole people are resampled. When n_gof is positive, a parametric bootstrap gives a p-value for the Kolmogorov-Smirnov distance. Values that are zero, negative, not finite or outside the range are left out and counted in n_excluded, so state the range when you report a fit: the estimates depend on it.

The families are f(x) ∝ x^-beta ("powerlaw"), exp(-x / kappa) ("exponential"), a log-normal with parameters mu and sigma ("lognormal"), (x + r0)^-beta ("shifted_powerlaw") and (x + r0)^-beta exp(-x / kappa) ("truncated_powerlaw", as in González et al., 2008). Comparing the last two shows whether the cutoff adds anything beyond the shift r0. A parameter that ends on the edge of its search range is listed in at_bound; for the truncated power law this typically means that the data have no cutoff.

Parameters:

Returns:

Raises:

Examples:

>>> import numpy as np
>>> from fastmob.measures.fitting import fit_distance_distribution
>>> rng = np.random.default_rng(1)
>>> jumps = rng.lognormal(mean=1.0, sigma=0.8, size=5000)
>>> fit = fit_distance_distribution(jumps, "lognormal", n_bootstrap=20)
>>> round(fit.parameters["mu"], 1), round(fit.parameters["sigma"], 1)
(1.0, 0.8)
>>> fit.at_bound
()

Fit several families to the same values and rank them by AIC.

Every family is fitted on the same range and bins, so their likelihoods, AIC and BIC are comparable. A small AIC difference (under about 2) means the data cannot tell the families apart.

A family with a parameter on the edge of its search range (at_bound) is a degenerate fit when the edge is not a legitimate limit of the family: the shape it reached is not the one the family is meant to describe (a shifted power law with beta at its upper limit is an exponential in disguise), so its AIC says little. r0 at its lower edge (a power law, or a power law with an exponential cutoff) and kappa at its upper edge (no cutoff) are legitimate limits, so those fits are ranked normally; see DistributionFit.limits. With degenerate_last=True (the default) those families are listed after the others, delta_aic is measured from the best family that is not degenerate, and akaike_weight is computed among the families that are not degenerate only (it is missing for the others). A negative delta_aic on a degenerate row means that its likelihood is higher, which is what the degeneracy buys; do not read it as support.

Parameters:

Returns:

Examples:

>>> import numpy as np
>>> from fastmob.measures.fitting import compare_distance_distributions
>>> jumps = np.random.default_rng(1).lognormal(1.0, 0.8, size=5000)
>>> table = compare_distance_distributions(jumps, ["exponential", "lognormal"])
>>> table.column("family").to_pylist()[0]
'lognormal'

Pick xmin as the lower end of the range that makes the fit closest to the data.

For each candidate xmin the family is fitted on [xmin, xmax] and the Kolmogorov-Smirnov distance between the fitted and the empirical distribution is computed; the candidate with the smallest distance is selected (Clauset, Shalizi and Newman, 2009). Fit fit_distance_distribution again with the selected xmin to get the estimates and intervals. The selection uses the data, so the intervals of that final fit do not include the uncertainty about xmin: report the selected value and how the estimate moves over the candidates (the table). The KS distance is computed on the bin edges of each fit, and a lower xmin leaves more values, so a family that does not describe the small values gets a high xmin.

Parameters:

Returns:

Raises:

Examples:

>>> import numpy as np
>>> from fastmob.measures.fitting import select_xmin
>>> x = np.random.default_rng(2).pareto(1.5, size=3000) + 1.0
>>> table = select_xmin(x, "powerlaw", n_candidates=8)
>>> bool(table.column("selected").to_pylist().count(True) == 1)
True

Test a family against the family that extends it.

The statistic is 2 (log L_general - log L_restricted). The restricted family is a limit of the general one with parameters on the boundary of their range (r0 -> 0, or the cutoff kappa -> infinity), so the null distribution is not chi-squared: it is a mixture of chi-squared distributions (Chernoff, 1954), which is used here (for the pair with two boundary parameters the mixture assumes that they are orthogonal, Self and Liang, 1987). That asymptotic p-value needs a range that does not clip the data: when xmax is the sample maximum the cutoff can hide behind the end of the data and the test has little power, and its size is wrong. With n_bootstrap above 0 the p-value comes instead from a parametric bootstrap: data are simulated from the restricted fit, both families are refitted, and the p-value is the share of statistics at least as large as the observed one. It is slower but does not rely on the asymptotic theory. The pairs that can be tested are powerlaw against shifted_powerlaw, shifted_powerlaw against truncated_powerlaw (does the data support a cutoff at all?) and powerlaw against truncated_powerlaw. The test treats the values as independent, so it overstates the evidence for clustered values.

Parameters:

Returns:

Raises:

Warns:

Examples:

>>> import numpy as np
>>> from fastmob.measures.fitting import fit_distance_distribution, likelihood_ratio_test
>>> x = np.random.default_rng(3).pareto(1.5, size=4000) + 1.0
>>> a = fit_distance_distribution(x, "powerlaw", n_bootstrap=0)
>>> b = fit_distance_distribution(x, "shifted_powerlaw", n_bootstrap=0)
>>> likelihood_ratio_test(a, b).test
'likelihood ratio'

Vuong test of two families that are not nested.

The test asks whether the two fitted families have the same expected log-likelihood per value. Under that null the standardised difference of the log-likelihoods is approximately standard normal (Vuong, 1989). A significant positive statistic favours first, a negative one second; a small statistic means that the data cannot choose, however large the AIC difference looks. The values are treated as independent, so the evidence is overstated for clustered values.

Parameters:

Returns:

Raises:

Examples:

>>> import numpy as np
>>> from fastmob.measures.fitting import fit_distance_distribution, vuong_test
>>> x = np.random.default_rng(1).lognormal(1.0, 0.8, size=4000)
>>> a = fit_distance_distribution(x, "lognormal", n_bootstrap=0)
>>> b = fit_distance_distribution(x, "exponential", n_bootstrap=0)
>>> vuong_test(a, b).favours
'lognormal'

Fit a lognormal distribution to per-user daily distinct-location counts.

staypoints may be a Staypoints object or an eager dataframe. When a Locations catalogue is supplied, its global or user-scoped location identities are validated before fitting. Timezone-aware starts are bucketed by their local wall-clock calendar day.

Fit a truncated power law to positive values using a log histogram.

Fitting uses a deterministic, parallel coarse-to-fine grid search over (r0, beta, kappa) implemented in Rust; the optimal c is solved in closed form in log space for every candidate. The log-spaced histogram the search optimizes over is also computed in Rust and returned alongside the fitted parameters.

Fit the universal visitation law from staypoints.

Uses supplied global locations directly, or assigns shared H3 cells when no catalogue is supplied; it then fits rho = mu * rf**(-eta).

Returns:

Evaluate log(c * (x + r0)**(-beta) * exp(-x / kappa)).