Flows between places

Question. How many people move between places, and how well do two flow tables agree?

You will get an origin-destination table of trips between H3 cells with per-area summaries, a spatial flow table on a grid of zones, and a comparison of two flow tables with the common part of commuters and of links.

Prerequisites. Population measures, which uses the same data and cleaning.

Data. Brightkite check-ins (Stanford SNAP, 2008 to 2010): the first 300,000 rows (about 4 MB of the 60 MB file) with fastmob.data.load_brightkite; users with fewer than 100 check-ins are dropped, which leaves about 306 users. Check-ins are voluntary and sparse. The grid used for zones is synthetic. The notebook stops if the download fails.

%pip install -q 'fastmob>=0.3.4' pandas geopandas shapely

Load and clean

The raw file contains placeholder and impossible coordinates. We drop check-ins at exactly latitude 0 and longitude 0 (a placeholder for "unknown") and the few rows with a latitude whose absolute value is 90 degrees or more, which is not a valid position. Always check coordinate ranges before a population analysis.

import fastmob
import numpy as np
from fastmob.data import load_brightkite
from fastmob.measures import collective

raw = load_brightkite(max_rows=300_000).to_pandas()

bad = ((raw["lat"] == 0) & (raw["lng"] == 0)) | (raw["lat"].abs() >= 90)
print(f"dropping {bad.sum()} check-ins with placeholder or invalid coordinates")
checkins = raw[~bad]
counts = checkins.groupby("uid").size()
checkins = (
    checkins[checkins["uid"].isin(counts[counts >= 100].index)]
    .sort_values(["uid", "datetime"])
    .reset_index(drop=True)
)
print(len(checkins), "check-ins by", checkins["uid"].nunique(), "users")

dropping 8327 check-ins with placeholder or invalid coordinates
283543 check-ins by 306 users

From check-ins to flows

trajectory_to_trips turns consecutive check-ins into trips between H3 cells (resolution 7, about 2.5 km), and trajectory_to_od counts them as an origin-destination table. od_matrix aggregates a trips table by origin and destination area; od_metrics_per_area summarises each area: trips inside it (MoveInside), arriving (InComing), leaving (OutGoing) and in total.

trips = fastmob.trajectory_to_trips(checkins, resolution=7)
od = fastmob.trajectory_to_od(checkins, resolution=7)
areas = trips.to_pandas().rename(
    columns={
        "origin_location_id": "origin_area",
        "destination_location_id": "destination_area",
    }
)
matrix = collective.od_matrix(areas)
per_area = collective.od_metrics_per_area(
    od, origin_col="origin", destination_col="destination"
)

print(len(trips), "trips;", len(od), "distinct origin-destination pairs")
print(matrix.head(3).to_string())
per_area.nlargest(3, "Total")

111610 trips; 45606 distinct origin-destination pairs
              origin_area    destination_area  count
22005  608006160485711871  608287684904878079      1
10918  608007117223231487  608663361365737471      1
35923  608007117223231487  608664125920247807      1

	area_code	InComing	OutGoing	Total
3100	608664132161372159	3438	3443	6881
3104	608664132228481023	2003	2004	4007
2617	608663361399291903	1858	1863	3721

The check-ins make 111,610 trips and 45,606 distinct origin-destination pairs at resolution 7. MoveInside is 0 for every area because trips within one cell are dropped by default. The busiest cell has 6,881 trips in total (3,438 arriving, 3,443 leaving), and arrivals and departures are close for the busiest cells, as expected when people return to where they started. od_matrix lists every origin-destination pair of areas with its trip count. The first rows are pairs travelled once, which is typical here: 111,610 trips over 45,606 pairs is about 2.4 trips per pair, so the matrix is sparse.

Flows between zones

For zone-based analysis, a TrajDataFrame converts to a FlowDataFrame given a tessellation (polygons with a tile_id). We use a synthetic 10 by 10 grid over the contiguous United States, and keep only check-ins inside it. get_flow reads one origin-destination flow, get_geometry returns a tile's polygon, to_matrix the full origin-destination matrix.

import geopandas as gpd
from shapely.geometry import box

n, (lat0, lat1, lng0, lng1) = 10, (25, 50, -125, -65)
grid = gpd.GeoDataFrame(
    {"tile_id": [str(k) for k in range(n * n)]},
    geometry=[
        box(
            lng0 + j * (lng1 - lng0) / n,
            lat0 + i * (lat1 - lat0) / n,
            lng0 + (j + 1) * (lng1 - lng0) / n,
            lat0 + (i + 1) * (lat1 - lat0) / n,
        )
        for i in range(n)
        for j in range(n)
    ],
    crs="EPSG:4326",
)
inside = checkins[
    checkins["lat"].between(lat0, lat1) & checkins["lng"].between(lng0, lng1)
]

flows = fastmob.TrajDataFrame(inside).to_flowdataframe(grid, self_loops=False)
table = flows.to_pandas()
top = table.nlargest(1, "flow").iloc[0]
matrix_z = flows.to_matrix()
one_flow = flows.get_flow(top["origin"], top["destination"])
tile = flows.get_geometry(top["origin"])

print(
    len(inside),
    "check-ins inside the grid;",
    len(table),
    "zone pairs with flow; matrix",
    matrix_z.shape,
)
print(
    "largest flow:",
    top["origin"],
    "->",
    top["destination"],
    "=",
    one_flow,
    "trips; origin tile polygon:",
    tile.wkt,
)

239549 check-ins inside the grid; 1117 zone pairs with flow; matrix (100, 100)
largest flow: 40 -> 50 = 1692 trips; origin tile polygon: POLYGON ((-119 35, -119 37.5, -125 37.5, -125 35, -119 35))

The grid has 1,117 zone pairs with flow; the largest, 1,692 trips, joins two neighbouring tiles on the California coast. get_geometry returns the tile's polygon as shapely geometry and to_matrix the full 100 by 100 matrix. The grid is synthetic and its tiles are several hundred kilometres wide.

Compare two flow tables

Split the users into two random halves and compare their flow tables. common_part_of_commuters (CPC) is the share of trips that both tables send along the same links (1 means identical, 0 completely different) and common_part_of_links (CPL) the share of links they share. The same two metrics exist for Trips tables. settings_from copies the tessellation and settings from one flow table to another.

users = inside["uid"].unique()
half = set(np.random.default_rng(0).permutation(users)[: len(users) // 2])

flows_a = fastmob.TrajDataFrame(inside[inside["uid"].isin(half)]).to_flowdataframe(
    grid, self_loops=False
)
flows_b = fastmob.TrajDataFrame(inside[~inside["uid"].isin(half)]).to_flowdataframe(
    grid, self_loops=False
)
cpc, cpl = (
    flows_a.common_part_of_commuters(flows_b),
    flows_a.common_part_of_links(flows_b),
)
flows_a.settings_from(flows_b)
trips_a = fastmob.trajectory_to_trips(inside[inside["uid"].isin(half)], resolution=7)
trips_b = fastmob.trajectory_to_trips(inside[~inside["uid"].isin(half)], resolution=7)
trips_cpc, trips_cpl = (
    trips_a.common_part_of_commuters(trips_b),
    trips_a.common_part_of_links(trips_b),
)
as_flow = trips_a.to_flow_dataframe()

print(f"zones:  CPC {cpc:.2f}, CPL {cpl:.2f}")
print(
    f"H3 trips: CPC {trips_cpc:.2f}, CPL {trips_cpl:.2f}; {len(as_flow)} H3 flow pairs for half A"
)

zones:  CPC 0.65, CPL 0.62
H3 trips: CPC 0.15, CPL 0.11; 20720 H3 flow pairs for half A

The two halves agree well at zone level (CPC 0.65: about two thirds of the trips follow links present in both) and poorly at the 2.5 km cell level (CPC 0.15, CPL 0.11). Agreement between two flow tables depends strongly on the spatial resolution, so always report the resolution next to CPC.

Interpretation and limits

Place ids come from a grid. Flows change with the grid size; state it, and test a second size.
Flows join consecutive check-ins of the same user, regardless of the time between them, so a trip may cover days.
Check-ins are not mobility data: flows describe where people chose to check in. The first 300,000 rows are not a random sample of users.
CPC and CPL compare two random halves of the same sample; a real comparison, such as model against data, uses the evaluation metrics (Fit and compare distributions).

Next steps

Evaluate a flow model against observed flows: Fit and compare distributions, Generate mobility with models.
Study when places are busy: Occupancy and phone records.

od_matrix · od_metrics_per_area · trajectory_to_trips · trajectory_to_od · get_flow · get_geometry · to_matrix · settings_from · common_part_of_commuters · common_part_of_links · to_flow_dataframe · common_part_of_commuters · common_part_of_links