Occupancy and phone records
Question. When is each place busy, which places share visitors, and how do I turn mobile-phone records into stays and trips?
You will get the average number of people at each place for each 10-minute slot of the day, the pairs of places visited by the same people, and the stays and trips estimated from a small set of call detail records.
Prerequisites. Population measures, which uses the same data and cleaning.
Data. Brightkite check-ins (Stanford SNAP, 2008 to 2010): the first 300,000 rows (about 4 MB of the 60 MB file) with fastmob.data.load_brightkite; users with fewer than 100 check-ins are dropped, which leaves about 306 users. Check-ins are voluntary and sparse. The call detail records are synthetic: six events of one invented person. The notebook stops if the download fails.
Load and clean
The raw file contains placeholder and impossible coordinates. We drop check-ins at exactly latitude 0 and longitude 0 (a placeholder for "unknown") and the few rows with a latitude whose absolute value is 90 degrees or more, which is not a valid position. Always check coordinate ranges before a population analysis.
import fastmob
import pandas as pd
from fastmob.data import load_brightkite
from fastmob.measures import collective
raw = load_brightkite(max_rows=300_000).to_pandas()
bad = ((raw["lat"] == 0) & (raw["lng"] == 0)) | (raw["lat"].abs() >= 90)
print(f"dropping {bad.sum()} check-ins with placeholder or invalid coordinates")
checkins = raw[~bad]
counts = checkins.groupby("uid").size()
checkins = (
checkins[checkins["uid"].isin(counts[counts >= 100].index)]
.sort_values(["uid", "datetime"])
.reset_index(drop=True)
)
print(len(checkins), "check-ins by", checkins["uid"].nunique(), "users")
When and where is each place busy?
To study occupancy we need stays. Check-ins have no duration, so we assume each check-in is a 30-minute stay (an explicit modelling choice, not data). generate_global_locations assigns the stays to a shared H3 grid (resolution 8). build_stvd then aggregates them into the average number of people at each place for each 10-minute slot of the day, averaged over the days of the week, interest_network (also available as collective.interest_network) lists pairs of places visited by the same people, and mean_area_volume does the same occupancy average for any visit table.
stays = fastmob.Staypoints(
pd.DataFrame(
{
"uid": checkins["uid"],
"lat": checkins["lat"],
"lng": checkins["lng"],
"started_at": checkins["datetime"],
"finished_at": checkins["datetime"] + pd.Timedelta(minutes=30),
}
)
)
places, stays_at = stays.generate_global_locations(h3_resolution=8)
stvd = stays_at.build_stvd(places)
shared = stays_at.interest_network(places)
busy = collective.mean_area_volume(
stays_at.to_pandas().rename(columns={"uid": "user_id"}),
area_col="location_id",
user_id_col="user_id",
start_col="started_at",
end_col="finished_at",
)
stvd_f = collective.build_stvd(stays_at, places)
shared_f = collective.interest_network(stays_at, places)
same = (
stvd.sort_values(list(stvd.columns))
.reset_index(drop=True)
.equals(stvd_f.sort_values(list(stvd_f.columns)).reset_index(drop=True))
)
print(
len(places),
"places;",
len(stvd),
"place-time cells;",
len(shared),
"linked place pairs; method and function agree:",
same,
len(shared_f) == len(shared),
)
print(busy.head(3).to_string())
shared.nlargest(3, "n_people")
18659 places; 291182 place-time cells; 2461194 linked place pairs; method and function agree: True True
area time_bin mean_volume
0 612509760157122559 08:20 0.142857
1 612509760157122559 08:30 0.142857
2 612509760157122559 08:40 0.142857
| location_id_a | location_id_b | n_people | |
|---|---|---|---|
| 182 | 613196570331971583 | 613196570376011775 | 71 |
| 9079 | 613196574729699327 | 613196570376011775 | 65 |
| 19781 | 613167731774062591 | 613167731786645503 | 63 |
Assigning the stays to a shared grid gives 18,659 places and 291,182 place-time cells, and 2.46 million pairs of places visited by the same people; the most strongly linked pair is shared by 71 people. Only the head of each table is shown. build_stvd called as a method and as a function give the same table. mean_area_volume gives, for each area and time-of-day slot, the mean number of people present. The first rows show 0.14, which is 1/7: one person present in that slot on one of the seven weekdays.
Mobile-phone records
Call detail records (CDR) give the cell tower a person was connected to at the time of each event, not their position. cdr_to_visitation_df collapses consecutive records at the same tower into a stay and estimates when each stay ended from the travel time to the next tower (avg_speed_kmh and circuity set the assumed speed and route winding); cdr_to_trips_df derives the trips between consecutive stays. The records below are synthetic: six events of one invented person at two invented towers.
from fastmob.preprocessing import cdr_to_trips_df, cdr_to_visitation_df
cdr = pd.DataFrame(
{
"user_id": ["u1"] * 6,
"timestamp": pd.to_datetime(
[
"2020-01-01 08:00",
"2020-01-01 08:15",
"2020-01-01 09:00",
"2020-01-01 12:00",
"2020-01-01 17:00",
"2020-01-01 22:00",
]
),
"venueId": [
"tower_home",
"tower_home",
"tower_work",
"tower_work",
"tower_work",
"tower_home",
],
"lat": [41.890, 41.890, 41.911, 41.911, 41.911, 41.890],
"long": [12.492, 12.492, 12.482, 12.482, 12.482, 12.492],
}
)
cdr_visits = cdr_to_visitation_df(cdr, avg_speed_kmh=30, circuity=1.2)
cdr_trips = cdr_to_trips_df(cdr_visits, avg_speed_kmh=30, circuity=1.2)
print(
cdr_visits[
["user_id", "area", "start_timestamp", "end_timestamp", "duration_minutes"]
]
.assign(duration_minutes=lambda d: d["duration_minutes"].round(1))
.to_string(index=False)
)
print(
cdr_trips[
[
"user_id",
"trip_number",
"origin_area",
"destination_area",
"duration_minutes",
]
]
.round(1)
.to_string(index=False)
)
user_id area start_timestamp end_timestamp duration_minutes
u1 tower_home 2020-01-01 08:00:00 2020-01-01 08:54:03.250349 54.1
u1 tower_work 2020-01-01 09:00:00 2020-01-01 21:54:03.250349 774.1
u1 tower_home 2020-01-01 22:00:00 NaT NaN
user_id trip_number origin_area destination_area duration_minutes
u1 1 tower_home tower_work 5.9
u1 2 tower_work tower_home 5.9
The estimated end of the home stay is 08:54, about six minutes before the next record at 09:00 (the estimated 5.9-minute trip at 30 km/h with a circuity of 1.2); the work stay is estimated to end at 21:54 for the same reason; the last stay has no end (NaT) because no record follows it. The two trips between the towers last 5.9 minutes each. These end times are model estimates built from the assumed speed, not observations, and a sparse CDR can hide whole trips.
Interpretation and limits
- The occupancy tables assume each check-in lasts 30 minutes. Their absolute values follow that assumption.
- Place ids come from a grid; occupancy and co-visitation change with the grid size. State it, and test a second size.
- Check-ins are not mobility data: the first 300,000 rows are not a random sample of users.
- The end of a stay from phone records is an estimate built from an assumed speed, and sparse records can hide whole trips.
Next steps
- Build contact networks from co-presence: Contact networks from stays.
- Compare occupancy patterns of two data sets: Fit and compare distributions.
Related API
build_stvd ·
mean_area_volume ·
build_stvd ·
interest_network ·
cdr_to_visitation_df ·
cdr_to_trips_df