About this project

A directed, weighted flow network built from every NYC yellow-taxi trip in 2015: 142,199,201 cleaned trips (97.4% of the raw rows) collapsed onto a 262-node, 42,347-edge graph whose nodes are TLC taxi zones and whose edge weights are annual trip counts. It started as a 2016-17 student project and was rebuilt in 2021 as a single reproducible pipeline.

Reproduce it

The whole pipeline is one command. It downloads roughly 2 GB of taxi parquet, cleans and aggregates it out-of-core, builds the graph, runs the analysis, and regenerates every figure. The environment is fully pinned, so a clean checkout should produce the same numbers.

pixi install
pixi run pipeline

Each stage is also runnable on its own, and each one is resumable and skips work it has already finished. The raw parquet lands in data/raw/ and is gitignored; the small aggregates in data/processed/ are committed, so the figures and analysis can be rebuilt without re-downloading.

pixi run download   # fetch the 12 monthly parquet files
pixi run geo        # build census tracts + taxi zones in EPSG:2263
pixi run ingest     # clean + aggregate 146M trips to a zone edge list (DuckDB)
pixi run graph      # build the weighted DiGraph; write node/edge tables
pixi run metrics    # degree, hub asymmetry, baseline community
pixi run temporal   # monthly volume, hour x weekday, cost vs duration
pixi run analysis   # gravity, backbone, centralities, communities
pixi run figures    # regenerate every figure (PNG + SVG)
pixi run summary    # print the headline measured numbers

Tooling

Nothing exotic, and everything pinned. Pixi manages the environment and the task graph above. DuckDB and Polars do the out-of-core cleaning and aggregation of 146 million rows. GeoPandas handles the spatial layer: dissolving the in-repo 2010 block shapefile into census tracts, building the taxi-zone geometries in EPSG:2263, and the zone-to-tract crosswalk. NetworkX and igraph carry the network science: the directed graph, centralities, the statistical backbone, and community detection with Leiden plus a seed-stability study. Math on the figures and Method page is rendered with KaTeX.

A note on the data

One honest caveat shaped the whole rebuild. The NYC TLC re-coded its historical trip data. The 2015 yellow parquet on the official mirror no longer carries pickup and dropoff latitude and longitude; it carries taxi zone ids (PULocationID / DOLocationID), and the original lat/long CSVs are no longer distributed. So the original project's central engineering problem, binning splintered points onto census tracts, is now solved upstream. We use the 263 TLC taxi zones as network nodes rather than census tracts. A taxi zone overlaps about 19 census tracts on average, so this is a real, deliberate drop in spatial resolution, recorded and quantified rather than hidden.

The deeper caveat is the data source itself. Yellow taxis over-sample Manhattan and the airports and under-sample the outer boroughs, so the network describes how the yellow-cab city moves, not how everyone moves. Every finding here should be read with that coverage bias in mind. The Method page states the cleaning filters and the model assumptions in full.

Read more

The full writeup, with the gravity model, the backbone, the community analysis, and the temporal decomposition, is in the paper. Every measured number, the decisions behind them, and how they reconcile with the original 2016-17 blog figures are in the modernization notes, which also record the eight-agent adversarial audit that re-derived the headline numbers from the raw parquet. All of it, including the pipeline and the preserved original notebooks under legacy/, is in the source repository.

Credit

This began as a 2016-17 student project on NYC taxi transportation networks, a pile of numbered Jupyter notebooks and a blog post. The 2021 modernization turned it into one reproducible pipeline with modern tooling, regenerated every original figure with real numbers attached, and added a research-grade network-science layer on top. The original notebooks, plots, and report are kept under legacy/ in the repository, both as provenance and so the before-and-after is honest.