Streaming FlatGeobuf Features into Report Context

FlatGeobuf’s whole design goal is that you never have to read all of it. When a report only needs the parcels inside one district but the source file holds an entire country, streaming lets you pull the relevant features one at a time — bounded by extent, capped in memory, and optionally fetched over HTTP range requests so the file never fully lands on disk. This technique is the incremental counterpart to the full-file reads in Geospatial Data Formats for Reporting Pipelines, and it is the reason GeoPackage vs FlatGeobuf for Reporting Pipelines recommends FlatGeobuf whenever a report reads a small slice of a large or remote extract.

Prerequisites

Python 3.10+ with pyogrio>=0.7, fiona>=1.9, and GDAL 3.6+
A FlatGeobuf (.fgb) input, local or served over HTTP with Accept-Ranges: bytes
The target extent as a bounding box in the file’s native CRS
Familiarity with generators and Python’s itertools; the output feeds the loop covered in Loop Mapping for Dynamic Attribute Tables

Step 1: Open the FlatGeobuf as a Feature Stream

fiona is the right tool for streaming: opening a collection returns an iterator that reads one feature per next() call rather than building a table. Combine it with a bbox so GDAL uses the Hilbert index to skip everything outside the extent.

Python

# stream_open.py
from __future__ import annotations

from collections.abc import Iterator
import fiona


def stream_features(
    path: str, bbox: tuple[float, float, float, float]
) -> Iterator[dict]:
    """Yield raw FlatGeobuf features intersecting bbox, one at a time."""
    with fiona.open(path) as src:
        # filter() pushes the bbox into GDAL's index-seek, not a Python scan
        yield from src.filter(bbox=bbox)

src.filter(bbox=...) is the load-bearing call: it delegates the spatial test to the packed R-tree in the file header, so the iterator only ever surfaces candidate features. Because stream_features is a generator, nothing is read until the caller iterates.

Step 2: Yield Serialised Records from a Bounded Generator

The template must not see raw geometry. Wrap the feature stream in a second generator that projects each feature down to the attributes the report renders and coerces nulls to a sentinel.

Python

# stream_records.py
from __future__ import annotations

from collections.abc import Iterator

SENTINEL = "—"  # em dash for missing values


def to_records(
    features: Iterator[dict], columns: list[str]
) -> Iterator[dict]:
    """Serialise a feature stream to flat records without buffering."""
    for feat in features:
        props = feat["properties"]
        yield {col: props.get(col) if props.get(col) is not None else SENTINEL
               for col in columns}

Chaining to_records(stream_features(...)) gives a lazy pipeline: memory use stays flat no matter how many features match, because each record is produced, consumed, and discarded before the next is read. This is the same has_features / empty-safe contract that Conditional Rendering for Missing Spatial Data relies on downstream — an empty stream simply yields nothing.

Step 3: Read a Remote File with HTTP Range Requests

The streaming model extends to remote files unchanged. Prefix the URL with GDAL’s /vsicurl/ handler and fiona issues HTTP range requests, fetching the header, index, and only the byte ranges covering the bbox.

Python

# stream_remote.py
from __future__ import annotations

from collections.abc import Iterator
import fiona


def stream_remote(
    url: str, bbox: tuple[float, float, float, float]
) -> Iterator[dict]:
    """Stream a bbox from a remote FlatGeobuf via HTTP range reads."""
    vsi = f"/vsicurl/{url}"
    with fiona.Env(CPL_VSIL_CURL_ALLOWED_EXTENSIONS=".fgb"):
        with fiona.open(vsi) as src:
            yield from src.filter(bbox=bbox)

Setting CPL_VSIL_CURL_ALLOWED_EXTENSIONS restricts the curl handler to .fgb requests, which avoids stray probes for sidecar files that do not exist. A multi-gigabyte national extract is now readable from a report worker that only ever downloads the district it draws.

Step 4: Build a Paginated Context Without Materialising the Dataset

Large attribute tables must not arrive as one giant list. Batch the record stream into fixed-size pages with itertools.islice, so the template renders one page at a time — the ingestion-side hook for Table Pagination Strategies for Large Attribute Tables.

Python

# paginate.py
from __future__ import annotations

from collections.abc import Iterator
from itertools import islice


def paginate(records: Iterator[dict], page_size: int = 100) -> Iterator[list[dict]]:
    """Group a record stream into fixed-size pages; never holds >1 page."""
    while page := list(islice(records, page_size)):
        yield page


def build_context(records: Iterator[dict], page_size: int = 100) -> dict:
    """Assemble a report context that carries a lazy page iterator."""
    pages = paginate(records, page_size)
    first = next(pages, [])
    return {
        "has_features": bool(first),
        "first_page": first,
        "remaining_pages": pages,   # still lazy — consumed by the template loop
        "page_size": page_size,
    }

The walrus while page := list(islice(...)) is the idiom: islice pulls exactly page_size records from the upstream generator, and the loop stops the moment a short (or empty) page signals exhaustion. build_context peeks the first page to set has_features without draining the rest, so a template can branch on emptiness yet still stream the remainder.

Key Parameters / Configuration Reference

Parameter	Type	Default	Effect
`bbox`	`tuple[float, float, float, float]`	—	Extent filter in the file’s native CRS; drives the index seek
`page_size`	`int`	`100`	Records per page; trades memory for iteration overhead
`columns`	`list[str]`	—	Attributes projected into each record; keep it minimal
`CPL_VSIL_CURL_ALLOWED_EXTENSIONS`	`str`	unset	Restricts range-read probing to `.fgb` for remote reads
`SENTINEL`	`str`	`"—"`	Replacement for null attribute values

Common Pitfalls

Calling list() on the stream too early. Wrapping the generator in list() — even accidentally, via len() — materialises the whole dataset and defeats streaming. Only paginate may buffer, and only one page at a time.
Reusing an exhausted generator. A generator yields once. If the template needs has_features and the rows, peek the first page (as build_context does) rather than iterating twice.
Forgetting the CRS of the bbox. filter(bbox=...) expects coordinates in the file’s own CRS. A WGS84 box against a projected FlatGeobuf silently matches nothing.
Range reads against a server without Accept-Ranges. Without byte-range support the /vsicurl/ handler falls back to a full download, quietly erasing the streaming benefit — verify the header in CI.

Verification

Assert that memory stays bounded and the stream is genuinely lazy:

Python

# verify_stream.py
from stream_open import stream_features
from stream_records import to_records
from paginate import build_context

records = to_records(stream_features("districts.fgb", (530000, 180000, 545000, 195000)),
                     columns=["site_id", "status"])
ctx = build_context(records, page_size=50)

assert ctx["has_features"] is True
assert len(ctx["first_page"]) <= 50, "first page must respect page_size"
# remaining_pages must still be a live iterator, not a materialised list
assert hasattr(ctx["remaining_pages"], "__next__")
print("Streaming context verified")

For a CI smoke test on the remote path, count fetched bytes via GDAL’s curl statistics and assert they are far below the full file size — proof the index seek and range reads did their job.

GeoPackage vs FlatGeobuf for Reporting Pipelines — why FlatGeobuf is the format that makes this streaming read possible
Choosing GeoJSON vs GeoPackage for Report Inputs — the sibling decision for text vs indexed binary inputs
Table Pagination Strategies for Large Attribute Tables — consume the paginated context safely across PDF page breaks
Parent: Geospatial Data Formats for Reporting Pipelines