Spatial Reporting Automated Docs & Maps

Parallel Batch Rendering with ProcessPoolExecutor

A nightly job that renders 900 spatial PDFs one after another wastes every core but one. PDF rendering is CPU-bound — Pango shapes text, Cairo rasterizes, Matplotlib draws figures — so the right tool is concurrent.futures.ProcessPoolExecutor, which sidesteps the GIL by running each render in a separate OS process. This guide, part of the Scheduled Batch Report Generation workflow, goes deep on the worker-pool mechanics that matter in production: initializing the heavy Jinja2 Environment once per worker, chunking to amortize inter-process overhead, capping memory with max_workers and worker recycling, and collecting per-report failures so one broken dataset never sinks the whole batch.

ProcessPoolExecutor render fan-out Diagram of a queue of report tasks distributed by a ProcessPoolExecutor to three worker processes. Each worker initializes a Jinja2 Environment once, renders its assigned tasks to PDF, and returns a result or an exception to the collector. task queue 900 report jobs worker 1 init Environment once → render worker 2 init Environment once → render worker 3 init Environment once → render collector as_completed → ok / error

Prerequisites

  • Python 3.10+ (concurrent.futures is in the standard library)
  • A single-report render function you can call in isolation — for example one wrapping weasyprint
  • Enough RAM headroom to hold max_workers concurrent renders; know your per-render peak RSS
  • Familiarity with passing data to templates from Loop Mapping for Dynamic Attribute Tables, since each worker renders the same template family

Step 1: Choose Processes Over Threads

ThreadPoolExecutor shares one interpreter and one GIL, so CPU-bound renders run effectively serially even with many threads. ProcessPoolExecutor forks separate interpreters, each with its own GIL, giving true parallelism on multiple cores. The trade-off is that arguments and return values are pickled across the process boundary — so pass small, picklable job descriptors, not open file handles or GeoDataFrames.

Python
# render_job.py
from __future__ import annotations
from dataclasses import dataclass


@dataclass(frozen=True, slots=True)
class RenderJob:
    """A small, picklable descriptor — NOT the data itself."""
    report_id: str
    dataset_path: str      # path, not a loaded GeoDataFrame
    out_path: str

Keeping RenderJob to plain strings means pickling is cheap and no unpicklable spatial object ever crosses the process boundary.

Step 2: Initialize Heavy State Per Worker

Building a Jinja2 Environment — loading templates, compiling macros, registering filters — is expensive. Doing it inside the per-task function repeats that cost 900 times. The initializer hook runs once per worker process at startup; stash the Environment in a module global the task function reads.

Python
# worker.py
from __future__ import annotations
from jinja2 import Environment, FileSystemLoader, select_autoescape

_ENV: Environment | None = None


def init_worker(template_dir: str) -> None:
    """Runs once per worker process, not once per task."""
    global _ENV
    _ENV = Environment(
        loader=FileSystemLoader(template_dir),
        autoescape=select_autoescape(["html", "xml"]),
    )


def render_one(job: "RenderJob") -> tuple[str, str]:
    import geopandas as gpd
    from weasyprint import HTML
    assert _ENV is not None, "worker not initialized"

    gdf = gpd.read_file(job.dataset_path)
    context = {"rows": gdf.drop(columns="geometry").to_dict("records")}
    html = _ENV.get_template("report.html").render(**context)
    HTML(string=html).write_pdf(job.out_path)
    return (job.report_id, job.out_path)

Amortizing the Environment across a worker’s whole task run is the single biggest throughput win — often larger than adding another core.

Step 3: Submit Tasks and Chunk the Work

executor.map with a chunksize batches several jobs into one IPC round-trip, which matters when individual tasks are short relative to pickling cost. For longer renders, submit + as_completed gives finer-grained result handling.

Python
# batch.py
from __future__ import annotations
from concurrent.futures import ProcessPoolExecutor, as_completed
from worker import init_worker, render_one
from render_job import RenderJob


def run_batch(jobs: list[RenderJob], template_dir: str, max_workers: int) -> None:
    with ProcessPoolExecutor(
        max_workers=max_workers,
        initializer=init_worker,
        initargs=(template_dir,),
    ) as pool:
        futures = {pool.submit(render_one, j): j for j in jobs}
        for fut in as_completed(futures):
            job = futures[fut]
            rid, out = fut.result()   # re-raises worker exceptions here
            print(f"rendered {rid} -> {out}")

A good starting chunksize for executor.map is len(jobs) // (max_workers * 4), which keeps every worker fed while still rebalancing near the end of the batch.

Step 4: Bound Memory With max_workers and maxtasksperchild

Each worker renders independently, so peak memory scales with max_workers. If one render peaks at 400 MB and you set eight workers, you need ~3.2 GB of headroom or the OOM killer strikes. Cap workers to min(os.cpu_count(), ram_budget // peak_rss). Long-running workers can also leak native memory (Cairo/Pango), so recycle them with max_tasks_per_child (Python 3.11+).

Python
import os

def safe_worker_count(peak_rss_mb: int, ram_budget_mb: int) -> int:
    by_cpu = os.cpu_count() or 2
    by_ram = max(1, ram_budget_mb // peak_rss_mb)
    return min(by_cpu, by_ram)

pool = ProcessPoolExecutor(
    max_workers=safe_worker_count(peak_rss_mb=400, ram_budget_mb=6000),
    initializer=init_worker,
    initargs=("templates",),
    max_tasks_per_child=50,   # recycle a worker every 50 renders to cap native leaks
)

On a constrained CI runner, this memory ceiling is why a runner-per-region matrix (see Scheduling Nightly PDF Runs with Cron and GitHub Actions) can be safer than one giant in-process pool.

Step 5: Collect Failures Without Losing the Batch

The whole point of batch rendering is resilience: one malformed dataset must not abort the other 899 reports. Catch each future’s exception individually and record it, then decide the exit code from the tally.

Python
# batch.py (resilient variant)
from __future__ import annotations
import logging
from concurrent.futures import ProcessPoolExecutor, as_completed

log = logging.getLogger("batch")


def run_batch_resilient(jobs, template_dir: str, max_workers: int) -> int:
    failures: list[tuple[str, str]] = []
    with ProcessPoolExecutor(max_workers=max_workers,
                             initializer=init_worker,
                             initargs=(template_dir,)) as pool:
        futures = {pool.submit(render_one, j): j for j in jobs}
        for fut in as_completed(futures):
            job = futures[fut]
            try:
                fut.result()
            except Exception as exc:                      # noqa: BLE001
                failures.append((job.report_id, repr(exc)))
                log.error("render failed for %s: %r", job.report_id, exc)

    for rid, err in failures:
        log.warning("FAILED %s -> %s", rid, err)
    return 1 if failures else 0

Returning a non-zero exit code only when there are failures lets the scheduler flag a degraded run while still keeping every successful PDF. Those survivors are then handed to durable storage, as covered in Versioning Generated PDFs in S3 Object Storage.

Key Parameters / Configuration Reference

Parameter Type Default Effect
max_workers int os.cpu_count() Concurrent processes; bound by RAM, not just cores
initializer callable None Runs once per worker to build the Environment
initargs tuple () Args passed to the initializer (template dir)
max_tasks_per_child int | None None Recycle a worker after N tasks to cap native leaks
chunksize (map) int 1 Jobs per IPC batch; raise for short tasks

Common Pitfalls

  • Rebuilding the Environment per task. Constructing the Jinja2 Environment inside render_one throws away the initializer’s benefit and dominates runtime. Build it once in init_worker.
  • Passing unpicklable arguments. A GeoDataFrame, open file, or DB connection cannot cross the process boundary and raises PicklingError. Pass paths and small descriptors; load data inside the worker.
  • Setting max_workers from cores alone. Eight renders at 400 MB each overrun a 4 GB runner. Compute the ceiling from ram_budget // peak_rss and take the min with core count.
  • Letting one exception kill the pool. Calling fut.result() without a try/except re-raises and, combined with the with block exit, can drop in-flight work. Catch per-future and tally failures.

Verification

Confirm the pool actually parallelizes and that a deliberately broken job is isolated rather than fatal:

Python
import time
from batch import run_batch_resilient

start = time.perf_counter()
exit_code = run_batch_resilient(jobs, "templates", max_workers=4)
elapsed = time.perf_counter() - start

# With 4 workers, wall time should be well under the serial sum.
print(f"exit={exit_code} wall={elapsed:.1f}s for {len(jobs)} reports")
assert exit_code in (0, 1)   # 1 only when some report failed, others still rendered

Compare the parallel wall time against a serial baseline: a healthy four-worker run on CPU-bound renders lands near one-third to one-quarter of the serial total, and a single injected failure returns exit code 1 while every other PDF is still written.