Spatial Reporting Automated Docs & Maps

Report Artifact Storage & Versioning

Once a pipeline renders spatial-report PDFs unattended, the hard questions move downstream: which version of last quarter’s flood-risk report did the client actually receive, can you prove the file has not been altered, how long must you keep it, and how do you hand someone a link without exposing an entire bucket of sensitive location data. Naming files report_final_v2.pdf answers none of these. This guide builds a storage layer where every artifact is content-addressed, immutable, catalogued with queryable metadata, distributed through expiring signed URLs, retained on a lifecycle policy, and promoted between environments by reference rather than re-render.

Prerequisites

  • Python 3.10+ with boto3>=1.34 for S3/MinIO access (the S3 API covers both)
  • An object store: AWS S3, or MinIO for self-hosted deployments — both speak the same API and support versioning and object lock
  • Deterministic renders: PDFs produced with a fixed SOURCE_DATE_EPOCH, as set up in Headless PDF Rendering in Docker Containers — content-addressed storage is only meaningful when identical inputs yield identical bytes
  • A batch producing artifacts: the run manifests from Scheduled Batch Report Generation become the source records this catalog ingests
  • A catalog store: a small relational table (SQLite, Postgres) or a DynamoDB table to index artifacts by report, version, and hash
Bash
pip install "boto3>=1.34"
# MinIO for local development, S3-compatible:
docker run -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001"

Pipeline Architecture

A rendered PDF is hashed, uploaded under its content-addressed key to an immutable bucket, and indexed in a metadata catalog. Consumers resolve the current version through the catalog and receive a short-lived signed URL; a lifecycle policy ages old versions to cheaper storage or expiry, and promotion copies an approved hash from staging to production.

Report Artifact Storage and Versioning Pipeline A rendered PDF is hashed, uploaded to an immutable versioned bucket, catalogued, distributed via signed URLs, aged by a lifecycle policy, and promoted by reference to production. Rendered PDF SHA-256 content key Object Store versioned object lock key = hash.pdf Metadata Catalog report · version · hash Lifecycle policy tier · expire old Distribution presigned URL expiring · private Consumer email · dash
The hash is the spine: it names the object, indexes the catalog, verifies downloads, and lets promotion copy exact bytes between environments rather than re-rendering.

Step-by-Step Implementation

Step 1: Compute a Content Hash for Each Rendered PDF

Hash the PDF bytes with SHA-256 and use a prefix of the digest as the object key. This makes the artifact self-verifying — any consumer can re-hash a download and confirm integrity — and naturally deduplicating, because identical inputs render to identical bytes and therefore the identical key.

Python
from __future__ import annotations

import hashlib
from pathlib import Path


def content_key(pdf_path: Path, report_id: str) -> str:
    """Return an object key namespaced by report and addressed by content hash."""
    digest = hashlib.sha256(pdf_path.read_bytes()).hexdigest()
    return f"reports/{report_id}/{digest}.pdf"

Content addressing depends entirely on deterministic rendering: if WeasyPrint stamps a live timestamp into the PDF, two identical reports get different hashes and the deduplication breaks. The SOURCE_DATE_EPOCH setup in Headless PDF Rendering in Docker Containers is the prerequisite that makes this step trustworthy.

Step 2: Upload to Immutable, Content-Addressed Object Storage

Enable bucket versioning and Object Lock, then upload under the content key. Because the key is derived from the bytes, an upload of already-present content is a harmless no-op; you can short-circuit it with a head_object check to save bandwidth.

Python
import boto3
from botocore.exceptions import ClientError

s3 = boto3.client("s3")
BUCKET = "spatial-reports-prod"


def upload_artifact(pdf_path: Path, key: str) -> bool:
    """Upload if absent. Returns True if newly stored, False if already present."""
    try:
        s3.head_object(Bucket=BUCKET, Key=key)
        return False  # content-addressed: identical key means identical bytes
    except ClientError as exc:
        if exc.response["Error"]["Code"] != "404":
            raise
    s3.put_object(
        Bucket=BUCKET,
        Key=key,
        Body=pdf_path.read_bytes(),
        ContentType="application/pdf",
        ObjectLockMode="GOVERNANCE",
        ObjectLockRetainUntilDate=_retain_until(days=1825),  # 5-year retention
    )
    return True

GOVERNANCE mode lets a privileged role lift retention if legally required, while blocking ordinary overwrites and deletes; use COMPLIANCE mode when even administrators must not be able to alter the object before expiry. The versioning + lock combination is what lets you promise that a report referenced by hash resolves to the same bytes years later.

Step 3: Record Artifact Metadata in a Queryable Catalog

The object store answers “give me these bytes”; the catalog answers “what is the current version of the Q2 flood report, who produced it, and from which inputs.” Record one row per stored artifact, linking report identity to the content hash and the run that produced it.

Python
import sqlite3
from datetime import datetime, timezone


def catalog_artifact(
    conn: sqlite3.Connection,
    report_id: str,
    key: str,
    run_id: str,
    template_version: str,
) -> None:
    digest = key.rsplit("/", 1)[-1].removesuffix(".pdf")
    conn.execute(
        """
        INSERT INTO artifacts (report_id, digest, key, run_id, template_version, stored_at, current)
        VALUES (?, ?, ?, ?, ?, ?, 1)
        ON CONFLICT(digest) DO NOTHING
        """,
        (report_id, digest, key, run_id, template_version,
         datetime.now(timezone.utc).isoformat()),
    )
    # Demote prior current versions of the same report.
    conn.execute(
        "UPDATE artifacts SET current = 0 WHERE report_id = ? AND digest != ?",
        (report_id, digest),
    )
    conn.commit()

The catalog ingests directly from the run manifests emitted by Scheduled Batch Report Generation, so every stored artifact traces back to the exact batch, template version, and input date that produced it.

Step 4: Distribute with Time-Limited Signed URLs

Never make a report bucket public — spatial reports routinely embed sensitive parcel, infrastructure, or personal location data. Keep the bucket private and mint a presigned URL per request, scoped to one object and expiring in minutes to hours.

Python
def presign(key: str, expires_seconds: int = 900) -> str:
    """One-object, time-limited download link — no bucket-wide exposure."""
    return s3.generate_presigned_url(
        "get_object",
        Params={"Bucket": BUCKET, "Key": key},
        ExpiresIn=expires_seconds,
    )

A dashboard or notification service resolves the current key from the catalog, then hands the consumer a fresh presigned URL. Because the link expires, a forwarded email or cached URL cannot be replayed indefinitely.

Step 5: Apply Lifecycle Retention and Promote Between Environments

A lifecycle policy ages superseded versions to cheaper storage and expires them past the retention horizon, controlling cost without manual cleanup. Object Lock still guarantees objects survive until their retain-until date regardless of the lifecycle rule.

JSON
{
  "Rules": [
    {
      "ID": "tier-and-expire-old-report-versions",
      "Filter": { "Prefix": "reports/" },
      "Status": "Enabled",
      "Transitions": [
        { "Days": 90, "StorageClass": "STANDARD_IA" },
        { "Days": 365, "StorageClass": "GLACIER" }
      ],
      "NoncurrentVersionExpiration": { "NoncurrentDays": 1825 }
    }
  ]
}

Promotion moves an approved artifact from staging to production by reference, not by re-rendering — copy the immutable object under its same hash key, then flip the production catalog’s current pointer. This guarantees production ships the exact bytes staging reviewed. The packaging and manifest conventions for bundling related outputs together are covered in Publishing Report Bundles to an Artifact Registry.

Production-Ready Script

This uploader ties the flow together: hash each PDF from a completed batch, upload immutably, catalog it, and print a presigned URL, all behind an argparse interface with logging and error handling. The bucket-specific versioning details for S3 in particular are expanded in Versioning Generated PDFs in S3 Object Storage.

Python
#!/usr/bin/env python3
"""
store_artifacts.py — content-address, upload, catalog and sign report PDFs.

Usage:
  python store_artifacts.py --dir output/ --report-id q2-flood --run-id 20260711-ab12cd34
"""
from __future__ import annotations

import argparse
import hashlib
import logging
import sqlite3
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path

import boto3
from botocore.exceptions import BotoCoreError, ClientError

logging.basicConfig(
    level="INFO", format="%(asctime)s %(levelname)s %(message)s", datefmt="%H:%M:%S"
)
logger = logging.getLogger("store")

BUCKET = "spatial-reports-prod"
CATALOG_DB = Path("catalog.sqlite")
s3 = boto3.client("s3")


def _retain_until(days: int) -> datetime:
    return datetime.now(timezone.utc) + timedelta(days=days)


def content_key(pdf: Path, report_id: str) -> str:
    digest = hashlib.sha256(pdf.read_bytes()).hexdigest()
    return f"reports/{report_id}/{digest}.pdf"


def ensure_catalog(conn: sqlite3.Connection) -> None:
    conn.execute(
        """
        CREATE TABLE IF NOT EXISTS artifacts (
            digest TEXT PRIMARY KEY,
            report_id TEXT NOT NULL,
            key TEXT NOT NULL,
            run_id TEXT,
            template_version TEXT,
            stored_at TEXT NOT NULL,
            current INTEGER NOT NULL DEFAULT 1
        )
        """
    )
    conn.commit()


def store_one(pdf: Path, report_id: str, run_id: str, conn: sqlite3.Connection) -> str:
    key = content_key(pdf, report_id)
    try:
        s3.head_object(Bucket=BUCKET, Key=key)
        logger.info("Already stored: %s", key)
    except ClientError as exc:
        if exc.response["Error"]["Code"] != "404":
            raise
        s3.put_object(
            Bucket=BUCKET, Key=key, Body=pdf.read_bytes(),
            ContentType="application/pdf",
            ObjectLockMode="GOVERNANCE",
            ObjectLockRetainUntilDate=_retain_until(1825),
        )
        logger.info("Stored: %s", key)

    digest = key.rsplit("/", 1)[-1].removesuffix(".pdf")
    conn.execute(
        "INSERT OR IGNORE INTO artifacts "
        "(digest, report_id, key, run_id, template_version, stored_at, current) "
        "VALUES (?,?,?,?,?,?,1)",
        (digest, report_id, key, run_id, "2026.07",
         datetime.now(timezone.utc).isoformat()),
    )
    conn.execute(
        "UPDATE artifacts SET current=0 WHERE report_id=? AND digest!=?",
        (report_id, digest),
    )
    conn.commit()
    return key


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description="Store and version report artifacts")
    parser.add_argument("--dir", required=True, type=Path)
    parser.add_argument("--report-id", required=True)
    parser.add_argument("--run-id", required=True)
    parser.add_argument("--sign-seconds", type=int, default=900)
    args = parser.parse_args(argv)

    conn = sqlite3.connect(CATALOG_DB)
    ensure_catalog(conn)

    pdfs = sorted(args.dir.glob("*.pdf"))
    if not pdfs:
        logger.error("No PDFs found in %s", args.dir)
        return 2

    for pdf in pdfs:
        try:
            key = store_one(pdf, args.report_id, args.run_id, conn)
            url = s3.generate_presigned_url(
                "get_object",
                Params={"Bucket": BUCKET, "Key": key},
                ExpiresIn=args.sign_seconds,
            )
            logger.info("Signed URL (%ds): %s", args.sign_seconds, url)
        except (BotoCoreError, ClientError) as exc:
            logger.error("Failed to store %s: %s", pdf.name, exc)
            return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

Edge Cases & Advanced Configuration

Deduplicating Identical Reports Across Runs

When two scheduled runs produce a byte-identical report — common when the underlying data did not change overnight — content addressing collapses them to one stored object automatically. The catalog still records both runs pointing at the same digest, so you preserve the audit trail (“this report was current for three consecutive nights”) without paying to store three copies.

Large Bundles and Multi-File Reports

A single logical report is often a bundle: the PDF, its source GeoJSON, a thumbnail, and a manifest. Hash the bundle’s manifest (which itself lists each member’s hash) to get a single addressable bundle version, then store members individually so unchanged members deduplicate. The registry-oriented packaging model for this is detailed in Publishing Report Bundles to an Artifact Registry.

Retention Conflicts with GDPR Erasure

Compliance-mode Object Lock and a right-to-erasure request can collide: locked objects cannot be deleted before expiry. Resolve this at design time by keeping personally identifying data out of the immutably stored PDF where possible, or by using GOVERNANCE mode plus a documented privileged-deletion process rather than COMPLIANCE mode for reports that may contain personal data.

Cross-Region and Multi-Environment Catalogs

When staging and production live in different accounts or regions, keep a catalog per environment and promote by copying the object and inserting a catalog row — never by pointing production at a staging URL. Reports whose large attribute tables were paginated per Table Pagination Strategies for Large Attribute Tables can be sizable, so prefer a same-region server-side copy_object over download-and-reupload to avoid egress cost.

Troubleshooting

Symptom Likely Cause Resolution
Every nightly run creates a new version of an unchanged report Non-deterministic render (live PDF timestamp) Set SOURCE_DATE_EPOCH; verify identical inputs hash to the same key
AccessDenied when uploading with Object Lock fields Bucket versioning or Object Lock not enabled Enable versioning and Object Lock at bucket creation; they cannot be added retroactively
Consumers report a broken/expired link Presigned URL past its ExpiresIn window Resolve the current key from the catalog and mint a fresh URL per request
Cannot delete an object during cleanup Object Lock retention still active Wait for retain-until, or use a GOVERNANCE-mode privileged bypass; never public-ACL as a workaround
Production shows a different report than staging approved Re-rendered per environment instead of copying bytes Promote by copying the immutable hash-keyed object; flip the catalog current flag
Storage cost grows unbounded No lifecycle policy on noncurrent versions Add tiering transitions and NoncurrentVersionExpiration to the lifecycle config

Frequently Asked Questions

Why name report PDFs by content hash instead of a timestamp or sequence number?

A content hash makes the artifact self-verifying and deduplicating: identical inputs produce the identical key, so re-runs never create spurious versions and any consumer can confirm a download is intact by re-hashing it. Timestamps and sequence numbers collide, hide duplicates, and cannot detect corruption. For content hashing to work the render itself must be deterministic — set SOURCE_DATE_EPOCH so the PDF creation date does not perturb the bytes.

How do I make stored report artifacts immutable?

Enable object versioning on the bucket and apply an S3 Object Lock retention policy in compliance or governance mode, or the MinIO equivalent, so a key cannot be overwritten or deleted before its retention period elapses. Combined with content-addressed keys, immutability guarantees that a report referenced by hash today resolves to exactly the same bytes months later — essential for audit and regulatory reporting.

How should I distribute report PDFs without making the bucket public?

Keep the bucket private and hand out time-limited presigned URLs generated per request. A presigned URL grants read access to one object for a short window without exposing credentials or the rest of the bucket, so you can embed a link in an email or dashboard that expires automatically. Never make a report bucket public — spatial reports frequently contain sensitive location data.

How do I promote a report artifact from staging to production?

Promote by reference, not by re-rendering. Because the artifact is content-addressed, promotion is copying the immutable object (or just its catalog pointer) from the staging bucket to the production bucket under the same hash key, then updating the production catalog to mark that version current. Re-rendering in each environment risks byte drift; copying the exact reviewed bytes guarantees production ships what staging approved.

Detailed Guides in This Section

Parent: CI/CD Scheduling & Automation for Spatial Report Pipelines

Conclusion

Treating each generated report as a content-addressed, immutable object — indexed in a catalog, distributed through expiring signed URLs, aged by a lifecycle policy, and promoted by reference — turns a directory of ambiguously named PDFs into an auditable, verifiable artifact history. That storage discipline is the durable end of the CI/CD Scheduling & Automation for Spatial Report Pipelines workflow, guaranteeing that every report a client received can be reproduced, verified, and traced back to the exact run that produced it.

Explore this section

Publishing Report Bundles to an Artifact Registry A single PDF rarely travels alone. A finished spatial deliverable is often a bundle: the report itself, the map images and fonts it embeds, the source data snapshot, and a manifest tying them… Versioning Generated PDFs in S3 Object Storage Once a spatial report is rendered, the question is not where it goes but which version a stakeholder is looking at three months later during an audit. S3 answers this with object versioning,…