SpatiaLite & GeoPackage Automation with Python

Pinning GDAL, PROJ and GEOS Versions in CI

Pin the image, not the requirements file. A locked requirements.txt fixes which Python packages install and says nothing about the GDAL, PROJ and GEOS…

Pin the image, not the requirements file. A locked requirements.txt fixes which Python packages install and says nothing about the GDAL, PROJ and GEOS builds they bind to — and those three libraries are where the answers come from. Assert the versions during the image build, so a base-image change fails the build rather than quietly altering results.

This page belongs to the Testing & CI for Spatial Pipelines guide.

Why This Matters

The Python packages in a spatial stack are thin. shapely calls GEOS, pyproj calls PROJ, fiona and pyogrio call GDAL — and each of those libraries has its own release cadence, its own behaviour changes, and its own data files. Two runs with byte-identical Python dependencies can produce different geometry, different coordinates and different validity verdicts because the layer underneath moved.

The changes are real and specific. make_valid output changed shape across GEOS releases. PROJ revises published transformations, so the same authority code resolves to different parameters. GDAL changes default layer creation options between minor versions. None of these is a bug; all of them change what your pipeline produces.

Prerequisites

  • A container image or an equivalently controlled environment
  • pyproj, shapely and either fiona or pyogrio installed
  • A CI system that can build and cache images
  • A decision about which versions the project supports

Primary Method

dockerfile
# Pin the whole stack. The digest is what makes this reproducible.
FROM ghcr.io/osgeo/gdal:ubuntu-small-3.8.4@sha256:0f6e5a9d…

# System libraries pinned to exact package versions
RUN apt-get update && apt-get install -y --no-install-recommends \
        libsqlite3-mod-spatialite=5.0.1-3 \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.lock /tmp/requirements.lock
RUN pip install --no-deps --require-hashes -r /tmp/requirements.lock

# Assert at BUILD time — a base-image change fails here, not in the tests
RUN python - <<'PY'
from packaging.version import Version
import pyproj, shapely
from osgeo import gdal

EXPECT = {"GDAL": "3.8.4", "PROJ": "9.3", "GEOS": "3.12"}

assert gdal.__version__.startswith(EXPECT["GDAL"]), gdal.__version__
assert pyproj.proj_version_str.startswith(EXPECT["PROJ"]), pyproj.proj_version_str
assert shapely.geos_version_string.startswith(EXPECT["GEOS"]), shapely.geos_version_string
print("stack pinned:", gdal.__version__, pyproj.proj_version_str, shapely.geos_version_string)
PY

Two details do the work. Pinning the base image by digest rather than by tag means a rebuilt upstream tag cannot change your stack. Asserting inside a RUN means a mismatch fails the image build, where the cause is obvious, rather than the test suite, where it looks like a code regression.

What each level of pinning actually fixesPinning the requirements file fixes which Python packages install and nothing below them. Pinning the base image tag fixes the libraries until that tag is rebuilt upstream. Pinning the base image by digest fixes the libraries permanently. Only the digest makes the stack reproducible, and only an assertion inside the build proves it stayed that way.requirements.lock onlyfixes the wheels · GDAL, PROJ and GEOS still float with the hostbase image by tagfixes the libraries until the tag is rebuilt upstream, which happens silentlybase image by digest, asserted at buildfixes the libraries permanently · a change fails the build with an obvious cause
Most projects are on the top row and believe they are on the bottom one.

Step-by-Step Walkthrough

1. Record what you actually have

Before pinning anything, capture the current stack, because that is the version set your existing expectations were formed against.

python
# stack.py — print the whole stack, for a lock file or an issue report
import sqlite3
import pyproj
import shapely
from osgeo import gdal

print(f"GDAL   {gdal.__version__}")
print(f"PROJ   {pyproj.proj_version_str}")
print(f"GEOS   {shapely.geos_version_string}")
print(f"SQLite {sqlite3.sqlite_version}")

conn = sqlite3.connect(":memory:")
conn.enable_load_extension(True)
conn.load_extension("mod_spatialite")
print(f"SpatiaLite {conn.execute('SELECT spatialite_version()').fetchone()[0]}")
print(f"PROJ data  {pyproj.datadir.get_data_dir()}")

SQLite belongs on that list even though nothing in the Python layer names it: mod_spatialite is compiled against a specific SQLite, and an interpreter embedding a different one produces the undefined-symbol failure described in Resolving mod_spatialite Load Errors Across Platforms.

2. Pin by digest, not by tag

bash
# Resolve the tag to a digest once, and commit the digest
docker pull ghcr.io/osgeo/gdal:ubuntu-small-3.8.4
docker inspect --format='{{index .RepoDigests 0}}' ghcr.io/osgeo/gdal:ubuntu-small-3.8.4
# ghcr.io/osgeo/gdal@sha256:0f6e5a9d…

A tag is a moving pointer. 3.8.4 names the GDAL version and says nothing about the base operating system, the PROJ build, or the data files — all of which change when the image is rebuilt for a security update, which happens regularly and without notice.

3. Include the PROJ data in the pin

The library version is only half of PROJ. The EPSG dataset ships alongside it and is revised several times a year, so two containers with identical PROJ versions can resolve the same authority code to different parameters.

python
# Record the EPSG dataset version, not just the PROJ version
from pyproj.database import get_database_metadata

print("EPSG dataset:", get_database_metadata("EPSG.VERSION"))
print("PROJ data version:", get_database_metadata("DATABASE.LAYOUT.VERSION.MAJOR"))

Where transformation accuracy matters, that figure belongs in the same assertion as the library versions — and in the provenance record the datum guide describes.

Everything that has to be pinned for a result to be reproducibleFour things, none of which the requirements file covers. The GEOS build decides predicate and repair behaviour. The PROJ build decides which transformation routes exist. The EPSG dataset shipped with PROJ decides what an authority code resolves to. The installed grid files decide which route is actually runnable. All four travel with the image rather than with the Python packages.GEOS buildpredicates, validity, make_validchanged shape across releasesPROJ buildwhich routes exist at alland which one is preferredEPSG datasetwhat an authority code meansrevised several times a yearinstalled gridswhich route is runnable herethe silent accuracy decision
None of these four appears in a Python lock file, and all four change what the pipeline produces.

4. Assert the same stack in the deployed image

CI and production frequently run different images, and the whole point of pinning is lost if only one of them is pinned. Run the version assertion as a startup smoke test in the deployed image too.

python
# entrypoint check — runs before the service accepts work
from stack_assert import assert_stack

assert_stack(gdal="3.8.4", proj="9.3", geos="3.12", epsg_dataset="v11.0")

5. Treat an upgrade as a deliberate change

When a version does move, expect output to move with it and check what changed rather than accepting it. The useful artefact here is a golden-output test: run the pipeline over a fixed input, compare against a committed result with a tolerance, and review any difference.

python
def test_pipeline_output_is_stable(built_container, golden_manifest):
    """A library upgrade may legitimately change this — the review is the point."""
    manifest = describe(built_container)

    assert manifest["layers"] == golden_manifest["layers"]
    for layer, area in manifest["total_area_m2"].items():
        assert area == pytest.approx(golden_manifest["total_area_m2"][layer], rel=1e-9)

Verification

python
# The assertion must actually fail on a wrong stack
import subprocess


def test_assert_stack_rejects_a_mismatch():
    result = subprocess.run(
        ["python", "-c",
         "from stack_assert import assert_stack; assert_stack(geos='2.0')"],
        capture_output=True, text=True,
    )
    assert result.returncode != 0, "assert_stack accepted an impossible version"
    assert "GEOS" in result.stderr
bash
# And the image must actually carry what the Dockerfile claims
docker run --rm "$IMAGE" python -c "
import pyproj, shapely
from osgeo import gdal
print(gdal.__version__, pyproj.proj_version_str, shapely.geos_version_string)
"

Running that against the built image, rather than trusting the Dockerfile, is what catches a multi-stage build that dropped a layer or an apt pin that silently resolved to something else.

Three places the version assertion earns its costAsserting during the image build fails the build when a base image changes, which is the cheapest place to notice. Asserting in the test suite catches an environment that was assembled differently from the image. Asserting at service startup catches a deployment running an image nobody expected. Each catches a different way the stack drifts.during the image builda changed base image fails here, with an obvious cause · cheapest to noticefirst test in the suitecatches an environment assembled differently from the imageat service startupcatches a deployment running an image nobody expected
The same three lines in three places; each one is the only thing that catches its own failure mode.

Alternative Approaches or Edge Cases

Conda environments. conda-lock produces a fully-resolved, platform-specific lock covering the C libraries as well as the Python packages, which gets most of the way to a pinned image without one. The remaining gap is the operating system underneath, which matters less for a build agent than for a deployed service.

Libraries rather than applications. A library installed by other people will meet whatever stack they have, so pinning is the wrong goal — the right one is a support matrix and CI that tests its boundaries. Two configurations, oldest and newest supported, catch most compatibility problems for a modest cost.

Wheels that vendor their own libraries. shapely and pyproj wheels on PyPI bundle GEOS and PROJ, so a pip install does pin those — but a system GDAL installed alongside brings its own copies, and which one loads depends on link order. Mixing the two is the most common cause of a stack whose reported versions do not match its behaviour; pick wheels or system packages and do not mix.

Troubleshooting

The reported version and the behaviour disagree

Cause: Two copies of a library are loaded — typically a wheel-bundled GEOS and a system GEOS pulled in by GDAL. The version string comes from one and the behaviour from the other. Fix: Use ldd on the compiled extension modules to see what each actually links against, and standardise on one source.

The image builds fine and CI fails on a version assertion

Cause: CI is not running the image — it installed requirements onto its own runner. Fix: Run the suite inside the image. A pinned image that the tests do not use pins nothing.

An upgrade changed geometry output and nothing else

Cause: A GEOS change, most likely in make_valid or a predicate at a boundary case. Fix: This is expected behaviour, not a regression to work around. Review the difference against the golden output, decide whether the new result is more correct, and update the golden file with the reason recorded in the commit.

Frequently Asked Questions

How often should the pin be moved?

On a schedule rather than on demand, so upgrades are a routine review instead of an emergency. Quarterly works well for most pipelines: often enough that each step is small and reviewable, rarely enough that it does not consume attention. What does not work is upgrading only when something forces it, because then the jump is large and every behavioural change arrives at once.

Is it worth pinning SQLite?

Yes, and it is the one most often forgotten. mod_spatialite is compiled against a particular SQLite, and the Python interpreter embeds its own — when those diverge the extension fails to load with an undefined-symbol error that looks nothing like a version problem. Recording both versions makes the diagnosis immediate.

What belongs in the golden output?

Aggregates rather than geometry: layer names, feature counts, total area per layer, the bounding box, the reference system. Those are stable under the representation changes a library upgrade legitimately makes, and they move when something real changes. Committing serialised geometry as golden output produces a test that fails on every upgrade for reasons that do not matter.