SpatiaLite & GeoPackage Automation with Python

How to Bundle PROJ Grid Files with an Application

Copy the grid files into a directory PROJ searches, point PROJDATA at it, and then assert at startup that the grid-based route is available — because if…

Copy the grid files into a directory PROJ searches, point PROJ_DATA at it, and then assert at startup that the grid-based route is available — because if it is not, PROJ falls back to a less accurate transformation and returns coordinates that are wrong by a metre or two with no warning at all. The assertion is the part that matters; the copying is trivial.

This page belongs to the Datum Transformations & Projection Accuracy guide.

Why This Matters

PROJ’s design is helpful in a way that becomes dangerous in a pipeline: when it cannot run the best available transformation, it runs the next best instead of stopping. On a build machine with the grid installed, your reprojection is accurate to a few centimetres. On a field device without it, the same code silently uses a seven-parameter approximation and lands a metre and a half away.

Nothing distinguishes the two outputs. Both containers claim the same target reference system, both open cleanly, and both look right on a map. The difference only appears when someone compares the two, or stands where the boundary is supposed to be.

Prerequisites

  • PROJ 7+ (the PROJ_DATA variable; older versions use PROJ_LIB)
  • pyproj 3.4+ to query which routes are available
  • The grid files for the transformations your data actually uses
  • A packaging step you control — a container image, an installer, an app bundle

Primary Method

python
# Assert that the grid-based route is available before doing any work
from pyproj.transformer import TransformerGroup


REQUIRED_ROUTES = [
    ("EPSG:27700", "EPSG:4326"),
    ("EPSG:4326", "EPSG:27700"),
]


def assert_grids_available(max_accuracy_m: float = 0.2) -> None:
    """
    Fail at startup if PROJ cannot run a transformation at least as accurate
    as max_accuracy_m for every route this application depends on.
    """
    problems = []
    for src, dst in REQUIRED_ROUTES:
        tg = TransformerGroup(src, dst)

        if not tg.transformers:
            problems.append(f"{src} -> {dst}: no usable transformation at all")
            continue

        best = tg.transformers[0]
        accuracy = best.accuracy if best.accuracy is not None else 999.0
        if accuracy > max_accuracy_m:
            missing = [
                g.short_name
                for op in tg.unavailable_operations
                for g in op.grids
            ]
            problems.append(
                f"{src} -> {dst}: best available is {accuracy} m "
                f"({best.description}); missing grids: {missing or 'none reported'}"
            )

    if problems:
        raise RuntimeError("PROJ grid check failed:\n  " + "\n  ".join(problems))

Both directions are listed deliberately. Grid availability is not symmetric in every configuration, and a pipeline that transforms out accurately and back approximately produces a round trip that does not close — a failure that looks like data corruption rather than a packaging problem.

The silent fallback a missing grid producesA reprojection request reaches PROJ, which looks for the most accurate route whose data is present. With the grid installed it uses the grid-based route, accurate to a few centimetres. Without it, PROJ does not fail — it selects the next best route, a seven-parameter transformation accurate to about a metre and a half, and returns coordinates with no warning. Both outputs claim the same target reference system.transform requestsource and target onlyPROJ selectsbest route it can rungrid present — ±5 cmwhat everyone assumes happensgrid absent — ±1.5 mno warning, no error, same label
There is no return value, log line or exception distinguishing the two branches — which is why the check has to be explicit.

Step-by-Step Walkthrough

1. Find out which grids you need

Ask PROJ rather than guessing. unavailable_operations names the grids and, usefully, gives a download URL for each.

python
from pyproj.transformer import TransformerGroup

tg = TransformerGroup("EPSG:27700", "EPSG:4326")

for op in tg.unavailable_operations:
    print(op.name)
    for grid in op.grids:
        print(f"    file: {grid.short_name}")
        print(f"    url:  {grid.url}")
        print(f"    open licence: {grid.direct_download}")

Run this on a machine that does not have the grids installed — on one that does, the list is empty and tells you nothing.

2. Fetch and vendor the files

Grids are ordinary data files, and most are small enough to commit alongside an application. Fetch them once, at build time, and store them in your own artefact rather than downloading at run time — a field device may have no network at exactly the moment it needs one.

bash
# Vendor the grids into the build context
mkdir -p vendor/proj
curl -fsSL -o vendor/proj/uk_os_OSTN15_NTv2_OSGBtoETRS.tif \
     https://cdn.proj.org/uk_os_OSTN15_NTv2_OSGBtoETRS.tif

# Record what you vendored, so the next build can verify it
sha256sum vendor/proj/*.tif > vendor/proj/SHA256SUMS

The checksum file is worth the two seconds. A grid that arrives truncated is a file PROJ will reject at load, and the failure message points at the transformation rather than at the download.

3. Put them where PROJ looks

PROJ searches its data directory, and PROJ_DATA overrides where that is. Setting it to a directory containing only your grids removes the rest of the PROJ data — so include the vendored grids alongside the library’s own directory rather than instead of it.

python
# Prepend the bundled directory to PROJ's search path
import os
from pathlib import Path
import pyproj

BUNDLED = Path(__file__).parent / "vendor" / "proj"

pyproj.datadir.append_data_dir(str(BUNDLED))
print("PROJ data dirs:", pyproj.datadir.get_data_dir())

append_data_dir is the safe form because it adds to the search path rather than replacing it. Setting the PROJ_DATA environment variable to the bundled directory alone is the classic mistake: the grids resolve and proj.db no longer does, so every reference-system lookup fails.

Adding to the PROJ search path rather than replacing itPROJ needs both its own data directory, which holds proj.db and the reference-system definitions, and any directory holding vendored grids. Appending the bundled directory keeps both resolvable. Setting PROJ_DATA to the bundled directory alone makes the grids resolve while proj.db does not, so every reference-system lookup fails instead.append — correctlibrary dir: proj.db, definitionsbundled dir: your gridsboth resolvableeverything worksreplace — the classic mistakePROJ_DATA = bundled dir onlygrids resolveproj.db does notevery CRS lookup failsthe second failure is at least loud, which makes it the easier of the two to diagnose
Appending costs nothing and removes an entire class of "it worked in development" packaging bugs.

4. Disable network fetching in production

PROJ can download grids on demand, which is convenient in development and unhelpful in a pipeline: it makes a build’s accuracy depend on network conditions at run time. Turn it off explicitly, so a missing grid is an error rather than a variable-latency download.

python
import pyproj

pyproj.network.set_network_enabled(active=False)
assert not pyproj.network.is_network_enabled()

With the network disabled and the grids bundled, the transformation route is fully determined by the artefact — which is the property that makes the output reproducible.

5. Run the assertion at startup

Call assert_grids_available() before the pipeline does any spatial work, and let it raise. A pipeline that fails in the first second with “missing grid: uk_os_OSTN15…” is a five-minute packaging fix; the same pipeline succeeding and producing metre-shifted coordinates is a week of investigation.

Verification

python
# In CI, and again as a startup smoke test in the deployed image
from pyproj.transformer import TransformerGroup

tg = TransformerGroup("EPSG:27700", "EPSG:4326")
best = tg.transformers[0]

assert "hgridshift" in best.definition or "gridshift" in best.definition.lower(), (
    f"expected a grid-based route, got: {best.description}"
)
assert (best.accuracy or 999) <= 0.2, f"route accuracy is {best.accuracy} m"
assert not tg.unavailable_operations, (
    f"{len(tg.unavailable_operations)} route(s) still unavailable"
)
print(f"grid route available: {best.description} ({best.accuracy} m)")

Pair that with the control-point residual check from Verifying a Reprojection with Known Control Points. The route assertion proves the right transformation is available; the residual check proves it is the one that actually ran.

Where the grid check belongs in the lifecycleThree placements. At build time, the check confirms the grids were vendored into the artefact and fails the build if not. At container start, it confirms the deployed image actually resolves them, catching a base-image change. Before a reprojection run, the control-point residual confirms the accurate route is the one that ran. Each catches a failure the others cannot see.build time — were the grids vendored?fails the build, not the deployment · cheapest place to noticecontainer start — does this image resolve them?catches a base-image change that moved the data directorybefore a run — did the accurate route actually run?control-point residual · the only check that measures the output
The first two check availability; only the third checks what happened.

Alternative Approaches or Edge Cases

Using the PROJ CDN with a local cache. PROJ can fetch grids on demand and cache them, which suits a long-running server on a reliable network. It does not suit a field device, a build that must be reproducible, or anything where the first run and the hundredth must behave identically. Where you use it, pre-warm the cache during the build so the artefact still ships with the data.

Old PROJ versions. PROJ 6 and earlier used NTv2 .gsb files and the PROJ_LIB variable; PROJ 7+ prefers GeoTIFF grids and PROJ_DATA. A bundle that supports both needs both formats and both variables, which is a reason to pin the PROJ version rather than to support a range.

Licensing. Not every grid is freely redistributable. grid.direct_download reports whether PROJ considers a grid openly available; where it is not, the grid must be obtained from the authority and the licence checked before it goes into an artefact you distribute.

Troubleshooting

CRSError: Invalid projection after setting PROJ_DATA

Cause: The variable was set to the bundled directory alone, so proj.db is no longer on the search path. Fix: Use append_data_dir, or include the library’s own data directory in the value.

Grids resolve locally and not in the container

Cause: The COPY in the image build put them somewhere PROJ does not search, or a multi-stage build dropped the layer. Fix: Print pyproj.datadir.get_data_dir() inside the container and compare it with where the files landed; add the startup assertion so this fails at deploy rather than at use.

The route is grid-based but the accuracy is still poor

Cause: A grid covering a different area — grids are regional, and PROJ will use one whose area of use does not contain your data if nothing better exists. Fix: Check the grid’s area of use against your extent, and confirm the residual against control points inside your working area rather than at the edge of the grid.

Frequently Asked Questions

How large are these files?

Most national grids are a few megabytes as GeoTIFF, and a bundle typically needs one or two. That is small against an application, and negligible against a tile pyramid — the size is almost never the reason not to bundle. The reasons that do come up are licensing and the assumption that the operating system will provide them, and only the first is a real constraint.

Should the grid ship inside the GeoPackage?

No — there is no place for it in the format, and a container is data rather than a transformation environment. What the container should carry is a record of which operation produced its coordinates, so a recipient can obtain the same grid and reproduce the result. The provenance table pattern in the parent guide is where that belongs.

Does this matter if everything is in EPSG:4326?

Only if the data reached 4326 from something else, which is nearly always. The transformation that produced the stored coordinates is the one that needed the grid, and if it ran without one, the container holds approximate coordinates regardless of what happens afterwards. Bundling matters at the point of conversion, not at the point of use.