SpatiaLite & GeoPackage Automation with Python

How to Batch Convert Shapefiles to GeoPackage with Fiona

Iterate over a source directory with pathlib.Path.glob("/.shp"), open each file with fiona.open(src, driver="ESRI Shapefile"), and write every layer into…

Iterate over a source directory with pathlib.Path.glob("**/*.shp"), open each file with fiona.open(src, driver="ESRI Shapefile"), and write every layer into a single GeoPackage using fiona.open(dst, "w", driver="GPKG", schema=src.schema, crs=src.crs) — wrapping the inner loop in a transaction batch to keep the output file consistent.

This page is part of the Fiona & OGR Driver Configuration guide, which covers explicit driver binding, schema enforcement, and CI-level environment pinning for the GDAL/OGR stack.

Why This Matters

Field GIS teams routinely receive Shapefiles that arrive as separate files per survey area, per date, or per data provider. Merging them into a single GeoPackage is the first step toward a queryable, portable, offline-ready spatial database. Manual conversion through desktop GIS is slow and error-prone; an automated Fiona batch pipeline processes hundreds of files in minutes, validates CRS consistency, detects geometry type mismatches, and logs failures without stopping the run.

Unlike GeoPandas-based conversion — which loads entire files into memory as DataFrames — Fiona streams features record by record. This makes Fiona the right tool on constrained field hardware where loading a large shapefile into a DataFrame would exhaust available RAM.

Prerequisites

  • Python 3.9+
  • fiona 1.9+ (pip install fiona[all] or conda install -c conda-forge fiona gdal)
  • GDAL/OGR 3.4+ compiled with GPKG and SQLite drivers enabled
  • Source Shapefiles with companion .shx, .dbf, and .prj sidecars present
  • Familiarity with Fiona & OGR Driver Configuration fundamentals

Primary Method

python
# Fiona 1.9+ / GDAL 3.4+ — batch Shapefile to GeoPackage conversion
from pathlib import Path
import fiona

def batch_shp_to_gpkg(src_dir: str, out_gpkg: str) -> int:
    """
    Convert every .shp in src_dir (recursively) into layers of a single GeoPackage.

    Returns the number of layers written.
    """
    src_root = Path(src_dir)
    dst_path = Path(out_gpkg)
    shapefiles = sorted(src_root.glob("**/*.shp"))

    if not shapefiles:
        raise FileNotFoundError(f"No .shp files found under {src_root}")

    layers_written = 0

    for shp in shapefiles:
        # Derive a safe layer name from the file stem
        layer_name = shp.stem.lower().replace(" ", "_").replace("-", "_")

        with fiona.open(str(shp), driver="ESRI Shapefile") as src:
            if len(src) == 0:
                # Skip empty layers rather than writing a schema-only shell
                continue

            write_mode = "w" if layers_written == 0 else "a"

            with fiona.open(
                str(dst_path),
                write_mode,
                driver="GPKG",
                layer=layer_name,
                schema=src.schema,
                crs=src.crs,
            ) as dst:
                for feature in src:
                    dst.write(feature)

        layers_written += 1

    return layers_written

The "a" (append) mode on subsequent layers adds new named layers to the same GeoPackage file without overwriting earlier ones. The first layer uses "w" to create the file; if it already exists, "w" truncates it, so remove or rename an existing output before re-running.

Step-by-Step Walkthrough

1. Collect and validate source files

python
from pathlib import Path
import fiona

src_root = Path("/data/field_surveys")
shapefiles = sorted(src_root.glob("**/*.shp"))

for shp in shapefiles:
    # Verify the minimum required sidecars are present
    for ext in (".shx", ".dbf"):
        if not shp.with_suffix(ext).exists():
            raise FileNotFoundError(
                f"Missing {ext} sidecar for {shp} — convert aborted"
            )
print(f"Found {len(shapefiles)} shapefiles")

Missing .shx or .dbf files cause Fiona to raise DriverError mid-conversion. Checking upfront produces a clear error message before any output is written.

2. Inspect the first file to establish a reference schema

python
# Use the first shapefile as the reference for geometry type and CRS
with fiona.open(str(shapefiles[0])) as ref:
    ref_crs = ref.crs
    ref_geom = ref.schema["geometry"]
    print(f"Reference CRS: {ref_crs}")
    print(f"Reference geometry: {ref_geom}")

When combining layers into a single GeoPackage, each layer can have its own schema. If you intend to merge all features into a single layer, the geometry types and CRS must be compatible across all source files — validate this before writing.

3. Convert with explicit driver binding

Passing driver="ESRI Shapefile" on read and driver="GPKG" on write prevents Fiona’s extension-based auto-detection from selecting the wrong parser, which can happen when filenames have unusual casing or when GDAL’s virtual filesystem is active.

python
with fiona.open(str(shp), driver="ESRI Shapefile") as src:
    with fiona.open(
        str(dst_path),
        "a",
        driver="GPKG",
        layer=layer_name,
        schema=src.schema,
        crs=src.crs,
    ) as dst:
        for feature in src:
            dst.write(feature)

4. Handle geometry type mismatches

GeoPackage layers are typed: a layer declared as Polygon rejects MultiPolygon features. Fiona raises ValueError: Record's geometry type does not match collection schema when a mismatch occurs. Promote the schema geometry type to "MultiPolygon" (or the appropriate Multi* variant) when mixed geometry types are present in the source:

python
import fiona.transform

with fiona.open(str(shp), driver="ESRI Shapefile") as src:
    schema = dict(src.schema)
    # Promote to Multi variant to accept both Polygon and MultiPolygon features
    if schema["geometry"] == "Polygon":
        schema["geometry"] = "MultiPolygon"

    with fiona.open(str(dst_path), "a", driver="GPKG",
                    layer=layer_name, schema=schema, crs=src.crs) as dst:
        for feature in src:
            dst.write(feature)

5. Reproject on-the-fly to a common CRS

When source Shapefiles use different coordinate reference systems, reproject each feature to the target CRS before writing. Use fiona.transform.transform_geom for geometry-level reprojection:

python
import fiona
import fiona.transform

TARGET_CRS = "EPSG:4326"

with fiona.open(str(shp), driver="ESRI Shapefile") as src:
    src_crs = src.crs_wkt

    with fiona.open(
        str(dst_path), "a", driver="GPKG",
        layer=layer_name, schema=src.schema, crs=TARGET_CRS,
    ) as dst:
        for feature in src:
            geom = fiona.transform.transform_geom(
                src_crs, TARGET_CRS, feature["geometry"]
            )
            dst.write({**feature, "geometry": geom})

Reprojection adds CPU cost per feature; for large datasets, pre-validate that most source files already use the target CRS and only reproject outliers.

6. Log failures and continue

A production batch pipeline should not abort on a single corrupt Shapefile. Use a try/except to collect failures and continue:

python
failures = []

for shp in shapefiles:
    layer_name = shp.stem.lower().replace(" ", "_")
    try:
        with fiona.open(str(shp), driver="ESRI Shapefile") as src:
            with fiona.open(
                str(dst_path), "a", driver="GPKG",
                layer=layer_name, schema=src.schema, crs=src.crs,
            ) as dst:
                for feature in src:
                    dst.write(feature)
    except Exception as exc:
        failures.append((shp, str(exc)))
        continue

if failures:
    for path, err in failures:
        print(f"FAILED: {path}: {err}")

Validation

After conversion, verify the output GeoPackage is valid and all expected layers are present:

python
import fiona

with fiona.open("output.gpkg", driver="GPKG") as dst:
    print(fiona.listlayers("output.gpkg"))

# Inspect each layer
for layer in fiona.listlayers("output.gpkg"):
    with fiona.open("output.gpkg", layer=layer, driver="GPKG") as lyr:
        print(f"{layer}: {len(lyr)} features, CRS={lyr.crs.to_epsg()}")

Also confirm the GeoPackage passes OGC compliance checks — see How to Validate GeoPackage OGC Compliance.

Common Failure Modes

DriverError: unable to open ESRI Shapefile

Missing .shx sidecar or the .shp file path has a typo. Verify all sidecars exist before opening. On case-sensitive filesystems (Linux), Field_Survey.SHP and field_survey.shp are different files.

ValueError: Record's geometry type does not match collection schema

The source contains mixed geometry types (e.g., Polygon and MultiPolygon). Promote the schema geometry type to the Multi* variant before opening the destination layer.

DriverError: Failed to open GeoPackage

Fiona cannot create the output file because the parent directory does not exist, or an existing output file is locked by another process. Ensure the output directory exists (Path(dst_path).parent.mkdir(parents=True, exist_ok=True)) and no other process has the .gpkg file open.

PROJ: proj_create_from_database: Cannot find proj.db

The PROJ_LIB environment variable does not point to the PROJ data directory. Set it before running: export PROJ_LIB=$(proj --searchpath 2>/dev/null | head -1) or the equivalent for your GDAL installation.