SpatiaLite & GeoPackage Automation with Python

Batch Convert Shapefiles to GeoPackage with Fiona

Open each .shp file with fiona.open(src, driver="ESRI Shapefile"), write its features into a named GeoPackage layer with fiona.open(dst, "a",…

Open each .shp file with fiona.open(src, driver="ESRI Shapefile"), write its features into a named GeoPackage layer with fiona.open(dst, "a", driver="GPKG", layer=name, schema=src.schema, crs=src.crs), and wrap the feature loop in explicit serialization batches — producing an OGC GeoPackage 1.2-compliant output from an entire directory of source Shapefiles in a single pipeline run.

This page is part of the Spatial Data Serialization Patterns guide, which covers WKB, GeoJSON, and batch-serialization strategies for SpatiaLite and GeoPackage.

Why This Matters

A Shapefile dataset spans at least four companion files (.shp, .shx, .dbf, .prj). One missing sidecar corrupts the entire layer silently. When a field team delivers 50 survey areas as 50 separate Shapefile sets, manual desktop conversion is error-prone and slow. An automated Fiona batch pipeline:

  • Processes hundreds of files in minutes
  • Validates CRS consistency across all sources
  • Detects and logs geometry type mismatches before they reach the database
  • Consolidates all layers into a single, portable, self-contained GeoPackage
  • Produces a machine-readable failure log so the team can fix corrupt source files without re-running the entire batch

Unlike GeoPandas-based pipelines that load each file entirely into memory, Fiona streams features record by record — making it the correct choice for large Shapefiles on constrained field hardware.

Conversion touches four things per source file, and only the first is about geometry. The other three are exactly where a batch quietly diverges from what the operator expected:

The four things a per-file conversion has to settleFour decisions per source. The geometry type must be compatible with the destination layer, usually by promoting to the multi-part variant. The coordinate reference system must be either consistent across sources or reprojected to a common target. The attribute schema must be reconciled where sources disagree on columns. The layer name must be derived deterministically, because a name collision merges two datasets silently.geometry typepromote to the Multi variantor the first mixed file raisescoordinate reference systemconsistent, or reprojectednever assumed from coordinatesattribute schemasources disagree on columnsreconcile, or lose the extraslayer namingderived deterministicallya collision merges two datasets
The bottom-right box is the quietest failure: two directories containing parcels.shp become one layer with no warning.

Prerequisites

  • Python 3.9+
  • fiona 1.9+ (pip install fiona[all] or conda install -c conda-forge fiona gdal)
  • GDAL/OGR 3.4+ compiled with GPKG and SQLite drivers enabled
  • Source Shapefiles with .shx, .dbf, and .prj sidecars present
  • PROJ_LIB and GDAL_DATA environment variables configured for CRS operations
  • Familiarity with Spatial Data Serialization Patterns fundamentals

Primary Method

python
# Fiona 1.9+ / GDAL 3.4+ — complete Shapefile-to-GeoPackage batch pipeline
from pathlib import Path
from typing import Optional
import fiona

def batch_convert(
    src_dir: str,
    out_gpkg: str,
    target_crs: Optional[str] = None,
    geometry_promotion: bool = True,
) -> dict:
    """
    Convert every .shp in src_dir to a named layer in out_gpkg.

    Args:
        src_dir:            Root directory to search for .shp files.
        out_gpkg:           Destination GeoPackage path (created or overwritten).
        target_crs:         If set (e.g. "EPSG:4326"), reproject all layers to this CRS.
        geometry_promotion: If True, promote Polygon schemas to MultiPolygon.

    Returns:
        {"converted": [...], "skipped": [...], "failed": [...]}
    """
    results = {"converted": [], "skipped": [], "failed": []}
    shapefiles = sorted(Path(src_dir).glob("**/*.shp"))

    if not shapefiles:
        raise FileNotFoundError(f"No .shp files found under {src_dir}")

    for idx, shp in enumerate(shapefiles):
        layer_name = shp.stem.lower().replace(" ", "_").replace("-", "_")
        mode = "w" if idx == 0 else "a"

        try:
            with fiona.open(str(shp), driver="ESRI Shapefile") as src:
                if len(src) == 0:
                    results["skipped"].append(str(shp))
                    continue

                schema = dict(src.schema)
                if geometry_promotion and schema["geometry"] == "Polygon":
                    schema["geometry"] = "MultiPolygon"

                dst_crs = target_crs or src.crs

                with fiona.open(
                    out_gpkg, mode,
                    driver="GPKG",
                    layer=layer_name,
                    schema=schema,
                    crs=dst_crs,
                ) as dst:
                    for feature in src:
                        if target_crs and src.crs != dst_crs:
                            import fiona.transform
                            geom = fiona.transform.transform_geom(
                                src.crs_wkt, target_crs, feature["geometry"]
                            )
                            dst.write({**feature, "geometry": geom})
                        else:
                            dst.write(feature)

            results["converted"].append(layer_name)

        except Exception as exc:
            results["failed"].append({"file": str(shp), "error": str(exc)})

    return results

Step-by-Step Walkthrough

1. Validate sidecars before opening

Missing .shx or .dbf sidecars cause DriverError mid-conversion. Check upfront:

python
for shp in shapefiles:
    for ext in (".shx", ".dbf"):
        if not shp.with_suffix(ext).exists():
            raise FileNotFoundError(
                f"Missing {ext} for {shp.name} — run will abort"
            )

A missing .prj does not raise an error but produces a geometry column with no CRS. The GeoPackage spec requires a valid CRS entry in gpkg_spatial_ref_sys; without it, desktop GIS tools cannot reproject the layer.

2. Use explicit driver binding

Always pass driver="ESRI Shapefile" on read and driver="GPKG" on write. Without explicit binding, GDAL uses file-extension heuristics, which fail on filesystems with case-insensitive naming, virtual filesystem overlays, or unusual extensions:

python
with fiona.open(str(shp), driver="ESRI Shapefile") as src:
    with fiona.open(out_gpkg, "a", driver="GPKG", layer=name,
                    schema=src.schema, crs=src.crs) as dst:
        ...

3. Resolve geometry type mismatches

GeoPackage layers declare a single geometry type. A layer declared as Polygon rejects MultiPolygon features with ValueError: Record's geometry type does not match collection schema. Promote the schema type before writing:

python
schema = dict(src.schema)
if schema["geometry"] in ("Polygon", "Unknown"):
    schema["geometry"] = "MultiPolygon"

For datasets where the source genuinely mixes geometry types (e.g., a survey where some zones are single polygons and others are multi-part), promotion prevents write failures without altering the stored geometry.

Deriving a layer name safelyA source path is reduced to a layer name in three steps: the file stem is taken, it is lower-cased and non-alphanumeric characters are replaced with underscores, and finally it is checked against the names already written. A collision must be resolved by qualifying with the parent directory rather than by appending to the existing layer, which would merge two unrelated datasets.take the file stemParcels-2026.shpnormaliseparcels_2026already written?check before writingon collision, qualify with the parent directorynever fall through to an append
Deterministic naming also means a re-run targets the same layers, which is what makes the batch idempotent.

4. Reproject on-the-fly when CRS varies across sources

Field data often arrives in local CRS projections. Reproject each feature to a common target CRS before writing:

python
import fiona.transform

TARGET_CRS = "EPSG:4326"

for feature in src:
    geom = fiona.transform.transform_geom(
        src.crs_wkt, TARGET_CRS, feature["geometry"]
    )
    dst.write({**feature, "geometry": geom})

transform_geom handles ring ordering and coordinate precision. It adds CPU cost per feature; for datasets where all sources already share the target CRS, skip this step.

5. Verify output layer count and feature counts

After conversion, confirm every expected layer is present and no features were silently dropped:

python
import fiona

output_layers = fiona.listlayers(out_gpkg)
print(f"Layers in output: {output_layers}")

for layer in output_layers:
    with fiona.open(out_gpkg, layer=layer, driver="GPKG") as lyr:
        print(f"  {layer}: {len(lyr)} features, CRS={lyr.crs.to_epsg()}")

Cross-check each layer’s feature count against the corresponding source Shapefile. A mismatch indicates features were skipped due to geometry type rejection or null geometry values.

6. Rebuild spatial indexes and validate OGC compliance

Fiona writes GeoPackage R-tree index entries during the feature-by-feature write loop. If you interrupted a conversion and resumed with "a" mode, the index may be stale. Rebuild it explicitly using ogrinfo or the SpatiaLite UpdateLayerStatistics() function, then validate OGC compliance — see How to Validate GeoPackage OGC Compliance.

Why streaming beats loading for a directory conversionTwo shapes for the same conversion. Loading each source into a frame and writing it holds one whole file in memory at a time, so peak memory tracks the largest source. Streaming feature by feature holds one feature at a time, so peak memory is effectively constant regardless of source size. On constrained field hardware only the second finishes.load then writeone whole file residentpeak tracks the largest sourcefails on the one big fileafter succeeding on forty small onesstream feature by featureone feature residentpeak is effectively constantsource size stops matteringthe shape Fiona is built for
The left-hand failure mode is the awkward one: it passes every test until the day a large source arrives.

Serialization Patterns Used Internally

Each call to dst.write(feature) triggers Fiona’s internal serialization pipeline:

  1. The GeoJSON-like Python dict is converted to an OGR feature object in the GDAL C layer.
  2. The geometry is encoded as ISO WKB with correct ring orientation enforced by OGR.
  3. The 8-byte GeoPackage Binary header is prepended.
  4. The blob is inserted into the GeoPackage layer table.
  5. The R-tree index entry is updated.

Understanding this pipeline matters when debugging geometry corruption: if a feature writes without error but fails ST_IsValid later, the source GeoJSON geometry had an orientation or self-intersection issue that OGR did not repair. Pre-validate with Shapely’s make_valid() before passing to Fiona — see How to Serialize Multipolygon Geometries to WKB in Python for the repair pattern.

Common Failure Modes

DriverError: unable to open ESRI Shapefile

Missing .shx sidecar or path mismatch. Validate sidecars before opening (step 1 above).

ValueError: Record's geometry type does not match collection schema

The source contains a geometry type not declared in the layer schema. Promote the schema geometry type to the Multi* variant (step 3 above).

DriverError: Failed to create GeoPackage

The output directory does not exist. Create it: Path(out_gpkg).parent.mkdir(parents=True, exist_ok=True).

PROJ: proj_create_from_database: Cannot find proj.db

The PROJ_LIB environment variable is unset or points to the wrong directory. Set it before running the pipeline: export PROJ_LIB=$(python3 -c "import pyproj; print(pyproj.datadir.get_data_dir())").

Layer appears empty after conversion

The source Shapefile had 0 features (len(src) == 0). The pipeline skips empty layers; check results["skipped"] to confirm which files were empty.

Frequently Asked Questions

Should each source become its own layer, or should they all merge into one?

Layer-per-source is the safer default: it preserves the provenance of each file, tolerates schema differences between sources, and makes a re-run of one file a targeted operation rather than a rebuild. Merging into a single layer is right only when the sources are genuinely tiles of one dataset with an identical schema and reference system — and even then, keeping a source column makes the merge reversible.

How do I make the batch re-runnable?

By making layer naming deterministic and the write mode explicit. If a name derives only from the source path, a second run targets exactly the same layers, and the choice between recreating and appending becomes a deliberate flag rather than an accident of which files happened to exist. The failure to avoid is a run that appends to whatever is already there, because two runs then double the feature count with nothing in the output to indicate it.

What should the batch do about sources with no features?

Skip them and record the skip. Writing a schema-only layer produces a table that every consumer will enumerate and none can use, and it is indistinguishable from a layer whose load failed silently. Recording the empty source in the run manifest keeps the information without putting an empty layer in the deliverable.

Is it faster to write all sources into one transaction?

Marginally, and it is usually the wrong trade. One transaction across the whole batch means a single failure discards every layer written so far, and it holds the write lock for the entire run. Committing per source keeps the lock hold short, makes the failure isolation meaningful, and costs one commit per file — which against the per-feature work of a conversion is negligible.