SpatiaLite & GeoPackage Automation with Python

Batch Convert Shapefiles to GeoPackage with Fiona

Open each .shp file with fiona.open(src, driver="ESRI Shapefile"), write its features into a named GeoPackage layer with fiona.open(dst, "a",…

Open each .shp file with fiona.open(src, driver="ESRI Shapefile"), write its features into a named GeoPackage layer with fiona.open(dst, "a", driver="GPKG", layer=name, schema=src.schema, crs=src.crs), and wrap the feature loop in explicit serialization batches — producing an OGC GeoPackage 1.2-compliant output from an entire directory of source Shapefiles in a single pipeline run.

This page is part of the Spatial Data Serialization Patterns guide, which covers WKB, GeoJSON, and batch-serialization strategies for SpatiaLite and GeoPackage.

Why This Matters

A Shapefile dataset spans at least four companion files (.shp, .shx, .dbf, .prj). One missing sidecar corrupts the entire layer silently. When a field team delivers 50 survey areas as 50 separate Shapefile sets, manual desktop conversion is error-prone and slow. An automated Fiona batch pipeline:

  • Processes hundreds of files in minutes
  • Validates CRS consistency across all sources
  • Detects and logs geometry type mismatches before they reach the database
  • Consolidates all layers into a single, portable, self-contained GeoPackage
  • Produces a machine-readable failure log so the team can fix corrupt source files without re-running the entire batch

Unlike GeoPandas-based pipelines that load each file entirely into memory, Fiona streams features record by record — making it the correct choice for large Shapefiles on constrained field hardware.

Prerequisites

  • Python 3.9+
  • fiona 1.9+ (pip install fiona[all] or conda install -c conda-forge fiona gdal)
  • GDAL/OGR 3.4+ compiled with GPKG and SQLite drivers enabled
  • Source Shapefiles with .shx, .dbf, and .prj sidecars present
  • PROJ_LIB and GDAL_DATA environment variables configured for CRS operations
  • Familiarity with Spatial Data Serialization Patterns fundamentals

Primary Method

python
# Fiona 1.9+ / GDAL 3.4+ — complete Shapefile-to-GeoPackage batch pipeline
from pathlib import Path
from typing import Optional
import fiona

def batch_convert(
    src_dir: str,
    out_gpkg: str,
    target_crs: Optional[str] = None,
    geometry_promotion: bool = True,
) -> dict:
    """
    Convert every .shp in src_dir to a named layer in out_gpkg.

    Args:
        src_dir:            Root directory to search for .shp files.
        out_gpkg:           Destination GeoPackage path (created or overwritten).
        target_crs:         If set (e.g. "EPSG:4326"), reproject all layers to this CRS.
        geometry_promotion: If True, promote Polygon schemas to MultiPolygon.

    Returns:
        {"converted": [...], "skipped": [...], "failed": [...]}
    """
    results = {"converted": [], "skipped": [], "failed": []}
    shapefiles = sorted(Path(src_dir).glob("**/*.shp"))

    if not shapefiles:
        raise FileNotFoundError(f"No .shp files found under {src_dir}")

    for idx, shp in enumerate(shapefiles):
        layer_name = shp.stem.lower().replace(" ", "_").replace("-", "_")
        mode = "w" if idx == 0 else "a"

        try:
            with fiona.open(str(shp), driver="ESRI Shapefile") as src:
                if len(src) == 0:
                    results["skipped"].append(str(shp))
                    continue

                schema = dict(src.schema)
                if geometry_promotion and schema["geometry"] == "Polygon":
                    schema["geometry"] = "MultiPolygon"

                dst_crs = target_crs or src.crs

                with fiona.open(
                    out_gpkg, mode,
                    driver="GPKG",
                    layer=layer_name,
                    schema=schema,
                    crs=dst_crs,
                ) as dst:
                    for feature in src:
                        if target_crs and src.crs != dst_crs:
                            import fiona.transform
                            geom = fiona.transform.transform_geom(
                                src.crs_wkt, target_crs, feature["geometry"]
                            )
                            dst.write({**feature, "geometry": geom})
                        else:
                            dst.write(feature)

            results["converted"].append(layer_name)

        except Exception as exc:
            results["failed"].append({"file": str(shp), "error": str(exc)})

    return results

Step-by-Step Walkthrough

1. Validate sidecars before opening

Missing .shx or .dbf sidecars cause DriverError mid-conversion. Check upfront:

python
for shp in shapefiles:
    for ext in (".shx", ".dbf"):
        if not shp.with_suffix(ext).exists():
            raise FileNotFoundError(
                f"Missing {ext} for {shp.name} — run will abort"
            )

A missing .prj does not raise an error but produces a geometry column with no CRS. The GeoPackage spec requires a valid CRS entry in gpkg_spatial_ref_sys; without it, desktop GIS tools cannot reproject the layer.

2. Use explicit driver binding

Always pass driver="ESRI Shapefile" on read and driver="GPKG" on write. Without explicit binding, GDAL uses file-extension heuristics, which fail on filesystems with case-insensitive naming, virtual filesystem overlays, or unusual extensions:

python
with fiona.open(str(shp), driver="ESRI Shapefile") as src:
    with fiona.open(out_gpkg, "a", driver="GPKG", layer=name,
                    schema=src.schema, crs=src.crs) as dst:
        ...

3. Resolve geometry type mismatches

GeoPackage layers declare a single geometry type. A layer declared as Polygon rejects MultiPolygon features with ValueError: Record's geometry type does not match collection schema. Promote the schema type before writing:

python
schema = dict(src.schema)
if schema["geometry"] in ("Polygon", "Unknown"):
    schema["geometry"] = "MultiPolygon"

For datasets where the source genuinely mixes geometry types (e.g., a survey where some zones are single polygons and others are multi-part), promotion prevents write failures without altering the stored geometry.

4. Reproject on-the-fly when CRS varies across sources

Field data often arrives in local CRS projections. Reproject each feature to a common target CRS before writing:

python
import fiona.transform

TARGET_CRS = "EPSG:4326"

for feature in src:
    geom = fiona.transform.transform_geom(
        src.crs_wkt, TARGET_CRS, feature["geometry"]
    )
    dst.write({**feature, "geometry": geom})

transform_geom handles ring ordering and coordinate precision. It adds CPU cost per feature; for datasets where all sources already share the target CRS, skip this step.

5. Verify output layer count and feature counts

After conversion, confirm every expected layer is present and no features were silently dropped:

python
import fiona

output_layers = fiona.listlayers(out_gpkg)
print(f"Layers in output: {output_layers}")

for layer in output_layers:
    with fiona.open(out_gpkg, layer=layer, driver="GPKG") as lyr:
        print(f"  {layer}: {len(lyr)} features, CRS={lyr.crs.to_epsg()}")

Cross-check each layer’s feature count against the corresponding source Shapefile. A mismatch indicates features were skipped due to geometry type rejection or null geometry values.

6. Rebuild spatial indexes and validate OGC compliance

Fiona writes GeoPackage R-tree index entries during the feature-by-feature write loop. If you interrupted a conversion and resumed with "a" mode, the index may be stale. Rebuild it explicitly using ogrinfo or the SpatiaLite UpdateLayerStatistics() function, then validate OGC compliance — see How to Validate GeoPackage OGC Compliance.

Serialization Patterns Used Internally

Each call to dst.write(feature) triggers Fiona’s internal serialization pipeline:

  1. The GeoJSON-like Python dict is converted to an OGR feature object in the GDAL C layer.
  2. The geometry is encoded as ISO WKB with correct ring orientation enforced by OGR.
  3. The 8-byte GeoPackage Binary header is prepended.
  4. The blob is inserted into the GeoPackage layer table.
  5. The R-tree index entry is updated.

Understanding this pipeline matters when debugging geometry corruption: if a feature writes without error but fails ST_IsValid later, the source GeoJSON geometry had an orientation or self-intersection issue that OGR did not repair. Pre-validate with Shapely’s make_valid() before passing to Fiona — see How to Serialize Multipolygon Geometries to WKB in Python for the repair pattern.

Common Failure Modes

DriverError: unable to open ESRI Shapefile

Missing .shx sidecar or path mismatch. Validate sidecars before opening (step 1 above).

ValueError: Record's geometry type does not match collection schema

The source contains a geometry type not declared in the layer schema. Promote the schema geometry type to the Multi* variant (step 3 above).

DriverError: Failed to create GeoPackage

The output directory does not exist. Create it: Path(out_gpkg).parent.mkdir(parents=True, exist_ok=True).

PROJ: proj_create_from_database: Cannot find proj.db

The PROJ_LIB environment variable is unset or points to the wrong directory. Set it before running the pipeline: export PROJ_LIB=$(python3 -c "import pyproj; print(pyproj.datadir.get_data_dir())").

Layer appears empty after conversion

The source Shapefile had 0 features (len(src) == 0). The pipeline skips empty layers; check results["skipped"] to confirm which files were empty.