SpatiaLite & GeoPackage Automation with Python

Validating Geometry Before a Field Sync

Check isvalid on the device, at the moment a feature is captured, and again on the delta just before it is pushed. Both checks are cheap because they run…

Check is_valid on the device, at the moment a feature is captured, and again on the delta just before it is pushed. Both checks are cheap because they run on a handful of geometries rather than a layer, and together they mean the central store never receives something it will have to refuse — which is the only outcome that keeps a surveyor’s afternoon from being wasted.

This page belongs to the Geometry Validity & Topology Repair guide. Where the other pages there repair a container after the fact, this one is about not needing to.

Why This Matters

A bulk repair pass is a reasonable response to a container you inherited. It is a poor response to a container your own pipeline produces, because it treats the symptom every cycle and never the cause — and because by the time the pass runs, the person who could have fixed the feature in ten seconds has moved on to a different site.

There is also a sync-specific cost. If the central store validates on apply and rejects what fails, a device that pushed a batch containing one bad geometry gets the whole batch refused, retries it, and gets it refused again. The device has no way to make progress, and the change log grows behind an entry that will never be accepted. Validating before the push turns that permanent stall into a single feature flagged for attention.

Prerequisites

Primary Method

python
# Validate a delta before it is pushed; hold back what would be refused
from shapely import from_wkb
from shapely.validation import explain_validity


def screen_delta(rows: list[tuple[int, bytes]]) -> tuple[list, list]:
    """
    Split (fid, gpb_blob) rows into (pushable, held_back).
    Held-back rows carry the reason, so the device can show it to the operator.
    """
    pushable, held = [], []
    for fid, blob in rows:
        if blob is None:
            pushable.append((fid, blob))          # a NULL geometry is a valid state
            continue
        geom = from_wkb(gpb_to_wkb(blob))
        if geom.is_empty:
            held.append((fid, "geometry is empty"))
        elif not geom.is_valid:
            held.append((fid, explain_validity(geom)))
        else:
            pushable.append((fid, blob))
    return pushable, held

Holding a row back rather than dropping it is the important detail. The feature stays on the device, stays in the change log, and stays visible to the operator — it simply does not enter this batch. When the geometry is corrected, the next cycle picks it up with no special handling.

Screening a delta so one bad feature cannot stall the batchA delta of changed rows is screened before it is sent. Valid rows form the batch that is pushed and acknowledged. Invalid rows are held on the device with the reason attached, remain in the change log, and are offered again on the next cycle once corrected. Without the screen, the whole batch is refused by the central store and the device retries it indefinitely.the deltarows above the watermarkscreenis_valid, per rowpushed and acknowledgedthe watermark advancesheld with a reasonstays in the log, offered again
The held rows keep their place in the change log, so correcting one is an ordinary edit rather than a recovery procedure.

Step-by-Step Walkthrough

1. Validate at capture, not only at push

The cheapest possible moment is when the operator finishes drawing, because that is the only moment they can fix it without going back.

python
# Called by the capture UI when a feature is committed
def accept_capture(geom) -> tuple[bool, str]:
    if geom.is_empty:
        return False, "The shape has no area — close the outline before saving."
    if not geom.is_valid:
        reason = explain_validity(geom)
        if reason.startswith("Self-intersection"):
            return False, "The outline crosses itself. Redraw the crossing section."
        return False, f"The shape is not usable: {reason}"
    return True, ""

Translating the GEOS reason into an instruction is what makes this useful in the field. “Self-intersection[512345 187654]” tells a developer everything and a surveyor nothing; “the outline crosses itself” tells the surveyor exactly what to do.

2. Screen the delta before the push

Capture-time validation catches what the device created. It does not catch what arrived by some other route — a layer copied on, an edit made in a desktop tool, a feature imported from a partner. The pre-push screen is the backstop.

python
rows = conn.execute("""
    SELECT p.fid, p.geom
    FROM parcels p
    JOIN _sync_log l ON l.table_name = 'parcels' AND l.fid = p.fid
    WHERE l.version > ?
""", (watermark,)).fetchall()

pushable, held = screen_delta(rows)
if held:
    print(f"{len(held)} feature(s) held back:")
    for fid, reason in held:
        print(f"  fid={fid}: {reason}")

3. Advance the watermark only past what was accepted

This is the step that makes the screen safe. If the watermark advances past a held row, that row is never offered again and the edit is silently lost.

python
# The new watermark is the highest version among the rows actually pushed
held_fids = {fid for fid, _ in held}
accepted_versions = [
    v for fid, v in versions_by_fid.items() if fid not in held_fids
]
new_watermark = min(
    max(accepted_versions, default=watermark),
    # never advance past the lowest held row, or it is skipped forever
    (min(versions_by_fid[f] for f in held_fids) - 1) if held_fids else float("inf"),
)

The min against the lowest held version is the whole safety property: the watermark stops just below the first row that could not be sent, so every later row is re-offered next cycle rather than being assumed delivered.

Where the watermark stops when a row is held backSeven change-log versions. Versions 101 to 103 pushed successfully. Version 104 failed the screen and is held. Versions 105 to 107 are valid but sit above the held row. The watermark advances only to 103, so 105 to 107 are offered again next cycle alongside a corrected 104. Advancing to 107 instead would mark the held row as delivered and lose it permanently.Watermark stops below the first held rowv101 okv102 okv103 okv104 heldv105v106v107watermark advances to 103valid, but re-offered next cycleadvancing to 107 would lose v104 permanentlythe cost of the conservative rule is a few rows re-sent
Re-sending three valid rows is cheap. Marking an unsent row as delivered is not recoverable.

4. Surface held rows to the operator

A held row that nobody sees is a lost edit with extra steps. The device needs to show it, and the sync status needs to say the batch was partial.

python
# Persist the screen result so the UI can list it
conn.execute("""
    CREATE TABLE IF NOT EXISTS _sync_held (
        table_name TEXT NOT NULL,
        fid        INTEGER NOT NULL,
        reason     TEXT NOT NULL,
        noticed_at TEXT NOT NULL,
        PRIMARY KEY (table_name, fid)
    )
""")
conn.executemany(
    "INSERT OR REPLACE INTO _sync_held VALUES ('parcels', ?, ?, datetime('now'))",
    held,
)

5. Keep the central-store check as well

The device screen is not a substitute for validating on apply. It reduces refusals to near zero; it does not make them impossible, because a device may be running an older build, or none at all. The central check stays as the authority, and the device check exists so that authority almost never has to say no.

Three places a validity check earns its costAt capture, the check costs one geometry and the operator can fix the problem immediately, which is where it is cheapest. Before the push, the check costs one delta and prevents the batch being refused. On apply at the central store, the check costs one delta and is the authority that protects the shared container. Each catches what the one before it missed, and only the last is mandatory.at capture — one geometrythe only moment the operator can still fix it · cheapest by a wide marginbefore the push — one deltacatches what arrived by other routes · stops a batch being refusedon apply — one delta, at the centrethe authority · mandatory, because a device may be running an older build
Only the bottom layer is required. The two above it exist so it almost never has to reject anything.

Verification

Prove the screen rejects something it must, and that the watermark rule holds.

python
# The screen must hold back a known-invalid geometry
from shapely.geometry import Polygon
from shapely import to_wkb

bowtie = Polygon([(0, 0), (2, 2), (2, 0), (0, 2), (0, 0)])
pushable, held = screen_delta([(1, gpkg_wrap(to_wkb(bowtie))), (2, gpkg_wrap(good_wkb))])

assert len(held) == 1 and held[0][0] == 1
assert len(pushable) == 1 and pushable[0][0] == 2
print("screen verified")
sql
-- After a partial push, the watermark must sit below the lowest held version
SELECT (SELECT watermark FROM _sync_state WHERE peer = 'central') AS wm,
       (SELECT min(version) FROM _sync_log l
         JOIN _sync_held h ON h.fid = l.fid AND h.table_name = l.table_name) AS lowest_held;

The second query is the assertion worth putting in the sync’s own test suite: wm < lowest_held must hold after every partial push, and a violation means edits are being silently dropped.

Alternative Approaches or Edge Cases

Repairing on the device instead of holding. Tempting, and usually wrong for field data. An automatic repair changes what the surveyor recorded, in the field, with nobody looking — and the surveyor is the one person who knows what the shape was meant to be. Hold and prompt; repair automatically only where the data is machine-generated and no human judgement was ever involved.

Validating only the geometry that changed. The screen already does this by construction, because a delta only contains changed rows. What it does not cover is a geometry that was valid when captured and became invalid through a later reprojection — for that, the check belongs after the transformation, not after the capture.

Devices with no geometry library. Where the device cannot run Shapely, the fallback is a cheaper structural screen: ring closure, minimum vertex count, coordinates inside the survey extent. It catches less, and it catches the cases that matter most — an unclosed ring and a coordinate in the wrong hemisphere are both common and both trivially detectable without a topology engine.

Troubleshooting

The same feature is held on every cycle

Cause: Working as designed — the geometry has not been corrected. Fix: Confirm the held list is actually visible in the device UI. A row that is held repeatedly and never surfaced is the failure mode this whole design exists to avoid.

The watermark never advances

Cause: The lowest held version is at or below the current watermark, so the conservative rule pins it in place. This happens when a row that was already pushed later becomes invalid through an edit. Fix: Compute the held minimum from rows above the watermark only; a held row below it has already been delivered and its current state is a separate correction.

Held rows accumulate after a reprojection

Cause: The transformation moved vertices past one another, creating self-intersections in previously valid geometry — the failure described in Datum Transformations & Projection Accuracy. Fix: Validate after the transformation rather than before it, and treat a sudden rise in held rows as a signal about the transformation rather than about the capture.

Frequently Asked Questions

Does the screen slow the push down?

Barely. A delta is tens or hundreds of rows, and a validity test on a field polygon is well under a millisecond — the screen is invisible next to the network round trip that follows it. The cost people worry about is validating the whole layer, which is a genuinely expensive full scan; the delta screen is a different operation with a different price.

What should the operator see when a feature is held?

The feature, on the map, marked as not yet sent, with a sentence saying why in ordinary language. What they should not see is a count, a log line, or a technical reason string. The design goal is that a held feature looks like an unfinished task rather than like an error, because that is what it is.

Should the central store repair what it receives?

Only if it can do so without changing meaning, and self-intersection repair does change meaning — it decides which of two readings of an ambiguous shape is correct. A central store that repairs silently accumulates data nobody chose. Rejecting, with the reason returned to the device, keeps the decision with the person who has the context.