Parallel ice inserts to the same table causes data loss

[subkanthi]

This happens when there is a conflict

nd)
2026-05-05 17:48:01 [main] INFO o.a.i.SnapshotProducer > Committed snapshot 2088013820286580968 (MergeAppend)
2026-05-05 17:48:01 [main] WARN o.apache.iceberg.util.Tasks > Retrying task after failure: sleepTimeMs=208 Commit failed: Requirement failed: branch main has changed: expected id 7393143700390627738 != 3084219558005623831
org.apache.iceberg.exceptions.CommitFailedException: Commit failed: Requirement failed: branch main has changed: expected id 7393143700390627738 != 3084219558005623831
	at org.apache.iceberg.rest.ErrorHandlers$CommitErrorHandler.accept(ErrorHandlers.java:94)
	at org.apache.iceberg.rest.ErrorHandlers$CommitErrorHandler.accept(ErrorHandlers.java:85)
	at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:216)
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:300)
	at org.apache.iceberg.rest.BaseHTTPClient.post(BaseHTTPClient.java:88)
	at org.apache.iceberg.rest.RESTClient.post(RESTClient.java:113)
	at org.apache.iceberg.rest.RESTTableOperations.commit(RESTTableOperations.java:159)
	at org.apache.iceberg.BaseTransaction.lambda$commitSimpleTransaction$3(BaseTransaction.java:428)
	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
	at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
	at org.apache.iceberg.BaseTransaction.commitSimpleTransaction(BaseTransaction.java:424)
	at org.apache.iceberg.BaseTransaction.commitTransaction(BaseTransaction.java:319)
	at com.altinity.ice.cli.internal.cmd.Insert.run(Insert.java:269)
	at com.altinity.ice.cli.Main.insert(Main.java:625)
	at com.altinity.ice.cli.Main.lambda$main$4(Main.java:1069) [9 skipped]
	at com.altinity.ice.cli.Main.main(Main.java:1076) [1 skipped]
2026-05-05 17:48:02 [main] INFO o.a.i.SnapshotProducer > Committed snapshot 6122489581799268749 (MergeAppend)
2026-05-05 17:48:02 [main] WARN o.apache.iceberg.util.Tasks > Retrying task after failure: sleepTimeMs=203 Commi

[subkanthi]

INFO o.a.i.SnapshotProducer - Committed snapshot 1616831077323646245 (MergeAppend)
MergeAppend - MergeAppend is the operation name reported by Apache Iceberg's MergeAppend snapshot producer (class org.apache.iceberg.MergeAppend, returned by Table.newAppend() in the default configuration). It's one of several "append" implementations Iceberg can use for adding new data files to a table.

FastAppend (Table.newFastAppend() / operation append): writes one new manifest file containing only the newly added data files. Cheap to commit, but produces lots of small manifests over time.
MergeAppend (Table.newAppend() / operation append historically, but logged as the producer name MergeAppend): writes a new manifest for the added files, and if existing manifests are below a size threshold, it rewrites/merges them with the new one to keep the manifest list compact.

Committed snapshot 1616831077323646245 -new snapshot ID.

Commit failed: Requirement failed: branch main has changed: expected id 7900474310346959375 != 1209248601415194206 - Default RETRY of 4

Documented here: https://aws.amazon.com/blogs/big-data/manage-concurrent-write-conflicts-in-apache-iceberg-on-the-aws-glue-data-catalog/

Step 1: ice CLI does a commit

              appendOp.commit();
               txn.commitTransaction();

Step 2:

Optimistic Concurrency Control vs Lock
OCC - Clients will receive a 409 rejection that that the ids dont match, the client has to retry, this could be a large number of retries depending on the number of concurrent writers to the same table.
Lock - A etcd based distributed lock that locks the resource for a single writer, this will reduce the retries on the writer(ice) CLI side.

[subkanthi]

apache/iceberg#15001