Move extents failover

[ohadbitt]

Move extents failure of one batch - should revert change if its a partial success or increase the number of tries after first success

[k-tomak]

We use spark driver to insert data in exactly once semantics: we mark data batches with unique tags on ingest and add ingestIfNotExists in ingestion properties.
There is a problem in KustoClient finalizeIngestionWhenWorkersSucceeded method. If spark cluster fails, we can stay in an intermediate state, some extents with our unique tag will be ingested, but some will not because of batching strategy in this method.
I suppose problem with a lot of extents ingestion can be solved only by cluster scale out.

[k-tomak]

Do you have any news about this request?
We can not upgrade to the next version of library because it isn't safety.

[ohadbitt]

This actually shouldnt concern most people - with newer version - with retries implemented and the ability to set the option
KUSTO_MAXIMAL_EXTENTS_COUNT_FOR_SPLIT_MERGE_PER_NODE "maximalExtentsCountForSplitMergePerNode"
this shouldn't be a problem.

[ag-ramachandran]

@ohadbitt since are now attempting retries in move extents as well, should we go ahead and close this task