CONNECTOR SDK REFERENCE6 of 10
Batch Handler
Process large datasets efficiently using batching, pagination and checkpoints.
OverviewProcess FlowBatch ContextConfigurationHandler ExamplePartial FailuresMonitoringBest Practices
At a glance
High throughput — optimized for volumeResumable — checkpoint & resumeReliable — retry & deduplicationObservable — progress & metrics
When to use
Initial full data loads
Incremental synchronizations
Large exports and migrations
Backfills and reprocessing
Cross-system data reconciliation
Batch context
typescriptCopy
interface BatchContext {
jobId: string; tenantId: string;
config: BatchConfiguration;
logger: StructuredLogger;
checkpoint: CheckpointStore;
http: HttpClient; metrics: Metrics;
cancelToken: CancelToken;
}Key considerations
Idempotency
Checkpoints
Rate limits
Ordering
Resource usage
Batch process flow
1. Initialize
Start batch job
2. Fetch
Read page of data
3. Transform
Map to Mediloop
4. Send
Write in batches
5. Checkpoint
Save cursor
6. Complete
All pages processed
Batch handler example (TypeScript)
typescriptCopy
export const batchHandler: BatchHandler = async (ctx) => {
let cursor = await ctx.checkpoint.getCursor(ctx.jobId);
let processed = 0, succeeded = 0, failed = 0;
while (!ctx.cancelToken.isCancelled()) {
const page = await fetchPage(cursor, ctx);
if (page.items.length === 0) break;
const mapped = await mapToMediloop(page.items, ctx);
const result = await writeInBatches(mapped, ctx);
await ctx.checkpoint.save(cursor.next,{processed,succeeded,failed});
cursor = page.nextCursor;
}
return { processed, succeeded, failed };
};Example batch configuration
jsonCopy
{
"batch": {
"pageSize": 500,
"maxConcurrency": 4,
"maxRetries": 5,
"backoff":{"strategy":"exponential"},
"checkpoint":{"enabled":true,"intervalSeconds":60}
}
}Metrics & progress
Total records1,254,320
Processed1,123,450
Succeeded1,110,230
Failed13,220
Duration00:18:42
Throughput1,000 rec/s