From experience, when syncing large datasets (50k+...
# office-hours
o
From experience, when syncing large datasets (50k+ records), this usually isn’t a hard public limit but more of an internal deep pagination constraint or performance safeguard on the /call endpoint. APIs that paginate using createdAtLt can sometimes return empty arrays if the timestamp reference isn’t being updated precisely to the last record’s value (including milliseconds), or if multiple records share the same timestamp. This are the things that might likely happen, • There’s likely an internal soft cap on deep pagination performance • createdAtLt pagination can become unreliable at scale • Cursor-based pagination (if available) is always more stable for 100k+ records • Best practice for full backfill is batching by date ranges (e.g., month-by-month) instead of one continuous deep crawl. I will suggest you do the historical sync in time-window batches (for example: Jan 2023 → Feb 2023 → Mar 2023, etc.), then switch to incremental syncing based on latest createdAt. I can help you design a clean and reliable backfill strategy so this doesn’t break again.