Migrations
Migrations built for millions of objects — and re-syncs that finish in seconds
A migration now runs at any tree size without exhausting a worker's memory, scans a server in seconds, and re-syncs by looking only at what actually changed.
A migration used to hold two things in the worker's memory at once: the entire list of files at the source, and the record of everything it had already copied. On a tree with millions of files that adds up to gigabytes — enough to run a worker out of memory before a byte moved. Both now stream through the database instead of the worker, so a job of tens of millions of objects runs in flat, predictable memory.
Two more improvements you'll feel immediately:
- •Scanning a server is now near-instant. Instead of walking a deep tree one directory at a time, we ask the machine for its whole file list in a single pass. A scan that took minutes on a large tree now takes seconds. (Servers that don't support it fall back to the previous method automatically.)
- •Re-syncs only look at what changed. A scheduled job now asks the source for the files modified since its last run, so a re-sync where nothing changed finishes almost instantly rather than re-listing the entire tree. A full scan still runs at least daily so deletions are noticed.
Nothing about how you set up a job changes — existing jobs simply get faster and lighter on their next run.