All updates
Migrations

Migrations built for millions of objects — and re-syncs that finish in seconds

A migration now runs at any tree size without exhausting a worker's memory, scans a server in seconds, and re-syncs by looking only at what actually changed.

A migration used to hold two things in the worker's memory at once: the entire list of files at the source, and the record of everything it had already copied. On a tree with millions of files that adds up to gigabytes — enough to run a worker out of memory before a byte moved. Both now stream through the database instead of the worker, so a job of tens of millions of objects runs in flat, predictable memory.

Two more improvements you'll feel immediately:

  • Scanning a server is now near-instant. Instead of walking a deep tree one directory at a time, we ask the machine for its whole file list in a single pass. A scan that took minutes on a large tree now takes seconds. (Servers that don't support it fall back to the previous method automatically.)
  • Re-syncs only look at what changed. A scheduled job now asks the source for the files modified since its last run, so a re-sync where nothing changed finishes almost instantly rather than re-listing the entire tree. A full scan still runs at least daily so deletions are noticed.

Nothing about how you set up a job changes — existing jobs simply get faster and lighter on their next run.