Jobs
Back up tens of millions of objects — fast
Large S3-to-S3 backups now run in a flat, steady memory footprint, copy small files in parallel for dramatically faster syncs, survive a deploy mid-run, and tell you how long each run took.
Big buckets deserve big backups. This release makes S3-to-S3 replication ready for serious scale:
- •No size ceiling. A backup now streams its source-vs-destination comparison through the database instead of holding the whole object list in memory, so a job over tens of millions of objects runs in a small, steady footprint — the out-of-memory wall on very large buckets is gone.
- •Small files fly. Runs made of many small objects — logs, thumbnails, documents — now copy in parallel, scaling the number of transfers to the work at hand. In our own testing a large sync went from roughly 1.6 MB/s to 30+ MB/s on the same hardware.
- •Deploy-safe. If we ship an update while your backup is running, the job hands off cleanly and resumes on its own — a routine deploy no longer interrupts a long-running copy.
- •Know how long it took. Backup and migration completion emails now include a "Finished after" time (e.g. "1h 30m").
There's nothing to turn on — every replication and migration job is faster and sturdier automatically.