Move a Linux server into Amazon S3

Every Linux shop already has a way to do this, and it is usually a cron job wrapping a copy tool. It works until the tree gets big enough that walking it takes longer than the window, or until the box is rebooted mid-run and nobody notices for a week.

The harder problem is knowing what the state is. A copy that ran last night either finished or it did not, and the log says which — but "what is on the server and not in the bucket, right now" is a question a script cannot answer without doing the whole walk again.

1. 1.Add the server under Credentials → Servers with its address and an SSH key, and press Test Connection.
2. 2.Connect the AWS account — the one-click least-privilege role, or a key — and choose the destination bucket.
3. 3.Create a migration job with the source path. The first run walks the tree and copies it.
4. 4.Re-run on a schedule. Later runs carry only what changed, so the pass before cutover is minutes.
5. 5.Cut over when a run copies nothing, which means both sides agree.

• •The tree is walked with find over one connection, not by opening a directory at a time. On a deep tree over a slow link that is the difference between seconds and a long wait — and it is why the walk is not what makes a large migration slow.
• •Re-runs can go incremental on the walk itself, using a watermark from the previous run so only files modified since then are examined. A full scan is still taken periodically, because only a full scan can notice that something was deleted at the source.
• •Concurrency adapts to the server. Some SSH servers refuse beyond a certain number of channels; BucketPilot finds the ceiling the machine will accept rather than pinning a number and failing.
• •Symlinks and special files are not objects. What goes into the bucket is files; the shape of the tree becomes key prefixes.
• •One address to allow through the firewall — every connection comes from 3.111.107.194, so the SSH port need not be open to the internet.

• •No agent, and no package to install. SSH in, copy out, revoke the key when you are done.
• •Incremental reruns and, where the server supports it, an incremental walk as well.
• •A log per file, naming anything that failed and why.
• •Retries and resume — transient errors back off, and a run whose worker dies is continued rather than restarted.
• •A schedule so the server and the bucket stay in step until the switch.

Access is an SSH key you install and can remove; host keys are verified on first use on Pro and above. During a run, files stream from the server into the bucket and no copy of their contents is kept by BucketPilot.

Adding and testing a server is not plan-gated. Using one as a migration source is Pro, which the 14-day trial includes. Full detail: Security & privacy.

• •You are retiring a server, or moving its data somewhere applications can reach it directly.
• •You want the tree and the bucket in step while you migrate whatever reads from it.
• •You want a per-file record of the move rather than an exit code.

When it is not. If the server is staying and you want protection from deletion, you want the same job on a schedule as a backup, with the understanding that a backup never removes anything at the destination. If what you actually need is to search what is on the server, a server can be indexed and searched without moving anything.