Search across every S3 bucket at once

The S3 console filters the page of the prefix you are looking at. That is already thin for one bucket. Across thirty buckets in four accounts it stops being a search problem and becomes a scripting problem: list everything, somewhere you can query it, and keep that somewhere current.

So people write the script. It works, and then it becomes the thing that is out of date — because listing thirty buckets nightly is slow, and nobody notices when it stops.

1. 1.Connect each account — one credential per AWS account, plus Cloudflare R2 or Google Cloud Storage on Business, and any servers you have.
2. 2.Index the buckets you want searchable. Press Index and BucketPilot walks the bucket in the background; a few million objects take minutes.
3. 3.Keep the index current — hourly or daily auto-sync, or from S3 Inventory for very large buckets, and uploads made through BucketPilot land in the index as they happen.
4. 4.Search the whole workspace at once, with wildcards and filters, and sort by any column.
5. 5.Act on what you find — open it, download it, copy it somewhere else, or change its storage class.

• •It is an index, not a live list. Search answers from what was walked, which is why it is fast and why freshness is a setting rather than a guarantee. A bucket on hourly auto-sync is at most an hour behind; a bucket nobody has re-indexed is as old as its last run.
• •There is a ceiling per plan on how many objects are indexed across the workspace: one million on Starter, ten million on Pro, fifty million on Business. Hitting it pauses new indexing rather than breaking what is already there.
• •Very large buckets can be read from S3 Inventory instead of listed, which is an AWS feature and so applies to S3 buckets only. R2 and Google Cloud Storage buckets are always listed directly.
• •Servers are in the same index. A Linux or Windows machine connected over SSH can be indexed and searched next to the buckets, which is usually how "where is that file" gets answered honestly.
• •Filenames are what is matched. Search is over object names and their attributes — size, type, storage class, folder — not the contents of the files.

• •Wildcards and filters by type, size and storage class, across every indexed bucket at once.
• •Duplicates matched on content, with the groups wasting the most listed first and a link to every copy.
• •Folder sizes and per-object cost, so "what is in here" comes with "and what it costs".
• •The same search over servers you have connected, not only buckets.
• •A REST API for the same queries, if the answer needs to go somewhere else.

The index holds metadata — object names, sizes, timestamps, storage classes and the folder they sit in. Object contents are not read and not stored. That is the whole reason a search can cover tens of millions of objects without holding your data.

Each account is connected with its own least-privilege credential, and who can see which bucket is controlled per bucket and per folder. Full detail: Security & privacy and Access levels.

• •You have more buckets than you can hold in your head, in more than one account.
• •Someone asks where a file is and the answer currently requires a script.
• •You are auditing what exists before a migration, a clean-up or a cost exercise.

When it is not. If you need to search inside documents rather than across their names, this is not that — it indexes metadata, deliberately. And if you need an answer that is accurate to the second, a list against the bucket is the honest way to get it; an index trades a little freshness for being able to answer at all.