S3VFS mounts one bucket: every object becomes a file and every / in a key a
folder. It works with Amazon S3 and any S3-compatible server, such as MinIO,
Ceph or SeaweedFS. Set up credentials as in S3 setup.
Install
Configure
endpoint_url and, if the
server needs it, path_style=True. A mount takes MountMode.READ, WRITE or
EXEC.
Fields
Seconds a request may wait to connect, or between two reads of its answer, before it fails; a download that keeps flowing is never cut off.
AWS region requests are signed for and sent to. Unset leaves it to the AWS SDK’s environment variables and profile.
Named profile in the shared AWS config and credentials files that supplies credentials when no access key pair is given.
Endpoint URL of an S3-compatible server. Unset uses
AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL when set, else AWS’s endpoint for region; an endpoint set only in an AWS profile is not read.Address the bucket in the URL path (
<endpoint>/<bucket>/<key>) instead of as a subdomain of the endpoint.Key prefix inside the bucket that the mount treats as its root; keys outside it are not visible. Leading slashes are dropped and a trailing
/ is added.Content-Type stored with every object the mount writes. Unset sends none, so the store applies its own default.Authentication
Credentials come from the first of:aws_access_key_idandaws_secret_access_key, withaws_session_tokenfor temporary credentials.aws_profile, a profile in~/.aws/credentials.- The AWS credential chain: environment variables, then the instance or container role.
s3:ListBucket on the bucket, s3:GetObject to read, and
s3:PutObject and s3:DeleteObject to write. An object the credentials cannot
read answers Permission denied. Without s3:ListBucket, a missing file
answers Permission denied too, since S3 then hides which keys exist.
Layout
/s3/data/file.txt is the key data/file.txt. Folders come from the / in
keys; S3 stores no folder objects.
Behavior
- Key prefix:
key_prefix="users/42/"mounts only that part of the bucket, so/s3/notes.mdis the keyusers/42/notes.md.
Integration
S3 answers these commands through its own API, so fewer objects are read. See Integration.md5sum: the object’s ETag, when it is the MD5 (a single-part upload, unencrypted or under SSE-S3).find: one recursive listing.du: one recursive listing.ls -Randtree: one recursive listing.
VFS.md
The VFS.md an agent reads with S3 mounted at/<path>.
<mode> is the mount’s mode: read, write or exec. The commands shown are
those of a write mount.
- Rendered
- Raw
Virtual Filesystem
Supported Unix Commands for VFSUnix commands work on each path as on a local filesystem, as far as the path supports them:
Limitations of Unix Commands for VFSWhere a path differs from a local filesystem:
Run
man <cmd> or <cmd> --help for one command’s usage.