> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mirage.strukto.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# VFS Integration

> Answer shell commands from a service's own search, queries and metadata.

A VFS can answer some commands from its service's search, queries or metadata
instead of reading files. `head` can show the first lines without downloading
the whole file.

Each capability is a set of methods a VFS overrides, and the commands they make
cheaper:

| Capability | Methods | Commands |
| - | - | - |
| [Search](https://docs.mirage.strukto.ai/home/integration/search.md) | `files_containing`, `lines_containing` | `grep`, `rg` |
| [Lines](https://docs.mirage.strukto.ai/home/integration/lines.md) | `first_lines`, `last_lines`, `line_count`, `follow` | `head`, `tail`, `wc -l`, `tail -f` |
| [Checksums](https://docs.mirage.strukto.ai/home/integration/checksums.md) | `checksum` | `md5sum`, `sha256sum` and other hash commands |
| [Walks](https://docs.mirage.strukto.ai/home/integration/walks.md) | `read_tree` | `find`, `du`, `ls -R`, `tree` |

Built-in VFSs already implement these where their service allows. A
[custom VFS](https://docs.mirage.strukto.ai/home/setup/vfs/new.md) overrides the methods it can answer.
Mirage still handles options and output, and reads or lists files whenever a
method returns `None`.

## Built-in VFSs

These tables list which built-in VFSs answer each command from their service.
A command or VFS not listed reads the files as usual.

### Search

A recursive `grep` or `rg` asks the service which files contain the text, and
reads only those files.

| VFS | Searches it speeds up | Files the search can miss |
| - | - | - |
| ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/gdrive-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=94291d56fedde3d667a564b1917de57a)[Google Drive](https://docs.mirage.strukto.ai/python/vfs/gdrive.md#integration) | Whole-word searches (`grep -rw`, `rg -w`) | Docs, Sheets and Slides files, Office files, PDFs, archives, the end of large files, recent edits |
| ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/gdocs-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=8f0b2acc75092c5e90345b99edbe7e8c)[Google Docs](https://docs.mirage.strukto.ai/python/vfs/gdocs.md#integration), ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/gsheets-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=4c029bc90a958b8d4867c1f7756755fc)[Sheets](https://docs.mirage.strukto.ai/python/vfs/gsheets.md#integration), ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/gslides-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=2a433b751118e00c76c7ab96ba8ced11)[Slides](https://docs.mirage.strukto.ai/python/vfs/gslides.md#integration) | Whole-word searches | Words only in fonts, links, formulas or titles; recent edits |
| ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/gcal-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=1904e1a1d24572d9a2663ae910cb5882)[Google Calendar](https://docs.mirage.strukto.ai/python/vfs/gcal.md#integration) | Whole-word searches | None |
| ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/dropbox-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=7701d8372130b5e8818af52655c6ecab)[Dropbox](https://docs.mirage.strukto.ai/python/vfs/dropbox.md#integration) | Whole-word searches | Office files, PDFs, archives, the end of large files, recent edits; every file's contents below the Professional plan |
| ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/box-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=70a2254be6ebf081ee4934b27d41dcf5)[Box](https://docs.mirage.strukto.ai/python/vfs/box.md#integration) | Whole-word searches | Office files, PDFs, archives, the end of large files, recent edits |
|  [OneDrive](https://docs.mirage.strukto.ai/python/vfs/onedrive.md#integration),  [SharePoint](https://docs.mirage.strukto.ai/python/vfs/sharepoint.md#integration) | Whole-word searches | Office files, PDFs, archives, the end of large files, recent edits |
| ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/slack-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=3179a7f44417e57a2d5ea310b04234f9)[Slack](https://docs.mirage.strukto.ai/python/vfs/slack.md#integration) | Whole-word searches, with a user token | Recent messages; bot names and link previews |
| ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/trello-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=85c19ea2096091fb3921f6e9e65e3943)[Trello](https://docs.mirage.strukto.ai/python/vfs/trello.md#integration) | Whole-word searches | Recent edits |
| ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/linear-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=e3b137d0009673120cbfb7f54318569f)[Linear](https://docs.mirage.strukto.ai/python/vfs/linear.md#integration) | Whole-word searches on issues; every search on other files | Recent issue edits |
| ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/gmail-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=31cc2874ffc7fed853ce07a178a70b16)[Gmail](https://docs.mirage.strukto.ai/python/vfs/gmail.md#integration) | Every search for ASCII text in messages. Attachments are always read and searched. | None |
|  [Email (IMAP)](https://docs.mirage.strukto.ai/python/vfs/email.md#integration) | Every search for ASCII text in messages. Attachments are always read and searched. | Parts of words, on a server that indexes whole words |
|  [GitHub](https://docs.mirage.strukto.ai/python/vfs/github.md#integration) | Whole-word searches on the default branch, with a credential | Recent pushes |
| ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/chroma-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=f88a09204bc193ee701e27df1d6bd492)[Chroma](https://docs.mirage.strukto.ai/python/vfs/chroma.md#integration) | Every search, except with `-i` | None |
| ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/qdrant-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=d0a9d59496aaf2e8ac8b33f5776f8830)[Qdrant](https://docs.mirage.strukto.ai/python/vfs/qdrant.md#integration) | Every search for ASCII text | None |
| ![](https://mintcdn.com/struktoai/smCV-z8Dromu2yrM/images/lancedb-logo.svg?fit=max\&auto=format\&n=smCV-z8Dromu2yrM\&q=85\&s=ed7108fc0fe317897d9d632d50530380)[LanceDB](https://docs.mirage.strukto.ai/python/vfs/lancedb.md#integration) | Every search | None |
| ![](https://mintcdn.com/struktoai/yJz6XpKlFIY24Ipv/images/ssh-logo.svg?fit=max\&auto=format\&n=yJz6XpKlFIY24Ipv\&q=85\&s=5b8b0b5b86d3dd4a2d7bd03dbf66a140)[SSH](https://docs.mirage.strukto.ai/python/vfs/ssh.md#integration) | Every search, on a server that runs commands | None |
|  [Machine Files](https://docs.mirage.strukto.ai/python/vfs/machine.md#integration) | Every search | None |
| ![](https://mintcdn.com/struktoai/wbn_tJeahvJMuT6I/images/sqlite-logo.svg?fit=max\&auto=format\&n=wbn_tJeahvJMuT6I\&q=85\&s=8cf1e3eb2c2d13057741075c271b8b1c)[SQL with SQLite](https://docs.mirage.strukto.ai/python/vfs/sql.md) | Every search for ASCII text | None |

* A search a VFS does not speed up reads the files itself, up to the mount's
  [`walk_limit`](https://docs.mirage.strukto.ai/home/integration/search#walk_limit), and says so on stderr
  when it stops there. It never leaves a file out silently. Some patterns and
  flags always read every file; see
  [Search](https://docs.mirage.strukto.ai/home/integration/search.md#files_containing).
* A file the service's search leaves out is not read, so `grep` does not
  print a match in it. To read every file instead, set the mount's
  [`walk_limit`](https://docs.mirage.strukto.ai/home/integration/search#walk_limit) to `-1` on `"@all"`.

### One-file search

`grep` and `rg` on one table file fetch only the rows that may match, not the
whole table. The output is the same as reading the file.

| VFS | File | How |
| - | - | - |
| ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/postgres-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=aca36d421c1424970f4f74eb0a3831b4)[PostgreSQL](https://docs.mirage.strukto.ai/python/vfs/postgres.md#integration) | `rows.jsonl` | A `LIKE` query |
| ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/mongodb-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=b2b0a224242b5d843a468f880febb2d9)[MongoDB](https://docs.mirage.strukto.ai/python/vfs/mongodb.md#integration) | `documents.jsonl` | A filter on the server |
| ![](https://mintcdn.com/struktoai/i_m5DIPdui2Jlopk/images/airtable-logo.svg?fit=max\&auto=format\&n=i_m5DIPdui2Jlopk\&q=85\&s=2b429baafe9b4e57a07cf90f0c4252f1)[Airtable](https://docs.mirage.strukto.ai/python/vfs/airtable.md#integration) | `records.jsonl` | A `filterByFormula` query |

### Lines

`head`, `tail`, `wc -l` and `tail -f` ask the service for the lines or the
count, without reading the whole file.

| VFS | Commands | How |
| - | - | - |
| ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/postgres-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=aca36d421c1424970f4f74eb0a3831b4)[PostgreSQL](https://docs.mirage.strukto.ai/python/vfs/postgres.md#integration) | `head -n`, `tail -n`, `wc -l`, `tail -f` | SQL queries (`LIMIT`, `COUNT(*)`); `tail -f` polls for new rows |
| ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/mongodb-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=b2b0a224242b5d843a468f880febb2d9)[MongoDB](https://docs.mirage.strukto.ai/python/vfs/mongodb.md#integration) | `tail -n`, `wc -l`, `tail -f` | Queries on the server; `tail -f` follows a change stream |
| ![](https://mintcdn.com/struktoai/i_m5DIPdui2Jlopk/images/airtable-logo.svg?fit=max\&auto=format\&n=i_m5DIPdui2Jlopk\&q=85\&s=2b429baafe9b4e57a07cf90f0c4252f1)[Airtable](https://docs.mirage.strukto.ai/python/vfs/airtable.md#integration) | `head -n` | Fetches only the first records |
| ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/notion-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=56d8acf991bbacdee6349192f8aa16ef)[Notion](https://docs.mirage.strukto.ai/python/vfs/notion.md#integration) | `head -n` | Renders only the first rows |
| ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/slack-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=3179a7f44417e57a2d5ea310b04234f9)[Slack](https://docs.mirage.strukto.ai/python/vfs/slack.md#integration), ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/discord-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=d18fb2b5008fbab5622dc98e1c487f46)[Discord](https://docs.mirage.strukto.ai/python/vfs/discord.md#integration) | `tail -f` | Each new message as it is posted |
| ![](https://mintcdn.com/struktoai/yJz6XpKlFIY24Ipv/images/ssh-logo.svg?fit=max\&auto=format\&n=yJz6XpKlFIY24Ipv\&q=85\&s=5b8b0b5b86d3dd4a2d7bd03dbf66a140)[SSH](https://docs.mirage.strukto.ai/python/vfs/ssh.md#integration) | `wc -l`, `tail -f` | The same command on the server |
|  [Machine Files](https://docs.mirage.strukto.ai/python/vfs/machine.md#integration) | `head -n`, `tail -n`, `wc -l` | The same command on the machine |

### Checksums

Hash commands print the hash the service stores, without reading the file.
Other files are hashed by reading them.

| VFS | Commands | Where the hash comes from |
| - | - | - |
| ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/s3-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=3a2c625e597531599eca1cdb67504e6b)[S3](https://docs.mirage.strukto.ai/python/vfs/s3.md#integration), [Cloudflare R2](https://docs.mirage.strukto.ai/python/vfs/r2.md#integration), [GCS](https://docs.mirage.strukto.ai/python/vfs/gcs.md#integration), [MinIO](https://docs.mirage.strukto.ai/python/vfs/minio.md#integration), [Ceph](https://docs.mirage.strukto.ai/python/vfs/ceph.md#integration), [DigitalOcean Spaces](https://docs.mirage.strukto.ai/python/vfs/digitalocean.md#integration), [Wasabi](https://docs.mirage.strukto.ai/python/vfs/wasabi.md#integration) | `md5sum` | The object's ETag, for a single-part upload |
| ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/gdrive-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=94291d56fedde3d667a564b1917de57a)[Google Drive](https://docs.mirage.strukto.ai/python/vfs/gdrive.md#integration) | `md5sum`, `sha1sum`, `sha256sum` | Drive's stored hash; not for Docs, Sheets or Slides |
| ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/box-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=70a2254be6ebf081ee4934b27d41dcf5)[Box](https://docs.mirage.strukto.ai/python/vfs/box.md#integration) | `sha1sum` | Box's stored hash |
| ![](https://mintcdn.com/struktoai/TeyGHfd6VCo9It4B/images/huggingface-logo.svg?fit=max\&auto=format\&n=TeyGHfd6VCo9It4B\&q=85\&s=fa988300c43fc55b54048f1c898796b2)[HF Models](https://docs.mirage.strukto.ai/python/vfs/hf_models.md#integration), ![](https://mintcdn.com/struktoai/TeyGHfd6VCo9It4B/images/huggingface-logo.svg?fit=max\&auto=format\&n=TeyGHfd6VCo9It4B\&q=85\&s=fa988300c43fc55b54048f1c898796b2)[Datasets](https://docs.mirage.strukto.ai/python/vfs/hf_datasets.md#integration), ![](https://mintcdn.com/struktoai/TeyGHfd6VCo9It4B/images/huggingface-logo.svg?fit=max\&auto=format\&n=TeyGHfd6VCo9It4B\&q=85\&s=fa988300c43fc55b54048f1c898796b2)[Spaces](https://docs.mirage.strukto.ai/python/vfs/hf_spaces.md#integration) | `sha256sum` | The LFS hash, for files in LFS |
| ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/nextcloud-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=8dc313df81e8df7e4a821c131e599348)[Nextcloud](https://docs.mirage.strukto.ai/python/vfs/nextcloud.md#integration) | `md5sum`, `sha1sum`, `sha256sum` | The hash the uploading client stored |
| ![](https://mintcdn.com/struktoai/yJz6XpKlFIY24Ipv/images/ssh-logo.svg?fit=max\&auto=format\&n=yJz6XpKlFIY24Ipv\&q=85\&s=5b8b0b5b86d3dd4a2d7bd03dbf66a140)[SSH](https://docs.mirage.strukto.ai/python/vfs/ssh.md#integration) | `md5sum` and the `sha*sum` commands | The same command on the server |
|  [Machine Files](https://docs.mirage.strukto.ai/python/vfs/machine.md#integration) | `md5sum` and the `sha*sum` commands | The same command on the machine |

### Walks

`find`, `du`, `ls -R`, `tree`, `grep -r` and `rg` get the whole folder tree in
one call, not one listing per folder.

| VFS | How |
| - | - |
| ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/s3-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=3a2c625e597531599eca1cdb67504e6b)[S3](https://docs.mirage.strukto.ai/python/vfs/s3.md#integration) and every S3-compatible store ([Cloudflare R2](https://docs.mirage.strukto.ai/python/vfs/r2.md#integration), [GCS](https://docs.mirage.strukto.ai/python/vfs/gcs.md#integration), [MinIO](https://docs.mirage.strukto.ai/python/vfs/minio.md#integration), [Ceph](https://docs.mirage.strukto.ai/python/vfs/ceph.md#integration), [DigitalOcean Spaces](https://docs.mirage.strukto.ai/python/vfs/digitalocean.md#integration), [Wasabi](https://docs.mirage.strukto.ai/python/vfs/wasabi.md#integration), [Alibaba OSS](https://docs.mirage.strukto.ai/python/vfs/aliyun.md#integration), [Backblaze B2](https://docs.mirage.strukto.ai/python/vfs/backblaze.md#integration), [OCI Object Storage](https://docs.mirage.strukto.ai/python/vfs/oci.md#integration), [QingStor](https://docs.mirage.strukto.ai/python/vfs/qingstor.md#integration), [Scaleway](https://docs.mirage.strukto.ai/python/vfs/scaleway.md#integration), [SeaweedFS](https://docs.mirage.strukto.ai/python/vfs/seaweedfs.md#integration), [Supabase Storage](https://docs.mirage.strukto.ai/python/vfs/supabase.md#integration), [Tencent COS](https://docs.mirage.strukto.ai/python/vfs/tencent.md#integration)) | One recursive listing |
| ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/gdrive-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=94291d56fedde3d667a564b1917de57a)[Google Drive](https://docs.mirage.strukto.ai/python/vfs/gdrive.md#integration) | One query per folder level |
| ![](https://mintcdn.com/struktoai/AvU64XZuiQaceY0a/images/dropbox-logo.svg?fit=max\&auto=format\&n=AvU64XZuiQaceY0a\&q=85\&s=7701d8372130b5e8818af52655c6ecab)[Dropbox](https://docs.mirage.strukto.ai/python/vfs/dropbox.md#integration),  [OneDrive](https://docs.mirage.strukto.ai/python/vfs/onedrive.md#integration),  [SharePoint](https://docs.mirage.strukto.ai/python/vfs/sharepoint.md#integration), ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/nextcloud-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=8dc313df81e8df7e4a821c131e599348)[Nextcloud](https://docs.mirage.strukto.ai/python/vfs/nextcloud.md#integration), ![](https://mintcdn.com/struktoai/TeyGHfd6VCo9It4B/images/huggingface-logo.svg?fit=max\&auto=format\&n=TeyGHfd6VCo9It4B\&q=85\&s=fa988300c43fc55b54048f1c898796b2)[HF Buckets](https://docs.mirage.strukto.ai/python/vfs/hf_buckets.md#integration) | One recursive listing |
|  [GitHub](https://docs.mirage.strukto.ai/python/vfs/github.md#integration) | One recursive tree listing |
|  [RAM](https://docs.mirage.strukto.ai/python/vfs/ram.md#integration), ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/redis-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=1f81a02282061ad43e3aa8f86b01434a)[Redis](https://docs.mirage.strukto.ai/python/vfs/redis.md#integration) | One scan of the store's keys |
| ![](https://mintcdn.com/struktoai/CsxMCKUyfoQQJYHB/images/mongodb-logo.svg?fit=max\&auto=format\&n=CsxMCKUyfoQQJYHB\&q=85\&s=b2b0a224242b5d843a468f880febb2d9)[GridFS](https://docs.mirage.strukto.ai/python/vfs/gridfs.md#integration) | One `fs.files` query |
| ![](https://mintcdn.com/struktoai/wbn_tJeahvJMuT6I/images/sqlite-logo.svg?fit=max\&auto=format\&n=wbn_tJeahvJMuT6I\&q=85\&s=8cf1e3eb2c2d13057741075c271b8b1c)[SQL](https://docs.mirage.strukto.ai/python/vfs/sql.md) | One query |
| ![](https://mintcdn.com/struktoai/yJz6XpKlFIY24Ipv/images/ssh-logo.svg?fit=max\&auto=format\&n=yJz6XpKlFIY24Ipv\&q=85\&s=5b8b0b5b86d3dd4a2d7bd03dbf66a140)[SSH](https://docs.mirage.strukto.ai/python/vfs/ssh.md#integration) | One `find` on the server |
|  [Machine Files](https://docs.mirage.strukto.ai/python/vfs/machine.md#integration) | One `find` on the machine |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.