HF Datasets

HfDatasetsVFS mounts a Hugging Face Dataset repo at a prefix such as /ds/. All reads are lazy: only the bytes you actually cat/head get transferred. The TypeScript backend mirrors the Python one and returns identical results. For credential setup, see HF Datasets Setup.

Node only: the VFS ships in @struktoai/mirage-node. It needs no native binding, though, because it speaks the Hub API over the runtime’s own fetch. The opendal binding belongs to HF Buckets, which is a different product.

Node

pnpm add @struktoai/mirage-node

import { HfDatasetsVFS, MountMode, Workspace } from '@struktoai/mirage-node'

const dataset = new HfDatasetsVFS({
  repoId: 'AlienKevin/SWE-ZERO-12M-trajectories', // "namespace/dataset-name"
  token: process.env.HF_TOKEN, // optional for public datasets
  // Optional:
  // endpoint: 'https://huggingface.co',
  // revision: 'main',
  // keyPrefix: 'train/',
})

const ws = new Workspace({ '/ds/': dataset }, { mode: MountMode.READ })

console.log((await ws.shell('ls -lh /ds/')).stdoutText)
console.log((await ws.shell("find /ds/ -name '*.parquet' | wc -l")).stdoutText)
console.log((await ws.shell('cat /ds/README.md | head -n 20')).stdoutText)

HfDatasetsVFS takes repoId in namespace/dataset-name form plus an optional access token. Public datasets need no token.

Reading, not writing

This mount is read-only, the way a github mount is. A Hub write is a commit, and a POSIX write cannot say where a commit ends, so echo >, rm, cp and mv are refused here rather than silently making one commit per file. The hf CLI is the write half: hf download --local-dir puts a copy on a ram or disk mount, which is an ordinary writable filesystem, and hf upload sends it back as a single commit.

Listings under `fresh`

Under read: fresh, a cached listing is checked against the commit the revision resolves to: one small check per command (revision/{rev}?expand[]=sha), after which every cached listing of the mount at that commit is served. A mount whose revision is a full 40- or 64-hex commit sha is checked the same way, with that one small request per command: a branch or tag named like the sha could take the name, and mirage does not assume which one the Hub resolves. With a keyPrefix, the stored version is that commit joined with the prefix, so mounts of different subtrees sharing an index never share a version. See listings under fresh.

Filesystem Layout

Maps dataset repo files (README, parquet shards, etc.) to virtual paths under the mount prefix. Parquet shards stream lazily via byte-range reads.

Shell Commands

Every read command in HF Buckets’ set works here, as do the text processing and path utilities, which only read. What does not is the File Operations group: this mount is read-only, so rm and touch are refused, as is any command asked to write into it.

Getting Started

Access

AI Agents

VFS

CLIs

Watch

Runtimes

Node

Reading, not writing

Listings under `fresh`

Filesystem Layout

Shell Commands

​Node

​Reading, not writing

​Listings under fresh

​Filesystem Layout

​Shell Commands

Node

Reading, not writing

Listings under `fresh`

Filesystem Layout

Shell Commands