# SkillPatch skill: huggingface-spaces This skill guides agents in building, deploying, and maintaining applications on Hugging Face Spaces across all three SDK types (Gradio, Docker, Static). It covers hardware tier selection including ZeroGPU and dedicated GPUs, CLI setup, searching for existing demos, and debugging Spaces. It is intended as a companion to the hf-cli skill. You (the agent) were given this URL and asked to use this skill. This is a **public** skill — no SkillPatch account, API key, or prior setup is required. Two ways to use it: **1. Use it right now** — the skill's complete file tree (SKILL.md + all reference files) is inlined below; read `SKILL.md` first, then follow it, consulting the other files as it directs. **2. Install the exact package onto disk** (recommended if you can run a shell — this reproduces the skill byte-for-byte, including any binary assets that can't be inlined): ```bash mkdir -p .claude/skills/huggingface-spaces curl -sSL https://skillpatch.dev/install_skill/huggingface-spaces | tar -xz -C .claude/skills/ ``` (`.claude/skills/` is Claude Code's convention; use whatever directory your agent loads skills from.) --- ## Skill files (9) - `SKILL.md` - `references/buckets.md` - `references/debugging.md` - `references/gradio.md` - `references/grants.md` - `references/inference-providers.md` - `references/known-errors.md` - `references/requirements.md` - `references/zerogpu.md` ### `SKILL.md` ````markdown --- name: huggingface-spaces description: Build, deploy, and maintain applications on Hugging Face Spaces — Gradio / Docker / Static SDKs, ZeroGPU and dedicated hardware, model loading, debugging, buckets, inference providers, community grants. Use whenever the user asks to create or host an app on Hugging Face, port code onto ZeroGPU, fix a Space that won't build or run, or otherwise work with `hf spaces …`, `@spaces.GPU`, Space README frontmatter, or the `spaces` Python package. --- # Hugging Face Spaces Hugging Face Spaces host machine-learning applications. There are 1M+ today; each Space is a git repo. This skill covers creating, building, debugging, and maintaining them. ## 0. Getting ready Before anything else: 1. Check the `hf` CLI is installed: `which hf`. If not, `pip install -U huggingface_hub`. 2. Check the user is logged in: `hf auth whoami`. If not, ask them to run `! hf auth login` in this session — they'll need a write-scoped token from https://huggingface.co/settings/tokens. 3. Note `whoami`'s `canPay` and `isPro` flags — they gate hardware choices below. The `hf-cli` skill teaches an agent every `hf` command and is the recommended companion to this one. Install it with `hf skills add hf-cli` (add `--claude --global` to install for Claude Code as well, user-level). ## 1. What a Space is A Space is a git repo with three possible SDKs: - **Gradio** — most Spaces. Python, fast iteration, supports ZeroGPU. - **Docker** — arbitrary container. Use when you need a non-Python stack or a pre-built template (Streamlit, Argilla, Shiny, etc. — full list at https://huggingface.co/docs/hub/spaces-sdks-docker). Does **not** support ZeroGPU. - **Static** — plain HTML, or a React/Svelte/Vue project built at deploy time. Use for in-browser ML (transformers.js / WebGPU / WebAssembly / onnxruntime-web), project pages, interactive reports, or Spaces that orchestrate other Spaces. No hardware needed. ### Hardware tiers Free, no creator cost: **`cpu-basic`** and **`zero-a10g`** (ZeroGPU). Static Spaces are also free and don't need hardware. **`cpu-basic`** — 2 vCPU / 16 GB. For data viz, API-proxy Spaces, small CPU-bound models. **ZeroGPU (`zero-a10g`)** — dynamic, per-request GPU allocation on NVIDIA RTX PRO 6000 Blackwell (sm_120). Two sizes: `large` (half MIG, 48 GB, 1× quota) and `xlarge` (full, 96 GB, 2× quota). Free for the Space creator; Space visitors consume their own daily quota (~5 min free / 40 min Pro / 60 min Enterprise). **Gradio-only**, **PyTorch-first**. Requires the creator to be on a PRO / Team / Enterprise plan. **Dedicated GPU** (T4, L4, A10G, L40S, A100, H200) — billed to the Space creator by the hour. List + pricing: `hf spaces hardware`. Only the creator can attach these, and only if `canPay=True`. Use when ZeroGPU genuinely doesn't fit — non-PyTorch main model with heavy init, very-large-model long-context inference, etc. If a non-PRO user has a use case that wants ZeroGPU, you can still build it: create a `cpu-basic` Space, code the app for ZeroGPU, push, then request a community grant. See [`references/grants.md`](references/grants.md). For the authoritative reference: https://huggingface.co/docs/hub/spaces-overview ## 2. Look for an existing demo first Before deciding how to build anything, search for prior art: ```bash hf spaces search "" --sdk gradio --limit 10 ``` If someone has built a similar Space, read its `app.py` and `requirements.txt` — that gives you the working pattern. Saves a lot of blind iteration. Mention to the user what you found before committing to an approach. ## 3. Decide SDK and hardware Follow the user's explicit request first. If they were vague: - **Default for a public ML demo**: Gradio + ZeroGPU. Use this unless something below applies. - **The model's only inference path is non-PyTorch** (ONNX / TF / JAX / vLLM as the MAIN model, with heavy init): dedicated GPU. - But: marginal non-torch tools (a small ONNX preprocessor, a TF utility) inside a torch-main pipeline are fine on ZeroGPU. The hijack only patches torch; init the non-torch lib inside `@spaces.GPU` and pay the short per-call init cost. - **Tiny / CPU-bound model, or API-proxy Space**: `cpu-basic` (`hardware`-free isn't applicable to Gradio). - **Browser-side ML or project page**: Static. - **Container with non-Python stack**: Docker. ### Sourcing the model - **GitHub repo** — clone locally to read structure. If it already has a Gradio demo, the minimal viable path is to adapt it onto ZeroGPU (see [`references/zerogpu.md`](references/zerogpu.md)). Otherwise: read the README + inference code, prefer the PyTorch path, estimate VRAM (bf16 ≈ `params_B × 2` GB; 48 GB fits ≤24B params at bf16, or much larger with quantization — see [`references/zerogpu.md`](references/zerogpu.md) for quantization on ZeroGPU). - **HF model repo** — read its README, follow any linked GitHub. - **Paper / blog post** — look for an official or unofficial implementation. Don't reimplement unless trivial or the user explicitly asks. - **Vague request** — search Spaces first; surface results. If the model genuinely won't fit, check **Inference Providers** as an alternative: see [`references/inference-providers.md`](references/inference-providers.md). This avoids hosting the model at all. ## 4. Create the Space ```bash hf repos create / --type space --space-sdk \ [--flavor zero-a10g|cpu-basic|] \ [--secrets KEY=val] [--env KEY=val] \ --public|--private|--protected \ --exist-ok ``` - `--space-sdk` is required. - `--flavor` selects hardware. `zero-a10g` is the (legacy) identifier for ZeroGPU. Omit for `cpu-basic`. Run `hf spaces hardware` for the full paid list and pricing. - Visibility: `--public` (anyone can view), `--private` (only you), `--protected` (app is reachable but git repo / Files tab is private). - `--secrets KEY=val` becomes an environment variable inside the Space and is **not** visible to visitors. Use for API keys, gated-repo tokens (`HF_TOKEN=hf_…`), etc. Can also be set later via `hf spaces secrets set KEY=val`. - `--env KEY=val` is **visible to visitors** — use only for non-sensitive config (`GRADIO_SSR_MODE=false`, `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, etc.). > Note: `hardware:` in the README YAML is silently ignored — hardware is only set via `--flavor` at creation, or later via `hf spaces settings --hardware `. ## 5. Build the app The Space now exists at `https://huggingface.co/spaces//` but is empty. ### README.md frontmatter Always required: ```yaml --- title: ... emoji: 🚀 # pick something representative colorFrom: blue # red|yellow|green|blue|indigo|purple|pink|gray (only these) colorTo: indigo sdk: gradio # gradio | docker | static sdk_version: 6.15.1 # latest stable unless you have a reason* app_file: app.py # gradio only (docker / static use Dockerfile / index.html) short_description: ... # ≤ 60 chars (server rejects longer) python_version: "3.12" # ZeroGPU officially supports 3.10.13 and 3.12.12 startup_duration_timeout: 30m # default; bump to 1h for big LLMs / heavy downloads --- ``` \* Reasons to use an older Gradio: a custom component pins it, or you're adapting an existing demo and don't want to rewrite for 5.x→6.x breaking changes. If you need a 5.x, pick `5.50.0` (latest of the series; still supports custom components). All frontmatter options: https://huggingface.co/docs/hub/spaces-config-reference ### Minimal ZeroGPU Gradio app ```python import spaces # MUST come before torch / diffusers / transformers import torch import gradio as gr from diffusers import DiffusionPipeline pipe = DiffusionPipeline.from_pretrained("", torch_dtype=torch.bfloat16).to("cuda") @spaces.GPU(duration=60) def generate(prompt): return pipe(prompt).images[0] gr.Interface(fn=generate, inputs=gr.Text(), outputs=gr.Image()).launch() ``` Three rules — full treatment in [`references/zerogpu.md`](references/zerogpu.md): 1. **`import spaces` before torch / any CUDA-touching import.** It monkey-patches `torch.cuda.*`; once CUDA is initialized in the main process, it's too late. 2. **Load the model at module scope, `.to("cuda")` eagerly.** ZeroGPU intercepts the call, packs weights to disk, and streams them into VRAM on the first `@spaces.GPU` entry. Lazy loading inside the decorator costs every user. 3. **Decorate the function Gradio binds.** Estimate `duration` to the realistic worst case (smaller = higher queue priority and tighter quota check). For input-dependent runtime, pass a callable. ### requirements.txt Short version: - **Do NOT list**: `gradio`, `spaces`, `huggingface_hub` (preinstalled and platform-managed; pinning them causes resolution failures or silently breaks the ZeroGPU runtime). - **Do list if you use them**: `torchvision`, `torchaudio` (not preinstalled), plus everything else (`diffusers`, `transformers`, `accelerate`, `sentencepiece`, …). - ZeroGPU only accepts torch `2.8.0`, `2.9.1`, `2.10.0`, `2.11.0`. Default to leaving torch unpinned (the runtime preinstalls the latest). Only pin when a dep forces it. - For prebuilt CUDA-extension wheels (`flash_attn`, `xformers`, `pytorch3d`, `nvdiffrast`, `diff_gaussian_rasterization`, `torchmcubes`): use the prebuilt Blackwell wheels at `https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/tree/main/wheels`. Full mapping + caveats in [`references/requirements.md`](references/requirements.md). ### Per-SDK depth - **Gradio patterns** (themes, `gr.Examples`, streaming, custom HTML components, `gr.Server`): [`references/gradio.md`](references/gradio.md). - **Docker**: https://huggingface.co/docs/hub/spaces-sdks-docker. Examples: `hf spaces list --filter docker`. - **Static**: https://huggingface.co/docs/hub/spaces-sdks-static. For built SPAs, set `app_build_command: npm run build` and `app_file: dist/index.html` in frontmatter. - **ZeroGPU specifics** (decorator semantics, sizing, AoTI, generators, concurrency, pickle / `gr.State` across the worker boundary): [`references/zerogpu.md`](references/zerogpu.md) — read this whenever the Space targets ZeroGPU. ## 6. Iterate on the Space, not locally Try to build a release candidate from the user quest locally and push it — then use the live URL as your test loop. The Space environment is the only one that matters; do not try to test locally. `python3 -m py_compile app.py` is the maximum local check worth doing before pushing. Once pushed, pick the cheapest update mechanism for each change — hot-reload for pure Python edits, `hf upload` for code-only files hot-reload can't touch, full rebuild only when `requirements.txt` / `Dockerfile` / README frontmatter actually changed. Full ladder + footguns (hot-reload poisoning factory reboot, runtime.sha lag, etc.) in [`references/debugging.md`](references/debugging.md). ## 7. Verify Don't trust `RUNNING` alone — the app can be running but broken. Four steps, in order: **A. Alive?** Stage + hardware: ```bash hf spaces info / --expand runtime ``` **B. Logs clean post-boot?** Read the run log to confirm startup finished without warnings or silent fallbacks: ```bash hf spaces logs / --tail 200 ``` Look for model-load completion, no import warnings, no "falling back to CPU" / dtype downgrade messages, no `RUNNING` masking a half-broken app. **C. API actually responds.** With logs still tailing in another terminal (`hf spaces logs / --follow`), call the endpoint: ```python from gradio_client import Client, handle_file import os c = Client("/", token=os.environ["HF_TOKEN"], httpx_kwargs={"timeout": 600}) print(c.view_api()) # discover endpoints — don't guess result = c.predict(..., api_name="/generate") ``` **D. Sniff output AND logs.** HTTP 200 ≠ correct output. Check both: ```python head = open(result, "rb").read(16) # glTF / \x89PNG / RIFF…WEBP / RIFF…WAVE / [4:8]==b"ftyp" → png/jpg/webp/wav/mp4 ``` And look at the run log emitted during the call — silent fallbacks (model snapping to a different size, missing optional dep, dtype downgrade) only show up there. Full smoke-test patterns (streaming endpoints, OAuth-gated Spaces, `gr.Server` custom routes): [`references/debugging.md`](references/debugging.md). ## 8. Permanent storage (buckets) Spaces are stateless — `/data` is wiped on restart. If the Space needs to persist user uploads, generations, logs, or interact with a long-lived store, mount a **bucket**: ```bash hf buckets create / # --private optional hf spaces volumes set / -v hf://buckets//:/data # read-write at /data ``` Buckets are paid storage; check `canPay` and confirm with the user. Full patterns (read-fast / write-durable, public bucket URLs, model-cache anti-pattern): [`references/buckets.md`](references/buckets.md). ## 9. When things break Order of operations: 1. Read the logs: `hf spaces logs --build --follow` (build error) or `hf spaces logs --follow` (runtime error). Find the **first** error, not the last. 2. Grep [`references/known-errors.md`](references/known-errors.md) for the error string. Check if this is a known issue before trying your own fix — most common ZeroGPU / Gradio / dependency errors have a 1–2 line fix there. 3. Iterate using the cheapest rung from [`references/debugging.md`](references/debugging.md). The vast majority of issues resolve with log-reading + smoke-test loops; interactive dev mode + SSH is a heavy-hammer last resort. If you solve an error that wasn't in the known-errors list, suggest the user PR it back to this skill so future runs benefit. --- ## Reference index | When to read | File | |---|---| | **How ZeroGPU works** + correct patterns (decorator, sizing, pickle, generators, real-time, AoTI) | [`references/zerogpu.md`](references/zerogpu.md) | | **Iterate + debug**: logs, rung ladder, smoke testing (and dev mode + SSH as a last resort) | [`references/debugging.md`](references/debugging.md) | | **Error-string lookup** — the single place for all error symptoms (Spaces, ZeroGPU, Gradio, deps) | [`references/known-errors.md`](references/known-errors.md) | | Pinning deps, picking wheels, torch-family alignment | [`references/requirements.md`](references/requirements.md) | | `gr.Examples` caching, themes, custom HTML components, `gr.Server` | [`references/gradio.md`](references/gradio.md) | | Persistent storage, public bucket URLs | [`references/buckets.md`](references/buckets.md) | | Community grant requests (non-PRO needing ZeroGPU) | [`references/grants.md`](references/grants.md) | | Provider proxy (zero-VRAM big LLM via Cerebras / Fireworks / Together / etc.) | [`references/inference-providers.md`](references/inference-providers.md) | ```` ### `references/buckets.md` ````markdown # Persistent storage with Buckets Spaces are stateless. All data is wiped on restart / rebuild. For state that must survive (user uploads, generations, dynamic feeds, logs, growing databases): mount an HF **Bucket** — S3-like object storage living at `hf://buckets//`. Buckets are paid (per-TB storage). Check `whoami.canPay` and confirm with the user before creating one. Pricing + free tier: https://huggingface.co/storage. Full docs: https://huggingface.co/docs/hub/storage-buckets. ## Create + attach ```bash hf buckets create / # --private optional hf spaces volumes set / -v hf://buckets//:/data ``` After this, writes to `/data/` in the Space are durable. Reads come from the bucket via the Xet storage backend. To make the bucket files publicly addressable: leave the bucket public. Public bucket files are served at `https://huggingface.co/buckets///resolve/` (HTTP 302 redirect to a signed CDN URL). The Space writes once and the public URL works forever — no streaming proxy needed. ## Write-durable, read-fast pattern For a feed-style Space (e.g. a community jam where users save generations and browse a public timeline), don't re-scan disk on every request. Module-level disk scan → in-memory list → every write appends to both: ```python import os, json, uuid from datetime import datetime, timezone BUCKET_ID = "/" BUCKET_URL = f"https://huggingface.co/buckets/{BUCKET_ID}/resolve" _feed = [] def _load_feed(): root = "/data/songs" if not os.path.isdir(root): return for sid in os.listdir(root): meta = f"{root}/{sid}/meta.json" if os.path.isfile(meta): _feed.append(json.load(open(meta))) _feed.sort(key=lambda s: s["created_at"], reverse=True) _load_feed() # one scan at startup @app.api(name="save", time_limit=60) def save(audio_bytes: bytes, title: str): sid = uuid.uuid4().hex[:12] d = f"/data/songs/{sid}"; os.makedirs(d, exist_ok=True) open(f"{d}/audio.wav", "wb").write(audio_bytes) meta = {"id": sid, "title": title, "url": f"{BUCKET_URL}/songs/{sid}/audio.wav", "created_at": datetime.now(timezone.utc).isoformat()} json.dump(meta, open(f"{d}/meta.json", "w")) _feed.insert(0, meta) # cache stays current — no re-scan return meta @app.api(name="feed", concurrency_limit=10) def feed(): return _feed[:50] # zero disk I/O ``` Reference Space using this pattern: https://huggingface.co/spaces/victor/ace-step-jam ## Anti-pattern: bucket as model-weights cache **Do NOT** `snapshot_download(..., local_dir="/data/weights")` and load checkpoints from there. Bucket I/O is S3-paced; reading a 22 GB `safetensors` from `/data` during `from_pretrained` stalls past any `@spaces.GPU` duration cap. For model weights, let HF Hub re-download to local container disk on each cold start. With `HF_HUB_ENABLE_HF_TRANSFER=1` (set in the runtime by default) this is fast — typically much faster than streaming the same bytes through bucket I/O at request time. Bucket I/O is fine for occasional metadata reads (the feed pattern above) or saving user information. It is *not* fine as the path your model loader streams gigabytes through every cold start. ## Cache redirects `/home/user/.cache` is read-only on ZeroGPU. Redirect transient caches at the top of `app.py`, before any library import that uses them: ```python import os os.environ.setdefault("HF_HOME", "/data/.cache/huggingface") # or /tmp on non-bucket Spaces os.environ.setdefault("HF_MODULES_CACHE", "/tmp/hf_modules") os.environ.setdefault("MPLCONFIGDIR", "/tmp/matplotlib") ``` Missing redirections fail silently or at first matplotlib / transformers / diffusers import. ## Write access from the Space The Space's `HF_TOKEN` secret needs write permission on the bucket. Set via Settings → Secrets in the Space UI, or `hf spaces secrets set HF_TOKEN=`. ## Security note Public bucket files are publicly accessible forever at their resolve URL. **Don't write PII** to a public bucket. If you need durable but private storage (e.g. per-user history requiring an HF login), keep the bucket private and gate reads through your Space's own auth. ```` ### `references/debugging.md` ````markdown # Debugging and iteration How to iterate cheaply, read logs, and smoke-test. Dev mode + SSH exists as a last resort (covered at the bottom). ## The rung ladder Pick the cheapest update mechanism that fits the change. Going one rung too high wastes 30 s – 15 min per cycle. | Rung | When | Command | Cost | |---|---|---|---| | 1. Hot-reload | Pure Python edit on a Gradio Space (SDK 6.1+), **no new deps** | `hf spaces hot-reload -f app.py` | seconds, no rebuild | | 2. `hf upload` | Code-only change hot-reload can't handle (`gr.Server`, Streamlit, Docker entrypoint, non-Python file) | `hf upload . --include ''` | 30–90 s app restart | | 3. Full rebuild | `requirements.txt`, `Dockerfile`, README frontmatter, or hardware change | `hf upload . && hf spaces logs --build --follow` | 1–15 min | | 4. Factory reboot | Container in inconsistent state (broken pip env, etc.) | `hf spaces restart --factory-reboot` | full rebuild + cold start | ### How hot-reload works `hf spaces hot-reload -f app.py` patches the running Python process in place via `jurigged` (vendored inside the `spaces` package), then commits the change to the repo with a hot-reload marker that tells the platform to skip its usual restart. Seconds, no rebuild. Applies cleanly to function-body changes and new top-level symbols. Does **not** rerun module-level imports or one-time init — the model was loaded when `app.py` first ran and jurigged won't re-execute that load. New pip deps, README frontmatter, `Dockerfile`, and hardware changes need a full rebuild (rung 3). Independent of dev mode. Marked experimental in the CLI. Requires Gradio SDK 6.1+. ### Footguns - **Hot-reload poisons factory reboot.** A commit that's only a hot-reload leaves runtime metadata that's valid only while the hot process is alive. `--factory-reboot` on top of one can fail with `fatal: could not read Username for 'https://huggingface.co'`. Recovery: push any normal `hf upload` commit (even a one-line no-op) first, then restart. - **`runtime.sha` lags repo SHA on restart.** `hf upload` succeeds → repo updates → `hf spaces info` keeps reporting the *previous* commit's SHA under `runtime` for several minutes while the new container loads. Poll `runtime.sha`, not just `stage`, and don't issue another restart until it flips. - **Concurrent uploads or restart-while-uploading collide.** Wait for one to finish. - **"Let me try locally first"** for anything that depends on the Space's Python / torch / CUDA env. The Space environment is the only one that matters. `python3 -m py_compile app.py` is the maximum local check worth doing before pushing. ## Reading logs ```bash hf spaces info --expand runtime # stage at a glance hf spaces logs --build --follow # build log, live hf spaces logs --follow # run log, live hf spaces logs --build --tail 500 # bigger window — default is small ``` Find the **first** error in the build log, not the last. Cascading errors after the first are noise. State machine (terminal states in bold): ``` BUILDING → APP_STARTING → RUNNING ↘ RUNTIME_ERROR ↘ BUILD_ERROR ↘ CONFIG_ERROR ``` For stage-specific lookups, see [`known-errors.md`](known-errors.md). ## Smoke-test patterns A Space isn't done until a `gradio_client` call against the live URL exercises the endpoint end-to-end. Four steps in order — keep `hf spaces logs --follow` running in another terminal throughout, so any silent fallback (model snapping to a different size, missing optional dep, dtype downgrade) surfaces. ### A. Alive? ```bash hf spaces info --expand runtime --format json \ | python3 -c "import json,sys; r=json.load(sys.stdin)['runtime']; \ print(r['stage'], r.get('hardware','?'))" # expect: RUNNING zero-a10g ``` If `requested_hardware` is `cpu-basic` when you wanted GPU, your `--flavor` was rejected silently. Fix with `hf spaces settings --hardware zero-a10g`. ### B. Logs clean post-boot? ```bash hf spaces logs --tail 200 ``` Confirm the model finished loading, no import warnings, no "falling back to CPU" / dtype-downgrade messages, no failing health checks the platform forgave. Do this before calling the API — many silent failures (a config typo loading the wrong model, a missing optional dep, a one-time init that errored but didn't crash boot) are only visible here. ### C. API actually functions? Default — sync `gr.Interface` / `gr.Blocks` / `gr.ChatInterface` / `gr.Server` `@app.api`: ```python from gradio_client import Client, handle_file import os c = Client("/", token=os.environ["HF_TOKEN"], httpx_kwargs={"timeout": 600}) # ≥ @spaces.GPU duration + 60s print(c.view_api()) # discover endpoints — don't guess api_name result = c.predict( handle_file("test.png"), # file inputs need handle_file() "short prompt", api_name="/generate", # matches @app.api(name=...) or the function name ) ``` **Streaming endpoints** (function uses `yield` or `TextIteratorStreamer`) — `.predict()` returns only the final value. Iterate chunks via `.submit()`: ```python job = c.submit("short prompt", api_name="/chat") for chunk in job: print(chunk, end="") # or job.result() for the final value ``` **`gr.Server` custom `@app.get/post(...)` routes** don't appear in `view_api()`. Hit them with plain HTTP: ```python import httpx r = httpx.post(f"https://.hf.space/your_route", json={...}, timeout=600, headers={"Authorization": f"Bearer {os.environ['HF_TOKEN']}"}) ``` **OAuth-gated Spaces** (`hf_oauth: true` + `gr.LoginButton`) — anonymous `Client` can't authenticate. Test interactively after sign-in, or capture a session token and pass via `httpx_kwargs={"headers": {...}}`. **MCP server mode** (`launch(mcp_server=True)`) — different protocol. Use an MCP client. ### D. Output bytes AND logs look right? HTTP 200 ≠ correct output. Sniff both the returned file and the run log emitted during the call. ```python head = open(path, "rb").read(16) # b'glTF...' → glb # b'\x89PNG' → png # b'\xff\xd8' → jpeg # b'RIFF...WEBP' → webp # b'RIFF...WAVE' → wav # head[4:8]==b'ftyp' → mp4 # b'ply\n' → ply ``` For text: non-empty, not all `...` (thinking-model leak), length reasonable. For images: returned dimensions match what was requested (some models snap to nearest preset). Look at the tailed run log alongside — silent fallbacks (model snapping resolution, missing optional dep falling back to a slower path, dtype downgrade) only show up there. ### What NOT to do - **Don't launch Playwright / headless browser** to verify backend logic. The Gradio UI calls the same API `gradio_client` does — one `predict` tests both. - **Don't build mock-mode + local-server harnesses** before pushing. Local-green ≠ Space-green. - **Don't smoke-test with full-budget inputs.** Smallest input that exercises the GPU code path — short prompt, small image, low step count. You're verifying wiring, not quality. ## Iterating on the Space, not locally The Space env is the only one that matters: Python, torch, CUDA, file paths, env vars, gradio version, the `spaces` hijack all differ from your laptop. Workflow: 1. Decide SDK + hardware. Write the smallest `app.py` / `Dockerfile` + `requirements.txt` + README frontmatter — just enough that the entry point loads. 2. Push immediately. Don't build a Playwright / mock harness first. 3. Once `RUNNING`: verify with `gradio_client` against the real Space. That's your test loop. 4. Iterate via the cheapest rung. `python3 -m py_compile app.py` is the maximum local check worth doing. ## Last resort: dev mode + SSH Use only when: - A failure is non-deterministic (device-side asserts, OOM under specific shapes, race conditions). - You need `CUDA_LAUNCH_BLOCKING=1` or `gdb` to localize a CUDA error. - You'd burn 4+ build cycles trying variations from outside. Reading logs + grepping [`known-errors.md`](known-errors.md) + a tight `gradio_client` smoke loop solves the vast majority of issues. Dev mode is a heavy hammer. ### Prerequisites 1. **PRO / Team / Enterprise plan** — dev mode is a paid feature. 2. **An SSH key registered on the user's HF profile.** Without this, SSH refuses the connection. If the user doesn't have one yet, they need to: - Generate a keypair locally: `ssh-keygen -t ed25519 -f ~/.ssh/hf_dev -N ''` (no passphrase keeps automation simple; user can pick differently if they prefer). - Add the **public** key (`~/.ssh/hf_dev.pub`) at https://huggingface.co/settings/keys. - Keep the **private** key (`~/.ssh/hf_dev`) on the machine they'll SSH from. 3. **The Space must be in `RUNNING` or `RUNTIME_ERROR`** before dev mode lets you in — not `BUILD_ERROR`. If it's in build error, push a stub `app.py` that boots cleanly first (e.g. `import gradio as gr; gr.Interface(lambda: 'ok', None, 'text').launch()`), then enable dev mode. ### Enable No `huggingface_hub` Python wrapper yet — use the REST endpoint: ```bash curl -s -X POST \ -H "Authorization: Bearer $HF_TOKEN" \ -H "Content-Type: application/json" \ "https://huggingface.co/api/spaces///dev-mode" \ -d '{"enabled": true}' ``` ### SSH in ```bash ssh -i ~/.ssh/hf_dev -o BatchMode=yes -o StrictHostKeyChecking=accept-new \ -@ssh.hf.space ``` Username is `-` **lowercase**, with `-` replacing `/`. Dots in the Space name become hyphens too. ### Inside the VM It's a normal container at `/home/user/app/`. You can edit files, `pip install`, run repros, call `@spaces.GPU`-decorated functions interactively (they get a real GPU window). **`nvidia-smi` in the bare terminal will fail** with `NVML: Unknown Error`. Expected — ZeroGPU only exposes the real GPU inside `@spaces.GPU` calls. Don't assume the GPU is broken. **Edits in `/home/user/app/` don't survive** a restart, sleep, or dev-mode-disable. Only commits persist. ### Smoke-test the fix inside the container Before exiting dev mode, verify the fix actually works under a real GPU window: ```bash cat > /home/user/app/_devtest.py <<'PY' import spaces, torch from app import predict # or whatever your @spaces.GPU function is print(predict()) PY python3 _devtest.py ``` ### Persist + exit Commit + push from inside the container (`git config user.email / user.name` first; the HF git remote works). Then disable dev mode: ```bash curl -s -X POST -H "Authorization: Bearer $HF_TOKEN" \ -H "Content-Type: application/json" \ "https://huggingface.co/api/spaces///dev-mode" \ -d '{"enabled": false}' ``` **Factory-reboot** to apply the pushed state (in dev mode the Space won't rebuild on commits): ```python from huggingface_hub import HfApi HfApi(token=HF_TOKEN).restart_space("/", factory_reboot=True) ``` Then re-run the outside-the-container smoke test. Dev-mode success does **not** guarantee post-rebuild success — different image, different process tree. ```` ### `references/gradio.md` ````markdown # Gradio for Spaces Patterns and quirks specific to running Gradio inside a Space. Assumes you're already comfortable with stock Gradio components and `gr.Blocks` / `gr.Interface`. For deeper Gradio API guidance — components, layouts, event listeners, chatbots, the Gradio 5→6 migration — use the dedicated `huggingface-gradio` skill. Install it with `hf skills add huggingface-gradio` (add `--claude --global` to also install for Claude Code, user-level). For ZeroGPU-specific decorator + worker semantics, see [`zerogpu.md`](zerogpu.md). ## Themes and layout - Default theme preference: `gr.themes.Soft()`. Alternatives: no theme, or `gr.themes.Citrus()`. Pick once and don't over-style. - For apps that don't need full-width, constrain with CSS so they're readable on 4K displays: ```python CSS = """ #col-container { max-width: 1100px; margin: 0 auto; } .dark .gradio-container { color: var(--body-text-color); } """ with gr.Blocks(theme=gr.themes.Soft(), css=CSS) as demo: ... ``` The dark-mode override fixes a recurring Gradio bug where dark text inherits unset colors. - Width-cap with `!important` if Gradio 6's own breakpoints fight you — target `main`, `.gradio-container`, and the inner fillable wrapper, otherwise the width caps but goes flush-left. ## Minimal layout most demos converge to ```python with gr.Row(): prompt = gr.Textbox(show_label=False, placeholder="…", container=False, scale=4) run = gr.Button("Run", variant="primary", scale=1) output = gr.Image(...) with gr.Accordion("Advanced settings", open=False): ... ``` `container=False` on the inline Textbox removes the default outer border for a tighter look. ## `gr.Markdown` for the intro Be succinct. Title, one-line description, a links section. Don't over-explain how it works — that's what the model card is for. ## `gr.Examples` The pattern that doesn't trip people up: ```python gr.Examples( examples=[ ["a cat sitting on a windowsill", 42], ["mountains at sunset, photorealistic", 7], ], inputs=[prompt, seed], outputs=output, fn=generate, cache_examples=True, cache_mode="lazy", ) ``` Why these flags: - `cache_examples=True` makes example clicks instant (no re-inference per visitor). - **`cache_mode="lazy"`** — caches on first user click for each example. **Required on ZeroGPU** (Gradio's default on ZeroGPU is already lazy via `GRADIO_CACHE_MODE=lazy`). Eager would pre-run every example at app startup, but ZeroGPU has no GPU attached at startup — it'd fail and burn the creator's daily quota. - `cache_examples=True` silently disables `run_on_click` / `run_examples_on_click`. If your app relies on click-only behavior, set `cache_examples=False`. The cache key is the **example row's file path**, not a content hash. Regenerating an asset in place serves the stale cached output forever. If you replace example files, bump a `cache_version` marker or wipe `.gradio/cached_examples//`. Hot-reload (rung 1) does **not** rebuild the cache. A `cache_version` bump needs a real commit + restart. ## Streaming and generators `gr.Interface(fn=...)` and `.click(fn=...)` both accept generator functions. Each `yield` pushes a new value: ```python @spaces.GPU(duration=120) def generate(prompt): yield gr.update(value=None, label="Starting…") for k in range(num_steps): yield gr.update(value=preview(k), label=f"Step {k+1}/{num_steps}") yield gr.update(value=final, label="Done") ``` Use `gr.update(label=...)` for status narration — feels like a status line without a separate component. **Caveat**: `gr.Progress(track_tqdm=True)` and `yield` partial outputs fight each other — pick one streaming mechanism. ## Useful UX bits - `progress=gr.Progress(track_tqdm=True)` in the GPU function gives a free tqdm-driven progress bar. - Pair a "Randomize seed" checkbox with the seed input, and write the actually-used seed back so users can re-run deterministically: ```python randomize = gr.Checkbox(label="Randomize seed", value=True) seed = gr.Number(label="Seed", value=0, precision=0) @spaces.GPU def gen(..., seed, randomize_seed): if randomize_seed: seed = random.randint(0, 2**31 - 1) seed = int(seed) yield ..., gr.update(value=seed) # write back into the Seed input ... run.click(gen, inputs=[..., seed, randomize], outputs=[..., seed]) ``` ## Custom HTML components Stock Gradio components cover 95% of cases. When they don't, `gr.HTML(...)` lets you build contextual custom UI without leaving the Gradio Space (no need to switch to Docker). Guide: https://www.gradio.app/guides/custom-HTML-components Example with a 3D camera-angle picker that makes sense in context: https://huggingface.co/spaces/multimodalart/qwen-image-multiple-angles-3d-camera Don't reach for this if a stock component covers the need. ## Custom frontends — `gr.Server` For fully custom frontends with their own HTML/JS, while keeping Gradio's queue + GPU scheduling: ```python from gradio import Server from fastapi.responses import HTMLResponse app = Server(title="my-app") @spaces.GPU(duration=60) def _run_gpu(prompt): return inference(prompt) @app.api(name="generate", concurrency_limit=1, time_limit=180) def generate(prompt: str) -> str: return _run_gpu(prompt) # @app.api wraps, doesn't stack with @spaces.GPU @app.get("/", response_class=HTMLResponse) async def homepage(): return open("index.html").read() demo = app # HF runtime expects `demo` if __name__ == "__main__": demo.launch(ssr_mode=False) ``` Required for the custom `/` route to actually serve: ```bash hf spaces variables add / --env GRADIO_SSR_MODE=false ``` `launch(ssr_mode=False)` is ignored on HF — must be the env var. Valid `@app.api` kwargs: `name`, `description`, `concurrency_limit`, `concurrency_id`, `queue`, `batch`, `max_batch_size`, `api_visibility`, `time_limit`, `stream_every`. **Don't stack `@spaces.GPU` and `@app.api`** on the same function — silently breaks request flow. Keep them on separate functions. **Two-ceiling coordination**: both `@spaces.GPU(duration=N)` and `@app.api(time_limit=M)` apply; the lower wins. Set `time_limit` to your max duration across modes — a too-low `time_limit` kills the request even if GPU duration would have allowed it. Hot-reload (rung 1 in [`debugging.md`](debugging.md)) does **not** work with `gr.Server` — always `hf upload` or commit. Reference Space: https://huggingface.co/spaces/huggingface-projects/rf-detr-realtime-webcam ## Slow-startup Gradio (big-model Spaces) For Spaces that take 10–20 min to load weights: - Set `startup_duration_timeout: 1h` in README frontmatter (default 30 min). - Disable SSR: `hf spaces variables add --env GRADIO_SSR_MODE=false`. Otherwise the SSR health check times out before the app finishes loading. ## Don't - `gr.Button(text="X")` — `text=` was removed; use `gr.Button("X")`. - `gr.Button(type="button")` — drop the kwarg. - `.click(..., _js=share_js)` — renamed to `js=`. - `.style(height=...)` — removed in gradio 4+. - `gr.ImageMask(brush_color=...)` — kwarg removed. - `demo.launch(mcp_server=True)` on gradio 4.x — only valid on 5+. ```` ### `references/grants.md` ````markdown # Community GPU grants When a non-PRO user has a good use case for ZeroGPU (open research demo, hobbyist project, educational tool, institutional showcase) and doesn't want to subscribe, they can request a free community grant from Hugging Face. ## The flow 1. **Build the Space.** Create it as `--flavor cpu-basic` (the user is not PRO, so creating with `--flavor zero-a10g` will fail at `create_repo`). Code the app for ZeroGPU anyway — `import spaces`, `@spaces.GPU`, module-scope `.to("cuda")`. The Space will technically run on CPU until the grant is approved, but it'll be ready to switch over instantly. In this mode you **can't iterate-with-real-inference** before the grant — the CPU Space won't actually run heavy compute. Get the app to BUILD cleanly and `RUNNING` (even if the runtime would OOM on real input), then submit. 2. **Submit a Community Tab discussion** on the Space. Title: ``` Apply for a GPU community grant: project ``` Pick the closest fit. Body: ``` Description of the app: one paragraph on what it does + who it's for. Justification: one paragraph on why this should run on ZeroGPU (open-source, research, educational, etc.). ``` If the user didn't give you a justification, a reasonable default is "Non-PRO wants to build a public ZeroGPU app — happy to provide more context if helpful." 3. **Wait.** Open and publicly-facing applications by researchers, tinkerers, and institutions are typically approved. Approval can take days. 4. **Once approved**, the Space is moved to ZeroGPU automatically — no code change needed. The user comes back and you can iterate / refine with real GPU access. ## When to suggest this - User is not on PRO but their use case is a clear fit for ZeroGPU (a public ML demo, not a private tool). - The model fits in `large` (≤ 48 GB VRAM at the chosen precision). ## When NOT to suggest this - Private / commercial / closed-source projects — push the user toward PRO instead. - The model genuinely needs dedicated paid hardware (huge LLM, vLLM/JAX/ONNX as main model with heavy init) — `canPay=True` users can use paid flavors directly. - The user is already PRO — they have ZeroGPU access; no grant needed. ## Posting the request programmatically ```python from huggingface_hub import HfApi api = HfApi(token="hf_...") api.create_discussion( repo_id="/", repo_type="space", title="Apply for a GPU community grant: Personal project", description="", ) ``` The Community Tab must be enabled on the Space (default — keep it on). ```` ### `references/inference-providers.md` ````markdown # Inference Providers — when not to host the model Some Spaces don't need a GPU at all. If the model is available through HF Inference Providers (Cerebras, Fireworks, Together, Replicate, OpenRouter, etc.), the Space can be a thin Gradio shell that proxies to a hosted endpoint: - Zero VRAM, no `@spaces.GPU`, no model download. - Works for models too large to fit on ZeroGPU (120B+). - Hardware can be `cpu-basic` — no GPU at all. ## When to use this pattern - **Stateless chat or text completion** with a big model. - **The user wants a public demo of a frontier-scale model** that obviously doesn't fit on a single 48 GB MIG. - **The user wants to ship something fast** without worrying about quantization / sharding. ## When NOT to use this pattern - The model isn't available on any Inference Provider. Check with: ```bash curl "https://huggingface.co/api/models//?expand[]=inferenceProviderMapping" ``` - The Space needs **custom decoding** (special sampling, tool use, retrieval, anything stateful or interactive across calls). - The Space needs **multimodal** beyond what the provider exposes. - The user explicitly wants to own the inference stack (model loading, decoding, performance tuning). For those, host the model yourself on ZeroGPU — see [`zerogpu.md`](zerogpu.md). ## Two billing modes Choose based on who pays for inference. ### Mode A — Space creator pays (simple) Set `HF_TOKEN` as a Space secret. The Space uses `InferenceClient` directly. Every visitor's call is billed to the Space creator's account. ```python import os, gradio as gr from huggingface_hub import InferenceClient client = InferenceClient(api_key=os.environ["HF_TOKEN"], provider="fireworks-ai") def chat(msg, history): return client.chat_completion( model="/", messages=[*history, {"role": "user", "content": msg}], max_tokens=512, ).choices[0].message.content gr.ChatInterface(chat).launch() ``` Use when you want users to "just click and try it" — no sign-in friction. Cost is on you. ### Mode B — Visitor pays (recommended for public demos) `gr.LoginButton` + `gr.load("models/...")` with `accept_token=button`. Each visitor signs in with their HF account; inference is billed to **their** account. ```python import gradio as gr with gr.Blocks(fill_height=True) as demo: with gr.Sidebar(): button = gr.LoginButton("Sign in") gr.load("models//", accept_token=button, provider="fireworks-ai") demo.launch() ``` README frontmatter needs: ```yaml hf_oauth: true hf_oauth_scopes: - inference-api ``` This is the **recommended pattern for public demos** — sustainable cost-wise, and visitors get to use their own provider quotas (which most have paid for or get free). ## Hardware `cpu-basic`. No GPU. Don't put `--flavor zero-a10g` — you'd waste a paid grant. ## Anti-pattern: `@spaces.GPU` wrapping a provider call If you do use Inference Providers, do **not** wrap the call in `@spaces.GPU`. The decorator reserves a GPU slot on your Space for the full `duration=`, but the function does no GPU work — just an HTTP call out. You burn your own ZeroGPU quota for nothing. A provider-proxy Space wants `cpu-basic` hardware and zero `@spaces.GPU`. ```` ### `references/known-errors.md` ````markdown # Known errors Check if this is a known issue before trying your own fix. Entries are keyed by the substring that actually appears in `runtime.errorMessage`, the build log, or a Python traceback — grep this file for the error you saw. If you hit something not listed here and figure out a fix, please ask your human to PR it back so future runs benefit. --- ## Build / config errors These come from the Space build pipeline before the app starts. Read with `hf spaces logs --build --tail 500` — find the **first** error, not the last. ### `CONFIG_ERROR: torch version in requirements.txt is not compatible with ZeroGPU` **Cause**: `requirements.txt` pins `torch==X.Y.Z` to a version outside the supported set (`2.8.0`, `2.9.1`, `2.10.0`, `2.11.0`). **Fix**: Unpin torch (preferred — the runtime preinstalls the latest supported version), or pin to one of the supported values. ### `Cannot install … because these package versions have conflicting dependencies` / `ResolutionImpossible` **Cause**: A dep conflicts with the Gradio SDK pinned by `sdk_version:` in README. Most commonly `pydantic`, `uvicorn`, `huggingface_hub`, or `jinja2` pinned to old values that the SDK no longer accepts. **Fix**: Unpin the offender. For `gradio[mcp]` specifically, `uvicorn>=0.31.1` and `pydantic>=2.11.10` are required. ### Build hangs in dependency resolution > 10 min **Cause**: pip backtracking through a deep version space. **Fix**: Pin the conflicting transitive dep. The `--build` logs will show which one. Bump `startup_duration_timeout: 1h` in README frontmatter if heavy downloads are expected. ### `ModuleNotFoundError: No module named 'pkg_resources'` **Cause**: setuptools 81 dropped `pkg_resources`; an old package's `setup.py` imports it. **Fix**: Bump or unpin the offender. Typical culprits: `deepspeed==0.15.x` (training-only — usually safe to drop from inference Spaces), `openai-whisper==20231117`. ### `400 Bad Request` from `/api/validate-yaml` during `create_repo` / `upload_file` **Cause**: README frontmatter failed server validation. Most common: `short_description` over the (undocumented) character cap — target ≤ 60. **Fix**: Shorten `short_description`. Long descriptions go in the README body. Also double-check `colorFrom`/`colorTo` are one of `red|yellow|green|blue|indigo|purple|pink|gray`. ### `403 Forbidden` from `create_repo` with `space_hardware="zero-a10g"` **Cause**: The user isn't on PRO / Team / Enterprise so the API rejects ZeroGPU at creation. **Fix**: Retry without `space_hardware=`. Keep `hardware:` out of README frontmatter (silently ignored anyway). The Space is created on CPU; point the user at PRO upgrade or [community grant](grants.md). ### `403 Forbidden` from `create_commit(..., create_pr=True)` **Cause**: Upstream Space has Discussions disabled. **Fix**: Ask the maintainer to enable Discussions, or push directly if you have write access. --- ## Startup / RUNTIME_ERROR These come from `hf spaces logs --tail 500`. ### `RuntimeError: CUDA has been initialized before importing the spaces package` **Cause**: Something triggered CUDA init in the main process before `import spaces`. Usually wrong import order; sometimes a third-party lib eagerly initializing CUDA at import time (e.g. `numba.cuda`). **Fix**: Reorder so `import spaces` is first. For numba-using stacks (NeMo, RAPIDS bits): ```python import os os.environ.setdefault("NUMBA_DISABLE_CUDA", "1") import spaces ``` ### `RuntimeError: No @spaces.GPU function detected during startup` **Cause**: The function bound to `.click(fn=...)` / `.submit(...)` isn't decorated. Decorating an inner helper doesn't count — the startup scan only walks Gradio's registered handlers. **Fix**: Decorate the function Gradio binds. If that conflicts with another decorator, wrap explicitly: ```python @spaces.GPU(duration=60) def gpu_inner(...): ... def gradio_handler(...): return gpu_inner(...) ``` (Or just decorate `gradio_handler` directly.) ### `ImportError: cannot import name 'HfFolder' from 'huggingface_hub'` **Cause**: Old gradio (`4.44` and similar) imports `HfFolder` from `huggingface_hub`, which was removed in recent hub releases. **Fix**: Two options. - Pin `huggingface-hub==0.25.0` in `requirements.txt` (keeps old gradio happy). - Bump `sdk_version` in README to `5.x` or `6.x` (also fixes a lot of other API breaks). If a Gradio custom component locks the major (`gradio-image-prompter`, `gradio_litmodel3d`, …), install it with `--no-deps` so its `gradio<5.0` requirement doesn't bind. ### `ImportError: cannot import name 'is_traceable_wrapper_subclass' from 'torch.utils._python_dispatch'` **Cause**: A dep with `torchaudio<2.1` / `torch<2` in its `setup.py` (e.g. `demucs`, `audiocraft`) downgraded torch silently. The build succeeded, the app booted, and `import spaces` then died on a missing torch symbol. **Fix**: Install the offender from `app.py` with `--no-deps` *before* `import spaces`: ```python import subprocess, sys subprocess.run([sys.executable, "-m", "pip", "install", "--no-deps", "git+https://github.com/facebookresearch/demucs"], check=True) import spaces ``` List its actual runtime deps (`dora-search einops julius lameenc openunmix pyyaml tqdm` for demucs) yourself in `requirements.txt`. ### `_pickle.UnpicklingError: Weights only load failed` **Cause**: `torch.load` weights-only default flipped to `True` in torch 2.6. Old checkpoints pickling numpy/object globals fail. **Fix**: For trusted upstream checkpoints, monkey-patch before the package import: ```python import torch _orig = torch.load torch.load = lambda *a, **k: _orig(*a, **{**k, "weights_only": k.get("weights_only", False)}) ``` ### Stuck at `ZeroGPU init – 10.0%` then 60 s timeout **Cause**: A library called `cuInit` in the parent process, poisoning the fork (most often `numba.cuda` via NeMo). The actual `@spaces.GPU` body never starts. **Fix**: `os.environ.setdefault("NUMBA_DISABLE_CUDA", "1")` as the first line of `app.py`, before `import spaces`. ### `RUNTIME_ERROR` right after long `APP_STARTING`, logs sparse **Cause**: Boot exceeded `startup_duration_timeout` (default 30 min). Big-model loads commonly trigger this. **Fix**: Bump `startup_duration_timeout: 1h` in README frontmatter. For Gradio 6 specifically, also set `GRADIO_SSR_MODE=false` via `hf spaces variables add --env GRADIO_SSR_MODE=false` to dodge SSR health-check timeouts during slow boot. ### `RUNNING` but the public URL returns 404 **Cause**: The Space is private. Anonymous Client / browser hits return 404. **Fix**: Authenticate. `gradio_client.Client(space, token=os.environ["HF_TOKEN"])`. The kwarg is `token=`, not `hf_token=`. ### `workload was not healthy after 30 min` **Cause**: Infra-side scheduling or a build that genuinely can't finish in time. **Fix**: Usually not actionable in code. Bump `startup_duration_timeout` if heavy downloads are expected; otherwise wait or report. ### Exit code 128 / containerd / scheduling failure **Cause**: HF infra glitch. **Fix**: `hf spaces restart --factory-reboot`. If it persists, retry later or report. Not fixable in code. --- ## Inference-time errors These appear in `hf spaces logs --follow` while a request is running. ### `ZeroGPU illegal duration` **Cause**: `@spaces.GPU(duration=N)` is larger than the visitor's tier per-call cap. **Fix**: Lower `N`. Tier caps live in the [ZeroGPU docs](https://huggingface.co/docs/hub/spaces-zerogpu). ### `ZeroGPU quota exceeded (X requested vs Y left)` **Cause**: The visitor's remaining quota < `requested duration`. The comparison is `requested vs remaining`, not `actual vs remaining` — a 10-second task left at the default 60 s blocks the user as soon as their remaining drops below 60 s. **Fix**: Lower `duration` to the realistic worst case. For input-dependent runtime, use a callable estimator. ### `RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at .../CUDACachingAllocator.cpp` **Cause**: Allocator fragmentation under transient memory spikes (high-res pixel-space ops, large attention activations, SR models, video DiTs). Not a clean OOM. **Fix**: Set expandable segments at the **very top** of `app.py`, before any torch import: ```python import os os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True") import spaces import torch ``` Usually a single-line fix that replaces lowering resolution or moving to `xlarge`. ### Call hangs forever on first `@spaces.GPU` entry **Cause**: The decorated function returned a CUDA tensor. Unpickling it in the main process triggers `torch.cuda._lazy_init()`, which ZeroGPU blocks. **Fix**: Convert to CPU before returning: `return tensor.cpu()` or `.cpu().numpy()`. For `gr.State`, scrub before yielding. ### `PicklingError` at call entry **Cause**: An argument crossing the fork boundary contains an unpicklable object — file handle, lock, lambda, closure, or `gr.SelectData`. **Fix**: Extract the picklable fields in a thin un-decorated wrapper, pass plain values to the `@spaces.GPU` function. For `gr.SelectData` specifically, pull out `evt.index[0]`, `evt.index[1]` etc. outside the decorator. ### `RecursionError` inside `gr.SelectData.__getattr__` **Cause**: Same as above — `gr.SelectData` doesn't survive pickle. **Fix**: Same — extract its fields before crossing the boundary. ### `CUDA error … flash_fwd_launch_template.h: no kernel image is available for execution on the device` (or `:188: invalid argument`) **Cause**: A Flash Attention 3 kernel was loaded — directly via `kernels-community/{flash-attn3,vllm-flash-attn3,sgl-flash-attn3}`, or indirectly via an old `xformers` wheel that auto-dispatched to FA3. FA3 has no Blackwell sm_120 build. **Fix**: Use `attn_implementation="sdpa"`, or `"flash_attention_2"` with the FA2 wheel from `multimodalart/zerogpu-blackwell-wheels`. For xformers, the prebuilt wheel from the same dataset auto-dispatches to FA2 — no monkey-patch. ### `NotImplementedError: sgl_flash_attn3 is only supported on sm80 and above with CUDA >= 12.3` **Cause**: `kernels-community/sgl-flash-attn3` rejects sm_120 at runtime despite the error wording. Same root cause as the FA3 entry above. **Fix**: Same — SDPA or FA2. ### `ImportError: cannot import name 'flash_attn_varlen_func' from 'flash_attn'` / model insists on `attn_implementation="flash_attention_2"` **Cause**: The model imports flash_attn at module top with no escape hatch, and the runtime doesn't ship it. **Fix**: Install the prebuilt `flash_attn-2.8.3-cp310-cp310-linux_x86_64.whl` from `multimodalart/zerogpu-blackwell-wheels`. Requires `python_version: "3.10"` in README (wheel is cp310 only). Real `flash_attn_2_cuda` satisfies xformers' `flash_attn_gpu` probe too. For transformers `AutoModel`-style configs that aren't a hard import, swap `attn_implementation="flash_attention_2"` → `"sdpa"`. Torch-native, zero deps. ### `selective_scan_cuda.so undefined symbol` / `_torchaudio.abi3.so undefined symbol` **Cause**: A direct-URL prebuilt CUDA wheel pinned to an old torch ABI (`cu12torch2.4cxx11abiFALSE`, etc.) — won't load on the current runtime. **Fix**: Drop the URL-pinned wheel from `requirements.txt`. Use a torch-current wheel from `multimodalart/zerogpu-blackwell-wheels`, kernels-community, or upstream's release page. ### `TypeError: 'dict' object is not hashable` inside `jinja2/utils.py:get` **Cause**: Old gradio (4.44) + modern starlette / jinja2 cache clash. **Fix**: Either pin `jinja2<3.2` + `starlette<0.40` (keeps old gradio), or bump `sdk_version` past 5.0. --- ## Smoke-test / client errors ### `Client.__init__() got an unexpected keyword argument 'hf_token'` **Cause**: Older `gradio_client` API used `hf_token=`; current uses `token=`. **Fix**: `Client(space, token=os.environ["HF_TOKEN"])`. ### `httpx.ReadTimeout` on `client.predict(...)` **Cause**: Default timeout too small for the GPU duration. **Fix**: `Client(..., httpx_kwargs={"timeout": 600})`. Set this to at least your `@spaces.GPU(duration=N)` plus 60 s. ### `404` on what looks like a valid endpoint name **Cause**: Streaming endpoint (function uses `yield`), or a custom `gr.Server` `@app.get/post` route that doesn't appear in `view_api()`. **Fix**: For streaming, use `client.submit(...).result()` (or iterate the job). For custom routes, use `httpx.post(base_url + "/route", ...)` directly — bypass `gradio_client`. ### Result file looks empty / dimensions wrong **Cause**: HTTP 200 ≠ correct output. The model snapped to a different size, or the wrong endpoint was hit. **Fix**: Sniff the returned file's magic bytes (`glTF`, `\x89PNG`, `RIFF…WEBP`, `RIFF…WAVE`, `[4:8]==b"ftyp"`) and check returned dimensions match what was requested. --- ## Submitting new entries If you hit an error not in this file and figure out the fix, please ask your human to PR it back to this skill. Format: - A 1-line heading with the exact error substring an agent would grep for. - **Cause**: one sentence on what triggered it. - **Fix**: concrete commands or code. If the fix needs more than 5 lines of narrative, point to another reference file (e.g. [`debugging.md`](debugging.md), [`zerogpu.md`](zerogpu.md)) for depth. ```` ### `references/requirements.md` ````markdown # requirements.txt for Spaces Rules for what to pin, what to leave alone, where to source CUDA wheels, and which torch-side-cars drift silently. ## What's preinstalled (do not list) The Gradio SDK base image already installs these on every hardware tier — listing them in `requirements.txt` causes resolution failures or, worse, lets pip silently drift the runtime out of compatibility: | Package | Pinning rules | |---|---| | `gradio` | Don't list. Locked by `sdk_version:` in README frontmatter; pinning here is ignored or breaks. | | `spaces` | Don't list. Platform-pinned; a user pin always loses. | | `huggingface_hub` | Don't list by default. Pin only as a workaround for old `gradio<5` that imports the removed `HfFolder` symbol (see [`known-errors.md`](known-errors.md)). | | `torch` | **Pinnable, but only within `{2.8.0, 2.9.1, 2.10.0, 2.11.0}`.** Anything outside causes `CONFIG_ERROR: torch version in requirements.txt is not compatible`. Default is to leave unpinned (runtime preinstalls 2.11), but pinning is appropriate when (a) a specific version is known-good for your model, (b) you're matching a CUDA-extension wheel's `torch2.X` tag, or (c) a dep would otherwise drag torch outside the supported set. When you pin torch, also pin `torchvision` / `torchaudio` to the matching minor — see the "Torch-family side-car drift" section below. | ## What to list Everything you actually `import`, including the often-forgotten: - `torchvision`, `torchaudio` — **not** preinstalled. Leave unpinned; pip resolves against the installed `torch` major.minor. - `accelerate` — needed whenever you use `device_map=`. Listing it also silences `low_cpu_mem_usage=False` warnings. - `sentencepiece` — required by most LLM tokenizers; rarely transitive. - `einops` — required by `flash_attn.layers.rotary` and many model repos. - Domain libs: `diffusers`, `transformers`, `safetensors`, `pillow`, `numpy`, etc. If a research repo ships a Python package directory (`models/`, `pipeline/`, …), just upload the directory with the rest of the Space — the whole repo root is importable as `/home/user/app`. **Do not** try to reference local paths from `requirements.txt`. ## Pinning torch ZeroGPU accepts only `2.8.0`, `2.9.1`, `2.10.0`, `2.11.0`. Default is unpinned (runtime preinstalls the latest). Pinning is fine — and sometimes warranted — within that set: - A specific torch is known-good for your model (numerics, attention kernel availability, etc.). - A direct-URL CUDA wheel encodes a `torch2.X` tag (see "Prebuilt CUDA wheels" below) — pin torch to match. - A dep's `setup.py` would otherwise downgrade torch outside the supported set. `2.8.0` is the safest fallback for old requirements that refuse modern torch. `2.10.0` / `2.11.0` is the sweet spot for new code. When you pin torch, also pin `torchvision` / `torchaudio` to the matching minor — see the side-car drift section. When a dep would silently downgrade torch (e.g. some forks of `demucs`, `audiocraft` pin `torchaudio<2.1`), install the offender from `app.py` with `--no-deps` rather than pinning torch around it: ```python import subprocess, sys subprocess.run([sys.executable, "-m", "pip", "install", "--no-deps", "git+https://github.com/facebookresearch/demucs"], check=True) import spaces # safe now — torch wasn't touched ``` List the offender's real runtime deps yourself in `requirements.txt`. ## Torch-family side-car drift `torchvision`, `torchaudio`, `torchcodec` are built against a specific `torch` major.minor. Listing them unpinned **usually** works, but two known drift patterns: - `torchaudio==2.11.0` (and later) **dropped its `Requires-Dist: torch==X.Y.Z` line**. With torch pinned to 2.10, pip silently resolves torchaudio to 2.11.0 and the import fails on ABI mismatch. - `torchcodec` declares no torch dependency in PyPI metadata at all. Verification after `pip install` or `uv lock --upgrade`: ```bash curl -s https://pypi.org/pypi///json \ | python3 -c "import json,sys,re; rd=json.load(sys.stdin)['info'].get('requires_dist') or []; \ print('\n'.join(x for x in rd if re.match(r'^torch(?![a-z])', x)) or '(no torch constraint)')" ``` When PyPI is silent, fall back to the project's README compatibility table (torchcodec's lives at https://github.com/pytorch/torchcodec). ## Prebuilt CUDA wheels — the Blackwell wheels dataset For CUDA-extension packages without an upstream wheel matching the ZeroGPU torch / cuda / cxx11-abi cell, use the canonical prebuilt wheels at: > https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels Wheels live at `wheels//--.whl`. Current cells: - `pt212-cu130-cp310` — built against torch 2.12 / CUDA 13.0 / Python 3.10. **Works on the live ZeroGPU runtime (torch 2.11)** for all packages below. - `pt212-cu130-cp312`, `pt212-cu130-cp313` — same matrix at other Python versions. - `pt28-cu128-cp310` — older fallback for Spaces stuck on torch 2.8 / CUDA 12.8. Reference by direct URL in `requirements.txt`: ``` https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/resolve/main/wheels/pt212-cu130-cp310/ ``` ### Per-package status Verified empirically against the live runtime + the real Spaces that previously shipped runtime patches: | Package | Wheel | Replaces | Caveats | |---|---|---|---| | `xformers` | `xformers-0.0.34+3da0fc92.d20260528-cp39-abi3-linux_x86_64.whl` | MEA→SDPA monkey-patch shim; Cutlass-force shim | `Requires: torch>=2.10`. Auto-dispatch picks FA2 (`fa2F@2.5.7-pt`) on sm_120. Classic Cutlass / FA3 still reject sm_120 but auto-dispatch never selects them now. | | `flash_attn` | `flash_attn-2.8.3-cp310-cp310-linux_x86_64.whl` | Committed `flash_attn/` stub package; `sys.modules["flash_attn"] = ...` injection | **cp310 only** — requires `python_version: "3.10"` in README. Needs `einops` for `flash_attn.layers.rotary`. Real `flash_attn_2_cuda` satisfies xformers' `hasattr(flash_attn.flash_attn_interface, "flash_attn_gpu")` probe. | | `pytorch3d` | `pytorch3d-0.7.9-cp310-cp310-linux_x86_64.whl` | Runtime `pip install git+...pytorch3d.git` inside `@spaces.GPU` | Needs `numpy`, `iopath`, `fvcore` listed. No torch pin in metadata; loads cleanly on torch 2.11. | | `nvdiffrast` | `nvdiffrast-0.4.0-cp310-cp310-linux_x86_64.whl` | Runtime build with `TORCH_CUDA_ARCH_LIST=12.0` | Needs `numpy`. `RasterizeGLContext` in 0.4.0 is a deprecation alias for `RasterizeCudaContext` — no headless-GL footgun. | | `diff_gaussian_rasterization` | `diff_gaussian_rasterization-0.0.0-cp310-cp310-linux_x86_64.whl` | Runtime build from `graphdeco-inria/diff-gaussian-rasterization.git` | **Upstream Inria API only** (returns 2-tuple `(color, radii)`). Does NOT match the ashawkey fork (4-tuple including alpha+depth) used by `ashawkey/LGM`, `dylanebert/LGM-mini`, etc. Forks need their own wheel. | | `torchmcubes` | `torchmcubes-0.1.0-cp310-cp310-linux_x86_64.whl` | Runtime `pip install git+...torchmcubes.git` | **sm_120 only** (no fatbin for older archs). Works on ZeroGPU / Blackwell; not portable to a dedicated T4 / L4 / A10G Space. | ### Pattern ``` # requirements.txt numpy einops https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/resolve/main/wheels/pt212-cu130-cp310/flash_attn-2.8.3-cp310-cp310-linux_x86_64.whl https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/resolve/main/wheels/pt212-cu130-cp310/xformers-0.0.34+3da0fc92.d20260528-cp39-abi3-linux_x86_64.whl ``` ```yaml # README frontmatter — pin Python to match wheel cell python_version: "3.10" ``` **Do not** install these from `@spaces.GPU` startup. The previous "subprocess.check_call pip install at first GPU acquire" pattern is now strictly worse than the wheel URL — slower cold start, eats `duration` budget, breaks reproducibility, and the build sometimes exceeds the `@spaces.GPU(duration=1500)` cap. ### When you need a wheel that's not in the dataset Three options, in preference order: 1. **kernels-community** — https://huggingface.co/kernels-community handles ABI matching for you. Often the simplest path; no version pinning needed. 2. **Upstream wheel matrix** — e.g. flash-attention's releases page ships a fairly complete `cu12 / torch / Python` matrix at https://github.com/Dao-AILab/flash-attention/releases. Pin `torch==X.Y.Z` in `requirements.txt` to match the wheel's `torch2.X` tag. 3. **Build it yourself and host on HF Hub.** Last resort — see [`debugging.md`](debugging.md) for the in-`@spaces.GPU` source-build pattern as a stopgap while a wheel is being built. ## Reading a CUDA wheel filename ``` flash_attn-2.8.3+cu130torch2.12cxx11abiFALSE-cp310-cp310-linux_x86_64.whl ``` | Tag | Meaning | |---|---| | `cu130` | CUDA major version (13.0) | | `torch2.12` | torch major.minor the wheel was compiled against | | `cxx11abiFALSE` | C++ stdlib ABI choice (`TRUE` or `FALSE`) | | `cp310-cp310` | CPython version (3.10) | ABI / symbol mismatches at any of these → `ImportError` on first import. Pin `torch` to match `torch2.X`. Set `python_version:` to match `cp3XX`. ## Don't pin `xformers` Leave bare in `requirements.txt` (or use the prebuilt URL above). Pip picks the wheel matching your installed torch. ## Don't pin `spaces` Even if a `uv export` produces it, exclude with `--no-emit-package spaces`. The platform always pins its own version. ## Specifically about Python version Pinning `python_version:` is effectively required: - ZeroGPU officially supports **3.10.13** and **3.12.12**. - The runtime default is 3.10. - Pinning to a `cp3XX` wheel matrix (e.g. `cp310` flash_attn wheel) forces matching Python. Both `"3.12"` and `"3.12.12"` forms are accepted in YAML. ```` ### `references/zerogpu.md` ````markdown # ZeroGPU Read this whenever the Space targets ZeroGPU (`zero-a10g` flavor). The SKILL.md's 3-rule summary is a starting point; this file covers the model in enough detail to debug and design. For numerical limits (per-tier daily quota minutes, runs-per-day caps, current backing GPU, supported Python / torch versions): https://huggingface.co/docs/hub/spaces-zerogpu. Those values change over time and are deliberately kept out of this skill. ## The mental model A ZeroGPU Space runs as **two processes**: - **Main web process** — long-lived. Imports `app.py`, launches Gradio. Holds no VRAM and, after the startup "pack" step, no model weights in RAM either. - **GPU worker** — short-lived. Forked per `@spaces.GPU` request (or reused if warm). Eventually killed by the ZeroGPU scheduler when another Space needs the slot. Your code never kills its own worker. `import spaces` monkey-patches `torch.cuda.*` in the main process so that `.to("cuda")` and `torch.cuda.is_available()` work at module scope **without** a real GPU attached. Module-level `model.to("cuda")` is intercepted: the tensor data physically stays in main-process RAM at this point, with a CUDA-presenting "fake" tensor registered alongside. At a startup "pack" step, the backend writes those original CPU tensors to disk via `O_DIRECT` and frees the RAM. After pack, main holds no weights anywhere. When a `@spaces.GPU` call lands, the scheduler routes it to a worker: - **Cold worker** — forked from the main process; torch is unpatched; real CUDA is initialized; weights are streamed disk → pinned host → VRAM via a double-buffered pipeline. This is the cold-start cost. - **Warm worker** — alive worker bound to the same slot; init is skipped; weights stay on VRAM from the previous call. A warm worker eventually dies when another Space needs the slot. Occasional cold starts on a low-traffic Space are normal. ## The three rules ### 1. `import spaces` before any CUDA-touching import ```python import spaces # FIRST import torch # then this ``` If something initializes CUDA before `import spaces`, the patch can't apply and you get `RuntimeError: CUDA has been initialized before importing the spaces package`. For libraries that eagerly init CUDA on import (e.g. `numba.cuda`, NeMo via numba), set the disable env *before* the import: ```python import os os.environ.setdefault("NUMBA_DISABLE_CUDA", "1") import spaces ``` ### 2. Load models at module scope, `.to("cuda")` eagerly ```python pipe = DiffusionPipeline.from_pretrained("...", torch_dtype=torch.bfloat16).to("cuda") ``` Do **not** lazy-load inside `@spaces.GPU`. The hijack is designed for module-level placement; deferring it puts tens of seconds of checkpoint I/O + dtype cast + GPU move inside every cold request. Use the **string `"cuda"`** — never an integer device id. ZeroGPU re-allocates device ids per request, so `.to(0)`, `device_map={"": 0}`, `torch.cuda.set_device(0)` silently break. For plain `from_pretrained` loads, use `.to("cuda")`, **not** `device_map="cuda"` (which routes through `accelerate.set_module_tensor_to_device` and calls `torch._C._cuda_init()` at load time, bypassing the hijack). The exception is loaders that are ZeroGPU-aware — notably the `bitsandbytes` quantization path; `from_pretrained(..., quantization_config=BitsAndBytesConfig(...))` works with `device_map="cuda"`. **Preloading multiple variants** (e.g. base + refiner, image + video model) is fine as long as their combined VRAM fits. Load all of them sequentially at module scope into a dict, then key per request. Don't unload/reload between requests — that puts the load cost back on the user. ### 3. Decorate the function Gradio binds ZeroGPU's startup scan walks Gradio's registered event handlers for `@spaces.GPU`-marked functions. If you decorate `inner_helper` but `click(fn=outer)` is what's wired up, you get `RuntimeError: No @spaces.GPU function detected during startup`. Always decorate the function passed to the event handler. ```python @spaces.GPU(duration=60) def generate(prompt): return pipe(prompt).images[0] btn.click(fn=generate, inputs=prompt_box, outputs=image_out) ``` ## Sizing duration `@spaces.GPU(duration=N)` means "reserve N seconds of GPU time." Two failure modes: - **`ZeroGPU illegal duration`** — `N` exceeds the visitor's tier cap. Lowering `duration` is the only fix. - **`ZeroGPU quota exceeded`** — the visitor's remaining quota is less than `requested`. Compared as `requested vs remaining`, not `actual vs remaining` — so a 10-second task left at the default 60 s blocks the user as soon as their remaining drops below 60 s. Smaller `duration` also ranks **higher** in the queue. Both reasons push toward declaring the realistic worst case, not a comfortable margin. **Pick the value — don't guess.** A too-high duration deploys cleanly then errors on the first call; too-low silently truncates. Methodology: 1. Ship with a placeholder (e.g. 180 s). 2. Instrument with `time.perf_counter()` and return the seconds in the response. 3. Run 2–3 representative calls via `gradio_client`. 4. Set `duration = round(measured_max × 1.4)`. For input-dependent runtime, pass a **callable**: ```python def _estimate(prompt, steps, *args, **kwargs): # Swallow extras with *args, **kwargs — Gradio passes progress= positionally # and a strict signature will raise "takes 5 positional arguments but 6 were given" return min(240, 60 + int(steps * 3.5)) @spaces.GPU(duration=_estimate) def generate(prompt, steps, ..., progress=gr.Progress(track_tqdm=True)): ... ``` ## Sizing memory: `large` vs `xlarge` `size="large"` (default) is half the backing card (48 GB on Blackwell). `size="xlarge"` is the full card (96 GB) and costs **2× quota** per second — plus higher queue waits. Use `large` unless the workload genuinely OOMs. Rough VRAM sizing: | Mode | Memory rule | 7B | 27B | 70B | |------|------------|------|------|------| | bf16 | `params × 2` GB | 14 GB ✓ large | 54 GB → xlarge | 140 GB → quant + xlarge | | int8 | `params × 1` GB | 7 GB ✓ large | 27 GB ✓ large | 70 GB → xlarge | | 4-bit (NF4 / int4) | `params × ~0.55` GB | 4 GB ✓ large | 15 GB ✓ large | 40 GB ✓ large | Numbers are for weights only; activations and KV cache add on top (significant for long context). ## Quantization ZeroGPU supports two quantization stacks: **`bitsandbytes`** (drop-in for transformers, well-trodden) and **`torchao`** (torch-native, newer, smaller install). Pick by what your model's `from_pretrained` actually wires up; if both work, default to `bitsandbytes` for transformers LLMs and `torchao` for diffusers. ### bitsandbytes (NF4 / int8) ```python import spaces, torch from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig bnb = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16, ) tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) model = AutoModelForCausalLM.from_pretrained( MODEL_ID, quantization_config=bnb, device_map="cuda", # OK here — bnb's loader is ZeroGPU-aware dtype=torch.bfloat16, ).eval() ``` This is the one case where `device_map="cuda"` is **safe** on ZeroGPU at module scope (bitsandbytes' loader path intercepts cleanly). For non-bnb loads, stick to `.to("cuda")`. `load_in_8bit=True` swaps the 4-bit block for int8 — same hijack-safe loader. Bigger but higher quality, no `compute_dtype` knob. ### torchao ```python import spaces, torch from diffusers import DiffusionPipeline from torchao.quantization import quantize_, Int8WeightOnlyConfig pipe = DiffusionPipeline.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16).to("cuda") quantize_(pipe.transformer, Int8WeightOnlyConfig()) # mutates in place ``` `torchao` is more flexible (fine-grained per-module quantization, `Int4WeightOnlyConfig`, `Float8WeightOnlyConfig`, etc.) and works with diffusers' `from_pretrained(..., quantization_config=TorchAoConfig(...))` integration too. No CUDA build dependency — installs as a wheel. ### Attention backends Default to `attn_implementation="sdpa"` — torch-native, works everywhere. Reach for an FA backend only when the upstream repo uses it by default or strongly recommends it. If it breaks on Blackwell, fall back to SDPA. **Flash Attention 2** works via the prebuilt wheel at `multimodalart/zerogpu-blackwell-wheels` ([`requirements.md`](requirements.md)). The wheel's real `flash_attn_2_cuda` also satisfies xformers' import-time probes. **Flash Attention 3** is not currently usable on Blackwell sm_120. `kernels-community/flash-attn3`, `vllm-flash-attn3`, and `sgl-flash-attn3` all fail with `no kernel image is available` or `NotImplementedError`. Use SDPA or FA2 instead. **xformers** is available via the prebuilt wheel — auto-dispatch picks FA2 on sm_120 with no monkey-patch. ## Concurrency Handlers run **concurrently by default**. Three rules: 1. **No mutable global state.** Handlers writing to a module-level dict / list race each other. 2. **No fixed output paths.** Two concurrent calls writing to `output.png` clobber each other (and leak data across users). Use `tempfile.NamedTemporaryFile(suffix=...)`. 3. **Read-only globals are safe** — models, tokenizers, configs loaded once and only read inside handlers. ## Process isolation and pickle `@spaces.GPU` runs in a separate fork. Arguments and return values cross via pickle: - **Only picklable objects** in/out. File handles, locks, lambdas, closures over unpicklable state → `PicklingError`. - **Never return CUDA tensors.** Unpickling in the main process triggers `torch.cuda._lazy_init()`, which ZeroGPU blocks → the call hangs. Convert to CPU first: `return tensor.cpu()` or `.cpu().numpy()`. - CPU tensors, numpy arrays, PIL Images, plain Python objects work fine. - `gr.SelectData` is a special case — its `__getattr__` recurses under pickle. Extract the fields you need (`evt.index[0]`, etc.) in a thin un-decorated wrapper, pass plain values to the `@spaces.GPU` function. ### `gr.State` across the fork `gr.State` is pickled on every yield. The handler receives a **copy**: - In-place mutations inside the fork are invisible to other handlers until you explicitly `yield` the mutated value back. - Yielding `gr.update()` for a state slot **skips** the update — other handlers continue to see pre-yield value. - For large state, minimize how often you yield it — ideally once at the end. - CUDA tensors inside state must be CPU-d before yielding (same `_lazy_init` issue). ## Generators and streaming `@spaces.GPU` supports generator functions — first-class for progressive UI updates: ```python @spaces.GPU(duration=120) def generate(prompt): yield gr.update(value=None, label="Starting…") for step in range(num_steps): latent = step_fn(...) yield gr.update(value=preview(latent), label=f"Step {step+1}/{num_steps}") yield gr.update(value=final_image, label="Done") ``` `gr.Progress(track_tqdm=True)` and `yield` compete with each other — pick one. For streaming previews **inside** a diffusers `callback_on_step_end`, use a thread + queue inside the decorator (forks share threads): ```python @spaces.GPU(duration=180) def generate(prompt, num_steps): q = queue.Queue() DONE = object() def cb(pipe, step, t, kw): q.put((step, taef1_preview(kw["latents"]))) return kw def run(): out = pipeline(prompt=prompt, num_inference_steps=num_steps, callback_on_step_end=cb, callback_on_step_end_tensor_inputs=["latents"]) q.put((DONE, out)) threading.Thread(target=run, daemon=True).start() while True: idx, payload = q.get() if idx is DONE: break yield gr.update(value=payload, label=f"Step {idx+1}/{num_steps}") ``` **Do not** use `ProcessPoolExecutor` / `multiprocessing.Pool` inside `@spaces.GPU` — the daemonic fork can't spawn children (`AssertionError: daemonic processes are not allowed to have children`). Threads only. ## Compilation `torch.compile` is **not supported** on ZeroGPU. Use PyTorch ahead-of-time inductor (AoTI), supported from torch 2.8+. Full guide: https://huggingface.co/blog/zerogpu-aoti. The `spaces` package exposes `aoti_capture`, `aoti_compile`, `aoti_apply`, `aoti_blocks_load` for the workflow. ## Local development **Do NOT** wrap `import spaces` in `try/except` with a no-op fallback. Off-ZeroGPU, the `spaces` package is *already* a true no-op — the heavyweight behavior is gated on `SPACES_ZERO_GPU=1`, set only on ZeroGPU. `@spaces.GPU` returns the undecorated function unchanged elsewhere. The Gradio base image installs `spaces` on every hardware tier, so a duplicate onto T4 / A10G / CPU works without code changes too. That said: **iterate ON the Space, not locally.** The Space environment (Python, torch, CUDA, drivers, env vars) differs from yours; passing local tests doesn't prove the Space works. Push early — even with the app not fully polished — and use the rung ladder ([`debugging.md`](debugging.md)) against the live URL. ## Allocator config for memory pressure If your workload hits transient allocation spikes (high-res pixel-space ops, large attention activations, SR models, video DiTs) and you see: ``` RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at .../CUDACachingAllocator.cpp ``` set expandable segments at the **very top** of `app.py`, before any torch import: ```python import os os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True") import spaces import torch ``` Often single-line fix for what looks like an OOM. See [`known-errors.md`](known-errors.md). ## Example caching `gr.Examples` defaults on ZeroGPU: - `cache_examples=True` - `cache_mode="lazy"` (eager would pre-run examples at startup, but no GPU is attached at startup) Don't override to `cache_mode="eager"` on ZeroGPU — it will fail or burn the creator's daily quota. The cache is keyed by example **file path**, not content hash: regenerating an asset in place serves the stale cached output. Bump a `cache_version` constant if you replace example files. ## Real-time sessions For real-time apps (webcam, audio streaming), the per-call fork model is too costly. ZeroGPU supports reusable "real-time sessions" — one GPU allocation amortized across many small requests. Reference Spaces: - https://huggingface.co/spaces/diffusers/unofficial-SDXL-Turbo-i2i-t2i - https://huggingface.co/spaces/huggingface-projects/rf-detr-realtime-webcam ## When things go wrong For specific error strings (CUDA init order, illegal duration, allocator asserts, PicklingError, returning CUDA tensors, …): [`known-errors.md`](known-errors.md). It covers the ZeroGPU-specific patterns alongside everything else and is the single error lookup for the skill. When the log endpoint can't explain a failure (device-side asserts, OOM under specific shapes, race conditions in pickle), dev mode + SSH is the last-resort tool — see [`debugging.md`](debugging.md). ````