# SkillPatch skill: huggingface-trackio

This skill enables ML experiment tracking using the Trackio library, covering metric logging via Python API, alert firing for training diagnostics, and metric retrieval via CLI. It supports real-time dashboard visualization with HF Space syncing, webhook-based alerting (Slack/Discord), and JSON output for automation. It also provides a structured autonomous ML experiment workflow for LLM agents to monitor and iterate on training runs.

You (the agent) were given this URL and asked to use this skill. This is a **public** skill — no SkillPatch account, API key, or prior setup is required. Two ways to use it:

**1. Use it right now** — the skill's complete file tree (SKILL.md + all reference files) is inlined below; read `SKILL.md` first, then follow it, consulting the other files as it directs.

**2. Install the exact package onto disk** (recommended if you can run a shell — this reproduces the skill byte-for-byte, including any binary assets that can't be inlined):

```bash
mkdir -p .claude/skills/huggingface-trackio
curl -sSL https://skillpatch.dev/install_skill/huggingface-trackio | tar -xz -C .claude/skills/
```

(`.claude/skills/` is Claude Code's convention; use whatever directory your agent loads skills from.)


---

## Skill files (4)

- `SKILL.md`
- `references/alerts.md`
- `references/logging_metrics.md`
- `references/retrieving_metrics.md`


### `SKILL.md`

````markdown
---
name: huggingface-trackio
description: Track and visualize ML training experiments with Trackio. Use when logging metrics during training (Python API), firing alerts for training diagnostics, or retrieving/analyzing logged metrics (CLI). Supports real-time dashboard visualization, alerts with webhooks, HF Space syncing, and JSON output for automation.
---

# Trackio - Experiment Tracking for ML Training

Trackio is an experiment tracking library for logging and visualizing ML training metrics. It syncs to Hugging Face Spaces for real-time monitoring dashboards.

## Three Interfaces

| Task | Interface | Reference |
|------|-----------|-----------|
| **Logging metrics** during training | Python API | [references/logging_metrics.md](references/logging_metrics.md) |
| **Firing alerts** for training diagnostics | Python API | [references/alerts.md](references/alerts.md) |
| **Retrieving metrics & alerts** after/during training | CLI | [references/retrieving_metrics.md](references/retrieving_metrics.md) |

## When to Use Each

### Python API → Logging

Use `import trackio` in your training scripts to log metrics:

- Initialize tracking with `trackio.init()`
- Log metrics with `trackio.log()` or use TRL's `report_to="trackio"`
- Finalize with `trackio.finish()`

**Key concept**: For remote/cloud training, pass `space_id` — metrics sync to a Space dashboard so they persist after the instance terminates.

→ See [references/logging_metrics.md](references/logging_metrics.md) for setup, TRL integration, and configuration options.

### Python API → Alerts

Insert `trackio.alert()` calls in training code to flag important events — like inserting print statements for debugging, but structured and queryable:

- `trackio.alert(title="...", level=trackio.AlertLevel.WARN)` — fire an alert
- Three severity levels: `INFO`, `WARN`, `ERROR`
- Alerts are printed to terminal, stored in the database, shown in the dashboard, and optionally sent to webhooks (Slack/Discord)

**Key concept for LLM agents**: Alerts are the primary mechanism for autonomous experiment iteration. An agent should insert alerts into training code for diagnostic conditions (loss spikes, NaN gradients, low accuracy, training stalls). Since alerts are printed to the terminal, an agent that is watching the training script's output will see them automatically. For background or detached runs, the agent can poll via CLI instead.

→ See [references/alerts.md](references/alerts.md) for the full alerts API, webhook setup, and autonomous agent workflows.

### CLI → Retrieving

Use the `trackio` command to query logged metrics and alerts:

- `trackio list projects/runs/metrics` — discover what's available
- `trackio get project/run/metric` — retrieve summaries and values
- `trackio list alerts --project <name> --json` — retrieve alerts
- `trackio show` — launch the dashboard
- `trackio sync` — sync to HF Space

**Key concept**: Add `--json` for programmatic output suitable for automation and LLM agents.

→ See [references/retrieving_metrics.md](references/retrieving_metrics.md) for all commands, workflows, and JSON output formats.

## Minimal Logging Setup

```python
import trackio

trackio.init(project="my-project", space_id="username/trackio")
trackio.log({"loss": 0.1, "accuracy": 0.9})
trackio.log({"loss": 0.09, "accuracy": 0.91})
trackio.finish()
```

### Minimal Retrieval

```bash
trackio list projects --json
trackio get metric --project my-project --run my-run --metric loss --json
```

## Autonomous ML Experiment Workflow

When running experiments autonomously as an LLM agent, the recommended workflow is:

1. **Set up training with alerts** — insert `trackio.alert()` calls for diagnostic conditions
2. **Launch training** — run the script in the background
3. **Poll for alerts** — use `trackio list alerts --project <name> --json --since <timestamp>` to check for new alerts
4. **Read metrics** — use `trackio get metric ...` to inspect specific values
5. **Iterate** — based on alerts and metrics, stop the run, adjust hyperparameters, and launch a new run

```python
import trackio

trackio.init(project="my-project", config={"lr": 1e-4})

for step in range(num_steps):
    loss = train_step()
    trackio.log({"loss": loss, "step": step})

    if step > 100 and loss > 5.0:
        trackio.alert(
            title="Loss divergence",
            text=f"Loss {loss:.4f} still high after {step} steps",
            level=trackio.AlertLevel.ERROR,
        )
    if step > 0 and abs(loss) < 1e-8:
        trackio.alert(
            title="Vanishing loss",
            text="Loss near zero — possible gradient collapse",
            level=trackio.AlertLevel.WARN,
        )

trackio.finish()
```

Then poll from a separate terminal/process:

```bash
trackio list alerts --project my-project --json --since "2025-01-01T00:00:00"
```

````


### `references/alerts.md`

````markdown
# Trackio Alerts

Alerts let you flag important training events directly from code. They are the primary mechanism for LLM agents to diagnose runs and iterate autonomously on ML experiments.

Alerts are printed to the terminal, stored in the database, displayed in the dashboard, and optionally sent to webhooks (Slack/Discord).

## Core API

### trackio.alert()

```python
trackio.alert(
    title="Loss divergence",                    # Short title (required)
    text="Loss 5.2 still high after 200 steps", # Detailed description (optional)
    level=trackio.AlertLevel.WARN,               # INFO, WARN, or ERROR (default: WARN)
    webhook_url="https://hooks.slack.com/...",   # Per-alert webhook override (optional)
)
```

### Alert Levels

| Level | Usage |
|-------|-------|
| `trackio.AlertLevel.INFO` | Informational milestones (checkpoints saved, eval completed) |
| `trackio.AlertLevel.WARN` | Potential issues (loss plateau, low accuracy, high gradient norm) |
| `trackio.AlertLevel.ERROR` | Critical failures (NaN loss, divergence, OOM) |

### Webhook Support

Set a global webhook URL via `trackio.init()` or the `TRACKIO_WEBHOOK_URL` environment variable. Alerts are auto-formatted for Slack and Discord URLs.

```python
trackio.init(
    project="my-project",
    webhook_url="https://hooks.slack.com/services/...",
    webhook_min_level=trackio.AlertLevel.WARN,  # Only send WARN+ to webhook
)
```

Per-alert override:

```python
trackio.alert(
    title="Critical failure",
    level=trackio.AlertLevel.ERROR,
    webhook_url="https://hooks.slack.com/services/...",  # Overrides global URL
)
```

Environment variables:
- `TRACKIO_WEBHOOK_URL` — global webhook URL
- `TRACKIO_WEBHOOK_MIN_LEVEL` — minimum level for webhook delivery (`info`, `warn`, `error`)

## Retrieving Alerts (CLI)

```bash
# List all alerts for a project
trackio list alerts --project my-project --json

# Filter by run or level
trackio list alerts --project my-project --run my-run --level error --json

# Poll for new alerts since a timestamp (efficient for agents)
trackio list alerts --project my-project --json --since "2025-06-01T12:00:00"
```

### JSON Output Structure

```json
{
  "project": "my-project",
  "run": null,
  "level": null,
  "since": "2025-06-01T12:00:00",
  "alerts": [
    {
      "run": "run-name",
      "title": "Loss divergence",
      "text": "Loss 5.2 still high after 200 steps",
      "level": "warn",
      "step": 200,
      "timestamp": "2025-06-01T12:05:30"
    }
  ]
}
```

## Autonomous Agent Workflow

The recommended pattern for an LLM agent running ML experiments:

### 1. Insert Alerts Into Training Code

Add diagnostic `trackio.alert()` calls for conditions the agent should react to:

```python
import trackio

trackio.init(project="hyperparam-sweep", config={"lr": lr, "batch_size": bs})

for step in range(num_steps):
    loss = train_step()
    trackio.log({"loss": loss, "step": step})

    if step > 200 and loss > 5.0:
        trackio.alert(
            title="Loss divergence",
            text=f"Loss {loss:.4f} still above 5.0 after {step} steps — learning rate may be too high",
            level=trackio.AlertLevel.ERROR,
        )

    if step > 500 and loss_delta < 0.001:
        trackio.alert(
            title="Training stall",
            text=f"Loss barely changed over last 100 steps (delta={loss_delta:.6f})",
            level=trackio.AlertLevel.WARN,
        )

    if math.isnan(loss):
        trackio.alert(
            title="NaN loss",
            text="Loss became NaN — training is broken",
            level=trackio.AlertLevel.ERROR,
        )
        break

trackio.finish()
```

### 2. Monitor Alerts

Alerts are automatically printed to the terminal when fired. If the agent is watching the training script's output (e.g. running in the foreground or tailing logs), it will see alerts immediately — no polling needed.

For background or detached runs, poll for alerts via CLI:

```bash
# Poll for alerts (run periodically)
trackio list alerts --project hyperparam-sweep --json --since "2025-06-01T00:00:00"
```

### 3. Inspect Metrics Around the Alert

When an alert fires, use `trackio get snapshot` to see all metrics at that point:

```bash
# Alert fired at step 200 — get all metrics in a ±5 step window
trackio get snapshot --project hyperparam-sweep --run run-1 --around 200 --window 5 --json

# Or inspect a single metric around the alert's timestamp
trackio get metric --project hyperparam-sweep --run run-1 --metric loss --around 200 --window 10 --json
```

### 4. React and Iterate

Based on alerts:
- **ERROR alerts** → stop the run, adjust hyperparameters, relaunch
- **WARN alerts** → inspect metrics with `trackio get snapshot ...`, decide whether to intervene
- **INFO alerts** → note progress, continue monitoring

### 5. Compare Across Runs

```bash
# Check metrics from previous runs
trackio get run --project hyperparam-sweep --run run-1 --json
trackio get metric --project hyperparam-sweep --run run-1 --metric loss --json

# Launch new run with adjusted config
python train.py --lr 5e-5
```

## Using Alerts with Transformers / TRL

When using `report_to="trackio"`, you don't control the training loop directly. Use a `TrainerCallback` to fire alerts:

```python
from transformers import TrainerCallback

class AlertCallback(TrainerCallback):
    def on_log(self, args, state, control, logs=None, **kwargs):
        if "trackio" not in args.report_to:
            return
        if logs and "loss" in logs:
            if logs["loss"] > 5.0 and state.global_step > 100:
                trackio.alert(
                    title="High loss",
                    text=f"Loss {logs['loss']:.4f} at step {state.global_step}",
                    level=trackio.AlertLevel.ERROR,
                )

trainer = SFTTrainer(
    model=model,
    args=SFTConfig(output_dir="./out", report_to="trackio"),
    callbacks=[AlertCallback()],
    ...
)
```

````


### `references/logging_metrics.md`

````markdown
# Logging Metrics with Trackio

**Trackio** is a lightweight, free experiment tracking library from Hugging Face. It provides a wandb-compatible API for logging metrics with local-first design.

- **GitHub**: [gradio-app/trackio](https://github.com/gradio-app/trackio)
- **Docs**: [huggingface.co/docs/trackio](https://huggingface.co/docs/trackio/index)

## Installation

```bash
pip install trackio
# or
uv pip install trackio
```

## Core API

### Basic Usage

```python
import trackio

# Initialize a run
trackio.init(
    project="my-project",
    config={"learning_rate": 0.001, "epochs": 10}
)

# Log metrics during training
for epoch in range(10):
    loss = train_epoch()
    trackio.log({"loss": loss, "epoch": epoch})

# Finalize the run
trackio.finish()
```

### Key Functions

| Function | Purpose |
|----------|---------|
| `trackio.init(...)` | Start a new tracking run |
| `trackio.log(dict)` | Log metrics (called repeatedly during training) |
| `trackio.finish()` | Finalize run and ensure all metrics are saved |
| `trackio.show()` | Launch the local dashboard |
| `trackio.sync(...)` | Sync local project to HF Space |

## trackio.init() Parameters

```python
trackio.init(
    project="my-project",           # Project name (groups runs together)
    name="run-name",                # Optional: name for this specific run
    config={...},                   # Hyperparameters and config to log
    space_id="username/trackio",    # Optional: sync to HF Space for remote dashboard
    group="experiment-group",       # Optional: group related runs
)
```

## Local vs Remote Dashboard

### Local (Default)

By default, trackio stores metrics in a local SQLite database and runs the dashboard locally:

```python
trackio.init(project="my-project")
# ... training ...
trackio.finish()

# Launch local dashboard
trackio.show()
```

Or from terminal:
```bash
trackio show --project my-project
```

### Remote (HF Space)

Pass `space_id` to sync metrics to a Hugging Face Space for persistent, shareable dashboards:

```python
trackio.init(
    project="my-project",
    space_id="username/trackio"  # Auto-creates Space if it doesn't exist
)
```

⚠️ **For remote training** (cloud GPUs, HF Jobs, etc.): Always use `space_id` since local storage is lost when the instance terminates.

### Sync Local to Remote

Sync existing local projects to a Space:

```python
trackio.sync(project="my-project", space_id="username/my-experiments")
```

## wandb Compatibility

Trackio is API-compatible with wandb. Drop-in replacement:

```python
import trackio as wandb

wandb.init(project="my-project")
wandb.log({"loss": 0.5})
wandb.finish()
```

## TRL Integration

When using TRL trainers, set `report_to="trackio"` for automatic metric logging:

```python
from trl import SFTConfig, SFTTrainer
import trackio

trackio.init(
    project="sft-training",
    space_id="username/trackio",
    config={"model": "Qwen/Qwen2.5-0.5B", "dataset": "trl-lib/Capybara"}
)

config = SFTConfig(
    output_dir="./output",
    report_to="trackio",  # Automatic metric logging
    # ... other config
)

trainer = SFTTrainer(model=model, args=config, ...)
trainer.train()
trackio.finish()
```

## What Gets Logged

With TRL/Transformers integration, trackio automatically captures:
- Training loss
- Learning rate
- Eval metrics
- Training throughput

For manual logging, log any numeric metrics:

```python
trackio.log({
    "train_loss": 0.5,
    "train_accuracy": 0.85,
    "val_loss": 0.4,
    "val_accuracy": 0.88,
    "epoch": 1
})
```

## Grouping Runs

Use `group` to organize related experiments in the dashboard sidebar:

```python
# Group by experiment type
trackio.init(project="my-project", name="baseline-v1", group="baseline")
trackio.init(project="my-project", name="augmented-v1", group="augmented")

# Group by hyperparameter
trackio.init(project="hyperparam-sweep", name="lr-0.001", group="lr_0.001")
trackio.init(project="hyperparam-sweep", name="lr-0.01", group="lr_0.01")
```

## Configuration Best Practices

Keep config minimal — only log what's useful for comparing runs:

```python
trackio.init(
    project="qwen-sft-capybara",
    name="baseline-lr2e5",
    config={
        "model": "Qwen/Qwen2.5-0.5B",
        "dataset": "trl-lib/Capybara",
        "learning_rate": 2e-5,
        "num_epochs": 3,
        "batch_size": 8,
    }
)
```

## Embedding Dashboards

Embed Space dashboards in websites with query parameters:

```html
<iframe 
  src="https://username-trackio.hf.space/?project=my-project&metrics=train_loss,val_loss&sidebar=hidden" 
  style="width:1600px; height:500px; border:0;">
</iframe>
```

Query parameters:
- `project`: Filter to specific project
- `metrics`: Comma-separated metric names to show
- `sidebar`: `hidden` or `collapsed`
- `smoothing`: 0-20 (smoothing slider value)
- `xmin`, `xmax`: X-axis limits

````


### `references/retrieving_metrics.md`

````markdown
# Retrieving Metrics with Trackio CLI

The `trackio` CLI provides direct terminal access to query Trackio experiment tracking data locally without needing to start the MCP server.

## Quick Command Reference

| Task | Command |
|------|---------|
| List projects | `trackio list projects` |
| List runs | `trackio list runs --project <name>` |
| List metrics | `trackio list metrics --project <name> --run <name>` |
| List system metrics | `trackio list system-metrics --project <name> --run <name>` |
| List alerts | `trackio list alerts --project <name> [--run <name>] [--level <level>] [--since <timestamp>]` |
| Get project summary | `trackio get project --project <name>` |
| Get run summary | `trackio get run --project <name> --run <name>` |
| Get metric values | `trackio get metric --project <name> --run <name> --metric <name>` |
| Get metric at step | `trackio get metric ... --metric <name> --step <N>` |
| Get metric around step | `trackio get metric ... --metric <name> --around <N> --window <W>` |
| Get all metrics snapshot | `trackio get snapshot --project <name> --run <name> --step <N>` |
| Get system metrics | `trackio get system-metric --project <name> --run <name>` |
| Show dashboard | `trackio show [--project <name>]` |
| Sync to Space | `trackio sync --project <name> --space-id <space_id>` |

## Core Commands

### List Commands

```bash
trackio list projects                                    # List all projects
trackio list projects --json                            # JSON output

trackio list runs --project <name>                      # List runs in project
trackio list runs --project <name> --json               # JSON output

trackio list metrics --project <name> --run <name>      # List metrics for run
trackio list metrics --project <name> --run <name> --json

trackio list system-metrics --project <name> --run <name>  # List system metrics
trackio list system-metrics --project <name> --run <name> --json

trackio list alerts --project <name>                       # List alerts
trackio list alerts --project <name> --run <name> --json   # Filter by run
trackio list alerts --project <name> --level error --json  # Filter by level
trackio list alerts --project <name> --json --since <ts>   # Poll since timestamp
```

### Get Commands

```bash
trackio get project --project <name>                    # Project summary
trackio get project --project <name> --json             # JSON output

trackio get run --project <name> --run <name>           # Run summary
trackio get run --project <name> --run <name> --json

trackio get metric --project <name> --run <name> --metric <name>  # Metric values
trackio get metric --project <name> --run <name> --metric <name> --json
trackio get metric ... --metric <name> --step 200                 # At exact step
trackio get metric ... --metric <name> --around 200 --window 10   # ±10 steps
trackio get metric ... --metric <name> --at-time <ts> --window 60 # ±60 seconds

trackio get snapshot --project <name> --run <name> --step 200 --json       # All metrics at step
trackio get snapshot --project <name> --run <name> --around 200 --window 5 --json  # Window
trackio get snapshot --project <name> --run <name> --at-time <ts> --window 60 --json

trackio get system-metric --project <name> --run <name>           # All system metrics
trackio get system-metric --project <name> --run <name> --metric <name>  # Specific metric
trackio get system-metric --project <name> --run <name> --json
```

### Dashboard Commands

```bash
trackio show                                              # Launch dashboard
trackio show --project <name>                           # Load specific project
trackio show --theme <theme>                            # Custom theme
trackio show --mcp-server                                # Enable MCP server
trackio show --color-palette "#FF0000,#00FF00"         # Custom colors
```

### Sync Commands

```bash
trackio sync --project <name> --space-id <space_id>     # Sync to HF Space
trackio sync --project <name> --space-id <space_id> --private  # Private space
trackio sync --project <name> --space-id <space_id> --force   # Overwrite
```

## Output Formats

All `list` and `get` commands support two output formats:

- **Human-readable** (default): Formatted text for terminal viewing
- **JSON** (with `--json` flag): Structured JSON for programmatic use

## Common Patterns

### Discover Projects and Runs

```bash
# List all available projects
trackio list projects

# List runs in a project
trackio list runs --project my-project

# Get project overview
trackio get project --project my-project --json
```

### Inspect Run Details

```bash
# Get run summary with all metrics
trackio get run --project my-project --run my-run --json

# List available metrics
trackio list metrics --project my-project --run my-run

# Get specific metric values
trackio get metric --project my-project --run my-run --metric loss --json
```

### Query System Metrics

```bash
# List system metrics (GPU, etc.)
trackio list system-metrics --project my-project --run my-run

# Get all system metric data
trackio get system-metric --project my-project --run my-run --json

# Get specific system metric
trackio get system-metric --project my-project --run my-run --metric gpu_utilization --json
```

### Automation Scripts

```bash
# Extract latest metric value
LATEST_LOSS=$(trackio get metric --project my-project --run my-run --metric loss --json | jq -r '.values[-1].value')

# Export run summary to file
trackio get run --project my-project --run my-run --json > run_summary.json

# Filter runs with jq
trackio list runs --project my-project --json | jq '.runs[] | select(startswith("train"))'
```

### LLM Agent Workflow

```bash
# 1. Discover available projects
trackio list projects --json

# 2. Explore project structure
trackio get project --project my-project --json

# 3. Inspect specific run
trackio get run --project my-project --run my-run --json

# 4. Query metric values
trackio get metric --project my-project --run my-run --metric accuracy --json

# 5. Poll for alerts (use --since for efficient incremental polling)
trackio list alerts --project my-project --json --since "2025-06-01T00:00:00"

# 6. When an alert fires at step N, get all metrics around that point
trackio get snapshot --project my-project --run my-run --around 200 --window 5 --json
```

## Error Handling

Commands validate inputs and return clear errors:

- Missing project: `Error: Project '<name>' not found.`
- Missing run: `Error: Run '<name>' not found in project '<project>'.`
- Missing metric: `Error: Metric '<name>' not found in run '<run>' of project '<project>'.`

All errors exit with non-zero status code and write to stderr.

## Key Options

- `--project`: Project name (required for most commands)
- `--run`: Run name (required for run-specific commands)
- `--metric`: Metric name (required for metric-specific commands)
- `--json`: Output in JSON format instead of human-readable
- `--step`: Exact step filter (for `get metric`, `get snapshot`)
- `--around`: Center step for window filter (for `get metric`, `get snapshot`)
- `--at-time`: Center ISO timestamp for window filter (for `get metric`, `get snapshot`)
- `--window`: Window size: ±steps for `--around`, ±seconds for `--at-time` (default: 10)
- `--level`: Alert level filter (`info`, `warn`, `error`) (for `list alerts`)
- `--since`: ISO timestamp to filter alerts after (for `list alerts`)
- `--theme`: Dashboard theme (for `show` command)
- `--mcp-server`: Enable MCP server mode (for `show` command)
- `--color-palette`: Comma-separated hex colors (for `show` command)
- `--private`: Create private Space (for `sync` command)
- `--force`: Overwrite existing database (for `sync` command)

## JSON Output Structure

### List Projects
```json
{"projects": ["project1", "project2"]}
```

### List Runs
```json
{"project": "my-project", "runs": ["run1", "run2"]}
```

### Project Summary
```json
{
  "project": "my-project",
  "num_runs": 3,
  "runs": ["run1", "run2", "run3"],
  "last_activity": 100
}
```

### Run Summary
```json
{
  "project": "my-project",
  "run": "my-run",
  "num_logs": 50,
  "metrics": ["loss", "accuracy"],
  "config": {"learning_rate": 0.001},
  "last_step": 49
}
```

### Metric Values
```json
{
  "project": "my-project",
  "run": "my-run",
  "metric": "loss",
  "values": [
    {"step": 0, "timestamp": "2024-01-01T00:00:00", "value": 0.5},
    {"step": 1, "timestamp": "2024-01-01T00:01:00", "value": 0.4}
  ]
}
```

## References

- **Complete CLI documentation**: See [docs/source/cli_commands.md](docs/source/cli_commands.md)
- **API and MCP Server**: See [docs/source/api_mcp_server.md](docs/source/api_mcp_server.md)


````