# SkillPatch skill: agent-platform-tuning

This skill provides procedural knowledge for fine-tuning Large Language Models (Open Models and Gemini Models) using Agent Platform's tuning service. It covers the full lifecycle from environment setup, dataset preparation, job configuration and submission, monitoring, and deployment. The skill includes a structured workflow decision tree to guide agents step-by-step through the tuning process.

You (the agent) were given this URL and asked to use this skill. This is a **public** skill — no SkillPatch account, API key, or prior setup is required. Two ways to use it:

**1. Use it right now** — the skill's complete file tree (SKILL.md + all reference files) is inlined below; read `SKILL.md` first, then follow it, consulting the other files as it directs.

**2. Install the exact package onto disk** (recommended if you can run a shell — this reproduces the skill byte-for-byte, including any binary assets that can't be inlined):

```bash
mkdir -p .claude/skills/agent-platform-tuning
curl -sSL https://skillpatch.dev/install_skill/agent-platform-tuning | tar -xz -C .claude/skills/
```

(`.claude/skills/` is Claude Code's convention; use whatever directory your agent loads skills from.)


---

## Skill files (11)

- `SKILL.md`
- `references/data_prep.md`
- `references/hf_datasets.md`
- `references/models.md`
- `references/requirements.txt`
- `references/tuning_guide.md`
- `scripts/calculate_cost.py`
- `scripts/cancel_tuning_job.py`
- `scripts/monitor_tuning_job.py`
- `scripts/prepare_dataset.py`
- `scripts/tune_open_model.py`


### `SKILL.md`

````markdown
---
name: agent-platform-tuning
metadata:
  category: AiAndMachineLearning
description: >-
  Agent Platform Model Tuning. Use when you need to fine-tune open models
  or Gemini models using Agent Platform infrastructure. Don't use for model
  training outside Agent Platform, model deployment to endpoints (use
  `agent-platform-deploy`), or managing serving endpoints (use
  `agent-platform-endpoint-management`).
---

# Agent Platform Model Tuning

## Overview

This skill provides procedural knowledge for fine-tuning Large Language Models
(both Open Models and Gemini Models) using Agent Platform's tuning service. It
covers the entire lifecycle from environment setup and data preparation to job
configuration, monitoring, and deployment.

## Workflow Decision Tree

1.  **Model Category Identification**: Has the user explicitly stated whether
    they want to tune an **Open Model** or a **Gemini Model**?

    -   **No** → **STOP**. Ask the user if they want to tune an Open Model or a
        Gemini Model. **CRITICAL EXCEPTION for Environment Setup Requests:** If
        the user is specifically asking for environment setup instructions (e.g.
        "What environment setup is needed?"), you **MUST** provide the full
        [Phase 0 environment setup](#phase-0) instructions in your initial
        response, *simultaneously* with asking clarifying questions about the
        model category.
    -   If the user provides a specific tuning purpose, you should
        recommend three models: one Open Model, one Gemini Model, and a third
        generally recommended choice. Briefly list the pros and cons of each
        (e.g., Gemini models might be more expensive, etc.). **CRITICAL:** You
        must read `references/models.md` during this step and only recommend
        models explicitly listed in that catalog. Do not recommend unsupported
        models like Mistral. Do not proceed with model configuration until the
        category is confirmed.
    -   **Yes** → Proceed.

2.  **Environment Check**: Has the environment (Auth, APIs, IAM, Venv) been
    initialized?

    -   **No** → Go to [Phase 0: Environment & IAM Setup](#phase-0).
    -   **Yes** → Proceed.

3.  **Dataset Status**: Is the dataset ready in JSONL format, **is its structure
    valid for tuning**, and is it uploaded to Google Cloud Storage?

        -   **No** → Go to [Phase 1: Dataset Preparation & Upload](#phase-1).
        -   **Yes** → Proceed.

4.  **Column Selection Confirmation**: Have you presented the columns to the
    user and confirmed the mapping?

    -   **No** → **STOP**. You must show samples and get user confirmation on
        column mapping as described in Phase 1.0 before proceeding.
    -   **Yes** → Proceed.

5.  **Configuration**: Has the user provided the target model and
    hyperparameters, or explicitly agreed to your recommendations?

    -   **No** → Go to
        [Phase 2: Model Configuration & Recommendation](#phase-2).
    -   **Yes** → Proceed.

6.  **Job Status**: Has the tuning job been submitted?

        -   **No** → Go to
            [Phase 3: Tuning Job Execution](#phase-3-tuning-job-execution).
        -   **Yes** → Proceed.

7.  **Job Completion**: Is the tuning job complete?

        -   **No** → Go to [Phase 4: Monitoring](#phase-4-monitoring).
        -   **Yes** → Proceed.

8.  **Deployment**: Has the tuned model been deployed (if required)?

        -   **No** → Go to [Phase 5: Model Deployment](#phase-5-model-deployment).
        -   **Yes** → Task Complete.

## Phase 0: Environment & IAM Setup {#phase-0}

Ensure the foundational environment is ready before proceeding.

### 0.1 Authentication & Project Context

-   Check if `gcloud` CLI is installed. If it is not installed, prompt the
    user for permission to install it before proceeding. If it is installed,
    update it:

```bash
gcloud components update --quiet > /dev/null 2>&1
```

-   Verify `gcloud auth list`. If not authenticated, run `gcloud auth login`.
-   Ensure `project` and `location` are known. Use `gcloud config get project`
    to retrieve the current project (and `gcloud config get compute/region` for
    region).
-   **CRITICAL: Ask for Confirmation.** You must prompt the user to confirm the
    retrieved project and region before proceeding, in case they want to switch
    to a different one.

### 0.2 Possible Locations

The following locations are available for tuning:

-   us-central1
-   europe-west4
-   us-west1
-   us-east5
-   asia-southeast1

No other values are supported for this section, ensure that the location is
listed above.

### 0.3 Enable APIs

Ensure `aiplatform.googleapis.com` and `storage.googleapis.com` are enabled.

```bash
gcloud services enable aiplatform.googleapis.com storage.googleapis.com \
    --project=YOUR_PROJECT
```

### 0.4 IAM Permissions

Verify the following identities have the required roles.

-   **Agent Platform Service Agent**:
    `service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com`
-   **Managed OSS Fine Tuning Service Agent**:
    `service-PROJECT_NUMBER@gcp-sa-vertex-moss-ft.iam.gserviceaccount.com`
-   **User Identity**: The account running the commands.

### 0.5 Virtual Environment

Create and use a virtual environment named `tuning_agent_venv` in the home
directory. Install dependencies from `references/requirements.txt`.

```bash
python3 -m venv ~/tuning_agent_venv
source ~/tuning_agent_venv/bin/activate
pip install -r references/requirements.txt
```

**CRITICAL AGENT INSTRUCTION:** You **MUST** ensure that every Python command or
script execution (e.g., `python3 scripts/...`, `pip install ...`) is prefixed
with the virtual environment activation command: `source
~/tuning_agent_venv/bin/activate &&`. Additionally, advise the user that every
single time they run a Python command, execute a script, or inspect data inline,
they **MUST** also activate this virtual environment first in their bash
execution. For example: `source ~/tuning_agent_venv/bin/activate && python3
...`. Do not run standalone `python3` commands without activating the
environment, as they will encounter `ModuleNotFoundError` issues.

## Phase 1: Dataset Preparation & Upload {#phase-1}

### 1.0 Dataset Discovery & Confirmation

-   **User-Provided Dataset Verification:** If the user specifies a dataset
    filename or path in their prompt, verify its existence in the workspace
    (e.g. via script execution or checking for typos).
    *   **If the file cannot be found anywhere**, you **MUST** inform the user
    that the dataset file does not exist or cannot be accessed. You **MUST**
    prompt the user to provide a valid dataset path. Alternatively, if candidate
    dataset files are found in the workspace during your search, you **MUST**
    present the candidates to the user and ask them to select one. You **MUST**
    stop tool execution immediately after reporting the missing file or
    presenting candidates, and wait for the user's response. Do **NOT** ask for
    80/20 validation split permission, and do **NOT** attempt to upload the
    dataset before receiving a valid dataset file selection from the user.
    *   **If the file is found and verified**, proceed to Step 1.1 Formatting &
    Validation below.
-   **Auto-Discovery: From User Bucket:** If the user does not have a dataset
    and no suitable alternative is found in the Hugging Face reference, offer to
    search the user's GCS buckets for potential training data. Prioritize
    searching for files with extensions like `.jsonl`, `.json`, `.csv`, and
    `.parquet`. If such files are found, read the first few lines/records of
    each to determine if they contain text-based data suitable for tuning
    (e.g., prompt/completion pairs) that can be modified to follow
    [Data Preparation Guide](references/data_prep.md) and is related to the
    tuning task requested. **DO NOT** search without prompting first.
-   **Auto-Discovery: From Task to Huggingface:** If the user has a specific
    task, refer to [Huggingface Datasets Reference](references/hf_datasets.md)
    and recommend a dataset from this if one exists. For each dataset
    recommended, provide some information about the dataset and provide some
    reasonable splits. > [!IMPORTANT] > **CRITICAL: Ask for Confirmation and
    Column Selection.** Do not proceed > with dataset preparation or upload
    until you perform the following > steps and get user confirmation: > 1.
    **Dataset and Split Confirmation:** Present the dataset and > available
    splits to the user and have them confirm which to use. > 2. **Column
    Selection (Hugging Face or Custom Datasets):** You must: > - Provide a list
    of all available columns in the selected dataset > split. > - **Show a few
    samples from the dataset** to help the user > understand the content and
    make the choice of columns. > - Recommend which columns should be mapped to
    `prompt` (or user > message) and `completion` (or assistant response),
    offering a few > reasonable options if applicable. > - Ask the user to
    confirm the column mapping or specify which > columns to use.

### 1.1 Formatting & Validation

-   **Conversion**: If data is in CSV, JSON, or Parquet, use
    `scripts/prepare_dataset.py` to convert.
-   **Validation Split Confirmation**: If the user only provides a training
    dataset, **you must prompt the user** to seek permission to split the
    training dataset 80/20 to form a validation dataset (using
    `--validation_split 0.2`). If they agree, proceed with the split. If they
    decline, just use the training dataset without a validation dataset.
-   **Validation**: If data is already in JSONL, validate it before uploading.
    Simply having a `.jsonl` extension is not enough. You must verify that the
    content schema is valid for tuning (e.g. correct system/user/model roles).

```bash
python3 scripts/prepare_dataset.py \
    --input my_data.jsonl \
    --format <messages|messages_gemini> \
    --validate_only
```

*(Use `--format messages` for open models and `--format messages_gemini` for
Gemini models.)* - Refer to [Data Preparation Guide](references/data_prep.md)
for required schemas.

### 1.2 Upload

Upload formatted `.jsonl` files to GCS using a unique directory (e.g., with a
datetime timestamp) to avoid overwriting outputs from different runs.
<!-- disableFinding(LINE_OVER_80) --> `bash
ARTIFACTS="gs://YOUR_BUCKET/tuning_agent_job_<datetime>/dataset.jsonl" gcloud
storage cp dataset.jsonl $ARTIFACTS` <!-- enableFinding(LINE_OVER_80) -->

## Phase 2: Model Configuration & Recommendation {#phase-2}

Help the user choose the best model and parameters. **Always seek user
confirmation before submitting the job.**

-   If the user does not specify a specific model in their prompt, calculate
    recommendations based on the **Models Catalog**.
-   **Prompt for Confirmation:** Present the recommended model to the user and
    ask for their confirmation before configuring hyperparameters.

### 2.1 Configuration

#### For Open Models

-   Recommend `tuning_mode`, `epochs`, `learning_rate`, and `adapter_size`
    based on the [Tuning Guide](references/tuning_guide.md) and model-specific
    baselines in the [Models Catalog](references/models.md).

### 2.2 Calculating Cost (Open Models Only)

-   We can calculate a rough estimate of cost of tuning based on the dataset and
    the selected model in the [Models Catalog](references/models.md):

    ```bash
    python3 scripts/calculate_cost.py \
        --input my_data.jsonl \
        --model MODEL_NAME \
        --tuning_mode TUNING_MODE \
        --epochs epochs
    ```

> [!NOTE]
> **Handling Missing Dataset Errors:** If `scripts/calculate_cost.py` fails
> because the dataset file (e.g. `my_data.jsonl` or `dummy_data.jsonl`) cannot
> be found, you **MUST** inform the user that the dataset file does not exist or
> cannot be accessed. You **MUST** prompt the user to provide a valid dataset
> path, and stop tool execution immediately to wait for their response. Do
> **NOT** retry or loop, do **NOT** invent a specific cost number, and do
> **NOT** prompt for job submission approval before receiving a valid dataset
> from the user.

-   **Prompt for Confirmation:** Present the recommended hyperparameter
    configuration and estimated cost to the user and ask for their approval
    before proceeding to job submission. Make sure to note that the estimated
    cost is just an estimate and can vary from actual billing costs.

## Phase 3: Tuning Job Execution {#phase-3-tuning-job-execution}

**CRITICAL Pre-Flight Check (GCS Verification):** Before you propose a
confirmation prompt or submit any tuning job, you **MUST** verify that the
specified training dataset GCS URI (e.g. `gs://dummy_bucket/dataset.jsonl` or
`gs://YOUR_BUCKET/...`) actually exists and is accessible. Run
`gcloud storage ls $DATASET_URI` (or `gsutil ls`).

*   **If the verification fails** (e.g. `BucketNotFound`, `404`, `AccessDenied`,
    or indicating a dummy/missing bucket), you **MUST** inform the user that the
    GCS bucket or dataset does not exist or cannot be accessed. You **MUST**
    prompt the user to provide a valid GCS URI for the dataset, and stop tool
    execution immediately to wait for their response. Do **NOT** propose a
    confirmation prompt and do **NOT** execute any tuning scripts before
    receiving a valid dataset URI from the user.
*   **If the verification succeeds**, proceed to propose the confirmation prompt
    below.

### For Gemini Models

Check if `scripts/tune_gemini_model.py` exists.

-   **If `scripts/tune_gemini_model.py` exists:** Submit the Gemini model tuning
    job using this script.

    ```bash
    python3 scripts/tune_gemini_model.py
    ```

-   **If `scripts/tune_gemini_model.py` does not exist:** Instruct the user to
    manually configure and submit the tuning job via the Google Cloud Console
    UI or using the Agent Platform SDK for Python.

### For Open Models

Submit the open model tuning job using `scripts/tune_open_model.py`. Identify
the model id using available models documentation at
<!-- disableFinding(LINE_OVER_80) -->
[documentation](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/open-model-tuning.md.txt).
<!-- enableFinding(LINE_OVER_80) -->

```bash
python3 scripts/tune_open_model.py \
    --project YOUR_PROJECT \
    --location YOUR_LOCATION \
    --base_model BASE_MODEL_ID \
    --train_dataset gs://YOUR_BUCKET/tuning_agent_job_<datetime>/dataset.jsonl \
    --output_uri gs://YOUR_BUCKET/tuning_agent_job_<datetime>/output \
    --epochs EPOCHS \
    --learning_rate LR \
    --tuning_mode MODE
```

> [!IMPORTANT]
> **Interactive Confirmation Required (Tier M):** Before proceeding with job
> submission, you **MUST** present the proposed command string showing all
> literal flags in a confirmation prompt to the user with 'Yes' and 'No'
> options.

> **CRITICAL:** When presenting this confirmation prompt to the user, you MUST
> output it as a direct plain text response and stop tool execution immediately.
> Do NOT call any command execution or interactive tools in the same turn, as
> unexpected tool calls may be auto-replied by the simulation harness and cause
> an infinite loop. Yield immediately for the user's reply.

## Phase 4: Monitoring {#phase-4-monitoring}

Monitor the job via the Cloud Console link provided in the script output.
Additionally, ask the user if they want you to monitor the job status for them
in the background. If they agree, execute `scripts/monitor_tuning_job.py` as
a background task to periodically poll the job status and notify the user to
show the status. If the user declines, leave it completely to the user to
check on the status.

## Phase 5: Model Deployment {#phase-5-model-deployment}

Once the tuning job is `SUCCEEDED`, deploy the model.

```bash
ARTIFACTS="gs://YOUR_BUCKET/tuning_agent_job_<datetime>/output/postprocess/node-0/checkpoints/final"
gcloud ai model-garden models deploy \
    --project=YOUR_PROJECT \
    --region=YOUR_LOCATION \
    --model="$ARTIFACTS" \
    --machine-type=MACHINE_TYPE \
    --accelerator-type=ACCELERATOR_TYPE \
    --accelerator-count=COUNT
```

> [!IMPORTANT]
> **Interactive Confirmation Required (Tier M):** Before proceeding with
> deployment, you **MUST** present the proposed command string showing all
> literal flags in a confirmation prompt to the user with 'Yes' and 'No'
> options.

> **CRITICAL:** When presenting this confirmation prompt to the user, you MUST
> output it as a direct plain text response and stop tool execution immediately.
> Do NOT call any command execution or interactive tools in the same turn, as
> unexpected tool calls may be auto-replied by the simulation harness and cause
> an infinite loop. Yield immediately for the user's reply.

Refer to [Models Catalog](references/models.md) for hardware recommendations for
specific open models.

## Resources

-   [Data Preparation Guide](references/data_prep.md)
-   [Models Catalog](references/models.md)
-   [Tuning Guide](references/tuning_guide.md)
-   `scripts/prepare_dataset.py`: Data conversion & validation.
-   `scripts/tune_open_model.py`: Open model tuning job submission.
````


### `references/data_prep.md`

````markdown
# Data Preparation for Agent Platform Model Tuning

Agent Platform Model Tuning requires training data in **JSON Lines (JSONL)**
format stored in Google Cloud Storage (GCS).

## Supported JSONL Formats for Open Models

### 1. Conversational (Messages) Format
Recommended for chat-based models (Llama 3.1/3.2/3.3 Chat, Gemma 3 IT, etc.).

```json
{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
    {"role": "assistant", "content": "The capital of France is Paris."}
  ]
}
```

### 2. Instruction (Prompt/Completion) Format
Suitable for base models or simple completion tasks.

```json
{
  "prompt": "Summarize the following text: [TEXT]",
  "completion": "[SUMMARY]"
}
```

## Dataset Requirements

- **File Type**: Must be `.jsonl`.
- **Encoding**: UTF-8.
- **Location**: Must be in a GCS bucket (e.g., `gs://my-bucket/train.jsonl`).
- **Validation Split**: A separate validation file is optional but recommended. It must be no more than 25% of the training dataset size.

## Bucket Considerations

If a bucket does not exist, create one in the same region as your tuning job:

```bash
gcloud storage buckets create gs://YOUR_BUCKET_NAME --location=YOUR_LOCATION
```

## Formatting Best Practices

1. **Quality over Quantity**: 100 high-quality examples often outperform 1,000 noisy ones.
2. **Consistency**: Use consistent formatting for system prompts and instruction styles.
3. **No Empty Values**: Ensure every example has a valid prompt/user message and
   completion/assistant response. Use the [preparation script](../scripts/prepare_dataset.py) to validate this.

````


### `references/hf_datasets.md`

````markdown
# Huggingface Dataset References

This document provides references for various datasets used in Agent Platform
Tuning.

## How to use datasets

When using a dataset for tuning, it is recommended to first analyze the dataset to understand its structure and content. For more information on data preparation and formatting, please refer to [Data Preparation for Tuning](data_prep.md).

To download the entire dataset, the following script can be used:

```python
from datasets import load_dataset
dataset = load_dataset("username/dataset_name")
```

To load a specific split, the following script can be used:

```python
from datasets import load_dataset
dataset = load_dataset("username/dataset_name", split="split_name")
```

For larger datasets, it might be helpful to subset the dataset and use only
portions of the dataset as required by the specific task requested. Either use
the splits defined in the huggingface website (ex. `default`) or offer to
partition the dataset for the user. Ensure that the user can see some examples
of the dataset before proceeding.

[!IMPORTANT]
**CRITICAL: Ask for Confirmation and Column Selection.**
Do not proceed with dataset preparation or upload until you perform the
following steps and get user confirmation:
1. **Dataset and Split Confirmation:** Present the dataset and available splits
    to the user and have them confirm which to use. Additionally, show a few
    samples to the user for preview.
2. **Column Selection:** The tuning entrypoint requires mapping source columns to the target format (either `prompt`/`completion` or `messages` format). You must:
    - Provide a list of all available columns in the selected dataset split.
    - Recommend which columns should be mapped to `prompt` (or user message) and `completion` (or assistant response), offering a few reasonable options if applicable.
    - Ask the user to confirm the column mapping or specify which columns to use.

## Datasets List

Each dataset includes a brief description of the dataset type and some usage
hints. The hints are **not** the only way to use these datasets but offer some
suggestions based on the corresponding dataset.

[!IMPORTANT]
Examine the hints since they include some information about the dataset. These
include comments about the contents of the dataset that may be pertinent to the
users request.

### General and Reasoning Tasks

#### Mathematical Reasoning

| Name | Description | Sample Count (Split) | Usage Hints |
|---|---|---|---|
| [open-r1/OpenR1-Math-220k](https://huggingface.co/datasets/open-r1/OpenR1-Math-220k) | A dataset of math problems and solutions. | 93,700 (default) - 220,000 (full) | The main columns are problem and solution. Some other helpful columns are answer, problem_type, question_type, and messages. |
| [AI-MO/NuminaMath-TIR](https://huggingface.co/datasets/AI-MO/NuminaMath-TIR) | A dataset for improving model performance in complex logic and calculations. | N/A | Good choice for mathematical reasoning. |

#### Instruction Following

| Name | Description | Sample Count (Split) | Usage Hints |
|---|---|---|---|
| [argilla/ifeval-like-data](https://huggingface.co/datasets/argilla/ifeval-like-data) | A dataset that involves instruction following. | 550,000 (default), 56,000 (filtered) | There are multiple languages in this dataset. Prompt the user with this dataset if specific languages are expected. Filter accordingly |
| [HuggingFaceTB/smoltalk2](https://huggingface.co/datasets/HuggingFaceTB/smoltalk2) | Enhancing broad instruction-following capabilities. | N/A | This needs to be subsetted. as the initial dataset is very large and covers a wide range of tasks. |

#### Multilingual Support

| Name | Description | Sample Count (Split) | Usage Hints |
|---|---|---|---|
| [CohereForAI/aya_dataset](https://huggingface.co/datasets/CohereForAI/aya_dataset) | Expanding linguistic capabilities across diverse languages. | N/A | Contains multi-language instruction following data. |

### Specialized and Technical Tasks

#### Programming & Coding

| Name | Description | Sample Count (Split) | Usage Hints |
|---|---|---|---|
| [ise-uiuc/Magicoder-Evol-Instruct-110K](https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K) | Code generation dataset. | 110,000 | Suitable for advancing coding capabilities. |
| [open-r1/datasets](https://huggingface.co/open-r1/datasets) | Specialized programming & reasoning data. | N/A | General source for open reasoning technical data. |

#### Tool Use & Integration

| Name | Description | Sample Count (Split) | Usage Hints |
|---|---|---|---|
| [gorilla-llm/Berkeley-Function-Calling-Leaderboard](https://huggingface.co/datasets/gorilla-llm/Berkeley-Function-Calling-Leaderboard) | Adhere to constraints and use external systems. | N/A | High-quality tool usage and function calling data. |
| [Bingguang/HardGen](https://huggingface.co/datasets/Bingguang/HardGen) | Evaluating handling complex tools and constraints. | N/A | Validated for tool use integration tasks. |


````


### `references/models.md`

```markdown
# Agent Platform Supported Models and Recommendations

This reference catalog provides technical specifications, tuning
recommendations, and deployment hardware requirements for supported models in
Agent Platform.

## Supported Models Catalog

> [!WARNING] **CRITICAL AGENT INSTRUCTION**
> Do NOT use this catalog to recommend a specific model to the user until they
> have explicitly confirmed their **Model Category** as Open Model.
> Furthermore, do NOT recommend any model that is not explicitly listed in this
> catalog, as the tuning service does not support it.

Available open models can be found in Google Cloud [documentation](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/open-model-tuning.md.txt).
This is the list of open models that are available for tuning; do not suggest
any other open models besides the one listed here.
Each model has some [limitations](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/open-model-tuning.md.txt) for tuning.

## Model Selection Guidelines

**Identify Task**: Check a few samples from the dataset to identify the task.

Choose a model family based on your task type:

- **Qwen**: Best for code generation or complex math-based tasks.
- **Gemma**: Optimized for chat-based interactions, creative writing and multilingual tasks.
- **Llama (Instruct)**: Strong general-purpose chat/instruction models.
- **Llama (Base/Scout)**: Best for continuation tasks or building custom instruction-tuned models.

**Complexity Heuristics**:

- **Simple (QA, Extraction)**: 1B - 3B models.
- **Intermediate (Summarization, Reasoning)**: 8B - 17B models.
- **Complex (Multi-turn, Tool use, Deep reasoning)**: 27B - 70B models.

## Baseline Hyperparameter Recommendations

These values are starting points and should be adjusted based on your dataset
size.

| Model | Tuning Mode | Learning Rate | Epochs | Adapter Size (PEFT) |
| :--- | :--- | :--- | :--- | :--- |
| Gemma 3 1B IT | Full | 2.0E-5 | 3 | N/A |
| Gemma 3 4B IT | Full | 1.0E-5 | 3 | N/A |
| Gemma 3 12B IT | Full | 1.0E-5 | 3 | N/A |
| Gemma 3 27B IT | PEFT | 2.0E-4 | 3 | 32 |
| Gemma 3 27B IT | Full | 2.0E-4 | 3 | N/A |
| Llama 3.1 8B | PEFT | 2.0E-4 | 3 | 16 |
| Llama 3.1 8B | Full | 2.0E-4 | 3 | N/A |
| Llama 3.1 8B Instruct | PEFT | 2.0E-4 | 3 | 16 |
| Llama 3.1 8B Instruct | Full | 2.0E-4 | 3 | N/A |
| Llama 3.2 1B Instruct | Full | 1.5E-6 | 3 | N/A |
| Llama 3.2 3B Instruct | Full | 1.0E-7 | 3 | N/A |
| Llama 3.3 70B Instruct | PEFT | 5.0E-5 | 3 | 16 |
| Llama 3.3 70B Instruct | Full | 5.0E-5 | 3 | N/A |
| Llama 4 Scout 17B 16E | PEFT | 2.0E-5 | 3 | 16 |
| Qwen 3 4B | Full | 7.5e-5 | 3 | N/A |
| Qwen 3 8B | Full | 5e-5 | 3 | N/A |
| Qwen 3 14B | Full | 4e-5 | 3 | N/A |
| Qwen 3 32B | PEFT | 2.0E-4 | 3 | 16 |
| Qwen 3 32B | Full | 2.5e-5 | 3 | N/A |
```


### `references/requirements.txt`

```
google-cloud-aiplatform==1.138.0
numpy==2.4.2
pandas==3.0.1
datasets==2.18.0
smart_open[gcs]==7.5.1
pyarrow==19.0.1
google-genai==1.73.1

```


### `references/tuning_guide.md`

```markdown
# Agent Platform Model Tuning Heuristics and Concepts

This guide details the core concepts of fine-tuning and provides heuristics for
adjusting hyperparameters based on your specific dataset.

## Core Tuning Concepts

### Open Models Tuning Modes

- **FULL**: Updates all parameters of the model. Requires more GPU memory and a larger dataset to avoid catastrophic forgetting.
- **PEFT_ADAPTER**: Parameter-Efficient Fine-Tuning. Only a small set of "adapter" weights are trained. Faster, uses less memory, and is less prone to overfitting on small datasets.

### Hyperparameters

- **Epochs**: Number of times the model sees the entire dataset.
- **Learning Rate**: Step size for optimization. Too high can cause instability; too low can lead to very slow convergence.
- **Adapter Size (Rank)**: For PEFT_ADAPTER, this determines the capacity of the adapters. Higher rank allows more complex learning but increases the risk of overfitting.

## Dataset Heuristics

The size and quality of your dataset should dictate your parameter choices. Refer to [Models Catalog](models.md) for baseline values, then adjust as follows:

### 1. Dataset Size Implications

| Dataset Size | Tuning Mode Recommendation | Learning Rate Adjustment | Epochs Recommendation |
| :--- | :--- | :--- | :--- |
| **< 100 examples** | PEFT_ADAPTER (Rank 8) | Lower than baseline | 1-2 |
| **100 - 1000 examples** | PEFT_ADAPTER (Rank 16/32) | Baseline | 3 |
| **> 1000 examples** | FULL or PEFT_ADAPTER (Rank 32) | Higher than baseline | 3-5 |

### 2. General Best Practices

- **Overfitting**: If validation loss starts increasing while training loss decreases, you are overfitting. Reduce epochs or decrease the learning rate.
- **Underfitting**: If both training and validation loss remain high, increase the learning rate or use more epochs.
- **Validation**: Always use a validation set to monitor performance. If not provided, a 10-20% split is highly recommended.
- **Checkpoints**: The final model is always saved to `<output_uri>/postprocess/node-0/checkpoints/final`.

## Hardware and Limitations
For specific hardware recommendations and sequence length limits per model, please refer to the [Models Catalog](models.md).

```


### `scripts/calculate_cost.py`

```
"""Calculate tuning cost for a given dataset and model."""

import argparse
import json
import sys
import smart_open

# Data from
# https://docs.google.com/spreadsheets/d/1pOXzfQBSCaKJYcemvRKv4b30qmUBx3yG28yScH-pnVI/edit?resourcekey=0-0kJGshytd3yrxB41YM4OFg&gid=0#gid=0
MODEL_DATA = {
    'Gemma 3 1B IT': {
        'Full': {'tokens_per_character': 0.231, 'cost_per_1m_tokens': 0.47},
    },
    'Gemma 3 4B IT': {
        'Full': {'tokens_per_character': 0.231, 'cost_per_1m_tokens': 1.14},
    },
    'Gemma 3 12B IT': {
        'Full': {'tokens_per_character': 0.231, 'cost_per_1m_tokens': 1.82},
    },
    'Gemma 3 27B IT': {
        'PEFT': {'tokens_per_character': 0.231, 'cost_per_1m_tokens': 6.83},
        'Full': {'tokens_per_character': 0.231, 'cost_per_1m_tokens': 6.83},
    },
    'Llama 3.1 8B': {
        'PEFT': {'tokens_per_character': 0.317, 'cost_per_1m_tokens': 0.67},
        'Full': {'tokens_per_character': 0.247, 'cost_per_1m_tokens': 0.67},
    },
    'Llama 3.1 8B Instruct': {
        'PEFT': {'tokens_per_character': 0.317, 'cost_per_1m_tokens': 0.67},
        'Full': {'tokens_per_character': 0.247, 'cost_per_1m_tokens': 0.67},
    },
    'Llama 3.2 1B Instruct': {
        'Full': {'tokens_per_character': 0.247, 'cost_per_1m_tokens': 0.28},
    },
    'Llama 3.2 3B Instruct': {
        'Full': {'tokens_per_character': 0.247, 'cost_per_1m_tokens': 0.61},
    },
    'Llama 3.3 70B Instruct': {
        'PEFT': {'tokens_per_character': 0.317, 'cost_per_1m_tokens': 6.72},
        'Full': {'tokens_per_character': 0.247, 'cost_per_1m_tokens': 6.72},
    },
    'Llama 4 Scout 17B 16E': {
        'PEFT': {'tokens_per_character': 0.295, 'cost_per_1m_tokens': 5.77},
    },
    'Qwen 3 4B': {
        'Full': {'tokens_per_character': 0.246, 'cost_per_1m_tokens': 1.35},
    },
    'Qwen 3 8B': {
        'Full': {'tokens_per_character': 0.246, 'cost_per_1m_tokens': 4.18},
    },
    'Qwen 3 14B': {
        'Full': {'tokens_per_character': 0.246, 'cost_per_1m_tokens': 8.46},
    },
    'Qwen 3 32B': {
        'PEFT': {'tokens_per_character': 0.246, 'cost_per_1m_tokens': 6.57},
        'Full': {'tokens_per_character': 0.246, 'cost_per_1m_tokens': 6.57},
    },
}


def count_characters(input_file: str) -> int:
  """Counts the characters in a jsonl dataset.

  It is expected that each line in the jsonl file is a json object
  with a "messages" key, which is a list of dictionaries. Each
  dictionary in the "messages" list should have a "content" key.
  This function counts the characters in the "content" field of each
  dictionary in the "messages" list.

  Args:
    input_file: Path to the input jsonl file.

  Returns:
    Total character count.
  """
  total_character_count = 0
  if not input_file.startswith('gs://') and '://' in input_file:
    raise ValueError(
        f'Unsupported file path: {input_file}. '
        'Only local paths and gs:// paths are supported.'
    )
  with smart_open.smart_open(input_file, 'r') as f:
    for line in f:
      data = json.loads(line)
      for message in data['messages']:
        content = message['content']
        total_character_count += len(content)
  return total_character_count


def calculate_cost(
    count: int,
    model: str,
    tuning_mode: str,
    epochs: int,
) -> float:
  """Calculates the tuning cost.

  Args:
    count: Total character count of the dataset.
    model: Model to use for tuning.
    tuning_mode: Tuning mode.
    epochs: Number of epochs.

  Returns:
    Estimated tuning cost.
  """
  model_data = MODEL_DATA[model][tuning_mode]
  tokens_per_character = model_data['tokens_per_character']
  cost_per_1m_tokens = model_data['cost_per_1m_tokens']
  num_tokens = count * tokens_per_character * epochs
  return (num_tokens / 1000000) * cost_per_1m_tokens


if __name__ == '__main__':
  parser = argparse.ArgumentParser(
      description='Calculate tuning cost for a given dataset and model.'
  )
  parser.add_argument('--input', help='Input jsonl file.', required=True)
  parser.add_argument(
      '--model',
      help='Model to use for tuning.',
      required=True,
      choices=MODEL_DATA.keys(),
  )
  parser.add_argument(
      '--tuning_mode',
      help='Tuning mode.',
      required=True,
      choices=['PEFT', 'Full'],
  )
  parser.add_argument(
      '--epochs',
      help='Number of epochs.',
      required=True,
      type=int,
  )
  args = parser.parse_args()

  if (
      args.model not in MODEL_DATA
      or args.tuning_mode not in MODEL_DATA[args.model]
  ):
    print(
        f'Error: Tuning mode {args.tuning_mode} not supported for model'
        f' {args.model}'
    )
    sys.exit(1)

  character_count = count_characters(args.input)
  cost = calculate_cost(
      character_count, args.model, args.tuning_mode, args.epochs
  )
  print(f'Total character count: {character_count}')
  print(f'Estimated tuning cost: ${cost:.2f}')

```


### `scripts/cancel_tuning_job.py`

```
"""Cancels a Agent Platform Supervised Tuning Job.

This script provides a function to cancel an ongoing SupervisedTuningJob
on Agent Platform given the project, location, and job ID.
"""

import argparse
import vertexai  # pytype: disable=import-error
from vertexai.tuning import sft  # pytype: disable=import-error


def cancel_job(project, location, job_id):
  vertexai.init(project=project, location=location)
  job_resource = f"projects/{project}/locations/{location}/tuningJobs/{job_id}"
  job = sft.SupervisedTuningJob(job_resource)
  print(f"Cancelling job: {job_resource}")
  job.cancel()
  print("Cancellation request sent.")


if __name__ == "__main__":
  parser = argparse.ArgumentParser(
      description="Cancel Vertex AI Supervised Tuning Job"
  )
  parser.add_argument("--project", required=True)
  parser.add_argument("--location", required=True)
  parser.add_argument("--job_id", required=True)

  args = parser.parse_args()
  cancel_job(args.project, args.location, args.job_id)

```


### `scripts/monitor_tuning_job.py`

```
"""Monitors a Agent Platform Supervised Tuning Job."""

import argparse
import logging
import time

from google import genai


def monitor_job(
    project: str, location: str, job_id: str, poll_interval_secs: int = 60
):
  """Monitors a Agent Platform Supervised Tuning Job.

  This function polls the job status until it reaches a terminal state.

  Args:
    project: The Google Cloud project ID.
    location: The Google Cloud location.
    job_id: The ID of the SupervisedTuningJob.
    poll_interval_secs: The interval in seconds to poll the job status.
  """
  job_resource = (
      f"projects/{project}/locations/{location}/tuningJobs/{job_id}"
  )

  logging.info("Starting monitoring for tuning job: %s", job_resource)
  logging.info(
      "View job in console: https://console.cloud.google.com/agent-platform/"
      "tuning/locations/%s/tuningJob/%s/monitor?project=%s",
      location,
      job_id,
      project,
  )
  with genai.Client(
      enterprise=True, project=project, location=location
  ) as client:
    while True:
      job = client.tunings.get(name=job_resource)
      status = job.state
      status_name = status.name if status else "unknown"

      if status_name in (
          "JOB_STATE_SUCCEEDED",
          "JOB_STATE_FAILED",
          "JOB_STATE_CANCELLED",
          "JOB_STATE_PARTIALLY_SUCCEEDED",
      ):
        logging.info("Job finished with terminal state: %s", status_name)
        break
      else:
        logging.info("Current job status: %s", status_name)

      time.sleep(poll_interval_secs)


if __name__ == "__main__":
  logging.basicConfig(
      level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s"
  )
  parser = argparse.ArgumentParser(
      description="Monitor Agent Platform Supervised Tuning Job"
  )
  parser.add_argument("--project", required=True)
  parser.add_argument("--location", required=True)
  parser.add_argument("--job_id", required=True)
  parser.add_argument(
      "--poll_interval_secs",
      type=int,
      default=60,
      help="Seconds to wait between polls",
  )

  args = parser.parse_args()
  monitor_job(args.project, args.location, args.job_id, args.poll_interval_secs)

```


### `scripts/prepare_dataset.py`

```
"""Module for preparing and validating datasets for Agent Platform model tuning."""

import argparse
import json
import logging
import os
import sys
from typing import Any

import datasets


def _validate_example(example: dict[str, Any], format_type: str) -> bool:
  """Validates a single example against the expected format."""
  if format_type == "messages":
    if "messages" not in example or not isinstance(example["messages"], list):
      return False
    for msg in example["messages"]:
      if not all(k in msg for k in ("role", "content")):
        return False
      if not msg["content"] or str(msg["content"]).strip().lower() == "nan":
        return False
  else:
    if not all(k in example for k in ("prompt", "completion")):
      return False
    for k in ("prompt", "completion"):
      if not example[k] or str(example[k]).strip().lower() == "nan":
        return False
  return True


def _format_row(
    row, format_type: str, prompt_col: str, completion_col: str
) -> dict[str, Any]:
  """Formats a single row into the expected JSON structure."""
  prompt_text = str(row[prompt_col])
  completion_text = str(row[completion_col])

  if format_type == "messages":
    return {
        "messages": [
            {"role": "user", "content": prompt_text},
            {"role": "assistant", "content": completion_text},
        ]
    }
  return {
      "prompt": prompt_text,
      "completion": completion_text,
  }


def validate_jsonl(file_path: str, format_type: str) -> bool:
  """Validates an existing JSONL file."""
  if not os.path.exists(file_path):
    logging.error("File not found: %s", file_path)
    return False

  valid_count = 0
  invalid_count = 0
  with open(file_path, "r", encoding="utf-8") as f:
    for i, line in enumerate(f):
      try:
        example = json.loads(line)
        if _validate_example(example, format_type):
          valid_count += 1
        else:
          invalid_count += 1
          logging.warning("Invalid format/empty content at line %d", i + 1)
      except json.JSONDecodeError:
        invalid_count += 1
        logging.warning("Invalid JSON at line %d", i + 1)

  logging.info("Validation complete for %s", file_path)
  logging.info("Valid: %d, Invalid: %d", valid_count, invalid_count)
  return invalid_count == 0


def convert_to_jsonl(
    input_file: str,
    output_file: str,
    format_type: str,
    prompt_col: str,
    completion_col: str,
    validation_split: float | None = 0.2,
):
  """Converts a file to JSONL format for Agent Platform tuning.

  Supports CSV, JSON, or Parquet files.

  Args:
    input_file: Path to the input CSV, JSON, or Parquet file.
    output_file: Path where the formatted JSONL file will be saved.
    format_type: Target format ("messages" or "prompt").
    prompt_col: Name of the column containing the prompt/user message.
    completion_col: Name of the column containing the completion/response.
    validation_split: Optional fraction of data to split for validation.
  """
  if not os.path.exists(input_file):
    logging.error("Input file not found: %s", input_file)
    sys.exit(1)

  try:
    if input_file.endswith(".csv"):
      dataset = datasets.load_dataset(
          "csv", data_files=input_file, split="train"
      )
    elif input_file.endswith(".json"):
      dataset = datasets.load_dataset(
          "json", data_files=input_file, split="train"
      )
    elif input_file.endswith(".parquet"):
      dataset = datasets.load_dataset(
          "parquet", data_files=input_file, split="train"
      )
    else:
      logging.error("Unsupported file format. Use .csv, .json, or .parquet")
      sys.exit(1)
  except Exception as e:  # pylint: disable=broad-exception-caught
    logging.exception("Failed to read input file: %s", e)
    sys.exit(1)

  for col in [prompt_col, completion_col]:
    if col not in dataset.column_names:
      logging.error(
          "Column '%s' not found. Available columns: %s",
          col,
          dataset.column_names,
      )
      sys.exit(1)

  # Remove rows with empty values in critical columns
  def is_valid(example):
    prompt_text = str(example[prompt_col]).strip().lower()
    completion_text = str(example[completion_col]).strip().lower()
    return (
        len(prompt_text) > 0
        and len(completion_text) > 0
        and prompt_text != "nan"
        and completion_text != "nan"
        and prompt_text != "none"
        and completion_text != "none"
    )

  initial_len = len(dataset)
  dataset = dataset.filter(is_valid)
  if len(dataset) < initial_len:
    logging.warning(
        "Dropped %d rows with empty or NaN values", initial_len - len(dataset)
    )

  def format_example(example):
    prompt_text = str(example[prompt_col])
    completion_text = str(example[completion_col])
    if format_type == "messages":
      return {
          "messages": [
              {"role": "user", "content": prompt_text},
              {"role": "assistant", "content": completion_text},
          ]
      }
    else:
      return {
          "prompt": prompt_text,
          "completion": completion_text,
      }

  formatted_dataset = dataset.map(
      format_example, remove_columns=dataset.column_names
  )

  if validation_split and validation_split > 0:
    if not (0 < validation_split < 1):
      logging.error("validation_split must be between 0 and 1")
      sys.exit(1)

    # Use the datasets library to perform the split as requested
    split_dict = formatted_dataset.train_test_split(
        seed=42, test_size=validation_split
    )
    train_ds = split_dict["train"]
    val_ds = split_dict["test"]

    val_output_file = output_file.replace(".jsonl", "_validation.jsonl")
    if val_output_file == output_file:
      val_output_file = output_file + ".validation.jsonl"

    train_ds.to_json(output_file, force_ascii=False, lines=True)
    val_ds.to_json(val_output_file, force_ascii=False, lines=True)

    logging.info(
        "Successfully saved %d training examples to %s",
        len(train_ds),
        output_file,
    )
    logging.info(
        "Successfully saved %d validation examples to %s",
        len(val_ds),
        val_output_file,
    )
  else:
    formatted_dataset.to_json(output_file, force_ascii=False, lines=True)
    logging.info(
        "Successfully saved %d examples to %s",
        len(formatted_dataset),
        output_file,
    )


if __name__ == "__main__":
  logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
  parser = argparse.ArgumentParser(
      description="Prepare or Validate dataset for Agent Platform Model Tuning"
  )
  parser.add_argument(
      "--input",
      help="Input CSV, JSON, Parquet, or JSONL file",
  )
  parser.add_argument(
      "--output",
      default="tuning_dataset.jsonl",
      help="Output JSONL file (only for conversion)",
  )
  parser.add_argument(
      "--format",
      choices=["messages", "prompt"],
      default="messages",
      help="Target format (messages or prompt/completion)",
  )
  parser.add_argument(
      "--prompt_col",
      help="Column name for prompt/user message (CSV/JSON/Parquet only)",
  )
  parser.add_argument(
      "--completion_col",
      help=(
          "Column name for completion/assistant response (CSV/JSON/Parquet"
          " only)"
      ),
  )
  parser.add_argument(
      "--validation_split",
      type=float,
      default=0.2,
      help="Fraction of data to use for validation (e.g. 0.2)",
  )
  parser.add_argument(
      "--validate_only",
      action="store_true",
      help="Only validate the input JSONL file without converting",
  )

  args = parser.parse_args()

  if args.validate_only:
    if not args.input:
      logging.error("--input is required for validation")
      sys.exit(1)
    success = validate_jsonl(args.input, args.format)
    sys.exit(0 if success else 1)
  else:
    if not all([args.input, args.prompt_col, args.completion_col]):
      logging.error(
          "--input, --prompt_col, and --completion_col are required for"
          " conversion"
      )
      sys.exit(1)
    convert_to_jsonl(
        args.input,
        args.output,
        args.format,
        args.prompt_col,
        args.completion_col,
        args.validation_split,
    )

```


### `scripts/tune_open_model.py`

```
"""Module for launching Agent Platform model tuning jobs."""

import argparse
import logging

from google import genai
from google.genai import types


def tune_open_model(
    project: str,
    location: str,
    base_model: str,
    train_dataset: str,
    validation_dataset: str | None,
    output_uri: str,
    epochs: int,
    learning_rate: float,
    tuning_mode: str,
    adapter_size: int | None = None,
) -> types.TuningJob:
  """Launches a Agent Platform model tuning job."""
  training_ds = types.TuningDataset(gcs_uri=train_dataset)
  validation_ds = (
      types.TuningValidationDataset(gcs_uri=validation_dataset)
      if validation_dataset
      else None
  )

  if tuning_mode == "FULL":
    mapped_tuning_mode = types.TuningMode.TUNING_MODE_FULL
  elif tuning_mode == "PEFT_ADAPTER":
    mapped_tuning_mode = types.TuningMode.TUNING_MODE_PEFT_ADAPTER
  else:
    raise ValueError(
        f"Unsupported tuning mode: {tuning_mode}. Supported modes are: FULL,"
        " PEFT_ADAPTER."
    )
  adapter_map = {
      1: "ADAPTER_SIZE_ONE",
      2: "ADAPTER_SIZE_TWO",
      4: "ADAPTER_SIZE_FOUR",
      8: "ADAPTER_SIZE_EIGHT",
      16: "ADAPTER_SIZE_SIXTEEN",
      32: "ADAPTER_SIZE_THIRTY_TWO",
  }
  mapped_adapter = adapter_map.get(adapter_size) if adapter_size else None

  config = types.CreateTuningJobConfig(
      epoch_count=epochs,
      learning_rate=learning_rate,
      validation_dataset=validation_ds,
      tuning_mode=mapped_tuning_mode,
      adapter_size=mapped_adapter,
      output_uri=output_uri,
      labels={"mg-source": "agent-platform-tuning-skill"},
  )
  with genai.Client(
      enterprise=True, project=project, location=location
  ) as client:
    tuning_job = client.tunings.tune(
        base_model=base_model,
        training_dataset=training_ds,
        config=config,
    )

  logging.info("Tuning job launched: %s", tuning_job.name)
  job_id = tuning_job.name.split("/")[-1] if tuning_job.name else "unknown"
  logging.info(
      "View job in console:"
      "https://console.cloud.google.com/agent-platform/tuning/locations/%s/tuningJob/%s/monitor?project=%s",
      location,
      job_id,
      project,
  )
  return tuning_job


if __name__ == "__main__":
  parser = argparse.ArgumentParser(
      description="Launch Vertex AI Model Tuning Job"
  )
  parser.add_argument("--project", required=True)
  parser.add_argument("--location", required=True)
  parser.add_argument("--base_model", required=True)
  parser.add_argument("--train_dataset", required=True)
  parser.add_argument(
      "--validation_dataset", help="Optional validation dataset URI"
  )
  parser.add_argument("--output_uri", required=True)
  parser.add_argument("--epochs", type=int, required=True)
  parser.add_argument("--learning_rate", type=float, required=True)
  parser.add_argument(
      "--tuning_mode", choices=("FULL", "PEFT_ADAPTER"), required=True
  )
  parser.add_argument(
      "--adapter_size",
      type=int,
      choices=(1, 4, 8, 16, 32),
      help="Adapter size for PEFT",
  )

  args = parser.parse_args()
  tune_open_model(**vars(args))

```