Gemma 4 Sudoku RL with Unsloth

This tutorial turns Unsloth’s Gemma 4 Sudoku notebook into one Compute function. Gemma writes a native Python Sudoku strategy, three reward functions score it, and GRPO updates a LoRA adapter from those scores. The default is a one-step smoke run. It downloads the public unsloth/gemma-4-E2B-it checkpoint, trains on an NVIDIA GPU, saves the adapter, checks every saved LoRA tensor, and returns JSON metrics before Compute terminates the machine.

gemma4_sudoku_rl.py is adapted from Unsloth’s Gemma 4 Sudoku notebook. This example is explicitly licensed LGPL-3.0-only; its SPDX header and source notice are part of the file. That license notice applies to this adapted example.

Before you start

Install Compute, sign in, and add prepaid credit:

curl -fsSL https://compute.cx/install.sh | sh
compute setup
compute credits add 10

You do not need a Hugging Face token for the default public model.

Save the example

Download the complete script:

curl -L https://raw.githubusercontent.com/theoriclabs/docs.compute.cx/main/examples/gemma4_sudoku_rl.py \
  -o gemma4_sudoku_rl.py

The file defines gemma4_sudoku_rl.py::train. Its Compute image starts from CUDA PyTorch. The function installs pinned Unsloth packages, then applies the newer Transformers and TRL versions used by the upstream notebook. Installing inside the function puts dependency output in the run log. GPU-only imports happen afterward. The core training setup is:

app = compute.App("gemma4-sudoku-grpo")
image = compute.Image.cuda_pytorch()


@app.function(gpu="H100-SXM", image=image, timeout=3600)
def train(
    model_id: str = "unsloth/gemma-4-E2B-it",
    max_steps: int = 1,
    dataset_size: int = 64,
    difficulty: int = 40,
    max_seq_length: int = 4096,
    lora_rank: int = 32,
    num_generations: int = 2,
    seed: int = 3407,
) -> dict:
    # Model loading, Sudoku rewards, GRPO training, and adapter verification
    # are implemented in the linked example file.
    ...

Inspect the payload

Run a local dry-run first:

compute run gemma4_sudoku_rl.py::train --gpu H100-SXM --dry-run

The output should list cuda_pytorch and the train entrypoint. A dry-run does not upload the file, request a quote, or create a machine.

Run one training step

An RTX 4090 has enough memory for the default E2B run. Vast.ai capacity is interruptible, so Compute may end the run if the provider reclaims the machine:

compute run gemma4_sudoku_rl.py::train \
  --provider vastai \
  --gpu RTX-4090 \
  --timeout 1800 \
  --wait

The CLI prints a quote and asks before spending credit. Keep the prompt for the first run. Use the reserved H100 path when you prefer steadier capacity:

compute run gemma4_sudoku_rl.py::train \
  --gpu H100-SXM \
  --timeout 1800 \
  --wait

Package installation and model download happen on the new machine, so the first log lines can take several minutes. --wait keeps the logs attached through training and teardown.

How the reward works

Each prompt asks Gemma to return def strategy(board, initial) in a Python code block. The generated function gets three scores:

function_works checks the exact signature and verifies that Unsloth can create a locked-down callable.
no_cheating rejects imports and gives them a heavy penalty.
strategy_succeeds runs the function against a generated puzzle. A complete solution earns 30.0; partial progress earns 0.2 per valid move.

Every completion in a GRPO group sees the same puzzle seed, which keeps comparisons within that group fair. The default group has two completions and uses the notebook’s BNPO loss settings.

Read the result

The final value is JSON. It includes the device and package versions, aggregate train_loss, per-step loss_history and reward_history, elapsed training time, and an adapter_verified flag. Inspect the histories instead of treating a successful exit as proof that the short training run improved. The exact metrics vary because generation and GPU training are stochastic. The example writes the adapter to /tmp/gemma4-sudoku-lora and opens adapter_model.safetensors before returning. The run fails if the file is empty or any adapter tensor is all zero.

The adapter path is inside the temporary machine. Compute v0.1 terminates that machine when train returns and does not expose artifact download for this workflow. This tutorial verifies the training path and returns metrics; the saved adapter does not persist after teardown.

Train for longer

The upstream notebook uses 60 steps and 1,000 copies of the prompt. Run that shape on reserved capacity with a longer timeout:

compute run gemma4_sudoku_rl.py::train \
  --gpu H100-SXM \
  --args '{"max_steps":60,"dataset_size":1000}' \
  --timeout 21600 \
  --wait

The quote uses the timeout as a spending ceiling. The machine still terminates as soon as the function returns.

General reinforcement learning

Run a smaller GRPO example on MI300X.

CLI run reference

Pass arguments, choose a GPU, and control the timeout.

Overview

Get started

Tutorials

Guides

CLI

Reference

Tinker training

Gemma 4 Sudoku RL with Unsloth

Before you start

Save the example

Inspect the payload

Run one training step

How the reward works

Read the result

Train for longer

General reinforcement learning

CLI run reference

​Before you start

​Save the example

​Inspect the payload

​Run one training step

​How the reward works

​Read the result

​Train for longer

General reinforcement learning

CLI run reference

Before you start

Save the example

Inspect the payload

Run one training step

How the reward works

Read the result

Train for longer