In this article, you will learn how to turn an open-weight language model into a decision model that classifies user intent, by fine-tuning Qwen3.5-4B on the CLINC150 dataset with Unsloth.
Topics we will cover include:
- What decision models are, and how Unsloth’s FastDecisionModel scores a fixed set of choices instead of generating text.
- How to convert CLINC150 into Unsloth’s decision format and fine-tune Qwen3.5-4B with 4-bit LoRA in Google Colab.
- How to evaluate, calibrate, test, and save the resulting intent classifier.
What if you could turn a regular language model into a decision model that chooses an answer instead of generating a long response?
Models like Jev, Laya, and Clef are designed to make structured decisions. Instead of generating text token by token, they select from predefined options and return probability estimates.
Unsloth now supports training decision models from open-weight language models such as Qwen and Gemma. Its official decision-model training guide explains the method and provides ready-to-run notebooks.
In this tutorial, I’m using Google Colab, Unsloth, and Qwen3.5-4B to build a custom intent classifier using the CLINC150 dataset. We will:
- Prepare the dataset
- Train the model using LoRA
- Evaluate its performance
- Test whether it can correctly identify what users want from their messages
By the end, you’ll have your own decision model that can classify user requests and return structured predictions with confidence scores. So, let’s get started.
What Are Decision Models?
Traditional language models generate responses one token at a time. However, many applications don’t need an explanation. They just need a decision.
For example, suppose a user sends the following message:
“Can you help me check the balance in my savings account?”
A traditional LLM might explain how to check an account balance. A decision model can instead classify the request:
|
1 2 3 4 |
{ "intent": "balance", "confidence": 0.97 } |
Decision models are useful for intent classification, content moderation, routing requests, sentiment analysis, and agent workflows where choosing an action matters more than generating text.
Unsloth’s FastDecisionModel converts a pretrained LLM into a decision model by adding a specialized decision head. The model processes the input and available choices and scores the possible answers without generating a conventional text completion.
It supports three main question types:
- Choice: Select one option from multiple possibilities
- Noul: Return a yes-or-no decision
- Score: Select a level from an ordered set of categories
For our experiment, we’ll focus on the Choice question type because we want the model to identify the intent behind a user’s message.
Getting Started with Unsloth and Qwen3.5
I’m running the entire experiment in Google Colab, so you can follow along without setting up CUDA or drivers locally. Start with Unsloth’s official decision-model guide or open the Qwen3.5-4B Decision Model notebook in Google Colab. Make a copy of the notebook in your Google Drive before making changes.
Under Runtime → Change runtime type, select an available NVIDIA GPU. I used a Tesla T4 with approximately 14.6 GB of GPU memory. The setup can run on that GPU, but training takes longer than it would on a newer GPU.
For this tutorial, we’ll use:
| Component | Selection |
|---|---|
| Base model | Qwen3.5-4B |
| Framework | Unsloth |
| Dataset | CLINC150 |
| Training method | 4-bit LoRA |
| Decision type | Choice |
| Environment | Google Colab |
Step 1: Installing Unsloth
I used the dependency installation cell from Unsloth’s Colab notebook. It installs uv, the compatible PyTorch and Transformers versions, Unsloth, and the required GPU packages. Run this in a Colab cell before loading the model.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
%%capture import os, importlib.util !pip install --upgrade -qqq uv if importlib.util.find_spec("torch") is None or "COLAB_" in "".join(os.environ.keys()): try: import numpy, PIL; _numpy = f"numpy=={numpy.__version__}"; _pil = f"pillow=={PIL.__version__}" except: _numpy = "numpy"; _pil = "pillow" !uv pip install -qqq \ "torch==2.8.0" "triton>=3.3.0" {_numpy} {_pil} torchvision "torchaudio==2.8.0" bitsandbytes xformers==0.0.32.post2 \ "unsloth_zoo[base] @ git+https://github.com/unslothai/unsloth-zoo" \ "unsloth[base] @ git+https://github.com/unslothai/unsloth" !uv pip install -qqq --no-deps "torchcodec==0.7.0" elif importlib.util.find_spec("unsloth") is None: !uv pip install -qqq unsloth !uv pip install --upgrade --no-deps "tokenizers>=0.22.0,<=0.23.0" trl==0.22.2 unsloth unsloth_zoo !uv pip install transformers==5.2.0 !unsloth install-kernels !uv pip install --no-deps --upgrade "torchao>=0.16.0" |
The cell uses %%capture to keep the installation logs out of the notebook. Once it completes, continue to load the model.
Step 2: Loading Qwen3.5-4B
Next, we’ll load Qwen3.5-4B using FastDecisionModel instead of the usual language-model loader.
|
1 2 3 4 5 6 7 8 |
from unsloth import FastDecisionModel, DecisionTrainer, is_bfloat16_supported import torch model, tokenizer = FastDecisionModel.from_pretrained( model_name="unsloth/Qwen3.5-4B", max_seq_length=2048, load_in_4bit=True, ) |
We use 4-bit quantization to reduce memory consumption. The decision head is initialized for training, while the pretrained model provides the language understanding capabilities.
Output:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning. 🦥 Unsloth Zoo will now patch everything to make training faster! /usr/local/lib/python3.13/dist-packages/unsloth/models/rl.py:1531: UserWarning: Torchaudio's I/O functions now support per-call backend dispatch. Importing backend implementation directly is no longer guaranteed to work. Please use `backend` keyword with load/save/info function, instead of calling the underlying implementation directly. if getattr(module, "prepare_peft_model", None) is original: ==((====))== Unsloth 2026.10.10: Fast Qwen3_5 patching. Transformers: 5.2.0. \\ /| Tesla T4. Num GPUs = 1. Max memory: 14.563 GB. Platform: Linux. O^O/ \_/ \ Torch: 2.8.0+cu128. CUDA: 7.5. CUDA Toolkit: 12.8. Triton: 3.4.0 \ / Bfloat16 = FALSE. FA [Xformers = 0.0.32.post2. FA2 = False] "-____-" Free license: http://github.com/unslothai/unsloth Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored! Unsloth: Using float16 precision for qwen3_5 won't work! Using float32. model.safetensors.index.json: 100% 76.2k/76.2k [00:00<00:00, 6.13MB/s] Loading weights: 100% 723/723 [00:35<00:00, 86.48it/s, Materializing param=model.visual.pos_embed.weight] Download complete: 0.00B Reconstruction complete: 0.00B / 0.00B Fetching 13 files: 100% 13/13 [00:00<00:00, 415.24it/s] |
Step 3: Configuring LoRA
Rather than updating all the model’s parameters, we’ll use LoRA.
LoRA allows us to train a relatively small number of additional parameters while keeping most of the base model unchanged. The decision head itself is also trained.
|
1 2 3 4 5 6 7 8 |
model = FastDecisionModel.get_peft_model( model, r=16, lora_alpha=16, lora_dropout=0, use_gradient_checkpointing="unsloth", random_state=3407, ) |
A LoRA rank of 16 gives us a reasonable starting point for this experiment. Gradient checkpointing helps reduce GPU memory usage during training.
Preparing the CLINC150 Dataset
For this tutorial, we’ll use CLINC150, a dataset designed for intent classification and detecting requests that fall outside a system’s supported capabilities.
It contains 150 intent categories covering areas such as banking, travel, shopping, and everyday assistant tasks.
The full dataset includes 15,000 in-scope training examples, 3,000 validation examples, and 4,500 in-scope test examples, along with separate out-of-scope examples.
Step 4: Downloading and Exploring CLINC150
We’ll download the original dataset and inspect its structure.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
import requests import random from collections import Counter url = ( "https://raw.githubusercontent.com/clinc/oos-eval/" "master/data/data_full.json" ) response = requests.get(url, timeout=60) response.raise_for_status() data = response.json() print("Dataset splits:", list(data.keys())) print("Training examples:", len(data["train"])) print("Validation examples:", len(data["val"])) print("Test examples:", len(data["test"])) print("Sample:", data["train"][0]) |
Each example contains a user message and its corresponding intent label.
Output:
|
1 2 3 4 5 |
Dataset splits: ['oos_val', 'val', 'train', 'oos_test', 'test', 'oos_train'] Training examples: 15000 Validation examples: 3000 Test examples: 4500 Sample: ['what expression would i use to say i love you if i were an italian', 'translate'] |
Step 5: Converting CLINC150 into Decision Model Format
This is the most important part of the tutorial.
Unsloth expects each decision-training example to contain three fields:
state: The input that the model will analyzequestions: The decisions we want the model to makegold: The correct answers used during training
Unlike a standard classification dataset, we need to define the available choices explicitly.
To make the first experiment practical on Colab, we’ll select 10 intents from CLINC150. You can increase NUM_INTENTS to use more categories, although the larger choice set will increase the input length and training requirements.
|
1 2 3 4 5 6 7 8 9 10 |
random.seed(3407) all_intents = sorted({ label for _, label in data["train"] }) NUM_INTENTS = 10 selected_intents = random.sample(all_intents, NUM_INTENTS) print("Selected intents:", selected_intents) |
Output:
|
1 |
Selected intents: ['calories', 'routing', 'date', 'payday', 'ingredient_substitution', 'restaurant_suggestion', 'oil_change_how', 'pto_used', 'jump_start', 'time'] |
Now we’ll construct a Choice question using the selected intents.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
questions = { "intent": { "type": "choice", "instructions": ( "Identify the user's primary intent." ), "criteria": { intent: intent.replace("_", " ") for intent in selected_intents }, } } |
Next, we convert the original CLINC150 records into Unsloth’s required format.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
def convert_to_decisions(examples): return [ { "state": text, "questions": questions, "gold": {"intent": label}, } for text, label in examples if label in selected_intents ] train_rows = convert_to_decisions(data["train"]) val_rows = convert_to_decisions(data["val"]) test_rows = convert_to_decisions(data["test"]) print("Training rows:", len(train_rows)) print("Validation rows:", len(val_rows)) print("Test rows:", len(test_rows)) print(train_rows[0]) |
Each converted row now contains a natural-language user request, the set of available intents, and the correct intent label.
We deliberately exclude the out-of-scope records for this first experiment. Their detection is a separate task that we’ll discuss after evaluating the classifier.
Output:
|
1 2 3 4 |
Training rows: 1000 Validation rows: 200 Test rows: 300 {'state': 'on what date do i get paid', 'questions': {'intent': {'type': 'choice', 'instructions': "Identify the user's primary intent.", 'criteria': {'calories': 'calories', 'routing': 'routing', 'date': 'date', 'payday': 'payday', 'ingredient_substitution': 'ingredient substitution', 'restaurant_suggestion': 'restaurant suggestion', 'oil_change_how': 'oil change how', 'pto_used': 'pto used', 'jump_start': 'jump start', 'time': 'time'}}}, 'gold': {'intent': 'payday'}} |
Step 6: Building the Training Dataset
Now we’ll tokenize and prepare our examples using Unsloth’s built-in dataset builder.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
train_items, train_report = FastDecisionModel.build_dataset( train_rows, tokenizer, model, ) eval_items, eval_report = FastDecisionModel.build_dataset( val_rows, tokenizer, model, ) test_items, test_report = FastDecisionModel.build_dataset( test_rows, tokenizer, model, ) print("Training items:", len(train_items)) print("Validation items:", len(eval_items)) print("Test items:", len(test_items)) print("Skipped training decisions:", train_report["skipped"]) |
This method converts the raw examples into the format required by the decision head.
Notice that we’re using CLINC150’s original training, validation, and test partitions. This keeps the final test data separate from training and calibration.
Output:
|
1 2 3 4 |
Training items: 1000 Validation items: 200 Test items: 300 Skipped training decisions: 0 |
Training Our Decision Model
With the dataset ready, we can start fine-tuning Qwen3.5-4B.
Step 7: Evaluating the Model Before Training
Before training, let’s measure the model’s initial performance.
|
1 2 3 4 5 6 7 |
before_metrics = FastDecisionModel.evaluate( model, tokenizer, eval_items, ) print(before_metrics) |
Because the decision head is newly initialized, its initial predictions may be close to random guessing.
For our 10-class experiment, random guessing would achieve approximately 10% accuracy on a balanced dataset.
This gives us a baseline for comparison after training.
Output:
|
1 |
{'accuracy': 0.185, 'ece': 0.08158441890031098, 'loss': 2.2937265050411226, 'record_accuracy': 0.185} |
Step 8: Configuring Training
We’ll use Unsloth’s DecisionTrainer, which is designed specifically for fine-tuning decision models.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
from transformers import TrainingArguments trainer = DecisionTrainer( model=model, processing_class=tokenizer, train_dataset=train_items, eval_dataset=eval_items, args=TrainingArguments( per_device_train_batch_size=8, gradient_accumulation_steps=4, warmup_steps=10, max_steps=60, learning_rate=2e-4, lr_scheduler_type="cosine", weight_decay=0.01, bf16=is_bfloat16_supported(), fp16=not is_bfloat16_supported(), eval_strategy="steps", eval_steps=20, logging_steps=10, output_dir="outputs", report_to="none", seed=3407, ), ) |
We’re using 60 training steps to make the experiment manageable.
The official Unsloth notebook also uses 60 steps for its quick-training example. For a more complete run, you can increase the number of steps or train for multiple epochs.
Step 9: Fine-Tuning the Model
Now we can start training.
|
1 2 3 |
trainer_stats = trainer.train() print(trainer_stats.metrics) |
During training, Unsloth reports metrics such as training loss and runtime.
Output:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1 \\ /| Num examples = 1,000 | Num Epochs = 2 | Total steps = 60 O^O/ \_/ \ Batch size per device = 8 | Gradient accumulation steps = 4 \ / Data Parallel GPUs = 1 | Total batch size (8 x 4 x 1) = 32 "-____-" Trainable parameters = 64,511,492 of 3,072,856,068 (2.10% trained) Unsloth: Will smartly offload gradients to save VRAM! Unsloth: Double buffering enabled (parallel H2D + compute) for backward pass. [60/60 33:26, Epoch 1/2] Step Training Loss Validation Loss 20 0.626067 0.015350 40 0.034541 0.001252 60 0.001530 0.003902 Unsloth: Restored added_tokens_decoder metadata in /content/outputs/checkpoint-60/.unsloth-save-m_o4y1v1/tokenizer_config.json. {'train_runtime': 2260.5268, 'train_samples_per_second': 0.849, 'train_steps_per_second': 0.027, 'total_flos': 0.0, 'train_loss': 0.4899662705448767, 'epoch': 1.896} |
Evaluating the Fine-Tuned Decision Model
Now that training has finished, we want to understand how much the model improved.
Step 10: Evaluating Accuracy After Training
First, evaluate the model again on the validation set.
|
1 2 3 4 5 6 7 |
after_metrics = FastDecisionModel.evaluate( model, tokenizer, eval_items, ) print(after_metrics) |
Output:
|
1 |
{'accuracy': 1.0, 'ece': 0.003142179250717159, 'loss': 0.003883412495779339, 'record_accuracy': 1.0} |
We can compare the accuracy before and after training.
| Metric | Before Training | After Training |
|---|---|---|
| Validation accuracy | 18.5% | 100.0% |
| Validation expected calibration error (ECE; lower is better) | 0.0816 | 0.00314 |
| Validation loss | 2.2937 | 0.00388 |
| Training duration | — | 2,260.5 seconds (about 37.7 minutes) |
In this run, validation accuracy rose from 18.5% to 100% after fine-tuning. That is an excellent result for this small, deliberately selected 10-intent task, but it does not mean the model is 100% accurate across the full 150-intent dataset or on out-of-scope requests.
Step 11: Calibrating Confidence Scores
A decision model should not only select the correct answer. Ideally, its predicted probabilities should also reflect how often its predictions are correct.
Unsloth includes a built-in calibration method.
|
1 2 3 4 5 6 7 |
calibration_metrics = FastDecisionModel.calibrate( model, tokenizer, eval_items, ) print(calibration_metrics) |
Calibration can improve the reliability of the probabilities returned by the model.
One useful metric is ECE, which measures how closely predicted confidence aligns with observed accuracy. Lower ECE is better.
Output:
|
1 |
{'accuracy': 1.0, 'ece': 0.00195585876703267, 'loss': 0.00248109146952288, 'record_accuracy': 1.0, 'fitted_types': [0]} |
Step 12: Evaluating on Unseen Test Data
Finally, we’ll evaluate the trained model using CLINC150’s test split.
|
1 2 3 4 5 6 7 |
test_metrics = FastDecisionModel.evaluate( model, tokenizer, test_items, ) print(test_metrics) |
This is particularly important because the test examples were not used for training or confidence calibration.
Output:
|
1 |
{'accuracy': 1.0, 'ece': 0.0, 'loss': 0.0, 'record_accuracy': 1.0} |
On this held-out 300-example test split, the model classified all examples correctly. The zero loss and ECE are the reported outputs of this run, not a guarantee of perfect generalization. Test it on harder, ambiguous, and out-of-scope messages before deploying it.
Testing Our Decision Model with New Examples
Let’s see how the trained model behaves when we give it a new user message.
Step 13: Making a Prediction
We’ll define a simple function that accepts a user query and returns the model’s predicted intent.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
FastDecisionModel.for_inference(model) def predict_intent(text): answers = FastDecisionModel.predict( model, tokenizer, text, questions, ) result = answers["intent"] return { "intent": result["answer"], "probabilities": result["probabilities"], } |
Now test it with a few messages relevant to your selected intent categories.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
sample_texts = [ "how many calories are in a medium banana", "when do i get my next paycheck", "what can i use instead of buttermilk in pancakes", "my car battery is dead, how do i jump start it", "what time is it in tokyo right now", ] for sample_text in sample_texts: prediction = predict_intent(sample_text) print(sample_text) print(prediction) print() |
For each message, the model returns an intent label and a probability for each option.
Output:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
how many calories are in a medium banana {'intent': 'calories', 'probabilities': {'calories': 1.0, 'routing': 0.0, 'date': 0.0, 'payday': 0.0, 'ingredient_substitution': 0.0, 'restaurant_suggestion': 0.0, 'oil_change_how': 0.0, 'pto_used': 0.0, 'jump_start': 0.0, 'time': 0.0}} when do i get my next paycheck {'intent': 'payday', 'probabilities': {'calories': 0.0, 'routing': 0.0, 'date': 0.0, 'payday': 1.0, 'ingredient_substitution': 0.0, 'restaurant_suggestion': 0.0, 'oil_change_how': 0.0, 'pto_used': 0.0, 'jump_start': 0.0, 'time': 0.0}} what can i use instead of buttermilk in pancakes {'intent': 'ingredient_substitution', 'probabilities': {'calories': 0.0, 'routing': 0.0, 'date': 0.0, 'payday': 0.0, 'ingredient_substitution': 1.0, 'restaurant_suggestion': 0.0, 'oil_change_how': 0.0, 'pto_used': 0.0, 'jump_start': 0.0, 'time': 0.0}} my car battery is dead, how do i jump start it {'intent': 'jump_start', 'probabilities': {'calories': 0.0, 'routing': 0.0, 'date': 0.0, 'payday': 0.0, 'ingredient_substitution': 0.0, 'restaurant_suggestion': 0.0, 'oil_change_how': 0.0, 'pto_used': 0.0, 'jump_start': 1.0, 'time': 0.0}} what time is it in tokyo right now {'intent': 'time', 'probabilities': {'calories': 0.0, 'routing': 0.0, 'date': 0.0, 'payday': 0.0, 'ingredient_substitution': 0.0, 'restaurant_suggestion': 0.0, 'oil_change_how': 0.0, 'pto_used': 0.0, 'jump_start': 0.0, 'time': 1.0}} |
Saving Our Trained Decision Model
Once you’re happy with the results, you can save the LoRA adapters and the trained decision head.
|
1 2 |
model.save_pretrained("qwen-clinc150-decision") tokenizer.save_pretrained("qwen-clinc150-decision") |
If you want a merged model, Unsloth also supports saving a full model:
|
1 2 3 |
model.save_pretrained_merged( "qwen-clinc150-decision-merged" ) |
You can later reload the model using FastDecisionModel.from_pretrained() and use it in your own Python applications or supported decision-model serving workflows.
Conclusion
Decision models are useful when an application needs to make structured choices rather than generate long responses. With Unsloth, you can train one using an existing LLM and a relatively small dataset.
In this example, we trained Qwen3.5-4B on 10 CLINC150 intents and achieved 100% accuracy on the test set. The results are promising, but they are limited to this small experiment. You can use the same approach with your own dataset and explore how decision models fit into your applications.
No comments yet.