Knowledge distillation is the process of training a small model to reproduce the behavior of a large model. The large model (teacher) generates training data; the small model (student) learns from it. The student doesn't need to reason from first principles — it only needs to generalize the pattern the teacher demonstrated.
This works well when the teacher's task is well-defined and the output format is narrow. Sinc JSON decomposition is exactly that:
formula, T, and a fragments array of exactly 6 objectsn (integer 0-5), t (band name), x (text content)fragments[3].x (CONSTRAINTS) must be the longest band by character countThe teacher (Haiku) was given the sinc-LLM specification and asked to decompose diverse prompts. I then trained Qwen2.5-7B on the resulting (prompt, JSON) pairs. The student doesn't need to understand why CONSTRAINTS should be longest — it just needs to have internalized that pattern from enough examples.
I wrote a data generation script that calls the Haiku API with a fixed system prompt explaining the sinc-LLM framework. Each call passes one raw prompt and collects the sinc JSON output. 120 calls at $0.002 each = $0.24 total data generation cost. I realized I could buy my entire training dataset for less than a cup of coffee.
import anthropic
client = anthropic.Anthropic()
SYSTEM = """You are a sinc-LLM scatter engine. Given any raw user prompt,
decompose it into a sinc JSON object with this exact structure:
{
"formula": "x(t) = Σ x(nT) · sinc((t − nT) / T)",
"T": "specification-axis",
"fragments": [
{"n": 0, "t": "PERSONA", "x": "..."},
{"n": 1, "t": "CONTEXT", "x": "..."},
{"n": 2, "t": "DATA", "x": "..."},
{"n": 3, "t": "CONSTRAINTS", "x": "..."}, // MUST be longest band
{"n": 4, "t": "FORMAT", "x": "..."},
{"n": 5, "t": "TASK", "x": "..."}
]
}
Output ONLY valid JSON. No preamble, no explanation."""
def generate_example(prompt: str) -> dict:
msg = client.messages.create(
model="claude-haiku-4-5",
max_tokens=1024,
system=SYSTEM,
messages=[{"role": "user", "content": prompt}]
)
return {"prompt": prompt, "completion": msg.content[0].text}Each generated example is saved to training_data.jsonl, one JSON object per line. I validated every generated example before including it in the training set: parse the JSON, check all 6 bands are present, verify CONSTRAINTS is the longest band.
I selected 120 prompts to cover the full distribution of real user inputs. This is the most important part of the data generation process — I learned this the hard way from an earlier draft that was too skewed toward technical prompts. A model trained only on long technical prompts will fail on one-word inputs. A model trained only on casual prompts will underperform on detailed specifications.
The distribution we used:
Qwen2.5-Instruct uses ChatML as its native format. Each training example must be formatted as a complete conversation turn — system prompt, user message, and assistant response — using the ChatML special tokens. Getting this right matters: using a different format will train the model to respond in a way that misaligns with its inference-time tokenization.
def format_example(prompt: str, completion: str) -> str:
return (
f"<|im_start|>system\n"
f"You are a sinc-LLM scatter engine. Decompose the user prompt "
f"into 6 sinc frequency bands as valid JSON. CONSTRAINTS must be "
f"the longest band. Output only JSON.\n"
f"<|im_end|>\n"
f"<|im_start|>user\n"
f"{prompt}\n"
f"<|im_end|>\n"
f"<|im_start|>assistant\n"
f"{completion}\n"
f"<|im_end|>"
)Training hyperparameters: 3 epochs, batch size 2, gradient accumulation 4 (effective batch 8), learning rate 2e-4, cosine decay schedule, warmup steps 5. LoRA rank 16, alpha 16, targeting all 7 projection matrices in the attention and MLP blocks.
Beyond the output format, three structural invariants define a valid sinc JSON:
Invariant 1: Exactly 6 bands, always. n=0 through n=5, in order, no duplicates, no missing entries. The model must output all 6 even when the input provides zero information about most bands.
Invariant 2: CONSTRAINTS (n=3) is always the longest band. By character count, fragments[3].x must exceed the character count of every other band's x value. This is the most frequently violated invariant in zero-shot decompositions from base models.
Invariant 3: TASK (n=5) is exactly one sentence. Not a paragraph, not a list. One atomic, imperative sentence specifying the action to take. The model must resist the temptation to add caveats or sub-tasks.
The Haiku-generated training examples enforce all three invariants in every single example. After 3 epochs of training, the student model passes invariant checks on 9/10 validation prompts without post-processing. What surprised me was that Invariant 2 — CONSTRAINTS always longest — was the hardest for base models but the easiest to teach through distillation.
Edge cases are where fine-tuned models typically fail. A model trained only on well-formed prompts will generate malformed JSON or produce an error response when it encounters "???" as input.
I deliberately included 10 edge case examples in the training set. Here's how Haiku handled "???" — and what the student model learned:
{
"formula": "x(t) = Σ x(nT) · sinc((t − nT) / T)",
"T": "specification-axis",
"fragments": [
{"n": 0, "t": "PERSONA", "x": "A clarification-seeking assistant prepared to resolve ambiguity before proceeding."},
{"n": 1, "t": "CONTEXT", "x": "The user has submitted a non-linguistic input consisting only of question marks, indicating confusion, a request for clarification, or a test of the system boundary."},
{"n": 2, "t": "DATA", "x": "Input received: '???'. No additional context. No prior conversation history."},
{"n": 3, "t": "CONSTRAINTS", "x": "Do not invent intent where none is present. Do not assume the user wants any specific action. The response must acknowledge the ambiguity explicitly. Do not ask multiple clarifying questions at once — ask the single most useful one. Do not produce a complex or lengthy response. Do not fail silently or produce an empty output. The system must handle this gracefully and return a useful response regardless of input quality."},
{"n": 4, "t": "FORMAT", "x": "A single short question or prompt to the user requesting clarification."},
{"n": 5, "t": "TASK", "x": "Ask the user what they need help with."}
]
}The key insight: the CONSTRAINTS band is still the longest, still operationally grounded, still specific to the edge case. The model didn't produce empty bands or refuse to output JSON. The training signal for these edge cases teaches the student to handle non-standard inputs gracefully rather than failing.
After training, I ran 10 held-out validation prompts through the model and checked each output against the invariants:
def validate_sinc_json(output: str) -> tuple[bool, str]:
try:
data = json.loads(output)
except json.JSONDecodeError as e:
return False, f"Invalid JSON: {e}"
fragments = data.get("fragments", [])
if len(fragments) != 6:
return False, f"Expected 6 bands, got {len(fragments)}"
band_names = [f["t"] for f in fragments]
expected = ["PERSONA","CONTEXT","DATA","CONSTRAINTS","FORMAT","TASK"]
if band_names != expected:
return False, f"Band order mismatch: {band_names}"
# CONSTRAINTS invariant
lengths = {f["t"]: len(f["x"]) for f in fragments}
if not all(lengths["CONSTRAINTS"] > lengths[k]
for k in lengths if k != "CONSTRAINTS"):
return False, "CONSTRAINTS band not longest"
if "formula" not in data:
return False, "Missing formula field"
return True, "PASS"Results: 9 passed, 1 failed. The failure was on a non-linguistic edge case ("???") where the model produced valid JSON with all 6 bands but omitted the top-level formula field. All other invariants passed. I added a one-line patch in the API server to inject the formula if it's missing:
if "formula" not in result:
result["formula"] = "x(t) = Σ x(nT) · sinc((t − nT) / T)"With the patch, all 10 validation prompts pass. The model has correctly internalized the CONSTRAINTS length invariant — this was the hardest invariant to train for, and it holds in 10/10 cases. The 9/10 pre-patch result is strong for 107 seconds of training on 120 examples. I didn't expect to get there in one shot.
AI Transform on sincllm.com runs the fine-tuned sinc-scatter model — Haiku's knowledge, 7B parameters, zero marginal cost.
Try AI Transform →