What is weight-space programming? AI now writes model weights

MIT researchers had GPT-6 Astra read, fix and write neural network weights with no training. What weight-space programming is, what it beat, where it fails.

One edit to 0.016% of a small language model's weights lifted its math score from 7.9% to 48.1%, with no training run. MIT researchers had GPT-6 Astra, a frontier LLM, write that edit by reading the weights directly.

That is weight-space programming: a general-purpose LLM reads and writes a neural network's weights with no optimizer and no gradient descent, guided by a plain-language spec. On Nov 2, four MIT CSAIL researchers showed it. Training is an indirect way to write weights, and their first results suggest a frontier model can write them directly.

The post, "Sparks of Weight-Space Programming" (Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, all MIT CSAIL), gives Astra the weights, sometimes a scene description or a task's rules and source code, sometimes a Python sandbox. It never gives a gradient.

One direction removed, six times the math score

The authors read the gain as mostly unblocking: Qwen3.5-0.8B now finishes answers it could already work out, and it needed the full 50-pass budget to get there.

Qwen's own model card warns Qwen3.5-0.8B is prone to "loops compared to other Qwen3.5 models, which may prevent it from terminating generation properly." Of the base model's first 96 practice answers, 85 ran to the 32,768-token limit.

Astra worked in a Python notebook with 10, 25 or 50 forward-pass budgets, using practice prompts from the GSM8K and MATH training sets. It traced the looping to one direction, a single pattern written by the MLPs of layers 15 and 19. It subtracted that direction from 64 output columns in each, 131,072 weights in all, or 0.016% of the model.

IFEval, which tests instruction following, rose too. That fits the authors' reading: a model that stops looping finishes more answers. Lower budgets barely helped. At 10 and 25 passes, MATH-500 reached only 12.3% and 14.1%, and the token-limit share stayed at 87.2% and 86.4%. Even at 50 passes, 33.1% of answers still hit the limit.

What you can check today, and what it costs

The logs repo holds everything sent to and from Astra, all code and all evaluations, so you can replay the Qwen result yourself.

The edited model is on Hugging Face, and only two tensors differ from base. Our read: the practical use today is diagnosing and surgically repairing small models.

It is also slow. Coverage of the logs by dailyaibrief.com and clauding.de reports that the successful 50-pass run took 158 conversational turns and 80 Python runs over 20.3 hours, 4.7 of them lost to a service outage. That is a day of agent work standing in for a training run, and no fine-tuning cost is given to compare it with.

A skeptic can call the Qwen edit a bug fix for a documented loop. None of the coverage we read tests whether a decoding-side fix, such as a repetition penalty, would match it.

Astra can read weights

With no list of games to choose from and no tools, Astra named the game each of four PPO policies was trained on, from 17,253 raw weights per network.

For Breakout, Astra noticed that first-layer weights concentrate along the bottom row, where a paddle lives.

A harder test hid the first and last layers of three 7.2M-parameter language models trained on Python code, Japanese romaji and English. From the four middle blocks alone, Astra named the language every time and found every induction head (a copying head), matching the measured ground truth with no false positives. Behavior is readable from the numbers.

Astra can fix weights

In one reply, Astra changed 3.5% of a stuck policy's weights and took its success rate from 0% to 100% over 1,024 episodes.

In a Kinetix car-ramp task, a PPO policy had converged to a local minimum: it drove straight at the ramp and stayed stuck. Astra received the weights, a scene description and one recorded episode, and had no tools.

Its edit added a small path through a few units per layer. The path reads position and speed, backs the car up past the red strip, then hands control back to the trained policy.

Astra wrote networks that beat 100M-step PPO

Written from rules and source code alone, Astra's MinAtar networks beat PPO trained for 100M steps in all three games.

Astra got no data, loss, gradients, tools or rollouts. Against 100M-step PPO the scores were Breakout 517.4 versus 15.8, Seaquest 53.1 versus 5.6 and Asterix 53.2 versus 25.8.

Against PPO at 500M steps, Astra still wins Breakout by 20x, reaches 60% of PPO's Seaquest score, and nearly ties on Asterix.

In 9x9 Go, with a 1.5M-parameter network too big to fit in context, a budget of 50 forward passes was enough to beat the equal-sized PPO network trained on 20 million games.

Where it fails, and the argument about the hint

Weight edits fixed Qwen's loop and gave Gemma no established gain.

Gemma 3 270M rarely loops: 8.5% of its answers hit the token limit, against 93% for Qwen. MATH-500 started at 6.8% and four edits scored 7.0%, 7.6%, 7.7% and 7.0%. None was established, and Astra said so itself. Its errors are ordinary arithmetic and algebra mistakes, with no single component to switch off. In the authors' words: "Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked."

In a separate test, two Gemma copies were fine-tuned identically, one correct and one sycophantic. From weights alone, Astra failed in all 4 attempts. Given attention patterns too, it picked the sycophant in all 3 pairs, so some hidden behavior shows only in activations.

Then there is the hint. Researcher Lucas Beyer read the assignment prompt, as Turing Post reported on Nov 2. It told Astra that if Qwen "repeats itself and never finishes, finding and fixing what drives that repetition counts." Coauthor Jayden Teoh replied that the team gave the short assignment via Claude Code. There is a fair counterpoint: Qwen's model card had already flagged the loop, so pointing at it was reasonable.

Hugging Face's Ahsen Khaliq rebuilt the edited model from the logs within a day, re-ran the evaluation and shipped the demo. HF's independent run of the full MATH-500 set scored 51.0%. That confirms the edit works. How much credit Astra deserves for finding it, given the hint, stays open, and the prompt is public.

Known limits:

• One frontier LLM was tested, on models up to 0.8B parameters.
• It is a blog post with logs, which has not been peer reviewed. The authors call it "sparks" and "very early."
• The Qwen edit left 33.1% of answers at the token limit, and Gemma gained nothing established.
• The assignment prompt pointed Astra at the loop.

Numbers come from the authors' post and logs, plus the HF model card for the 51.0% re-run. Coverage: The Register and VentureBeat.

The horizon: written, not trained

Every item on the authors' what-comes-next list is a hypothesis, and the benchmark meant to test them is still being built.

The authors list these five ideas as their own speculation about what comes next:

• Auditing without evaluation: finding backdoors or hidden capabilities without running inference.
• Cheaper optimization: one-shot initialization that skips dataset curation and training loops.
• Zero-data steering and unlearning without catastrophic forgetting, since the Qwen edit touched only 0.016% of the weights.
• Recursive self-improvement: a model inspecting and improving its own weights.
• Cross-architecture merging: grafting a circuit between different architectures, with no averaging.

The authors are building a benchmark and harness to make this "a rigorous, reproducible engineering discipline." Watch for a replication above 0.8B parameters, and for a second model whose fault is one removable direction, which Gemma's was not.

Astra wrote weights directly and beat trained networks on small tests. If that holds at scale, the next model upgrade might be written rather than trained.