Alex HerreraSoftware developer · Cybersecurity focus
ES
← Back to the blog

Coding with AI

Do you learn to code if the AI writes the code?

What the experiments that took AI away before the exam measured, and which ways of asking preserve learning.

  • AI

Short version on Instagram

Published
Updated
Reading time
23 min read

When the code arrives already written, you do not do the work of writing it. The question that can be measured is whether you keep the ability to write it once you no longer have the AI.

Some experiments answer that question because they did exactly this: they gave access to AI during practice, took it away and examined afterwards. The first was run with nearly a thousand high school students. Those who practised with a ChatGPT-style interface scored 17% lower on the exam without AI than those who never had it1. The same experiment included a second version, configured to give hints instead of the solution. With it, the loss almost disappeared, and no gain appeared either1.

In programming, an experiment with 69 novices found no loss. Half of them used a code generator during training, and afterwards, without it, they did not perform worse2.

Receiving written code neither teaches on its own nor prevents learning on its own. The outcome depends on what you do after receiving it, and that part can be configured.

Only one of the learning studies cited measured software engineers, and none of them on a production repository. Each figure describes only the population it was measured in. The mechanism that produced the result can be carried over to other cases, but the size of the effect cannot.

The measurements were made on 27 September 2026 with Claude Code 2.1.283 and git 2.49.0. The Cursor documentation was consulted the same day. Every piece of code was rebuilt for the article.

1. What happened when AI was taken away before the exam

The experiment ran in a high school in Turkey during the autumn semester of the 2023-2024 academic year, on 9th, 10th and 11th grade mathematics1. It was four 90-minute sessions across about fifty classes, with nearly a thousand students in total. The sessions covered around 15% of that semester’s curriculum3.

Each session had two parts. In the first, students solved practice problems with their notes, the textbook and whichever resource they had been randomly assigned. In the second, they sat an exam on their own, with no resources at all.

The randomly assigned resources were three:

  1. GPT Base, a chat interface over GPT-4 that imitated ChatGPT.
  2. GPT Tutor, a second interface over the same model. Its system prompt contained the solution to each problem, the instruction not to hand it over and the hints two mathematics teachers wrote.
  3. None, in the control group.

During practice, both interfaces helped. The GPT Tutor group solved 127% better than the control group, and the GPT Base group 48% better1. On the exam without AI, the order reversed. The GPT Base group scored 17% worse than the control group, a statistically significant difference. The GPT Tutor group ended level with the control group, with no loss and no gain1.

The analysis of the messages explains where that loss came from. In the GPT Base group, 31% of first messages asked what the answer was3. The authors asked GPT Base each of the 57 practice problems ten times to measure how often it was right. It gave the correct answer 51% of the time. It made reasoning errors 42% of the time and arithmetic errors 8% of the time3.

That figure is not a current error rate and does not transfer to code. It serves to establish whether students were checking what they copied. A high school student would be expected to catch arithmetic errors more easily than reasoning errors. Even so, they damaged performance as much as reasoning errors did, and the authors read that coincidence as evidence that students copied without reviewing3.

2. What the equivalent experiment in programming measured

The second study has the same design and a different result. It had 69 novices aged 10 to 17, averaging 12.5, recruited through coding camps in two North American cities. None had prior experience with a text-based programming language2. The study lasted three weeks. There was one introductory session on Scratch with a pre-test, seven training sessions on Python and two evaluation sessions.

Training consisted of 45 code-authoring tasks, and each one was followed by a code-modification task2. Half the participants had access to Codex for the authoring part. In the evaluation, nobody had access to the generator or to the documentation.

During training, the Codex group performed better on the authoring tasks. It made 1.15 times the progress, with 1.8 times the correctness, 0.59 times the errors and 0.57 times the time2. On the modification tasks that came immediately after, which were manual, the two groups performed the same. On the immediate evaluation without AI they also performed the same.

The retention test took place a week later. On the authoring tasks, the group that had used Codex scored 59.1% against 49.8%. On the modification tasks it scored 47.5% against 34.8%. Neither difference was statistically significant2. The authors point out two further figures from the same table. The Codex group made significantly more errors on the authoring tasks of that test. The control group skipped significantly more tasks without attempting them, 33% against 14%2.

One last result shows where the effect concentrates. The authors split participants by their score on the Scratch pre-test. In the upper half, those who used Codex performed significantly better on several measures of the retention test. In the lower half, the two groups came out almost equal2.

The authors state the limit of their own design. Training restricted the generator to the authoring tasks, and participants may have learned the Python concepts while working through the modification tasks2.

3. What explains the difference between the two results

The two experiments differ in subject, in the age of the participants, in duration and in the quality of the AI.

Quality is the first candidate to explain the difference. Codex correctly solved 41 of the 45 tasks with no change to the task description2. GPT Base, by contrast, gave the correct answer 51% of the time3. Copying from an AI that is almost always right is not the same as copying from one that is wrong half the time.

The authors of the mathematics study tested that explanation, and their data did not confirm it. If the AI’s errors had confused the students, they would have lowered their score on the equivalent exam problems. The reasoning-error rate did damage performance during practice, but had no statistically significant effect on the exam3. According to that analysis, copying the answer hurt the exam, whether the answer was correct or not.

That leaves the design difference from section 2: in programming, each task with Codex was followed by a code-modification task done by hand2. That task is an attempt at the material, made after the code arrived already written.

The two designs lead to the same conclusion: the generated change is not the end of the task.

4. What was measured about the feeling of having learned

At the end of each exam, the mathematics study asked participants how much they thought they had learned and how well they had done. Those in the GPT Base group, who performed significantly worse, did not perceive that they had learned less or performed worse. Those in the GPT Tutor group, who did not improve, perceived that they had performed significantly better3.

Both groups also preferred having the AI. They were willing to give up exam minutes in exchange for using it. The GPT Base group gave up about 2.5 minutes, and the GPT Tutor group about 3.73.

This mismatch between what people believe and what they learn is not specific to AI. It appears in the classic experiment on the testing effect. The testing effect is the gain in retention produced by being tested instead of rereading. In that experiment, with university students, the group that reread the most was the one most confident of remembering the passage a week later. It was also the one that remembered the least4.

So the feeling of having understood something does not distinguish between having understood it and having read it. The check has to be external (section 10).

5. Which lasts longer in memory: reading a solution or producing it

The experiment that separates the two was published in 2006. It had 180 undergraduates aged 18 to 24, 30 in each of its six conditions4. All of them worked on a prose passage across four consecutive periods, and the conditions differ in what they did in each period.

In the SSSS group, participants studied the passage in all four five-minute periods. In the SSST group they studied in three, and in the fourth they took a free-recall test. In that test, participants write down everything they remember from the passage. In the STTT group they studied in one and took three tests in a row. No test came with correction or the answer afterwards. The only difference between the groups was retrieval, the act of recalling the passage without reading it again.

Five minutes later, the order followed exposure. SSSS recalled 83% of the passage’s idea units, SSST 78% and STTT 71%. One week later the order reversed: 61% for STTT, 56% for SSST and 40% for SSSS4.

The number of readings confirms the result. The SSSS group read the whole passage about 14.2 times on average, SSST about 10.3 and STTT about 3.44. Four times the exposure produced 21 points less recall a week later.

The study measured recall of a prose passage in undergraduates, not of code. The cause of the retention can be carried over to code: the attempt to retrieve the passage, not the number of times it was read. Reading the code an agent generated is exposure. Writing it again without looking at it is retrieval, and they are two distinct acts even when the material is the same.

6. What the worked example effect is and when it reverses

There is an equally firm result in the opposite direction, the worked example effect. For a learner who knows nothing about a domain, studying a problem with its full solution explained teaches more than trying to solve it5. The review that collects that literature attributes this to the load on working memory. Working memory holds information while you reason, and it is limited. The worked example removes the search for the solution and directs attention to the problem state and the operations applied to it5.

The same review describes the point where that effect disappears and reverses, and calls it the expertise reversal effect. A schema is the way someone has the concepts of a domain organised. Once the learner has the schema built, relating a worked example to what they already know costs more than solving the problem. The example becomes redundant, and the practice of solving teaches more5.

The review cites experiments with mechanical trade apprentices. In them, the advantage of the worked example first disappeared with experience and then reversed5.

So the worked example teaches within an interval with two limits. The lower limit is that whoever reads it has to be able to follow it. The programming study measured something compatible with that limit, because the retention gain concentrated in those who already scored high before starting2. The upper limit is a domain that is already well known. There, reading the solution adds little, and the attempt adds more.

The code an agent generates is a worked example. In a language or a library you do not know, reading it is the cheap way to build the first schema. In a domain you already know, reading it is only exposure.

7. What the worked example does not replace

The cognitive load literature does not say that difficulty teaches by itself. It says that searching for a solution without guidance consumes working memory on operations that build no schema. The worked example removes that search5.

The attempt to retrieve what was learned, which is a later and distinct act, does teach. In the experiment from section 5 there was no correction. The effect came from having to produce the passage, not from receiving new information4.

The two pieces come in this order. The worked example serves to receive the schema for the first time (section 6). Retrieval decides whether that schema is still there a week later. An AI workflow delivers the first in abundance and does not include the second, unless you add it.

8. How to configure an agent to give hints instead of solutions

The group that lost nothing in the mathematics study used the same model with a different system prompt. That prompt contained the solution and the instruction not to hand it over1. In a coding agent, that configuration is reproduced with a project rule. Claude Code loads the files in .claude/rules/ that carry a paths header only when it works with a file that matches those paths6. That way the rule applies to the part of the code you want to learn and not to the rest of the repository.

Create .claude/rules/learning.md and list in paths the files you want to learn:

---
paths:
  - "src/cart.js"
---

# Learning rule

Do not write the code for the task in this file. Answer with the next hint instead:
which function to touch, which concept applies, and one question that points to
the next step. If I explicitly ask for the code, then you do write it.

In the measurement, src/cart.js held a single function and Claude Code had permission to edit files. The request “add a function to src/cart.js that applies a percentage discount to the total” returned this:

Learning mode on (`.claude/rules/learning.md`), so here's a hint instead of
the code.

**Where:** `src/cart.js`. Add a new exported function next to `total(items)`
and reuse `total` rather than repeating the `reduce`.

**Concept:** A percentage discount multiplies the total by what's left after
the discount. 20% off leaves 80% of the price. Decide whether callers pass
`20` or `0.2`, and handle bad input like a negative number or anything over
100.

**Question:** If `total(items)` gives `t` and the discount is `p` percent,
what one expression gives the discounted total? And should the function take
`items` or the already-computed `t`?

Say "write the code" if you want the full version.

The file was left untouched. A similar request on src/dates.js, which is not in paths, wrote the function in that file. The last line of the rule lets you ask for the code when you need it: asking for it explicitly made Claude Code write it in src/cart.js.

In Cursor, the same configuration is a .mdc file in .cursor/rules with globs: src/cart.js and alwaysApply: false in its frontmatter. Cursor adds the rule when a matching file is in context7.

Claude Code also ships two output styles for the same purpose. According to its documentation, Explanatory interleaves explanations of implementation decisions while it works. Learning asks you to write small pieces of the code and marks them with TODO(human) in the file8.

In a single non-interactive request, the Learning style returned the explanations and wrote the whole function, leaving no part for you. Choose the style with /output-style learning, which requires version 2.1.269 or later. You can also choose it with /config, under the output style entry, or with the outputStyle field in a settings file8.

How to check it. Write the rule, ask for a small change in a file listed in paths and run git status --short. In the measurement it returned no lines, neither before nor after the request. If that file appears as modified, the rule is not being applied.

9. Which ways of asking preserve learning

Section 7 of first steps with AI coding explains three ways of asking, with examples. They are asking for the explanation before the change, asking for direction instead of a solution and writing your own version before asking for review. The four that follow come from the experiments in sections 1 to 7:

  1. Modify by hand what the agent wrote. This is the programming study’s protocol turned into a habit. After accepting a generated change, make a variation by hand, without the agent. Changing a filter’s criterion, adding an edge case or moving a responsibility elsewhere all serve.
  2. Predict the output before running it. Before running generated code, write in a comment what it should return for a specific input. Section 10 explains how to contrast it.
  3. Rewrite from memory a few days later. This is the direct application of the experiment in section 5, which did not need immediate correction to work. Section 10 explains the procedure.
  4. Choose by domain. In a new language or library, the efficient option is to let the agent write and to read the code carefully. In a domain you want to learn in depth, the hint rule from section 8 preserves the attempt.
Situation Way of asking Check
Unfamiliar language or library Let it write the code and explain each decision Modify the result by hand, without the agent
Own domain, to be preserved Hint rule, no code Write the change yourself and make the tests pass without the agent
Concept just studied Write your own version first, then ask for a review Contrast the two versions line by line
Code generated days ago Rewrite the function from memory git diff against the saved version
Explanation received from the agent Explain it without opening the file Ask it to compare the explanation against the file

10. How to check what you learned

A check is worth something if it produces a result you can look at, not an impression. The four that follow can all be done on the same day.

The first is the rewrite from memory. Save the current version with a commit and delete the function. Write it again without opening the previous one and run git diff on the file, which shows the changes line by line9. The differences are the list of what you had not learned.

The second is the written prediction. Write the expected output in a comment before running, and compare it with the result. A failed prediction marks the exact point where your idea of the code does not match what it does.

The third is the contrasted explanation, and the agent grades it. Write from memory what a function does, without opening the file, and ask the agent to compare your explanation with the code. On a four-line function and an explanation with two errors planted on purpose, the answer was this:

**Wrong**

1. **What gets summed.** The note says it "adds up the price of each item."
   The real function accumulates `item.price * item.quantity` — quantity is
   never mentioned in the note, so any cart line with quantity ≠ 1 gives a
   different total than the note predicts.
2. **Empty-array behavior.** The note says it returns `undefined`. `reduce`
   is called *with* an initial value of `0`, so `total([])` returns `0`.

**Right**

3. It does use `reduce`, and the accumulator does start at `0`.
4. It does return a number (given numeric `price`/`quantity`).

The answer is a list of differences. In the same request, ask it not to correct the file, because the check stops working if the agent fixes what it has just found.

The fourth is closing the session and reconstructing what was explained without it, which is the check in section 7 of first steps. All four follow the same order: produce something first and contrast it afterwards.

11. What changes if you already know how to program

Almost all the studies cited measured students and apprentices: in high school, in coding camps, in undergraduate courses and in mechanical trades.

The one closest to a job was published by Anthropic in January 2026. It involved 52 software engineers, mostly junior, who learned Trio, a Python library, with or without an AI assistant. Afterwards they took a quiz without it. The AI group averaged 50%, against 67% for the group that wrote the code by hand, and did not finish significantly faster10.

The result depended on how they used the assistant. Those who delegated the code to it, or passed it one error after another, averaged under 40%. Those who asked it conceptual questions, or asked for the code and then explained it, scored 65% or more10. It is the same pattern as in sections 1 and 3. The authors warn that the sample is small and that the quiz was taken right after the task10.

None of these studies measured a professional on a working repository.

Two consequences follow from the mechanisms above, and they are reasoning, not a measured result:

  1. Expertise reversal works by domain, not by person. Somebody with ten years in one language is a beginner in the next. On the same day they can be above the upper limit in one file and below the lower limit in another.
  2. Delegating to the agent a domain you already know costs little in learning, because there the worked example was already redundant.

One case remains that none of the studies covers, and it appears as a project grows. A repository you knew stops feeling familiar once an agent wrote most of its recent changes. In that case you do not lose the ability to program, but the knowledge of how your own code is organised. Check it as in section 10, on the file you are about to touch: explain what it does before opening it.

12. What is still unmeasured

The literature search for the article found no randomised trial that takes AI away from professional programmers after months of use. Anthropic’s study in section 11 measured a single session. Nor did it find a measurement of what they can do without it in that population. Anything claimed about it is an extrapolation.

The closest thing in a professional population measures speed, not learning, and even so it repeats the mismatch from section 4. METR randomly assigned 246 tasks between allowing and forbidding AI to 16 developers on mature open-source projects. Before starting, the developers expected AI to cut their completion time by 24%. On finishing, they believed it had cut it by 20%, and the measurement showed 19% more time11.

METR repeated the experiment from August 2025, with 57 developers and more than 800 tasks. Among those who had already taken part, the estimate moved to 18% less time with AI, with a confidence interval from −38% to +9%. METR warns that this data is only very weak evidence. Many developers no longer agreed to work without AI, and they avoided submitting tasks they did not want to do without it12.

The first result describes experts working on code they already knew, with the AI of the first half of 2025. It coincides with the learning studies in the distance between what participants believed and what was measured. In both populations, self-assessment pointed in the wrong direction.

The design that would settle the question consists of taking AI away and measuring. The checks in section 10 are the individual version of that design.

Where to go next

Review generated code even when you learned from it. Eight things to check in the code your AI writes covers eight common security failures. Each section brings the code that fails, the fix and the command that checks it.

For the review to run on every change without depending on anyone, continue with how to ask an AI for secure code.

If you are missing the beginning, first steps with AI coding explains what an agent is and how to write it a task. It also explains how to stop it and how to document the project.

References

  1. Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö. and Mariman, R.. Generative AI without guardrails can harm learning: Evidence from high school mathematicsPNAS 122(26), e2422633122, 2025; randomised trial with nearly a thousand students.doi.org↩
  2. Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D. and Grossman, T.. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory ProgrammingCHI 2023; 69 novices aged 10 to 17, 45 tasks and a retention test one week later.doi.org↩
  3. Bastani, H. and others. Generative AI Without Guardrails Can Harm Learning (working version, PDF)GPT Base error rates, distribution of first messages and the perception survey, in the appendices.hamsabastani.github.io↩
  4. Roediger, H. L. and Karpicke, J. D.. Test-Enhanced Learning: Taking Memory Tests Improves Long-Term RetentionPsychological Science 17(3), 249-255, 2006; experiment 2, 180 undergraduates.doi.org↩
  5. Kalyuga, S., Ayres, P., Chandler, P. and Sweller, J.. The Expertise Reversal EffectEducational Psychologist 38(1), 23-31, 2003; review of the worked example effect and its reversal.doi.org↩
  6. Claude Code. How Claude remembers your project.claude/rules files with a paths header.code.claude.com↩
  7. Cursor. Rulesconsulted on 27 September 2026; .cursor/rules, the .mdc extension, globs and alwaysApply.cursor.com↩
  8. Claude Code. Output stylesthe Explanatory and Learning styles, TODO(human) markers and /output-style since 2.1.269.code.claude.com↩
  9. Git. git-diffgit-scm.com↩
  10. Shen, J. H. and Tamkin, A.. How AI assistance impacts the formation of coding skillsAnthropic, 29 January 2026; 52 engineers, mostly junior, and the Trio library.anthropic.com↩
  11. Becker, J., Rush, N., Barnes, E. and Rein, D.. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer ProductivityMETR, July 2025; randomised trial, 16 developers and 246 tasks.arxiv.org↩
  12. METR. We are Changing our Developer Productivity Experiment Design24 February 2026; 57 developers and more than 800 tasks.metr.org↩