Who keeps the skill when AI does the drafting?

You may have had enough of AI productivity studies, but a new working paper published by the National Bureau of Economic Research in the States is worth your time. The authors and a team of Google researchers switched the AI off for part of their study, and the results were very interesting.

Approach

The team recruited 133 patent lawyers at 11 US intellectual property firms and randomly gave two-thirds of them access to a custom AI drafting assistant for three months. The rest carried on without. Everyone completed a patent drafting task at 10 days and another at 90 days, with AI available to the treatment group. Then, at 90 days, everyone did a third task without AI: marking up a flawed patent application, which is a part of the job that runs entirely on judgement. Blinded patent attorneys at an independent firm scored all the work.

Findings

Across the board, the drafts got better with AI switched on. Treated lawyers scored 0.34 standard deviations higher at 10 days and 0.38 at 90 days, and the improvement showed up on every dimension the raters assessed. Junior lawyers gained roughly twice as much as seniors, which matches the findings in studies of writing, customer service and software.

With AI switched off, the pattern flipped. Senior lawyers who had spent three months drafting with AI marked up patents better than senior lawyers who had not, but junior lawyers showed no gain at all.

At least for now, we can’t say for sure that the juniors didn’t learn because only 25 finished the mark-up task, about seven of them in the control group. So we can say with some confidence that seniors improved, but mark the juniors as ‘unmeasured’ rather than ‘unchanged’.

That said, I suspect that we’ll see more and more research pointing to the same dual-phenomenon.

If you already know what you’re doing, AI is an upgrade and it can have a durable effect. It catches your repetitive mistakes or lazy patterns, and you can onboard those lessons.

But if you’re new to a discipline, there’s the real danger that AI will take the lead. It’s the ‘talent’ and you are its assistant, not the other way around. Writers know that we learn from the painful ‘blank page’ and the grind of failure. Good judgement has to be earned the hard way, but without thoughtful guardrails, AI makes things easy and undermines learning.

Adding to the problem is that the AI-treated juniors reported the largest self-reported gains, on both speed and satisfaction, but that didn’t show up in their unassisted work. Their sense of their own improvement tracked the accomplishments of the AI, not a skill they had developed.

What do we do with this?

• If you have someone devoted to learning and organisational evaluation, share this study.
• If there’s any self-reported element in your evaluations of AI and productivity, get rid of it.
• Deconstruct how junior people build skill and judgement at your business and work backwards into an approach that gives them the opportunity to grow organically. Soon enough, their critical thinking, judgement and experience will dictate success or failure.