Is more engagement good? What 28,000 recorded lessons told us about coaching

01The number everyone watches

If you build a product for teachers, you end up watching the same dashboard as everyone else: active users, sessions a week, how many came back. We watched it too. Our coach lives on WhatsApp. A teacher records a lesson as a voice note, and a report comes back with a score and one move to try next time. From July to October the recordings kept coming (some teachers recorded two or three times a week, some once), and the score printed at the top of each report climbed month on month. By the usual measures it was working, and working better the more a teacher used it.

There's a question underneath all of those numbers: does more of it buy better teaching? I mean a child in a classroom being taught better in October than in July. More recordings don't count as an answer. Neither does a higher score from the system that wrote the advice, because the system that wrote the advice also wrote the score, and it can't be the judge of its own work. I think most teams building for teachers have the same blind spot.

We had checked once, and the answer was too thin to build a program on. So we went and looked again, in two separate ways.

02What we had

The teachers work in Islamabad's government schools, and between July and October 2026 they recorded 28,403 lessons. Every recording became a transcript, mostly Urdu with some English, and every transcript a one-page report with a score and a single 'try this tomorrow'. Each report names one move to try next; the rest of the report usually mentions several. 1,484 teachers recorded long enough for us to watch them change over the term. Most recorded on 1 or 2 days a week. That spread is what lets us ask the question at all: 702 recorded on 2 days a week or more, 652 on about 1 day, and if more coaching makes better teachers, the first group should pull ahead.

A first pass at this question, earlier this month, had said they don't. It read every lesson blind on two rubrics, but it left real doubts. Were the recordings themselves changing over the term? Were the subjects, or the mix of teachers who kept going? We didn't want to build a program on that. This is the recheck. We wrote the plan and fixed the bars before joining any score to a date. A bar is the size a change had to reach before we'd call it real. The recheck was free to come back and say the first pass was wrong.

03How we checked

We ran two separate checks, both blind. In the first, an AI reader scores every lesson from the transcript alone. In the second, an AI judge is shown two lessons by the same teacher, dates hidden, and says which is better taught.

1 · Every lesson re-scored, with nothing but the transcriptThe reader is given a lesson as text and nothing else: no date, no teacher's name, no idea whether this is the teacher's 1st recording or 20th. It scores the lesson on four rubrics. Two are outside standards, an audio version of the World Bank's Teach and the OECD's classroom rubric. One is FICO, the classroom-observation rubric our coaching reports are built on. The fourth is a 12-move rubric we wrote ourselves, to be sensitive to exactly the moves our coach asks for: wait time, pair work, open questions, specific feedback and so on. Then we compare each teacher with their own earlier lessons. Lessons before the first report against lessons after it, and their lessons month by month from then on. Then we split teachers by how many days a week they usually recorded, twice or more against about once, and look for the frequent group's line climbing faster month on month.

The unit for all of this is the spread between teachers (a statistician's standard deviation; the charts use that name). If you lined up 100 teachers by their score, a change of a tenth of that spread would move the middle teacher up about 4 places. We set that as the bar for 'meaningful' before we looked at anything, for the first‑report step and for improvement per month alike. Every estimate comes with a 95% range, the span the true value very likely sits in, and we wrote down that the range had to sit above zero too. And because we test 12 moves at once, a single move has to clear that bar even after allowing for the luck of testing many things.

One thing nearly fooled us. Teachers record more of each lesson as the term goes on, about 1.6 minutes more a month, and a recording that reaches the practice and the close scores higher than one that stops mid-explanation. Whole-recording scores creep up for that reason alone. So every slope allows for recording length, and as a second check we score only the first 15 minutes of a recording, a window that getting longer can't touch.

2 · Two lessons side by side, dates hiddenThe judge sees two lessons by one teacher, order shuffled, and picks the better taught. Every pair is judged both ways round, 17,078 judgments in all. Control pairs tell us how much a judge invents. Show it the same lesson twice and it should favour neither, and it doesn't. Then show it two lessons a few days apart from later in the term, when the teacher already had several reports and nothing new had happened. We call this the quiet week. The judge leans toward the later one by about 3 in 100. That's the judge imagining, and a real change has to clear it.

Readers and judges are AI models (Claude Sonnet, with a sample audited by Claude Opus). Both checks run the same kind of model over the same transcripts, so they can share its blind spots. No human coach has read a pair yet. We built the listening packet, costed it at 9–20 hours per coach, and held it back for now. Until a human coach reads a sample, that's the study's biggest gap.

04 · Finding 1Teachers improved, once

Lessons after the first report score about a tenth of the spread higher than the same teacher's lessons before it, on all four rubrics. That's about 4 places in a line of 100. The blind judges see it too. Across the first report they picked the later lesson 13 more times in 100 than the earlier one; across a quiet week, 3 more. Most of the step is there in the very next lesson, inside the first week. On our rubric it survives every exclusion we tried: short recordings, re‑sent audio, lessons with no audible students, all of them at once.

Then, in the minutes every recording covers, the line goes flat. On Teach and FICO the slope from the first report onward is indistinguishable from zero, and so is our own rubric inside the first 15 minutes of each recording. The judges, shown pairs either side of the second report, lean +3, no different from the quiet week. Over whole recordings, our rubric and the OECD's creep up a little, by about +0.06 a month. We can't separate that creep from teachers recording more of each lesson, so we print it as 'cannot tell' rather than claim it either way.

The +0.13 our rubric gives for the single before-and-after step is a different cut of the data from the weekly points above. On our rubric the line drifts a little higher over the following weeks, inside its range and over whole recordings, which is the creep I just described. On Teach it drifts back toward zero after a month, and the data can't tell a plateau from a fade.

I'd call the step fairly sure. All four rubrics see it, it survives every exclusion on ours, and the blind judges agree. What isn't proven is that the report caused it. In 87% of these pairs the earlier lesson is the teacher's first recording ever, so the step measures the first report and the act of coming back to record a second time, together. That first recording is also the patchiest one a teacher makes, so we checked: the step is still there in the first 15 minutes only (+0.13 on our rubric), and with short, re‑sent and plain-transcript recordings left out. We tried to separate the report from the return with same-day recordings, and the data can't tell. The judges' timing leans one way, which I come back to below, and a lean isn't proof. The teachers who recorded once and never came back start 0.1–0.2 of a spread lower than the rest, and nothing here speaks for them.

05 · Finding 2More often wasn't more

This was the question we set out to answer.

Teachers in the twice-a-week group recorded on 9 days in a typical 4 weeks. The weekly group recorded on 5. Almost double the recording days, and no faster improvement. For each extra recording day a week, the gain per month sits between −0.04 and +0.03 on the four rubrics, and no 95% range reaches the +0.10 we set as meaningful. On no rubric, and not in the 15‑minute window either, is the twice-a-week group measurably ahead of the weekly group. Teachers who started fast in their opening weeks weren't better later on. The frequent recorders did start a touch higher, about 0.12 of a spread, and allowing for who kept recording doesn't change the picture. Whichever way we cut it, recording more often after the first report bought nothing that recording weekly didn't.

A third group, 132 teachers who recorded about once a fortnight, is left off the chart. It's too small to read, its ranges are about twice as wide, and which way it leans depends on the rubric.

06 · Finding 3What changed in classrooms

A tenth of a spread is a small number. What does it look like in a full classroom?

One routine, mostly. On our rubric the move that clears the bar is pair or group work, and the OECD's rubric, the other one that scores it as a move of its own, agrees. On Teach and FICO what clears sits beside it (FICO's prior knowledge; Teach's perseverance and FICO's supportive environment clear too, but only under our covariates, not the first study's), and their own collaboration items lean up without clearing. Pair or group work is the only move on our rubric that clears. Calling on children who hadn't volunteered, and open questions, lean the same way without clearing.

When a first report asked for pair work, a pair task ran in 37 of 100 next lessons against 9 of 100 before, and where it appeared it mostly stayed. It arrived in its simplest form, though. In about 4 of every 5 lessons where pairs actually worked together, nobody reported back. In another 14 of the 100 the pair was announced and never happened, and only 7 of the 100 had a structured task with children explaining their thinking afterwards.

Alongside it, a few more 'why' questions reached a child's own words. Where open questions were the ask, a child's reasoning was heard in 45 of 100 next lessons against 37 before. That's a smaller change than it sounds, because open questions in some form were already in 73 of 100 of these teachers' lessons before their first report.

This is what the step sounded like in a maths lesson, translated from Urdu. Even and odd numbers, taught before the first report and again after it:

The report had asked for open questions and 10 seconds of wait. The why questions came. The wait didn't; later in the same lesson: 'Why is it even? It has a 1, it has an 8. Think a little. Okay, there is an 8 at the end.' Both judges preferred the later lesson.

And an Urdu lesson where chorus turned into sentences:

The other half of the picture matters more, I think. The moves the reports asked for most never moved. Wait time came up in 86 of 100 reports and didn't change by any of three measures we tried. On our rubric it even leans up a little, but the second of the five rungs for wait counts any 3‑second gap, and 88 in 100 lessons had one before any report. A deliberate, held wait showed up in 2 of 20 lessons where it was the ask, and the measured gap rate didn't change. What we heard instead was the move being named: one teacher told the class 'I'm giving you one second to think', and gave them one second. Building on a child's answer, specific feedback, tasks at different levels, children checking each other's work: all of them came up in roughly half of all reports, and none of them moved on our rubric.

And it wasn't everyone. In about 1 first‑report pair in 4 the later lesson was clearly worse on the rubric, and the judges agreed it was worse.

07Why more didn't help

We tested the boring explanations first, the ones that would have let the coach off the hook: the mix of subjects shifting, recordings changing or being re‑sent, the better teachers dropping out, reports arriving late. All four are present in the data and none of them moves the line, so the flat line is real in the minutes every recording covers. Three others, the advice repeating itself, too little time, and the lesson's format deciding what shows, this data can't rule in or out.

The coach itself is our best explanation for the shape. A report names one move to try, and from the first report onward 6 in 10 named open questions, which most lessons already had. By the 16th report, the move it asked for was already in the lesson 80% of the time (55% at the first). Being asked barely mattered. Across 18,828 report-to-lesson steps, an asked-for move showed up in the next lesson 0.4 percentage points more often than one nobody asked for. Pair work, at +4.7, was the exception. The reports didn't remember either. 83 in 100 kept the focus of the one before, almost none referred back to what the last one had said, and a third contradicted themselves between the 'focus' and the 'one thing to do next'. So the lesson after report 12 looks like the lesson after report 11: the report mostly asked for something the lesson already had, and being asked made almost no difference.

If a report mostly asks for what the lesson already has, more reports a week only means more of that ask, and there's nothing for frequency to add. I'm fairly sure that loop is the reason. Not certain: the teachers whose ask never changed are too few to tell.

Recording more did grow the vocabulary. On a rough count we haven't validated, moves named aloud went from 0.09 per lesson at a teacher's first recording to 0.34 by the 16th, and 'I'll give you time to think' with no gap behind it nearly tripled. As teachers clocked up recordings they learnt the words for the moves. A move is scored on a ladder whose upper rungs need a child to actually do something, and on those rungs nothing changed after that first step.

08What this means if you build for teachers

I'd take three things from this.

So here is what we're recommending. Ask for one recording a week. Spend field time getting each teacher to a first delivered report and a second recording, rather than nudging the already-recording toward a third. Then assign schools at random to a push that gets a teacher a first report within a week, followed by weekly or twice-weekly recording, so that next time we can measure if the report itself causes the step.

09What we still don't know

• Does the report cause the step, or the act of returning? Lessons recorded within a day of the report are preferred more strongly by the judges than lessons recorded a week later, which leans toward the report, but a lean isn't proof. The randomised test is for this.
• Does it keep going? 168 teachers were active for 8 weeks or more. They lean upward on two of the four rubrics, and the group is too small to call.
• Would a human coach agree with the model judges? The packet exists, and it hasn't been sent.
• Is the creep on our own rubric over whole recordings better teaching, or more teaching recorded? The extra minutes are real teaching (practice and the close), but this data can't say whether a teacher who now records the close has started teaching it or started recording it.
• Wait time by ear. A transcript shows gaps, but not whether a silence was held on purpose, and that's the move the reports asked for most.