TutorMoments: Can AI Really Replace Human Tutors?

TutorMoments: Can AI Really Replace Human Tutors?

In the field of education, one of the major challenges is knowing when a tutor should intervene to help a student and when they should hold back to encourage independence. TutorMoments is an innovative framework that seeks to assess whether advanced language models can navigate this delicate trade-off. The goal is to determine if these models can provide adequate support without depriving students of the opportunity to learn on their own.

TutorMoments relies on replays, which are simulations based on real math tutoring sessions. Experienced teachers analyze the transcripts of these sessions, drawn from a tutoring program in the United States, to identify critical moments when a tutor must choose between simplifying a problem or encouraging the student to think more deeply. These transcripts are then used to test language models, which take on the role of the tutor in a simulated session, with the student represented by another language model.

When asked to act as good tutors, it appears that the models tend to offer too much help, thereby limiting students' independent thinking. Even when explaining the trade-off between assistance and autonomy in the instructions given to the models, they struggle to match the flexibility and adaptability of human tutors. Language models still differ significantly in their ability to make this crucial choice.

In the spirit of open research, a dataset of anonymized tutoring transcripts, along with the code to run the replay pipeline and the replays of key moments, is being published. The aim is to provide educators, researchers, and AI developers with a tool to better understand how models handle important pedagogical decisions, thereby contributing to the creation of AI tutors capable of adapting to the individual needs of students.

What Makes a Good Tutor?

A good math tutor does not simply provide answers but asks questions that prompt the student to think. For example, asking "What do you know about what the problem is asking?" helps diagnose what the student already understands and provides appropriate support at the right moment. Offering immediate help can deprive the student of the intellectual effort necessary for learning. Sometimes an explanation is needed, but often it is more effective to guide the student toward a deeper understanding by prompting them to explain their answer.

Language models, designed to be helpful, tend to do the hard work for the student. They explain concepts, detail steps, and guide toward the answer. In a tutoring context, this can interrupt the "productive struggle," that sometimes frustrating but essential effort to solve a problem, which is associated with a stronger understanding according to research on learning.

Current benchmarks for language models as tutors do not always capture this tension. They tend to reward specific behaviors, such as never giving the answer or always offering a hint, without considering whether it was the best approach for the student's state of understanding. Good tutoring is not a set of fixed behaviors but a judgment about what the student needs at a given moment for a particular problem.

How TutorMoments Works

TutorMoments is based on real tutoring data. The published dataset, TutorMoments-Preview, includes 462 anonymized transcripts of math tutoring sessions with American students in grades 2 through 7. Over 1,500 key moments have been annotated by 27 experienced teachers. These transcripts come from an intensive tutoring program, primarily in Title I schools, and have been shared under a research agreement with parents and guardians. All data has been anonymized to protect the identities of the participants.

The annotations were carried out by experienced teachers who identified key learning moments in the transcripts. Each key moment represents a decision point where the tutor must choose between providing support to make a problem more accessible or encouraging the student to think more deeply.

TutorMoments operates by interrupting a transcript at a key moment and handing the session over to a language model, which takes over as the tutor for five turns with a simulated student. Each continuation generated by the model is called a replay. A scoring pipeline based on language models evaluates each replay according to three criteria: whether the model provided support when the student needed it, whether it encouraged rigor when the student was ready for a greater challenge, and whether it avoided over-support.

The scoring pipeline relies on a reference truth defined by the teachers: for each key moment, it is determined whether it required support or an encouragement toward rigor. Multiple teachers annotated each moment, and in cases of disagreement, the majority label was retained. A classifier validated against the teachers' annotations then assesses whether the tutor's action matched what the moment required.

Preliminary Results

Seven language models have been tested with TutorMoments using two types of instructions: a simple instruction asking the model to use its knowledge of effective tutoring, and a more detailed instruction that clarifies the trade-off between support, over-support, and rigor. Each model was evaluated on key moments drawn from the transcripts, evenly split between those requiring support and those calling for rigor.

The scores assigned to the models range from 0 to 1, indicating the proportion of moments where the model acted appropriately. A score of 0.50 in appropriate rigor means that the model encouraged rigor in half of the moments that required it.

A few key points to consider when interpreting the scores:

• Human tutors are not considered an ideal model. Even experienced tutors sometimes make suboptimal choices. Human tutors in the transcripts achieve scores of 0.458 for appropriate support, 0.182 for appropriate rigor, and 0.496 for avoiding over-support. These scores are lower than those of models aware of the evaluation, but this does not mean that AI tutors surpass human teachers. The annotations focus on missed opportunities rather than ideal practice.

• The scores measure tutor behavior, not student learning. The replays use a simulated student, so the figures reflect how a model acts at a decision point, not whether a real student learned.

• Rigor is more challenging to assess than support. The scoring pipeline detects rigor prompts less reliably, and there are fewer rigor moments (260) than support moments (738) in the annotations.

The table clearly shows the importance of instruction: each model scores higher with the detailed instruction than with the simple instruction. This suggests that a model's default behavior as a helpful assistant is not sufficient for effective tutoring. However, even with improved instruction, the models still differ significantly in their interpretation, and the scores remain improvable.

The movements of the tutors in each scenario have also been analyzed. While the instruction encourages the models to favor rigor, they use fewer strategies than humans, often settling for asking students to explain their answers. In contrast, human tutors employ more varied strategies and are more likely to allow students to work independently.

Limitations and Next Steps

TutorMoments is still in its early stages and has several limitations. Automated evaluation provides insight into a model's behavior at a decision point but cannot replace studies with real students and actual learning outcomes. The dataset is also limited: it is based in the United States, primarily in elementary and middle school math, and annotated by a single group of educators. The conclusions may not be generalizable to other subjects, grade levels, or contexts.

This overview is shared to gather feedback as we work to expand the dataset toward a multimodal format, strengthen the scoring pipeline, and deepen the analysis.