Bocconi: ChatGPT Polit, Diversifying Causal Reasoning

A randomized experiment conducted at Bocconi University with over 1,000 first-year students distinguishes the contributions of AI and causal reasoning training. ChatGPT enhanced the form and coherence of responses, while the reasoning exercise stimulated originality. Together, these two levers accumulate and challenge traditional evaluation methods.

More polished submissions raise questions about the value of the final grade

AI enables students to produce highly refined responses, but focusing solely on the final output provides less insight into their actual understanding. Researchers believe that assignments may need to evolve to better measure originality. They note that a traditional rubric can reward clarity and structure while neglecting the novelty of ideas. When AI brings a submission closer to an expert standard, the final grade reflects less of the process and reasoning involved. The results underscore the importance of criteria that value originality, argumentation, and consideration of multiple avenues, beyond merely conventional or polished responses.

Access to ChatGPT: nearly a point gain and more ideas

Access to ChatGPT improved the quality and coherence of the work, with a gain of nearly one point on a five-point scale. The submissions contained more ideas, followed a clearer logic, and aligned more closely with recommendations produced by experts. Students did not delegate the entire task; they formulated queries, evaluated responses, and selected the final content. Thus, AI helped produce more polished outputs, rich in proposals and better structured.

Causal reasoning: clearer explanations, more distinct ideas

The critical thinking exercise pushed students to clarify why their recommendations might work or fail. This approach did not improve scores on the rubric, which focused on two marketing objectives: awareness and usage of the university shop. An increased variety of ideas, more distinct from those of peers, was observed. This training broadened the range of options considered and strengthened the questioning of assumptions.

Combining AI and training: cumulative effects across multiple measures

The group that had access to both ChatGPT and underwent the reasoning exercise reaped the benefits of both approaches. Its variety of ideas matched that of students trained in reasoning alone, while its scores on the rubric and the number of ideas aligned with those of the group with AI only. The outputs from the combined group also displayed stronger logical coherence and more evidence of seeking explanations and questioning assumptions.

A randomized protocol to isolate effects

Over 1,000 first-year students at Bocconi were randomly assigned to four groups: access to ChatGPT based on GPT-4o, causal reasoning training, a combination of both, or neither. The case addressed involved marketing recommendations for the university's merchandise shop. Causal reasoning, developed through a game, examples, questions, and feedback, aimed to connect causes and effects and clarify the conditions for success or failure. Human graders scored on a five-point scale, while automated analysis measured the number and variety of ideas, markers of causal reasoning, and similarity to three experts. This randomized protocol allowed for the isolation of the specific effects of AI, training, and their combination.

Evolving assessment practices in a growing research field

Researchers place these results within an expanding body of research on the impact of AI and the organization of learning. Many educators are already considering adaptations to assignments and assessments. This approach is part of a decades-long trajectory of the joint evolution of tools and educational practices. The observations accumulated here confirm that AI helps improve responses, that critical thinking broadens ideas, and that both approaches complement each other in preparing students.