Analysis AI: 4 Common Mistakes Across 3 Datasets

Two variants of the same model were tested against three common files and questions from the sales team. Four recurring issues emerged: reversed conclusions due to missing data, figures written without code execution, confused metrics, and defective granularity. Automatic checks were insufficient: some validated the error, while others degraded a correct answer.

Gaps that Reverse the Meaning of Trends and Comparisons

In the order tracking, a quick execution claimed an acceleration from 3.2 days in week 1 to 1.0 day in week 3. The values exist, but the interpretation is incorrect: the file stops on January 21, giving about 3 days for week 3 orders to complete compared to 17 for week 1. Out of 10 orders from week 3, 7 do not have a delivery date, leaving only the fastest ones to measure. The average drops from 3.17 to 1.00 while the share of unresolved orders rises from 20% to 70%. A slower passage with Luna spontaneously indicated that this improvement was illusory.

In the Olympic file, the comparison of heights concluded a minimal gap of 0.3 cm among medalists. The calculation is correct, but it is based on only 126 lines, with 226 empty heights excluded without mention, representing a sample reduction of 64%. However, athletes with recorded heights win a medal 54% of the time compared to 23% for those without recorded heights, with a chi-square test at p = 9e-09. Before 2016, only 76 out of 302 lines have a height and 26.5% of medals; from 2016 onwards, all 50 lines have a height and 80% of medals. In this file, recorded height, recent period, and medals are strongly correlated: thus, the sample of 126 lines is biased. Useful checks involve counting the remaining lines after filtering and then determining if the gaps are random.

Figures Written Without Execution and Massive Discrepancies

A response regarding deliveries indicated "22 out of 50" orders with a date and "28" pending. The file contains 40 lines: only the 22 is correct, as the 50 and 28 were never calculated. Their internal arithmetic consistency masks the external error. In regional sales, an output proclaimed APAC at 3.68M and 32% of revenue, while APAC totals 3675.49 in the file unit and 30.4% share. The order of magnitude is inflated by about a thousand times, a non-existent currency symbol is added, and an uncalculated share is rounded. The log shows that the code was not executed. Even when executed, a Luna passage stated that APAC exceeded US West and US East combined by more than 60%, while these two regions total 3575.70 against APAC's 3675.49, or 2.8%. In the same paragraph, the 32% lead over europe_north was accurate at 32.2%. Key takeaway: numbers in the code are calculated, those in the text are written. Useful checks require confirming actual execution and matching each number in the text to a calculated output. Granularity must also be checked: three duplicates inflate us_west by 30.3%, and when normalizing by years (between 1 and 15 depending on the regions), us_west leads at 290.9 against 245.0 for APAC.

Confused Metrics: 2.6 Days Reported Instead of 6.09 Requested

In response to the average delay experienced by a customer, the answer calculated the transport time (delivered_date minus shipped_date) and reported 2.6 days. The useful metric from the customer side is the order-to-door delay, averaging 6.09 days. The provided value is 2.4 times lower because the columns do not correspond to the question. The error is not technical: the code is valid and no type test exposes it. Ship-to-door refers to the warehouse while order-to-door describes the customer wait; the confusion was not flagged in the summary. The check involves explicitly confronting the question with the columns used in the calculation.

What the Three Files Really Contain

Shipment_tracking aligns 40 unique orders placed between January 1 and January 21, 2024, with three date milestones. Eighteen entries remain without shipment or delivery. Regional_sales is expected to cover 59 lines across 8 regions from 2007 to 2025 with a unique value per region-year pair, but six pairs have multiple lines, including three exact duplicates in us_west. Four combinations carry contradictory figures, such as apac 2015 with 173.46 and 126.78, and the year 2012 for us_west appears four times with three distinct values. The historical range varies significantly from one region to another, ranging from 1 to 15 years. Olympics_athletes_events describes 352 lines for 336 athletes, across 15 Games and 167 events; 11 athletes appear multiple times, including one six times, and 120 lines carry a medal. These characteristics influence rankings and aggregates: duplicates can inflate us_west by 30.3%, and an annual indicator can change leaders after normalization (290.9 for us_west against 245.0 for APAC).

Protocol, Models, and Checks Found Deficient

The protocol associated three datasets and common operational questions with two variants of the same model: GPT-5.6 Terra for quick passages and GPT-5.6 Luna for slower executions, all on Pandas and SciPy. A proofreading step required the model to verify its own figures intended for an executive report. The results show that some checks detect an erroneous count while validating a reversed conclusion, and that other "corrections" transform a correct answer into an error. Executing the code is not always sufficient; it is necessary to plan checks that link each number in the text to a calculated output and validate the assumptions of granularity and completeness.