Microsoft Copilot Premium: A Promising Yet Disappointing AI

⚡

Key Takeaways

1Microsoft's Copilot agents, designed to assist with research and analysis, did not produce useful results during testing.

2Despite massive investments in AI, the features of Microsoft 365 and Windows have often generated incorrect information.

3A test with the Analyst agent showed initial promise but ultimately failed to deliver a functional file.

💡Why it matters — High expectations for Microsoft's AI are being tested, revealing gaps in the effectiveness of its current tools for professional users.

Copilot Agents Put to the Test

Microsoft's Copilot agents, designed to facilitate research and analysis, have been tested to evaluate their effectiveness. However, the results have not lived up to expectations, as the agents failed to provide useful solutions. The troubleshooting process with Copilot turned out to be a waste of time, without delivering concrete resolutions to the encountered issues.

Microsoft has invested heavily in its AI features, building data centers and acquiring licenses for large language models from OpenAI, Anthropic, among others. At the same time, the company is striving to develop its own internal alternatives to language models. Microsoft's stated goal is to transform Windows and Microsoft 365 into an "agentic operating system," capable of automating tedious professional tasks such as memo writing, presentation creation, meeting organization, and automating routine tasks.

Disappointing Results

In recent weeks, the use of integrated AI features in Microsoft 365 and Windows for daily professional tasks has been tested. Although Copilot has shown promising capabilities at times, the results have often been marred by misinformation, hallucinations, and false leads, resulting in significant time loss.

A Frustrating Experience with the Analyst Agent

Microsoft has been hounding me for months to switch to its new Microsoft 365 Premium plan, which offers higher usage limits for AI and exclusive agents. In the interest of science, I paid $10 to upgrade an unused account for a month to try them out myself. The Analyst agent was tested with a spreadsheet used to track income and expenses, aiming to improve its design.

After several exchanges, the agent suggested optimizations for formulas and consolidating duplicate tables. It even proposed creating a dashboard using only formulas and pivot tables. Copilot told me, "If you want, I can sketch a clean dashboard layout (exact cells and sections) tailored to your data so you can build it in about 15 minutes." However, there was just a small limitation: I would need to create one of the pivot tables myself, but that task would take less than 10 seconds. When asked to build the Excel file, the agent failed to provide a functional link to download the modified file. Copilot informed me that it had created a modified workbook and provided a link that was not clickable. After several unsuccessful attempts, Copilot admitted that the link was not functional in the user interface.

An Unfruitful Search

The Researcher agent from Microsoft 365 Premium was asked to provide a concise explanation of the advantages and disadvantages of Microsoft 365 Premium. The response received was confusing, asking for clarification on the specific plan in question, without providing clear information.

Misplaced Trust

Microsoft's AI chatbots, despite their apparent confidence, failed to resolve the encountered issues. During an attempt to connect to a computer via Remote Desktop, Copilot suggested solutions that proved ineffective. The solution was ultimately found by manually inspecting the connection settings.

Conclusion

While Copilot may one day achieve general artificial intelligence, for now, even a semblance of common sense seems out of reach. Expectations for these AI tools remain high, but current results show that there is still a long way to go to achieve optimal efficiency.