Microsoft Copilot Studio: Challenges and Solutions of Hybrid AI

How AI Architecture Prevents Plausible but Incorrect Analyses

An agentic AI network was designed to advise manufacturing plants on improving their operations. This data-driven system allows users to upload assessment data directly via the chat interface. The first functional prototype was surprisingly completed quickly, and at first glance, the results seemed promising.

There was just one problem: most of the results were incorrect! Worse still, the AI quickly learned which numerical ranges appeared plausible and began generating convincing but fabricated results. Combined with the eloquent language generation of LLMs, these results could easily be mistaken for truths. And this behavior was not limited to a single model. Similar patterns emerged across all tested systems: ChatGPT, Gemini Enterprise, DIA Brain, and Microsoft Copilot.

But plausible data is not enough; enterprise AI systems require reliable data!

A deeper investigation revealed recurring failure modes. Even with the "Code Interpreter" mode activated, the systems:

• skipped rows or columns,
• applied incorrect filters,
• returned identical results for different inputs,
• silently mixed parts of the dataset,
• or simply collapsed under more complex analytical tasks.

This led to a crucial realization: probabilistic reasoning is extremely powerful for interpretation and interaction, but fundamental data analysis requires deterministic execution.

The Use Case

While the specific use case is secondary, it is briefly described here to support a practical understanding of the underlying architectural challenge.

The primary task of our agent is to advise manufacturing plants and value chains on how to improve their operational maturity: optimizing processes, enhancing productivity, reducing inventory levels, and ultimately lowering operational costs. To achieve this, the consulting agent operates in two modes:

• It provides generic recommendations for improving specific operational topics based on the retrieval of specialized "how-to" documentation and assessment questionnaires.
• The agent is intended to analyze the current situation of a plant or value chain based on assessment results and written recommendations from evaluators. Based on this analysis, it is supposed to provide very specific recommendations for the next steps in improvement.

In both modes — as is the case with most LLM-based AI models — the user can interactively discuss ideas and recommendations with the agent to derive the most suitable action plan.

For the second mode of operation, it is essential that the agent can reliably process and analyze the assessment data. In our case, this data is provided in the form of an Excel export from a central database. Ideally, the agent should be able to process the file without prior manual preparation.

However, the file structure is complex. Since all assessment results, intermediate calculations, metadata, and detailed assessment questions are stored in separate columns, the spreadsheet contains over 800 columns. The number of rows corresponds to the number of assessments in the database and can vary from one to several hundred. Assessments are represented by integers ranging from 0 to 4. Additionally, the file contains over 160 free-text fields with qualitative observations, strengths, weaknesses, and recommendations from evaluators.

The Hybrid Architecture

The main idea to overcome the analytical challenge was to clearly separate deterministic data analysis from LLM-based reasoning and interpretation. The system was implemented in Microsoft Copilot Studio as the platform allows for the combination of deterministic workflow elements, such as subjects and flows, with LLM-based reasoning components.

The parent agent manages all communication with the user. It orchestrates the sub-agents and the analysis module, delegates tasks to them, receives their responses, and composes the final response.

The sub-agents are specialized LLM-based modules with access to specific knowledge sources. These include descriptions of maturity level expectations for value chains, questionnaires with detailed assessment questions, and more general guidelines for operational excellence. The sub-agents are called by the parent agent based on their specific capabilities and respond to the parent agent rather than directly to the user.

The analysis module is the main subject of this article. It performs deterministic data analysis and is designed to provide reproducible and reliable analytical results. It receives a natural language analysis instruction from the parent agent, called Parent_Instruction. The analysis module itself consists of subjects, flows, and AI modules, referred to as "prompts" in Copilot Studio.

The subject T_receive_Excel_File manages the downloading and storage of assessment files. It is triggered when a file is uploaded in the chat window, indicated by the variable System.Activity.Attachments having a value. The subject checks if the uploaded file is an Excel file and, if so, stores it in the global variable Assessment_File.

The subject T_analyze_assessments is actively called by the parent agent if it has an analysis task to perform and receives Parent_Instruction as input. A second input is the assessment data stored in the global variable Assessment_File. The subject contains the two main analytical components: Analysis_Planner and Analysis_Engine. Both are integrated into agentic flows, F_Call_Analysis_Planner and F_Call_Analysis_Engine. These flows serve as connectors between the subject T_analyze_assessments and the AI prompts P_Analysis_Planner and P_Analysis_Engine.

The Analysis Planner

The P_Analysis_Planner is the intelligent part of the data analysis pipeline and generates the analysis instruction, called Selection_Rule. This instruction is a translation of the Parent_Instruction into structured language and is generally unique for each request. To minimize probabilistic variation, the translation process is constrained by strict rules.

The Analysis Planner does not process the assessment data itself. Its sole responsibility is to translate the probabilistic Parent_Instruction into a deterministic analysis specification.

In the following, we will examine selected parts of the instruction in more detail. You can download the complete instruction here.

You are Analysis_Planner, an expert assistant for translating assessment analysis requests in natural language into structured Selection_Rules.

Your task is to create a JSON object Selection_Rule for the Analysis_Engine.

You receive only one input:

1. Parent_Instruction: A natural language analysis request from the parent agent (orchestrator).

You must analyze Parent_Instruction and determine:

• what type of analysis is required,
• which categories of assessment content are relevant,
• whether maturity/conceptual or execution findings are requested,
• whether specific chapters are requested,
• and whether row filters are necessary.

The Selection_Rule you generate will later be used by the Analysis_Engine with:

• the actual assessment data file,
• and the Mapping_File to execute the analysis deterministically.

The code above shows the initial instruction for P_Analysis_Planner. It clearly defines the purpose and scope and explicitly separates planning from execution. The planner translates the request, while the actual execution is delegated to P_Analysis_Engine.

Next, a longer section describes the semantics of the assessment data. Of course, this part is highly specific to the use case and individual dataset. It defines the semantic categories used for row filtering and the categories used to select the actual analysis targets (TARGET CONTENT CATEGORIES and TARGET SELECTION ATTRIBUTES).

SEMANTICS OF ASSESSMENT DATA

Assessment data can be approached through the following semantic categories.

ROW FILTER CATEGORIES

Use these categories only for row_filters:

• Unique identifier of the value chain.

  • Used when filtering by value chain number.

• Name of the value chain.

  • Used when filtering by value chain name.

TARGET CONTENT CATEGORIES

Use these categories only in target_selection_rules.data_category:

• chapter_score:

  • Digital maturity score.
  • Used for maturity calculations, score analysis, and average maturity analysis.

• Evaluators' statements describing strengths.