AI in the Public Sector: Small Models, Big Challenges

The Rise of AI in the Public Sector: Adoption Under Constraints

Artificial intelligence (AI) is transforming various sectors, and public institutions are no exception. However, unlike the private sector, government organizations must navigate a complex landscape of security, governance, and operational constraints. These unique challenges require tailored solutions, and this is where small language models (SLMs) come into play. These models, adapted to the specific needs of agencies, offer a promising pathway for integrating AI into public environments.

A study conducted by Capgemini reveals that 79% of global public sector leaders express concerns about data security related to AI. This mistrust is understandable, given the sensitive nature of government data and the legal obligations surrounding it. Han Xiao, Vice President of AI at Elastic, emphasizes that agencies must be extremely cautious with the data they share over the network, which imposes significant limits on their information management.

The need to control sensitive information is one of many obstacles complicating the deployment of AI in the public sector, compared to the more flexible practices of the private sector.

Specific Operational Challenges in the Public Sector

In the private sector, AI development often relies on certain assumptions, such as continuous cloud connectivity, centralized infrastructure, limited model transparency, and minimal restrictions on data movement. For public institutions, these conditions are not always feasible. Government agencies must ensure that their data remains under control, that information is verifiable, and that operational disruptions are minimized.

Moreover, these institutions often have to operate in environments where internet connectivity is limited or non-existent. These complexities prevent many promising pilot projects from moving beyond the experimental stage. Han Xiao points out that many underestimate the operational challenges of AI in the public sector, where continuity of operations is crucial. An Elastic survey indicates that 65% of public sector leaders face difficulties in using data continuously and at scale.

Infrastructure constraints add an additional layer of complexity. Government organizations often struggle to obtain the graphics processing units (GPUs) necessary to train and run complex AI models. Unlike the private sector, the government does not frequently acquire GPUs, which constitutes a significant bottleneck.

The Advantage of Small Language Models

The strict requirements of the public sector render large language models (LLMs) unsuitable. In contrast, SLMs, which can be hosted locally, offer enhanced security and control. These specialized models utilize billions of parameters, rather than hundreds of billions, making them less resource-intensive than LLMs.

The public sector does not need to develop increasingly larger models hosted in centralized data centers. Empirical studies show that SLMs can match or even surpass LLMs in performance. They allow for efficient use of sensitive information while avoiding the operational complexity of large models. Han Xiao explains that while it is easy to use tools like ChatGPT for simple tasks, getting one's own large language models to function in a network-less environment is much more complex.

SLMs are designed to meet the specific needs of the departments or agencies that use them. Data is securely stored outside the model and is only accessible when queried. Carefully crafted prompts ensure that only the most relevant information is retrieved, providing accurate responses. By using methods like intelligent retrieval and source verification, AI systems can be tailored to the needs of the public sector.

Thus, the next step in AI adoption in the public sector may involve bringing the AI tool to the data, rather than transferring data to the cloud. Gartner predicts that by 2027, specialized small AI models will be used three times more than LLMs.

Enhancing Research Capabilities Through AI

When discussing AI in the public sector, many think of tools like ChatGPT. However, AI offers much broader opportunities, particularly in enhancing research capabilities. The public sector holds vast amounts of unstructured data, such as technical reports, procurement documents, meeting minutes, and invoices. Modern AI can provide results from mixed media, such as PDFs, scans, images, spreadsheets, and recordings, in multiple languages.

This data can be indexed by systems powered by SLMs to provide tailored responses and draft complex texts in any language while adhering to legal requirements. Han Xiao emphasizes that the public sector has a wealth of data but does not always know how to fully leverage it.

AI can also assist government employees in interpreting available data. A well-trained SLM can interpret legal standards, extract information from public consultations, support data-driven executive decision-making, and improve public access to administrative services. This can lead to significant improvements in how the public sector conducts its operations.

The Promise of Small Language Models for the Future

Focusing on SLMs shifts the discussion from model completeness to efficiency. LLMs incur high performance and computational costs and require specialized hardware that many public entities cannot afford. While they do require some capital expenditure, SLMs are less resource-intensive than LLMs, making them generally more affordable and reducing their environmental impact.

Public sector agencies are often subject to strict audit requirements, and SLM algorithms can be documented and certified as transparent. Some countries, particularly in Europe, also have privacy regulations such as GDPR, which SLMs can be designed to comply with.

Custom training data produces more targeted results, reducing errors, biases, and hallucinations to which AI is prone. Han Xiao explains that large language models generate text based on their training, which can lead to errors if the information is outdated. By forcing the model to work from verified sources, this issue can be mitigated.

Risks are also minimized by keeping data on local servers or even on a specific device. This is not about isolating data but ensuring strategic autonomy to enhance trust, resilience, and relevance.

By prioritizing task-specific models designed for environments that process data locally, and continuously monitoring performance and impact, public sector organizations can build sustainable AI capabilities that support concrete decision-making. Han Xiao advises starting not with a chatbot, but with research, as much of AI's intelligence lies in the ability to find the right information.