Building an AI Infrastructure: Five Keys for Universities

Infrastructure Strategies for AI in Higher Education

Higher education institutions are seeking to maximize the impact of artificial intelligence while ensuring strict governance, enhanced security, and operational efficiency. Here are five essential considerations for building a robust AI infrastructure.

Choosing the AI Consumption Model

Universities must choose between AI models hosted by providers and internal hosting. Hosted models are ideal when speed, elasticity, and compliance are crucial. In contrast, internal hosting is preferable for sensitive data, disconnected operations, or custom models. A hybrid approach is often adopted, requiring precise mapping of use cases based on data classification, latency, as well as output and completion timelines. It is also important to budget for pilots, training, and ongoing inference costs.

Preparing for Large-Scale AI Orchestration

To orchestrate AI at scale, it is crucial to have a comprehensive inventory of containers, continuous integration and continuous delivery, as well as observability. Multi-tenant isolation, GPU scheduling, secret management, and software invoice/workflow updates are key elements to validate. DevSecOps standards, trusted Internet connections 3.0, and zero-trust security plans must be aligned. In cases of low maturity, managed orchestration can be a good starting point.

Facility Capacity to Support AI

Facilities must be assessed for their capacity to handle AI workloads, particularly in terms of power density, cooling, and space for GPUs. It is also essential to examine the capacity of UPS systems and generators. Network throughput to systems and cloud exchanges, as well as physical security, are critical factors, along with supply chain delivery timelines and maintenance windows. If constraints exist, colocation or utilizing GPU capacity from providers may be considered, while modernizing the data center infrastructure.

Governance of AI Workloads

Governance of AI workloads should be based on the risk management framework from the National Institute of Standards and Technology. This includes human oversight, privacy, model traceability, dataset provenance, and model cards. It is essential to integrate these approvals into IT governance processes and campus security reviews. Monitoring for drift and bias, proper logging of prompts and outputs, as well as establishing rollback procedures are crucial. Procurement and vendor contract clauses must cover intellectual property, security, and incident response.

Expanding AI Without Overprovisioning

To effectively expand AI, it is advisable to start with a small, high-value use case. Resources should be adjusted based on actual usage, not peak estimates. Tracking re-billing costs and total cost of ownership is necessary to optimize training and inference. Expansion should be iterative, reusing models and pipelines across campus departments. It is also important to eliminate underutilized resources and continuously reassess build versus buy as offerings mature.