VMware Private AI Foundation: we bring the AI to your data

VMware Private AI Foundation with NVIDIA (PAIF-N) lets companies run AI workloads securely and efficiently on their own infrastructure.

VMware Private AI Foundation with NVIDIA (PAIF-N) lets companies run AI workloads securely and efficiently on their own infrastructure. The platform combines the strengths of VMware Cloud Foundation (VCF) with NVIDIA’s GPU technology and software ecosystem. PAIF is an add-on for VCF that makes RAG and generative AI services possible on-premises.

With VMware Private AI Foundation, all the data is processed on-premises. The point of PAIF is to give internal data science and MLOps engineers the infrastructure and the tooling they need to build new services on top of pre-trained LLMs. Provisioning that infrastructure, VMs, vector databases and Kubernetes clusters, happens automatically through a self-service portal.

The challenges

Governance around using AI tools such as ChatGPT, Microsoft Copilot and Perplexity in a company raises several problems:
- Unintended data leakage: people may upload internal company information into public AI systems by accident.
- Data protection risk: using external AI services risks confidential company data ending up in those systems’ training data.
- Access: restricting access to AI-generated content is difficult, because the AI can potentially reach all of its training data.
Hardware utilisation and management in on-premises AI solutions
- High capital cost: buying capable hardware, GPUs in particular, takes considerable money.
- Inefficient use of resources: without proper virtualisation and monitoring, the hardware ends up underused. Expensive GPU resources sit idle.
Complexity
- Technical challenges: AI software is often built on cloud-native technology with a large stack behind it.

VMware Private AI Foundation addresses those as follows:

Governance:
- On-premises processing: all the data is processed locally, which improves control and security.
- Secure data storage: vector databases such as PostgreSQL with pgvector, which support role-based access control (RBAC).
- LLM validation: LLMs can be validated, checked and stored centrally in the Harbor container registry. The Giskard framework allows further checks.
- Granular access control: PAIF implements access controls, so end users only reach the data they are entitled to.
Resource monitoring:
- vGPU slicing
- GPU monitoring
- Preemptable VMs: non-critical workloads can be tagged so they do not claim resources during HA events (test workloads).
Complexity:
- An integrated stack: VMware Cloud Foundation provides a complete virtualisation and container orchestration stack.
- Automated provisioning through Aria Automation
- Monitoring of the whole stack with Aria Operations
- DBaaS through Data Services Manager

VMware Private AI in detail

VMware Private AI Foundation (PAIF) builds on VMware Cloud Foundation (VCF) and is available as an add-on. Implementing PAIF requires the following.

Prerequisites

vSAN hosts (ReadyNodes or VCG-compatible hardware)
VMware Cloud Foundation licences
A VMware Private AI Foundation and NVIDIA add-on licence
An NVIDIA NGC API key for access to the nvcr.io registry
NVIDIA L40s, H100 or A100 cards (PCIe, SXM or NVL)

Building blocks

VMware vSphere
VMware vSAN
VMware NSX
VMware IaaS Control Plane and vSphere Kubernetes Services (VKS), formerly vSphere with Tanzu
VMware Data Services Manager
VMware Aria Automation, for provisioning AI Kubernetes clusters or AI VMs (deep learning VMs) automatically

What the PAIF add-on brings

AI workstations: deep learning VMs packaged by VMware with all the NVIDIA drivers and tools, Docker, PyTorch and TensorFlow
AI Kubernetes clusters: vSphere Kubernetes clusters with the NVIDIA GPU operator
Self-service through Aria Automation
An Aria Operations dashboard for monitoring the GPUs and their utilisation

GPU resources

GPUs are the foundation of AI services, and VMware vSphere makes efficient use of them through several technologies:

GPU slicing (vGPU): NVIDIA vGPU technology partitions a single physical GPU into several virtual GPUs, which different virtual machines then use. That happens either by time-slicing or through Multi-Instance GPU (MIG). Time-slicing shares the GPU’s capacity between the VMs; MIG partitions in hardware, so each VM gets isolated resources.
GPU reservations: VMware vSphere lets you reserve GPU resources for particular VMs, so critical applications always have the capacity they need.
GPU monitoring: through vSphere and Aria Operations, administrators watch GPU usage and performance, including metrics such as GPU utilisation and memory.

Which GPU to choose depends heavily on the use case and the LLM in play:

GPU sizing and performance guide

Deep learning VMs

Deep learning VMs are Ubuntu VMs packaged by VMware that serve developers as AI workstations. AI workstations are for the development process; once an application is ready for production, Kubernetes clusters take over. Deep learning VMs include:

PyTorch
TensorFlow
CUDA samples
DCGM exporter
Triton Inference Server
NVIDIA drivers
Docker

LLMs

Training an LLM from nothing simply takes too many resources, but fine tuning works perfectly well on a PAIF platform.

PAIF puts no fundamental limits on which LLMs can be used. Common ones are:

Llama 3
Mistral
Falcon

With PAIF you distribute ML models across deep learning VMs and Kubernetes clusters through a central Harbor container registry.

Central storage of validated LLMs
Validating the LLMs, with Giskard for instance
Harbor’s RBAC restricts access to sensitive data

RAG and vector databases

Vector databases solve problems that LLMs bring with them:

Accuracy: LLMs sometimes hallucinate and produce wrong or nonsensical results.
Currency: an LLM is only ever as current as its last training run, and retraining it for every new piece of data takes too many resources.
Traceability: LLMs often cannot reference or cite the sources of their answers, which raises questions about how reliable the information is.

This is where retrieval-augmented generation (RAG) comes in. RAG combines the intelligence of an LLM with the company’s own data, held in a vector database. That way the AI draws not only on its extensive training but on specific, current and reliable information from the company’s own data.

Data Services Manager

VMware Data Services Manager provisions vector databases for an AI use case simply. DSM can provision MySQL and PostgreSQL services automatically, and with pgvector a DSM-provisioned PostgreSQL can serve as a vector database.

Upgrades of MySQL and PostgreSQL are driven through DSM too, and a backup solution is built in that writes the backups to object storage (S3).

Self-service through Aria Automation

In the end, ML ops engineers and developers need a simple way to consume this infrastructure. Aria Automation provides a self-service portal where a developer orders the resources and gets them provisioned automatically. The self-service catalogue items for PAIF are, by default:

AI workstation (provisioning a deep learning VM)
AI Kubernetes clusters: vSphere Kubernetes clusters with the NVIDIA GPU operator
PostgreSQL pgvector database

A use case

For an example of an AI service, this post is worth reading: Summarize and chat service with VMware Private AI

It shows how VMware Private AI can be used to build a service that summarises data such as PDF, DOCX and PPTX files as well as audio and video, and lets you query the information in them through a chat.

The source code is on GitHub.

Conclusion

With VMware Private AI, the AI comes to your data rather than the other way round. PAIF lets you train existing models on internal data, on SharePoint or file shares for instance, to build something like an internal chatbot service.