Data Services for AI: Storage, Objects, and Databases
Data Services for AI: Storage, Objects, and Databases
AI pipelines have different data paths: shared training sets, high-throughput checkpoints, S3 data lakes, and transactional metadata. NCP-AI tests choosing the right Nutanix data service and protecting it.
Core services
- Nutanix Files: NFS/SMB shared file storage for datasets, home directories, and legacy ML tools.
- Nutanix Objects: S3-compatible object storage for data lakes, model artifacts, checkpoints, and backups.
- Nutanix Volumes: block storage for VM-based databases and high-IOPS workloads.
- Nutanix Database Service (NDB): database lifecycle automation-provisioning, cloning, patching, backup-for PostgreSQL, MySQL, SQL Server, Oracle, and similar engines.
- Nutanix Data Services for Kubernetes (NDK): CSI-based persistent volumes and snapshots for NKP workloads.
Match protocol to consumer: Kubernetes uses CSI/NDK, S3-native tools use Objects, and NFS is common for shared read-heavy datasets. Protect with snapshots and replication, and test restore rather than only backup.
Worked example
A training job reads a 20 TB image dataset and writes checkpoints every 10 minutes. Put the dataset in Files for shared parallel reads or Objects if the framework is S3-native; write checkpoints to Objects for durability and versioning; keep experiment metadata in NDB-managed PostgreSQL. The reasoning separates high-volume immutable data from transactional metadata.
Traps
- Assuming Objects implements every S3 API extension.
- Using NDB as a general file store for model weights.
- Ignoring NDK storage classes in Kubernetes PVCs.
- Designing backups without restore testing.