Accelerating AI Pipelines with Hammerspace

In this session, Floyd Christofferson and Chad Smith from Hammerspace will look at solutions to achieve HPC-class performance to feed GPU-based AI pipelines while leveraging data in place on existing storage resources. This session will give real-world examples of how customers have adapted their existing infrastructure to accommodate the performance levels needed for AI and other high-performance workflows.

Christofferson and Smith discuss how Hammerspace can accelerate AI pipelines by addressing the challenges of managing and accessing unstructured data across various storage systems and locations. They introduce the concept of a global data environment that leverages a parallel global file system, allowing data to remain in place while providing high-performance access necessary for AI workloads. They begin by explaining the silo problem in AI pipelines, where unstructured data is spread across multiple storage types and locations, making it difficult to aggregate without moving it to a new repository. Hammerspace’s solution allows for the assimilation of file system metadata from existing storage, enabling a global view and access to data without physically moving it. This approach prevents copy sprawl, maintains data governance, and avoids additional capital and operational expenses.

The session highlights the introduction of a new product, Hammerspace Hyperscale NAS, which provides HPC-class parallel file system performance using standard protocols and networking, without requiring proprietary clients or altering existing infrastructure. This solution is said to be storage agnostic and can accelerate existing third-party storage, making it suitable for enterprises looking to incorporate AI workflows without significant upfront investment. The duo provides real-world examples, including a hyperscaler with a large AI training and inferencing environment, where Hammerspace’s technology enabled scalability without altering the existing infrastructure. Another example is a visual effects customer who achieved the required performance for rendering without changing their storage infrastructure.