Introduction
Storage architectures are limiting our potential to gain the most from the Era of AI. Storage systems haven’t fundamentally changed for decades in their philosophy. Whereas data is increasingly distributed, shared, defined and utilized by metadata, highly voluminous and increasingly precious, storage architectures are still localized and HW defined, clumsy in their approach to multi-tenancy, oriented around data (not metadata), limited in their scale and with security as an afterthought.
Today’s data management challenges require an approach which takes this new paradigm and places these core characteristics as the starting point for a design: with ever growing pools of data assets, new performance requirements for AI and analytics and the demand for multi-cloud integration and edge generated data, the traditional hardware-centric approach to data management systems no longer makes sense. Flexibility must be the central tenet of any new approach, moving to a software-defined architecture for scalability, and take advantage of, rather than relying upon, hardware properties.
By focusing on a software-defined storage control plane, we can combine the flexible consumption models of cloud storage, with the cost and performance benefits of on-premise deployments AND protect all the data with a centralized data management regime. The storage control plane must meet the data where it needs to live, take advantage of any cost-effective or performance boosting hardware available, and spin-up capabilities on-demand.
Storage administrators can turn the current paradigm on its head. Instead of tuning their business and workflows to fit the capabilities of their storage and the data management environment, they can take a business-centric approach to data and leverage optimized storage capabilities whenever and wherever they are needed.
Welcome to the Future
SOFTWARE-DEFINED UNIFIED STORAGE
Until now, organizations have had to make a choice between cloud simplicity and on prem economics, and between hardware performance and software-defined flexibility. Existing storage architectures persist these trade-offs that force anti-business decisions and unnecessary cost and complexity while encouraging cumbersome data silos.
DDN Infinia gives fluid control of data from edge to core and core to cloud, while maximizing the capabilities of the underlying infrastructure, including storage, network and compute.
NO COMPROMISE MULTI-TENANCY AND QoS
Infrastructure sharing is a fundamental to cost optimization and efficiency – but corporate networks are subject to ever more creative security threats looking to steal data or compromise operations.
These two conflicting drivers lead organizations to implement scheduling tools to avoid resource conflicts, and byzantine security architectures to protect against possible cyber-attacks, both of which can compromise the performance and resilience of critical systems. Today’s storage architectures have retrofitted coarse data isolation with performance ceilings by separating out hardware resources in an attempt to help, but leading in confusion and inefficiency.
DDN Infinia has native multi-tenancy that securely isolates user data, and applies efficient quality of service algorithms across all tenants and subtenants entirely in software and ensuring data is protected in-flight, at rest and while shared between organizations.
WORKLOAD OPTIMIZED PERFORMANCE
Every workload is different – and while one type of storage system works well for small random I/O, another is highly effective for throughput-oriented workloads, and yet another is designed for low-cost long-term retention of data. These different attributes inevitably lead to data silos and management complexity.
DDN Infinia is different, ensuring all workloads are served equally well. This is done by managing the behaviour of each workload automatically and optimizing each depending upon their nature. DDN Infinia deploys multiple I/O Engines that move and place the data within the storage environment in the best way via DDN’s novel credit-based system I/O and QOS engine algorithm.
This means that DDN Infinia can cater to a range of requirements simultaneously, such as low latency requirements for smaller files, and high throughput for large data objects. No need for specialized hardware such as memory-class storage, nor to perform any specific application tuning.
SERVICE DELIVERY WITH AIOps
Hardware-centric IT infrastructure approaches can create complexity that is difficult to manage. Limited resources and changing priorities lead to longer IT service delivery times, making this approach unsustainable as infrastructure needs to become more distributed.
DDN Infinia eliminates this complexity, by making the services delivery framework a core part of the design. A central, agile management control plane makes it easy to oversee micro data centers, edge deployments and multi-cloud hybrid strategies.
A rich set of APIs and a wide set of ecosystem partnerships means simple integration into existing IT toolkits for service orchestration, management, audit, analytics and monitoring. Native AIOps intelligently roots out problems and maintains your service levels. DDN Infinia instantly adapts to strategic business priorities, no specialized expertise required.
Introducing DDN Infinia
DDN Infinia brings a new approach to data management, with native multi-tenancy, cross-protocol support, and cloud flexibility to solve the challenges that come with managing data in today’s world. Distributed data, distributed computing, diverse performance requirements and the sheer volume and complexity of unstructured data means the end to the traditional hardware-centric approach to storage design.
Created to simplify data management for the modern enterprise, DDN Infinia is easy to deploy, manage and scale from edge to datacenter to cloud. The Dynamic Storage Engine and core data services have been designed for performance and multi-tenancy support by DDN software engineering over the last 4 years and will be ready for deployments in July 2023.
Initial deployments will be concentrated on large Enterprise customers, where ease of installation, ease of management, and ease of cluster growth to support capacity and performance are key priorities. From the first release, DDN Infinia will support implementation of 100s of petabytes, with the ability for online expansion, non-disruptive upgrades, and a wide erasure coding protection system.
DDN Infinia is ideal for enterprise corporate file stores and archives, media and entertainment deployments, medical PACs (Picture Archive and Communication System), biotech research data storage and archives, and IT technology and service companies.
HPC and AI customers will also benefit from DDN Infinia’s scalable multi-tenant S3 object store and block driver support for Kubernetes and Open stack environments. Whether to back up HPC scratch space, store result files or long-term storage of experiment data, DDN Infinia is especially suitable for customers with strong data sharing requirements in complex multi-departmental environments.
A New Data Management Paradigm
DDN Infinia is a modern multi-tenant data management platform for S3 objects, along with Kubernetes and Open Stack data block management. As a next-generation data services platform that has been designed for multi-cloud deployment, DDN Infinia allows for more fluid control of data across enterprise organizational boundaries. DDN Infinia is built to overcome many critical challenges of traditional data management solutions and take advantage of new technology opportunities in storage and data management, while retaining the economic benefits and flexibility of future multi-cloud architectures.
With innovations in solid-state flash memory, and scale-up and out GPU architectures and ultra-fast networking, organizations and research institutions are starting to realise the potential and value of data in AI and Deep Learning applications. In order to achieve that goal, DDN Infinia breaks through these traditional barriers:
- Designed and architected to scale seamlessly to support the data management challenges that will be needed for the future distributed, high volume, high value data centric world;
- The world’s first multitenanted QoS that doesn’t introduce barriers to infrastructure efficiency using trustworthy, SW-powered SLAs;
- Security built in at every level, to protect data at rest, in flight and secure data sharing between collaborating groups and organizations;
- Integrated AIOps with autonomous storage management, to allow organizations to focus on value-oriented data services, and less on storage management business itself;
DDN INFINIA ARCHITECTURE
Unified Platform for Enterprise Shared Data
DDN Infinia is 100% software defined, with no dependencies on underlying hardware, yet able to take advantage of performance-optimized hardware components where available. There is no reliance on specific Flash media types or underlying platform resilience to protect data against power loss or hardware failure. DDN Infinia can run on fast appliances, or in containers, virtual machines, whether in the cloud, in a data center. DDN Infinia will be launched initially on DDN certified hardware.
NATIVE MULTITENANCY
Current storage systems implement Multi-Tenancy and Quality of Service through dedicating virtual servers, drives and/or networks to individual tenants. This is a poor approach because:
- It is not scalable
- It does not enable resources sharing, instead it enforces limits through physical limits
- It limits flexibility – what if two groups merge or want to share data. How easy is it to expand or shrink?
- There is no concept of subtenants – organizations want to enable their departments to manage their own allocation, without subtenants, that doesn’t work.
- It is fundamentally inefficient, allocating hardware instead of logically partitioning software.
DDN Infinia supports native multitenancy. All DDN Infinia data objects are all tagged with the tenant ID, and used throughout the platform to implement tenant isolation and to manage performance levels. DDN Infinia credit-based flow control model automatically separates your IOPs intensive mission critical workloads from your applications that might be disruptive.
This means that running DDN Infinia is like running your own simple cloud data environment: when a new group or department needs resources, just dial in performance and capacity allocation and DDN Infinia will do the rest, allocating the resources for the new tenant and subtenants, and provisioning individualized space for storing user projects and data.
Each user can only access their stored information, and each tenant’s data is inaccessible to all other tenants, and can only be reached with the cloud provider’s permission.
PERFORMANCE
DDN’s leadership in parallel filesystems is unmatched in terms of IOPs and throughput, and DDN Infinia leverages that expertise to ensure that all workload demands are well served. This is done by managing the end-to-end data path to optimize different application workloads, depending on their behavior.
DDN Infinia deploys multiple I/O Engines that receive and place the data within the storage environment in the optimal way, managing erasure coding and data replication along with optimized data structures to cater both to low latency needs and high throughput requirements simultaneously, without having to use special hardware (such as Optane) or doing any specific tuning.
SIMPLICITY
DDN Infinia brings a new level of simplicity to data management at scale. Many of the standard steps and complexities in managing storage systems are eliminated or simplified. On initial deployment, DDN Infinia leverages self-discovery and self-test to detect and recommend how to use the underlying components, so that building a secure storage cluster is a 10-minute operation. Similarly, system expansion, removing components, or performing upgrades, everything is managed with simplicity and without downtime. The full containerization of DDN Infinia’s storage services allows upgrades to happen in seconds.
Conclusion
Existing storage solutions fail to address the technical challenges of distributed data management for several reasons:
- Data Silos: Existing solutions may not have the capability to effectively integrate data from multiple sources and locations, leading to data silos that impede data access and analysis.
- Data Security: As data becomes more distributed, the risk of data breaches and security breaches increases.
- Data Governance: The governance of distributed data can be complex and time-consuming, requiring active data quality, accuracy, and compliance.
- Scalability: As data volume and variety increases, existing storage solutions may struggle to scale up to meet the demand.
- Latency: Distributed data can lead to increased data latency as data has to be pulled from different sources which can affect the performance of AI models that depend on real-time data.
DDN Infinia is ideal for customers who need the flexibility of managing their own object store, whether in micro data centers, edge deployments and across multi-cloud hybrid strategies.
