Explicit constructions of optimal-access MDS codes with nearly optimal sub-packetization (1605.08630v4)

Abstract: An $(n,k,l)$ MDS array code of length $n,$ dimension $k=n-r$ and sub-packetization $l$ is formed of $l\times n$ matrices over a finite field $F,$ with every column of the matrix stored on a separate node in a distributed storage system and viewed as a coordinate of the codeword. Repair of a failed node can be performed by accessing a set of $d\le n-1$ helper nodes. The code is said to have the optimal access property if the amount of data accessed at each of the helper nodes meets a lower bound on this quantity. For optimal-access MDS codes with $d=n-1,$ the sub-packetization $l$ satisfies the bound $l\ge r^{(k-1)/r}.$ In our previous work, for any $n$ and $r,$ we presented an explicit construction of optimal-access MDS codes with sub-packetization $l=r^{n-1}.$ In this paper we take up the question of reducing the sub-packetization value $l$ to make it approach the lower bound. We construct an explicit family of optimal-access codes with $l=r^{\lceil n/r\rceil},$ which differs from the optimal value by at most a factor of $r^2.$ These codes can be constructed over any finite field $F$ as long as $|F|\ge r\lceil n/r\rceil,$ and afford low-complexity encoding and decoding procedures. We also define a version of the repair problem that bridges the context of regenerating codes and codes with locality constraints (LRC codes), calling it group repair with optimal access. In this variation, we assume that the set of $n=sm$ nodes is partitioned into $m$ repair groups of size $s,$ and require that the amount of accessed data for repair is the smallest possible whenever the $d$ helper nodes include all the other $s-1$ nodes from the same group as the failed node. For this problem, we construct a family of codes with the group optimal access property. These codes can be constructed over any field $F$ of size $|F|\ge n,$ and also afford low-complexity encoding and decoding procedures.

• The paper presents explicit constructions for optimal-access MDS codes achieving near-optimal sub-packetization (l = r[n/r]) over any finite field, improving upon previous methods.
• It introduces a group repair concept for distributed storage, enabling optimal access during node recovery by using only nodes within the same group.
• The constructions lead to significantly smaller sub-packetization values, enhancing encoding/decoding efficiency and improving distributed systems like GFS and HDFS.

Optimal-Access MDS Codes with Nearly Optimal Sub-Packetization

In their paper, Ye and Barg focus on developing explicit constructions for Maximum Distance Separable (MDS) array codes that offer optimal data access during node recovery while aiming to minimize sub-packetization in distributed storage systems. MDS codes are integral in distributed systems as they ensure data recovery from node failures by storing parts of the codeword across multiple nodes. The efficient regeneration of failed nodes, which involves accessing and downloading minimal amounts of data from surviving nodes, is crucial for reducing system bottlenecks and optimizing storage utilization.

The authors present a construction for optimal-access MDS codes with a sub-packetization value that approaches the theoretical lower bound. They demonstrate that these codes can be explicitly constructed for any given number of storage nodes and recoverable nodes. The sub-packetization, denoted as $l = r[n/r]$, is at most a factor of two away from the optimal value, ensuring a significant improvement over previous explicit constructions. Importantly, these codes can be constructed over any finite field $|F| > r[n/r]$, making them versatile and adaptable to varying system specifications.

The paper introduces a "group repair" concept within distributed storage systems, which combines principles of regenerating codes and codes with locality constraints (LRC codes). Here, nodes are partitioned into repair groups. The authors establish a coding scheme that ensures minimal data access for node repair within these groups, termed as "group optimal access." The construction supports this property by enabling repair using only a subset of nodes from the same group, thus reducing I/O operations and network usage in node regeneration.

Among the robust numerical results, the paper presents the ability to construct MDS codes with optimal access properties using matrices of $l = r[n/r]$, which are significantly smaller sub-packetization values compared to earlier models, thereby reducing complexity and improving encoding and decoding efficiency.

The implications of this research extend to significantly enhancing distributed storage systems like the Google File System (GFS) and Hadoop Distributed File System (HDFS) by enabling efficient node recovery processes with minimized operations. The theoretical developments also pave the way for further optimization in the field of regenerating and locally repairable codes. Given the versatility of these codes over a finite field, future work could involve exploring improved constructions with even lower sub-packetizations or applications in more diverse system architectures.

The contributions of Ye and Barg strongly suggest a trend towards reducing computational overhead while maintaining robustness in distributed storage systems. As the demand for scalable and efficient storage systems grows, such constructions provide a foundational model for future research and development in data recovery protocols.