LAUSR.org creates dashboard-style pages of related content for over 1.5 million academic articles. Sign Up to like articles & get recommendations!

On Implementation of Parallel Map and Shuffle Phases for Coded Distributed Computing

Photo by britishlibrary from unsplash

In this letter, we consider the MapReduce-type distributed system and jointly design the Map and Shuffle phases of the coded distributed computing (CDC) scheme proposed by Li, et al. By… Click to show full abstract

In this letter, we consider the MapReduce-type distributed system and jointly design the Map and Shuffle phases of the coded distributed computing (CDC) scheme proposed by Li, et al. By delicately setting the order of file mapping and data shuffling, we implement the Map and Shuffle phases in parallel, and thus significantly reduce the overall task latency (time cost at the Map, Shuffle, Reduce operations and codes generation). Furthermore, our scheme enables all nodes to remove their auxiliary intermediate values once these values are no longer useful in the Shuffle phase, leading to great reduction in the storage cost of the system.

Keywords: shuffle phases; map shuffle; distributed computing; coded distributed; phases coded

Journal Title: IEEE Communications Letters
Year Published: 2022

Link to full text (if available)


Share on Social Media:                               Sign Up to like & get
recommendations!

Related content

More Information              News              Social Media              Video              Recommended



                Click one of the above tabs to view related content.