M5: Mastering page migration and memory management for CXL-based tiered memory systems

CXL has emerged as a promising memory interface that can cost-efficiently expand the capacity and bandwidth of a memory system, complementing the traditional DDR interface. However, CXL DRAM presents 2–3× longer access latency than DDR DRAM, creating a tiered-memory system that demands an effective and efficient page-migration solution. Although many page-migration solutions have been proposed for past tiered-memory systems, they have achieved limited success. In this work, first, we introduce a CXL-driven profiling solution that exploits the near-memory processing capability of an Intel CXL device to precisely and transparently count the number of accesses to every 4KB page and 64B word in CXL DRAM. Second, using the profiling solution, we uncover that (1) widely-used CPU-driven page-migration solutions often pick warm pages, and (2) certain applications have sparse hot pages, where only a small percentage of words in each of these pages are frequently accessed. Besides, we demonstrate that the performance cost of identifying hot pages is sometimes high enough to degrade the performance of applications. Lastly, we present M5, a platform designed to facilitate the development of CXL-driven page-migration solutions. M5 consists of the following two key components. (1) Hot-Page Tracker (HPT) and Hot-Word Tracker (HWT) track the top-K hot pages and words, respectively, in the CXL controller. HPT determines hotter pages than the CPU-driven hot-page migration solutions at practically no performance cost. HWT provides the ability to discern sparse hot pages, a capability lacking in CPU-driven page-migration solutions. (2) M5-manager offers software interfaces for users to synergistically use HPT and HWT with diverse policies that can identify more useful hot pages. On average, M5 can identify 31% hotter pages and offer 17% higher performance than the best CPU-driven page-migration solution, even with a simple policy.

paper

slides