AMD is pitching a piece of software called Mext that lets Epyc servers use NVMe flash in place of part of their DRAM. An AI model predicts which memory pages are about to be needed and moves them back into memory before the operating system or the application notices. AMD says that with half the DRAM replaced by flash, some workloads still retain up to 99% of their performance, and enterprise server costs can fall by as much as 83%.
Mext started out as an independent company; AMD acquired the technology in June this year. Gary Smerdon, Mext's former CEO and now AMD's vice president of business development, told The Register that the software is already deployed on the first 145 systems, and by his estimate customers have so far avoided roughly $5 million in DRAM purchases.
Why nobody dares turn swap on
Much of the memory a server reports as “in use” is not actually being used. Citing a Meta study, Smerdon said 50% to 80% of memory pages go unaccessed for five minutes, ten minutes or longer. In his words: “Five minutes is an eternity for an Epyc.”
Moving cold pages to a cheaper medium is the same idea as operating-system swap or the Windows page file. The catch is speed. By Smerdon's numbers, flash costs about 50 times less per bit than DRAM and draws about 30 times less power, but is about 500 times slower. Traditional swap fetches a page from disk only when a program asks for it, so latency spikes and service-level agreements are breached. Operators therefore disable it wherever they can.
Mext brings pages back ahead of demand. “We transparently make flash look like DRAM,” Smerdon summarized. In his example, a server that would normally be fitted with 2TB of memory can now be bought with 1TB, with Mext covering the other 1TB using flash.
A prediction model trained on the CPU, on the fly
The main prediction model is a Transformer derivative. Smerdon's analogy: a large language model predicts the next word, while this one predicts the next memory page that will be used. As long as a page is back in DRAM before it is accessed, the order can be a little off.
Memory-access patterns differ from one workload to the next, so pre-training a general-purpose model is not practical. Instead, Mext trains an initial model on a single Epyc core in under one second, then keeps retraining as the workload runs until the predictions are accurate enough. The whole process runs on the CPU alone, with no GPU required, so in theory any Epyc customer can use it.
That is also where the overhead sits. The Register notes that in virtualized environments the software occupies one core per virtual machine, so the more VMs there are, the more cores it consumes. Whether the same applies to containers is something AMD has not yet answered the publication on.
Performance depends on the workload
Smerdon says moving cold pages out has “no impact on performance at all,” while the figures AMD itself measured are more conservative:
- Synopsys PrimeTime (an EDA tool used in chip design): with half the memory replaced by flash, performance is 95% of an all-DRAM system; with only 25% DRAM and 75% flash, it drops to 80%.
- Apache Spark: with half the DRAM replaced by flash, performance stays at 99%.
EDA software is extremely memory-hungry, but at any given moment the data actually being processed by the compute cores is often only a few MB, which is why Mext was first tried on it. Support later widened to latency-sensitive scenarios such as databases and agent inference pipelines. No figures for the latter have been published this time, and no third-party retests have appeared yet.
A back-of-the-envelope look at whole-server cost
Smerdon says memory conservatively accounts for 60% to 70% of the cost of a large-memory server, and 80% to 90% for very-large-memory machines. Using his 2TB example as a rough calculation: assuming memory is 70% of total system cost, cutting half the DRAM and replacing it with flash that costs about one-fiftieth as much per unit, and leaving out the Mext license fee, brings the total cost down by a little over 30%. The 83% ceiling presumably corresponds to configurations where memory makes up a larger share and more of it is swapped out; AMD has not published how it arrives at the figure.
Licensing is currently a blank. Mext runs only on AMD hardware and will be sold under license alongside Epyc servers through OEMs such as HPE; pricing has not been announced. On Linux, only a kernel driver needs to be installed. AMD also offers a tool called RAM View to help enterprises work out which machines and workloads are suited to switching Mext on.
The AI data-center build-out keeps memory supply tight, and AMD is betting that customers will trade a few percentage points of performance for a large cut in their memory budget. By its plan, the company is in phase one, working with OEMs to make machines “Mext ready.” Phase two is when licenses go on sale and RAM View comes preinstalled on Epyc machines shipped from the factory.
Sources: The Register interview with Gary Smerdon, CocoLoop, Meta memory-access research; the performance ratios, the 145-system deployment count and the 83% cost reduction are all AMD's figures.