Memory Hierarchy

The CPU runs tens to hundreds of times faster than main memory, and that gap — not the processor itself — is the dominant limit on how fast a program runs. The standard response is not to build equally fast memory, which would be prohibitively expensive and small, but to build a hierarchy: fast and expensive memory close to the CPU, cheap and large memory further away, with hardware moving data between the levels automatically so that the programmer never has to say where anything lives.

This chapter builds that hierarchy from the bottom. Memory is classified by the properties that actually matter for design, the layers are laid out and the resulting speed–capacity–cost trade-off is made explicit, and the semiconductor devices that make up each layer are introduced before the question of how the CPU is wired to main memory is answered.

The middle section is devoted to Cache, where the interesting design choices live: how a block maps into a cache line (direct, fully associative or set associative), what happens when the set is full and a replacement is needed, and how writes are handled — write-through or write-back, and who is responsible for keeping the two levels consistent. Multi-level caches are then treated as the natural extension of the same idea.

Virtual memory completes the picture. The chapter explains how a process can use addresses far larger than physical memory, compares paging, segmentation and the segment-page scheme, and introduces the TLB as the hardware cache that keeps page table lookups from dominating every memory access. Read across the chapter, the theme is constant: performance comes from predicting correctly and paying as little as possible when the prediction fails.