#124 What is a Cache?

Caches exist throughout computing systems. They're not something we strictly need. They're something that speeds up their operation. In this episode, we'll explain how they do that's. Welcome to COPEC Explain Software, the podcast where we make computing intelligible. In this episode, we're going to be discussing caches. Not C-A-S-H-C-A-C-H-E. Why don't we start with just a definition? Sure. If I had to formally define what a cache is, I'd call it a piece of temporary, typically fast storage used for the retrieval of expensive calculations or high latency resources. So we're talking about little bits of memory throughout a system that are used to make the system operate faster.

What do you mean by an expensive calculation or a high latency resource?

Yeah, an expensive calculation might be something that requires a lot of CPU instructions or operating across a lot of different portions of memory just to get one small result back. That calculation is going to be very CPU intensive, and it's going to take a lot of time, and maybe we don't want to spend all that time every time we need the result of the calculation. So it'd be nice if we already had that result available to us in some small bit of memory.

And what about a high latency resource?

High latency resource might be something that it takes a while to get. It's not something we're actually calculating. It's just something that we're retrieving. Maybe it's a file that it's going to be hard to find. Maybe it's something that's across the network that's going to take a while when we ask for it for it to come back to us.

And caches can exist on hardware or in software.

Right. Which do you want to talk about first?

Let's start with hardware.

When people hear hardware cache, the first thing they typically think about is a CPU cache. CPUs actually have multiple layers of memory. There's not just the main memory that the CPU accesses right on the chip. A CPU is going to have several layers of cache in addition to, of course, the registers. The registers are where the actual arithmetic takes place. But then there will be caches that have values from main memory retrieved within them so that we don't have to make the long jump all the way to main memory. Going into one of those caches is going to be a lot faster than making the trip all the way to main memory and back. The cache is memory that's already there on the CPU. Instead of all that latency of going to main memory.

Can you give us an example of how the CPU cache might be used?

All right, let's say we're doing a calculation that involves a lot of memory, but some of it is focused on just a small subset of that memory. If that memory can fit within the confines of the cache, then perhaps we can do that calculation a lot faster because we don't have to make all those trips back and forth in the main memory. Let me give you an example. Ram on a typical laptop today might be 16GB. The CPU cache might be 16 megabytes. If we're doing a calculation across a large file, maybe the file is a gigabyte, but we're repeatedly coming to just maybe, let's say, five megabytes of that file. There's five megabytes of that file that we're doing some more intense calculations on. Potentially all five of those megabytes might fit into the CPU cache. So the first time that we want to access those values, we do have to go to main memory because we have to get them and then load them into the CPU cache. But then the second, 3rd, 50th time that we're doing continual calculations on that five megabytes, it's already there on the CPU, so we don't have to make that big trip. So that's what a CPU cache enables. It enables fewer trips to main memory, and it enables that data that we're calculating with to be right there, close to where the instructions are actually operating, so they can quickly be shoved in and out of registers.

Another commonplace for a cache might be a hard drive.

Yeah, and it's really the same principle. To really illustrate, let's talk about the old style spinning rust hard drives that hopefully nobody has anymore. And you all have solid state drives today, but those old spinning disk hard drives, it could take quite a while to actually seek and find a particular file that you wanted to manipulate. Those hard drives might hold terabytes of data, but you just want that one 1 MB file that you want to get to. Well, they would typically have some random access memory, some Ram on the hard drive for recently accessed files. So the first time you access the file yeah, the first time you open it, you have to go through those terabytes of spinning disk memory to find it. But the second time, it's already there in that little bit of cache, that little Ram on the hard drive so that it can get found a lot more quickly. And then it's likely that you're going to keep working on the same file, right? When you open a file, you're going to keep working on it continuously. And so it's already there in that faster memory. So we're going to have a lot faster writes to it, reads to it, and then occasionally the hard drive can take the change data in the cache and put it back out into the spinning disk portion.

All right, let's dive into software caches with some specific examples. Why don't you start with the Web browser?

Sure. Most people are familiar that their Web browser has a cache. So the sites you've recently been to are going to have some JavaScript, some HTML. They're going to have some images. Maybe they even have some sounds or some custom font files. And your web browser doesn't just show them to you the second you're looking at the site, it actually downloads them and stores them on your computer so that if you go back to that site a little bit later, it won't have to go across the network and get all of them again. These caches can really build up. You can have gigabytes of data stored in these caches, old websites you've been to, all the resources associated with them. The reason is that it speeds up going back to them. If you're a web developer, this can actually be a problem. Sometimes it's happened to me that I update something on a site and then on my own web browser I'm like, hey, that doesn't look updated because I'm still seeing an old version of the site because the old version was cached. So sometimes it's helpful to actually go and clear this cache. Maybe because you're a web developer and you need to see the latest version. Or maybe you're a user and it's just taking up a lot of space on your hard drive.

All right, what about how a social media app or social media uses cache?

I've worked on some social media apps in the past, and one easy performance hack is to cache profile photos because you see the same profile photo over and over again, right? A user has their profile photo on their profile page, but then every time they post a comment, every time they post a new post, their little picture is going to appear next to it. There's no reason to go download it 50 times for every post that the person is browsing through. It's better if we cache it. If we just downloaded it once, we store it somewhere in temporary memory, usually on disk somewhere, but maybe in Ram. And then every time that user's photo needs to be shown, we already have it here on the device. We don't have to go across the network to the server and get that photo again.

All right, so now walk us through a news app cache.

Yeah. Another reason you might want to have a cache is because you might get into situations where the data is actually unavailable. For example, maybe you're using a news app and you're walking and you're reading the news as you walk. People do that. Maybe they shouldn't be. Whatever. There's an elevator. You walk into the elevator and there's no network connection. You can't get a WiFi signal in the elevator. You can't get a connection to a cell tower in the elevator. You still want to keep reading the story you were reading and maybe even a couple more stories as the elevator is going. Right. So it makes sense if the news app is actually caching the stories, keeping little copies of them in temporary memory so that the program can keep operating when the network connection is gone.

That makes sense. So you can still read when you're in a dead spot.

Exactly. So we've talked about a lot of different scenarios where you might want to have a cache, but of course there are many more. If I had to summarize here's the two things I think everyone should know about caches. We have caches in order to speed up the operation of retrieval because maybe a resource is going to take a while to get or to already have a pre computed calculation.

All right, let's put all of these examples together in one final example.

Yeah, let's say a Maps app. Okay, so you're creating an app to allow people to get directions from one place to another. There's some difficult calculations in there. For example, how many miles is it from the place you're starting to the place you're going? And maybe later on the person is going to calculate that same connection because they go to some of the same places. It would be good if we already knew in terms of miles how far that place is away. We might have to recalculate in terms of distance and the best routes because of traffic or because there's an accident somewhere or a road is closed. But the distance between two places as the bird flies, for example, is not going to change. So it might good to already have that calculation cached so we don't have to do it again. Another way a cache might be used in a Maps app is sometimes there's going to be different little photos associated with different places that are shown on the map. Well, it'd be good if we already had those photos available to us so we don't have to redownload them every time we see the same place. And a more recent feature that a lot of Maps apps are incorporating is offline Maps. That's like our example with the news app where you go into the elevator, right? What if you're driving somewhere and you're in a dead spot? It would be good if the maps themselves are actually cached offline on your device so that you can still see maybe not an up to date version of the map, but at least some kind of map when you're in that dead zone. So we see how multiple different caches can make the usability of an app like a Maps app way better.

So we decided to talk about caches after our last episode, hash tables. Why did we pick that order?

Software caches in particular. But hardware caches can be too are often implemented with similar principles to a hash table. Or even hash tables themselves are used to implement them. So remember, with a hash table, on average, we have constant time retrievals and updates of information that makes them ideal for something like a cache. And oftentimes with caches we don't even worry about having every possible thing that we could want to look up. So if we're using a hash table and we don't have everything in it that's actually okay. And the hash table will tell us very quickly that something is not in it.

As a software developer, what's important to keep in mind when it comes to caches?

I think caches are an opportunity. Usually the first time you implement a program, you might not actually implement all the different spots that you can put a cache in, but you realize, well, there's opportunities here for performance optimizations by adding a cache. I gave you the example of when I worked on social media apps when I was first working on them, a new one. I wouldn't think too much about how to optimize, not having to download resources because I wanted to get the basic design right. I wanted to get the core code architecture right. But then once I had that, it was easy to slip in. As long as I was using a pretty modular architecture, it would be easy to take a single component and say, hey, this part of the app is responsible for retrieving profile photos. Well, because I kept that really modularized, I can just slip in this extra layer there that says, oh, the first time I retrieve it, put it in a cache, and the next time I look for it, well, let's see if it's already there. So as long as I'm building my code in a very modular way, it's easy to later on go in and slip these caching layers where they're going to optimize performance. Thanks for listening to us this week, Rebecca. How can people get in touch with us on X?

We're at COPEC explains K-O-P-E-C-E-X-P-L-A-I-N-S-I want to.

Remind you to follow us on Spotify or Apple podcasts or your podcast player of choice. Leave us a review if you enjoy the podcast and we'll see you in a couple weeks.

Bye. It's.