One of my clients recently cross graded from a 9600 Pro to what he believed was a 9800 Pro. The card in question turned out to be the 9800 SE, 128 bit edition, basically a down grade in the long run. So, I attempted to explain the differences between the cards, and one of the subjects was the number of bits listed for Memory Bus Width.
A normal Radeon 9800 Pro has a 256bit memory bus, while the 9800 SE normally has a 128-bit bus. So this is the analogy I gave the client.
Imagine you have a 20x20 room. You have 8 big strong guys in the room who build vehicles. They can build a truck, a sports car, or a compact car. The Room is the graphics card, and the number of guys inside are the number of shaders available in the card. In the case of the 9600 and 9800, they only have 4 guys inside this room, instead of 8, so they only have half the power available.
Our Memory Bus Width though, is the door in the room. The size of our door impacts how many parts can be taken in and out. A 128bit bus width would be a typical 4 foot wide door. A 256bit Bus would be two double doors, and a 64bit bus would be half a door.
Now, if you want a compact car, it's going to be very easy to move the small parts in and out regardless of the size of the door. Even with only half a door, our guys could probably build a Geo Metro.
In computer terms, this would be running in Low Resolution with Low Detail in a game. There is not much going on, so the card is able to keep up easily. However, if you increase the size of the door, you don't see any benifit.
Now, if you want a sports car, say like a Mustang, you'll need to move larger parts in and out of the door. With only half a door, it'll be hard to fit the engine block in, and the frame would probably have to be assembled inside the room, so our big strong guys will be working harder. Our guys would also have to wait longer for parts to be shuttled in and out with a limited amount of space.
In computer terms, this would be turning up the resolution and detail level. The card is going to have to work harder to display the image, but if it has a low memory bandwith, or a small door, then the Graphics card will be spending a lot of time just waiting for more information. The higher the Bus Width, the more information that can be transferred.
If you want a Big Ole Truck, something with a v12 Hemi inside, you'll need a massive door to get the engine block inside. That big door will also come in handy moving the frame inside. Go with a smaller door though, and you'll have to break everything down on one side of the door, pass it through, and then re-assemble it in the room.
In computer terms, this is turning the resolution and detail levels all the way up. The card is going to be put to work to put everything together. But, if it can't transfer the information in and out, it's going to wind up waiting on the information to transfer. So, a big door, or high memory bus width will help.
Now, this analogy can be built on by talking about Memory Speeds, or how quick parts can be sent through the door, or by talking about Shader Efficiency, how much work the guys inside can actually do in a given time.
But, I hope it gives a rough idea on why Memory Bus Width Matters. This may also help explain upgrades that were not really upgrades. For example, if you had a 9800 Pro and you upgrade to an x1600 Pro, you would have more of a cross-grade.