parallel processing - Do multiple CPUs compete for the same memory bandwidth?

Question

In a multi-CPU machine, do the different CPUs compete for the same memory bandwidth, or do they access DRAM independently?

In other words, if a program is memory bandwidth limited on, say, a 1-CPU 8-core system, would moving to a 4-CPU 4*8-core machine have a chance to speed it up (assuming the CPUs and DRAM are comparable)?

Answer 1

The answer to your main question is: Depends. What does it depend on? It depends on the camp your set up falls in, and technically speaking there are two.

In the first camp, which is known as Shared-Memory Multicore, the answer to your question would be "Yes". With this model, if you will, multiple processors having multiple cores share memory by way of a common bus (which is where you would get your bottleneck) and other than that, there is nothing connecting the CPUs together. This is the category/camp/ model where the typical consumer-grade computer falls in.

In the second camp, known as Distributed-Memory Multicore, the answer to your question is "No". This hardware-setup-scenario means that each processor has its own private memory but there is a bus connecting the processors together directly. The protocol for making this scenario possible is Message Passing Interface. This also means that the group of CPUs physically don't have to be in the same box or room as the RAM they access. You probably won't find this kind of set up in a home. Think research facilities, labs, universities, mid-large businesses etc..

To answer your second question. The answer is also depends. And it depends because one needs to know if the program was written to make use of parallelism with the system's parallel execution possibilities. Although your consumer-grade computer having one or two processors shares a single memory bus, if the program was written with parallelism in mind then you will notice a performance increase. Otherwise, serial instructions coming from a program will be executed serially on just one core.

If you are into the nitty-gritty of multi-core processing, and how memory is accessed via a program, a good "gateway resource" to expand your cranium on is Flynn's Taxonomy). Just Google-ing it will take you down the rabbit hole, if you are interested.

Edit: To give credit where credit is due, I highly recommend Professional Parallel Programming in C# by Gaston C. Hillar. This delightful book has been the most revealing on the topic of parallelism for me in my short career. It helps clear the muddy water on distinctions between Parallel Programming and Multi-core Programming and the types of multi-core processing I've just mentioned, complete with diagrams!