Login/Sign Up

parallel programming

In computer science, parallel programming refers to writing code to make CPUs or multiple cores on single CPU execute the multiple instructions simultaneously. Whenever I read a question or answer on stackoverflow I get the sense that people are reasonably confusing multi-threading, asynchronous programming and parallel programming (or parallel computing).

To make the distinction clear, I will provide you different scenarios for different types of concurrent programming. (Oh shit, another term. Concurrent programming. Snap...)

First scenario is for parallel programming because it is easier to understand. If you have a single CPU with a single core then we cannot talk about parallel computation. (There are some exceptions, but the former assumption is quite sufficient for now). Parallel computation requires multiple pipelines for multiple instructions (machine code). This is not possible if not supported by hardware. Sorry.

Multi-threading happens when you have multiple "execution flows" (thread) running simultaneously. In the most basic terms, this can be done by single CPU and core by dividing the processing power for two different instruction lists. Instructions are never executed at the same time but feels like they are. Instead of writing ABC and DEF, you write ADBECF and it gives you the illusion of them happening at the same time. 

Although asynchronous programming have a lot in common with these guys, it is actually a little different paradigm. In this case, you don't need another thread or process. A method, statement or multiple of them are executed but not waited buy the mainstream execution of your program. You can either check the result of asynchronous routine either when you need it or when it is complete. By the way, asynchronous calls don't need another thread but it can be implemented and executed on another thread, as well.

Concurrency? It is just a broad term to include all the things above. 

To confuse you a little more, multi-threading programs can be used with asynchronicity paradigm and become multi-threading asynchronous programming. This is generally done by taking ready to use threads from a thread pool and using them for asynchronous calls. This can be achieved by either using event driven programming or state machine generation (mostly done by compilers. C# async await is an example for the latter one).

Multi-threading is also not a term referring to a single paradigm. There are two kinds of threads which deals with two different kinds of operations. IO bound operations such as networking, reading files, sql queries etc. And there is CPU bound operations such as large matrix multiplications, string manipulations, encryption/decryption and so forth. That is why some of them are suitable for async programming, other ones are multi-threading. 

Enough? I don't think so. Here is another thing. Multi-processing is a kind of multi-threading. Because in both cases there are multiple threads running independently at the same time. (Multi processing is a multi threading but it is not true for vice versa. Not every multi threading is multi processing.). Difference? As I stated in the first entry, one is truly parallel the other one is just an illusion of parallelism. Another difference is that int multi-threading (without multi processing) the threads share the program memory, because they are in the same process. (Yes. Every program is a process according to the operating system). But multi processes do not have a shared memory, that's why there are different methods for inter process communication.