Programmer Musings

Unintuitive Multithreading: Speed

This begins a short series of essays on what most programmers get wrong about multithreading. Over and over again, I've seen programmers make the same mistakes in multithreaded applications on multiple operating systems and in multiple languages. So I decided to give you the benefits of what little insight I have into the problem.

The first problem most programmers have with multithreading is extremely fundamental: multithreading an application does not speed it up. In fact, the decision to multithread a program is rarely about raw speed. I expect that some of the people reading this are going to decide that I have no idea what I'm talking about at this point. However, I can back up this statement.

Given a single CPU machine and process that is CPU-bound, adding threads must slow down the program. In addition to the work that we were already doing, we now have to spend extra time doing context switches. Therefore, for the CPU-bound program, threading is guaranteed to slow down the program. (Of course, this only applies to preemptive multithreading.)

If raw speed is not the issue, why would we multithread a program? On a single CPU system, there are only two real reasons to use threads.

• responsiveness
• resources that block

Responsiveness

The main reason that preemptive multitasking has become popular in recent years is responsiveness. One process or thread should not be able to lock up the whole machine when the user wants access. The average user does not care if some process running in the background takes a few seconds longer to run as long as the computer responds immediately when a key is pressed or the mouse is moved. Although the computer is actually doing things slower, it feels faster because the computer is more responsive.

Many years ago, I was working in medical research. There was a program written by one of the programmers that normally ran over the weekend. Unfortunately, the guy that wrote it did not write any output to the screen or disk until the program was finished running. So, on Monday morning, we were often confronted with a blank screen and we didn't know if the program was still running, locked up, or what. We had no feedback at all. (This was back in the days of DOS, so we couldn't do anything else while the program was running either.) Eventually we changed the program to add a little progress counter to the screen, so we could at least tell if the program was still running. We used to joke about how much faster this made the program. Even though we knew the extra output was slowing down the real work, the feedback made it seem faster.

This concept is what drives most multithreading development. We want instant feedback and responsive computers. Things can actually run slower, as long as we get these two things.

Blocking

The other reason for multithreading is that not all processes are CPU-bound. At some time, most programs must do something that blocks progress. We might need to read data from disk to do further calculations, or retrieve data from a database, or even get user input. While this process or thread is blocked, it would be nice to allow the CPU to do other work. Threading makes this possible. (Actually, we used to do this with a cruder form of multitasking years ago. But, threads do make it easier.)

With multithreaded systems, the CPU can ignore threads that are blocked and continue with threads that are ready for work. This makes better use of the CPU and allows us to appear to be doing more than one thing at a time. In fact, by carefully organizing our code to keep the processor busy, we can appear to be running faster by keeping the CPU busy. In this case, we are not really speeding up the code. We are just no longer wasting CPU cycles.

Multiple CPUs

Of course, the entire discussion above assumes a single CPU. If we have multiple CPUs in a system, the situation is only slightly different. We can get more work done at a time because there is more than one CPU, but the problem remains the same. We can not get more work done than we have CPUs.

In a later essay, I'll delve into a piece of advice I received a long time ago that you should never have more threads than you have CPUs (plus a few extra to to take advantage of blocking calls). This turns out to be like most multithreading advice, part right, but mostly wrong.