In the previous post we learned how Windows handles slow devices. Completion Port is an effective tool to handle I/O results. Completion Ports have Thread Pool like features, and it is not a coincident. Windows implements a Thread Pool, and it is partially based on Completion Ports. It may sound strange, because Thread Pool is a general pool of threads, which we can use for any kind of tasks. Completion Port is about I/O.

Completion Ports are not only for I/O

Well, it is not only the I/O Manager that can put Completion Packets into a queue of a Completion Port. We can do it, too. Of course, a Completion Packet created by us will not contain valid I/O result, but it will be good enough to trigger a thread of the Completion Port.

Windows Thread Pool does the same trick. When Windows Thread Pool assigns a thread to a Completion Port, that thread doesn’t look for I/O results. It looks for a method address put into the Completion Packet, and starts running that method. This way Windows Thread Pool can run custom methods.

The Completion Port of a Windows Thread Pool still can be used as a normal Completion Port, so it can be assigned to a File Object, and then every completed IRP will result in a Completion Packet, and the I/O Manager will enqueue this packet to the Completion Port. The Windows Thread Pool threads however still will be looking for the method address they need to execute, and they will know nothing about how to handle our I/O result. To solve this problem, when we want to use a Windows Thread Pool Completion Port for I/O, we need to use a special method to assign the File Object to the Completion Port. With this method we can specify a call back method, and the Thread Pool threads will call that method for us when a Completion Packet is ready to be processed. So using Windows Thread Pool for I/O, we can specify a custom handler method for every File Object, which is better than when we use the bare Completion Port. It is still not possible to define an individual callback method for every I/O operation, though.

Why do Windows Thread Pools have two set of threads?

Problems arise when we put something on the Windows Thread Pool (we make a Thread Pool Thread to call a callback) that performs I/O using the old Completion Routines. The problem is not the Completion Routine itself, the problem is that the operating system uses APC to call those Completion Routines, as we have seen it in the previous part. We know that APCs are executed when a thread starts waiting. So what is the problem, Thread Pool Threads wait a lot when no tasks to execute, don’t they? Yes they do, but for APCs, they need to wait in a special state, called Alertable Wait State. It is not that hard to achieve this, it is just that instead of calling Sleep(), we need to call SleepEx(), or instead of Waitxxx() we need to call WaitxxxEx(). This is how we can tell the operation system, that the thread is prepared to deal with APC-s.

The Windows Thread Pool Threads assigned to Completion Ports don’t wait in Alertable Wait State. This means that APCs would never run. No big deal, we may say, we won’t do I/O operations with Completion Routines, and then we don’t need APC. But it is not that obvious. We may use a library we bought that uses APC, and we may call this library from a Thread Pool Thread.

To make Windows Thread Pool support APC, it has a second set of threads besides the Completion Port threads. These threads wait in Alertable Wait State. If the programmer wants to do some operations that requires APC, then they use a special parameter when they put the task to the Thread Pool. In this case the Thread Pool will not be using the Completion Port but the second set of threads. These threads will enter Alertable Wait State after they finished with a task, so sooner or later the APCs will be processed.

These two sets of Windows Thread Pool threads have names. Because the non-Completion Port Threads are created to support APC, which is used by I/O operations with Completion Routines, these threads are called I/O Threads. The Completion Port threads are called Non-I/O Threads, which is funny because Completion Port is designed to process I/O results.

Why do .Net Thread Pools have two set of threads?

In case of .Net, when it was designed, they didn’t need to care about libraries using APC, because there were no libraries at all for .Net. .Net implements its own Thread Pool and it doesn’t have APC support, so there are no I/O Threads in it. Which is not true, because it has I/O Threads, but .Net calls Completion Port threads I/O Thread, that is the Non-I/O Thread for Windows Thread Pool. Confusing? It is, so let’s start it over.

.Net also uses two sets of threads, but not for the same reason as Windows Thread Pool. .Net uses Completion Ports, that is one set of the threads. Completion Port is powerful but it is not a universal tool. It is really good at processing I/O results. I/O result is not just reading a file. A Web Service accepting client requests generates I/O results every time a client is connecting. Completion Ports keep the thread count low. They start additional threads in special circumstances only. If we had only the Completion Port, and we put several CPU heavy tasks on it, then its few threads would start working on the calculations, and when I/O results came in, nothing would be able to process those. In general, different strategy is needed to handle computing related tasks and I/O tasks.

Because of this, .Net also has worker threads. It is an overloaded term, somebody calls every thread a worker thread if it is not the main thread. I will call this second set of thread pool threads worker threads. This second set of threads is supposed to used for calculation heavy (not I/O heavy) operations.

When we work directly with the .NET Thread Pool (QueueUserWorkItem), that goes for the worker threads. When we use an I/O class (e.g. FileStream) with asynchronous operations, those will be handled by the Completion Port. In case of Windows Thread Pool, we have the same API for I/O and computing, and the programmer needs to know how to invoke it to address the appropriate threads. In .Net we do not need to decide, the framework does it for us, it is harder to make mistakes.

In the next part we will investigate a little-bit how the framework decides when to add new threads to the thread pool.