Reusing ExecutorService inside itself - NOT GOOD

This week I ran into an interesting deadlock situation in my Java web app code. I have never faced it before and also never read any "Don't do it this way" warnings. So I thought I will share it as it was pretty interesting what happened.

Setup

I needed to parallelize the run of the web application in order to gain speed. Task was well parallelizable so I put in an implementation of an Executor service. Because the computer is expected to have 4 cores (for now) and the task is purely CPU intensive (no IO) I wanted to limit the workers in the thread pool so that I don't have unnecessary too many threads. So I picked 10 (out of the blue). The task can be parallelized at different levels and because I did not want to have one executorService per level I created a general one so that when any part of the codebase on the particular critical path need to paralyze it can take it and use it. I thought it was clever and first everything ran ok. Before I proceed lets look at how the situation looked:

The path of execution is starting at the top where it splits into two tasks that can run in parallel, this happens per the request arriving to the server. On the second level each task splits their work again into two so that in the end I utilize fully all 4 cores. Then all the tasks on 3rd level has to finish so that the tasks on the 2nd level are finished and the client's request is completed. On both the 2nd and 3rd level I used the same ExecutorService instance that had 10 threads limit.

The problem:

I tried to test it and everything looked nice and the speed per one request was improved. So I proceeded to putting a higher load (in terms of requests per second) on the web app. And again it was looking good and smooth so I thought GOOD WORK! Until I increased the load even more and the whole system completely deadlocked.

What went wrong:

First I was troubleshooting Semaphores and some locking I had inside but every time I put extreme load on the server it deadlocked. Finally I found why. When you put on the set up so many requests per second what happens is that many tasks are being forked on the 1st level before anything started being forked on the 2nd level. This leads to the ExecutorService workers being fully depleted at the 1st level and there are no more available workers. For a worked to finish it would mean that any task from the 2nd level has to finish which means that all of its sub tasks from 3rd level have to finish, but they cannot because there are no more available workers in the executor service to start execution on 3rd level.

How I fixed it:

For the sake of simplicity and because the performance was actually better than I needed I decided to remove the forking on the 2nd level and run it single threaded at that level. For the future I would probably just implement two different executor services per each level. So the lesson I learnt is never to use the same executor service "inside" itself.