Achieve parallel processing in php using shell_exec()

When people talk about parallel processing in PHP, it seems like a pretty much-complicated job.

The majority of developers use scheduler jobs, and that’s a good choice. We also use job queues in some cases.

But there are certain scenarios where we can make use of parallel processing effectively.

There are various ways to achieve above objective.

1. Multi-Thread:

The PHP language is not configured to be multi-threaded by default. However, there is an experimental PHP extension, pthreads, which supports multi-threading.

2. Parallel Processes

This can be done in 3 ways:

• Forking
• Executing Commands
• Piping

3. Distributed Parallel Processing:

Distributed computing allows us to distribute a problem over a network of computing devices so that we don’t have to rely solely on one computer to solve the problem.

How it works:

1. The Client App sends data to the Engine . (can be local or external a web service)
2. The MQ Engine stores the data “mostly in Memory and optionally in Database” inside a queue (you can define the queue name)
3. The Client App asks the MQ Engine for a data (message) to be processed them in order (FIFO or based on priority) “you can also request data from specific queue".

Some MQ Engines:

RabbitMQ
Beanstalkd
Gearman
Amazon SQS
IronMQ etc

Ref- https://stackoverflow.com/a/36440644/3309228

Now we will specifically discuss achieving parallel processing using shell_exec() command.

// Set data which needs to be passed to separate process running php file. 
$arguments = [
   'data' => [],
   'msg' => 'Test Message'
];
// Set Redirect output path to run php file in background rather than waiting for its output. It is compatible for both windows and linux platform. 
$redirectPath = strtoupper(substr(PHP_OS, 0, 3)) === 'WIN' ?
' > NUL' : ' > /dev/null 2>&1 &';
// Execure php file in shell and pass arguments using json encoded string. 
echo shell_exec('php Test.php' . ' ' . rawurlencode(json_encode($arguments)) .
' ' . $redirectPath);

As we can see above, whenever we want to start a task in parallel, we have to move the functionality into a separate php file and run it via shell. We need to make use of shell_exec() to do that. If we want to pass data, we can do so via command like arguments as written in above php code.

Since a new process will be started from within the main PHP process, we don’t want the running PHP process to wait for output of command line execution.
So we need to redirect the output to ‘> /dev/null 2>&1’ in the case of Linux and. ‘ > NUL’ in the case of the windows command line so that new processes can run in the background.

With shell_exec(), it is easy to run multiple php files at once. We can use it for glip/slack notifications, email notifications, or other CPU-intensive tasks. There is a need to add loggers in PHP files that run in the background, allowing easier error logging and debugging.

Thank you for reading this article. If you have any questions, please add comments below. You can connect me via Linkedin or Twitter