Home » BASH » Write a file-in-file-out program in bash and running processes in parallel

Write a file-in-file-out program in bash and running processes in parallel

Update on:
Oct 4, 2021

In this script, we use a concept called file-in-file-out (FIFO), also known as pipes, to pass along a parameter to several “worker” scripts. These workers operate in parallel (in other words, mostly independent of the master process), read an input, and execute a command. FIFOS are useful because they can reduce file system activities or input/output (IO), and data can flow directly to listeners or recipients. They are represented on the file system as files and are bidirectional—they can be read and written to at the same time.

Prerequisites

To create FIFOs, we use the mkinfo command to create what appears to be a file (everything is a file in Linux). This file has a special property, though, which is different than normal files and also different from the pipes we had been previously using: the pipes, in this case, can allow for multiple readers and writers!

As with any file, you can also provide permissions using the -m flag such as this: -m a=rw, or use the mknod command (this isn’t covered as it requires that you use a second command called chown to change permissions after creation).

Write Script:

To start this exercise, we will introduce two terms: leader and follower, or master and worker. In this case, the master (the central host) will create the workers (or minions). While the recipe is a bit contrived, it should make for an easy go-to template for a simple named pipes or FIFO pattern. Essentially, there is a master that creates five workers, and those newly created workers echo out what is provided to them through the named pipe:

To get started, open a new terminal and create two new scripts: master.sh and worker.sh.

In master.sh, add the following contents:

master.sh

#!/bin/bash 

FIFO_FILE=/tmp/WORK_QUEUE_FIFO 
mkfifo "${FIFO_FILE}" 

NUM_WORKERS=5 
I=0 

while [ $I -lt $NUM_WORKERS ]; do 
	bash worker.sh "$I" & 
	I=$((I+1)) 
done 

I=0 
while [ $I -lt $NUM_WORKERS ]; do 
	echo "$I" > "${FIFO_FILE}" 
	I=$((I+1)) 
done 

sleep 5 
rm -rf "${FIFO_FILE}" 
exit 0

In worker.sh, add the following contents:

worker.sh

#!/bin/bash 

FIFO_FILE=/tmp/WORK_QUEUE_FIFO 
BUFFER="" 
echo "WORKER started: $1" 

while : 
do 

read BUFFER < "${FIFO_FILE}" 
if [ "${BUFFER}" != "" ]; then
	echo "Worker received: $BUFFER" 
	exit 1 
fi 

done 

exit 0

In the terminal, run the following command and observe the output:

$ bash master.sh

How this script works:

The idea of this script is that if you have several repetitive tasks such as bulk operations and potentially multiple cores, you can perform tasks in parallel (often seen in the Linux world as Jobs). This recipe creates a single master that spawns several worker scripts into the background, which await input from the named pipe. Once they read input from the named pipe, they will echo it to the screen and then exit. Eventually, the master will exit too, removing the pipe along with it:

In step 1, we open a new terminal and create the two scripts: master.sh and worker.sh.

In step 2, we create the master.sh script. It uses two while loops to create n numbers of worker scripts with $I identifiers and then sends the same number of values to the FIFO before sleeping/exit.

In step 3, we create the worker.sh script, which echos an initialization message and then waits until $BUFFER is not empty (NULL, as it can be sometimes referred to). Once $BUFFER is full or rather, contains a message, then it echos it to the console and the script exits.

In step 4, the console should contain an output similar to the following:

$ bash master.sh 
WORKER started: 0 
WORKER started: 1 
We got 0 
We got 1 
WORKER started: 4 
We got 2 
WORKER started: 2 
We got 3 
WORKER started: 3 
We got 4

With the two scripts working in tandem over the FIFO, a numeric value is passed between them and the workers perform their work. These values or messages could easily be modified so that the workers execute commands instead!

Note:

Notice that the output can be in a different order. This is because Linux is not deterministic and spawning processes or reading from the FIFO might be blocked, or someone might get there before it (due to scheduling). Keep this in mind as the FIFO is also not atomic or synchronous—if you wish to designate which message goes to what host, you could create an identifier or messaging scheme.

Related Posts

Creating a config file and using it in tandem with your scripts

In this article, we are going to create a config file and use it in our shell script.PrerequisitesBesides having a terminal open, you need basic knowledge of creating scripts and config files.Write scriptNow, we are going to create a script and config file. The...

Calculating and reducing the runtime of a script

In this article, we are going to learn how to calculate and reduce the script’s runtime. A simple time command will help in calculating the execution time.PrerequisitesBesides having a terminal open, make sure you have the necessary scripts present in your...

Using Bash to monitor battery life and optimize it

In this article, we will learn about the TLP Linux tool. TLP is a command-line tool; it is used for power management and will optimize the battery life.PrerequisitesBesides having a Terminal open, you need to ensure that you have TLP installed on your system.How to do...

Creating a simple NAT and DMZ firewall using bash script

In this article, we will create a simple NAT firewall with DMZ using iptables.PrerequisitesBesides having a Terminal open, you need to ensure that iptables is installed in your machine.Write scriptWe will write a script to set up a DMZ using iptables. Create...

Follow Us

Our Communities

More on BASH

The Ultimate Managed Hosting Platform
Load WordPress Sites in as fast as 37ms!

0 Comments

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

five − three =

Shares