sed – non-interactive stream editor uses in Linux

May 9, 2021

Comments

Adsense

Adsense

Adsense

Adsense

The stream editor (sed) is a very popular non-interactive stream editor. Normally, whenever we edit files using the vi editor, we need to open the file using the vi command, then we interact with the file to see the content of the file on screen, then edit it, and then save the file. Using sed, we can type commands on the command line and sed will make the changes to the text file. sed is a non-destructive editor. sed makes the changes to the file and displays the content on screen. If we want to save the changed file, then we need to redirect the output of sed to the file.

The procedure to install sed is shown here.

For Ubuntu or any Debian-based distributions, enter the following command:

$ apt-get install sed

For Red Hat or any rpm-based distribution enter the following command:

$ yum install sed

To check the version of sed, enter the following command:

$ sed -V

Otherwise, enter this command:

$ sed --versionGNU sed version 3.02

Understanding sed

Whenever you use sed commands on a text file, sed reads the first line of the file and stores it in a temporary buffer called pattern spacesed processes this pattern space buffer as per commands given by the user. Then, it prints the output on screen. This line from the pattern space is then removed and the next line of the file is loaded in the pattern space. In this way, it processes all the lines one by one. This line-by-line processing is continued till the last line of the file. As the sed commands are processed in the temporary buffer or pattern space, the original line is not modified. Therefore, we say sed is a non-destructive buffer:

Understanding regular expression usage in sed

While using sed, regular expressions are enclosed in forward slashes, as grep and sed use regular expressions and meta-characters for searching patterns in the file. An example of this would be the following:

sed -n '/Regular_Expression/p' filenamesed -n '/Mango/p' filename

This will print lines matching the Mango pattern:

sed -n 's/RE/replacement string/' filenamesed -n 's/Mango/Apple/p' filename

This will find the line containing the Mango pattern and then the Mango pattern will be replaced by the Apple text. This modified line will be shown on screen and the original file will be unchanged.

The following is a summary of various meta-characters and their usage in sed:

Meta-character

Function

^

This is the beginning-of-line anchor

$

This is the end-of-line anchor

.

This matches one character, but not the newline character

*

This matches zero or more characters

[ ]

This matches one character in the set

[^ ]

This matches one character not in the set

(..)

This saves matched characters

&

This saves the search string so it can be remembered in the replacement string

<

This is the beginning-of-word anchor

>

This is the end-of-word anchor

x{m}

This is the repetition of the character x:m times

x{m,}

This means at least m times

x{m,n}

This means between m and n times

Addressing in sed

We can specify which line or number of lines the pattern search and commands are to be applied on while using the sed commands. If line numbers are not specified, then the pattern search and commands will be applied to all lines of the input file.

The line numbers on which commands are to be applied are called the address. The address can be a single line number or range of lines in which the starting number of the line and the ending number of the range will be separated by commas. Ranges can be composed of numbers, regular expressions, or a combination of both.

The sed commands specify actions such as printing, removing, replacing, and so on.

The syntax is as follows:

sed 'command' filename(s)

Here is an example:

$ cat myfile | sed '1,3d'

You could also use the following:

sed '1,3d' myfile

This will delete lines 1 to 3:

sed -n '/[Aa]pple/p' item.list

If the Apple or apple pattern is found in the item.list file, then those lines will be printed on screen and the original myfile file will be unchanged.

To negate the command, the exclamation character (!) can be used.

Here’s an example:

sed '/Apple/d' item.list

This tells sed to delete all the lines containing the Apple pattern.

Consider the following example:

sed '/Apple/!d' item.list

This will delete all the lines except the line containing the Apple pattern.

How to modify a file with sed

sed is a non-destructive editor. This means the output of sed is displayed on screen but the original file is unchanged. If we want to modify the file, then we can redirect the output of the sed command to the file. Deleting lines is illustrated in the following examples:

$ sed '1,3d' datafile > tempfile$ mv tempfile newfile

In this example, we have deleted lines 1 to 3 and stored the output in tempfile. Then, we have to rename tempfile to newfile.

Printing – the p command

By default, the action of the sed command is to print the pattern space, such as every line that is copied into the buffer, and then print the result of processing it. Therefore, the sed output will consist of all lines along with the processed line by sed. If we do not want the default pattern space line to be printed, then we need to give the -n option. Therefore, we should use the -n option and the p command together to see the result of the sed processed output.

Here is an example:

$ cat country.txt

The output is as follows:

Output:

Country     Capital     ISD CodeUSA         Washington  1China       Beijing     86Japan       Tokyo       81India       Delhi       91$ sed '/USA/p' country.txt

The output is as follows:

Output:

Country     Capital     ISD CodeUSA         Washington  1USA         Washington  1China       Beijing     86Japan       Tokyo       81India       Delhi       91

All the lines from the file are printed by default and the lines with the USA pattern are also printed:

$ sed -n '/USA/p' country.txt

The output is as follows:

Output:

USA  Washington  1

As we have given the -n option, sed has suppressed default printing of all lines from the country file but has printed the line that contains the text pattern USA.

Deleting – the d command

The d command is used to delete lines. After sed copies a line from a file and puts it into a pattern buffer, it processes commands on that line, and, finally, displays the contents of the pattern buffer on screen. When the d command is issued, the line currently in the pattern buffer is removed and not displayed, as follows:

$ cat country.txtCountry     Capital     ISD CodeUSA         Washington  1China       Beijing     86Japan       Tokyo       81India       Delhi       91$ sed '3d' country.txt

The output is as follows:

Output:

Country      Capital    ISD CodeUSA          Washington 1Japan        Tokyo           81India        Delhi       91

Here is the explanation.

The output will contain all the lines except the third line. The third line is deleted by the following command:

$ sed '3,$d' country.txt

The output is as follows:

Output:

Country       Capital   ISD CodeUSA           Washington      1

This will delete the third line to the last line. The dollar sign in the address indicates the last line. The comma is called a range operator:

$ sed '$d' country.txt

The output is as follows:

Output:

Country       Capital   ISD CodeUSA          Washington       1China         Beijing         86Japan         Tokyo           81

Here is the explanation.

This deletes the last line. All lines except lines will be displayed.

Here is an example:

$ sed '/Japan/d' country.txt

The output is as follows:

Output:

Country     Capital           ISD CodeUSA         Washington    1China       Beijing           86India       Delhi             91

The line containing the Japan pattern is deleted. All other lines are printed:

$ sed '/Japan/!d' country.txt

The output is as follows:

Japan       Tokyo         81

This has deleted all the lines that do not contain Japan.

Let’s see a few more examples with the delete command.

This will delete line 4 and the next five lines:

$ sed '4,+5d'

This will keep lines 1 to 5 and delete all the other lines:

$ sed '1,5!d'

This will delete lines 1, 4, 7, and so on:

$ sed '1~3d'

Starting from 1, every third line step increments. The number that follows the tilde is what is called the step increment. The step increment indicates the following:

$ sed '2~2d'

This will delete every other line, starting with line 2.

Substitution – the s command

If we want to substitute some text with new text, then we can use commands. After the forward slash, the regular expression is enclosed and then the text to be substituted is placed. If the g option is used, then substitution will happen globally, meaning that it will be applied to the full document. Otherwise, only the first instance will be substituted:

$ cat shopping.txt

The output is as follows:

Output:

Product     Quantity   Unit_Price  Total_CostApple       2           3          6Orange            2           .8         1.6Papaya            2           1.5        3Chicken     3           5          15Cashew       1           10         10$ sed 's/Cashew/Almonds/g' shopping.txt

The output is as follows:

Output:

Product          Quantity   Unit_Price  Total_Cost Apple       2           3           6Orange            2           .8         1.6Papaya            2           1.5         3Chicken     3           5          15Almonds     1           10          10

The s command has replaced Cashew with Almonds. The g flag at the end indicates that the substitution is to be applied globally. Otherwise, it will be applied to the first pattern match only.

The following substitution command will replace two-digit numbers at the end of the line with .5 appended to them:

$ sed 's/[0-9][0-9]$/&.5/' shopping.txt

The output is as follows:

Output:

Product     Quantity  Unit_Price  Total_CostApple       2          3            6Orange            2           .8          1.6Papaya            2          1.5          3Chicken           3          5           15.5Cashew            1         10           10.5

The ampersand in the search pattern represents the exact pattern found. This will be replaced by the exact pattern with .5 appended to it.

Range of selected lines the comma

To use sed effectively, we should be clear about how to define range. Range is typically two addresses in a file as follows:

  • Range with numbers:
'6d': range of line 6 
'3,6d': range from line 3 to 6 
  • Range with pattern:
'/pattern1/,/pattern2/ 
  • This will specify the range of all the lines between pattern1 and pattern2. We can even specify the range with a combination of both, that is, '/pattern/,6'. This will specify the range of lines between the pattern and line 6.

As mentioned, we can specify the range as numbers, pattern, or a combination of both, as shown here.

$ cat country.txtCountry     Capital           ISD CodeUSA         Washington  1China       Beijing           86Japan       Tokyo             81India       Delhi             91$ sed -n '/USA/,/Japan/p' country.txt

The output is as follows:

Output:

USA     Washington      1China   Beijing         86Japan   Tokyo           81

In this example, all the lines between addresses starting with USA and until the pattern Japan will be printed on screen, as shown here.

$ sed -n '2,/India/p' country.txt

The output is as follows:

Output:

USA      Washington    1China    Beijing         86Japan    Tokyo          81India     Delhi            91

In this example, line 2 to the pattern India, are printed on screen as shown here.

$ sed '/Apple/,/Papaya/s/$/**   Out of Stock   **/' shopping.txt

The output is as follows:

Output:

Product       Quantity  Unit_Price  Total_CostApple       2          3              6**       Out of Stock    **Orange            2          .8             1.6**     Out of Stock    **Papaya            2          1.5            3**       Out of Stock    **Chicken           3          5              15Cashew            1          10             10

In this example, for all the lines between the Apple and Papaya patterns, the end of line will be replaced by the ** Out of Stock ** string.

Multiple edits – the e command

If we need to perform multiple editing with the same command, then we can use the -e command. Each edit command should be separated by the -e command. sed will apply each editing command separated by -e on the pattern space before loading the next line in the pattern space:

$ cat shopping.txt

The output is as follows:

Output:

Product     Quantity  Unit_Price  Total_CostApple       2          3            6Orange            2          .8           1.6Papaya      2          1.5          3Chicken     3          5            15Cashew            1          10           10

This is an example:

sed -e '5d' -e 's/Cashew/Almonds/' shopping.txt

The output is as follows:

Output:

Product     Quantity  Unit_Price  Total_CostApple       2         3           6Orange            2         .8          1.6Papaya            2         1.5         3Almonds     1         10          10

Initially, the command for deleting the fifth line is called, then, the next substitution command to replace Cashew with Almonds is processed.

Reading from files – the r command

If we need to insert text from another file into a file, processed by sed, then we can use the r command. We can insert text from another file to the specified location:

Here is an example:

$ cat new.txt

The output will be:

Output:

*********************************    Apples are out of stock*********************************$ sed '/Apple/r new.txt' shopping.txt

The output is as follows:

Output:

Product     Quantity  Unit_Price  Total_CostApple       2         3           6*********************************    Apples are out of stock*********************************Orange      2         .8         1.6Papaya      2         1.5        3Chicken     3         5          15Cashew     1         10         10

This command has added the content of the new.txt file after the line containing the Apple pattern.

Writing to files – the w command

The sed command for writing is w. Using this command, we can write lines from one file to another file.

Here is an example:

$ cat new.txt 

The output is as follows:

Output:

new is a empty file$ sed -n '/Chicken/w new.txt' shopping.txt$ cat new.txt Chicken    3    5    15

After the w command, we specify the file to which we will perform the write operation. In this example, the line containing the Chicken pattern is written to the new.txt file.

Appending – the a command

The a command is used for appending. When the append command is used, it appends the text after the line in the pattern space in which the pattern is matched. The backslash should be placed immediately after the a command. On the next line, the text to be appended is to be placed.

Here is an example:

$ cat shopping.txt

The output is as follows:

Output:

Product     Quantity  Unit_Price  Total_CostApple       2          3           6Orange      2           .8         1.6Papaya      2          1.5         3Chicken     3          5          15Cashew      1         10          10$ sed '/Orange/a**** Buy one get one free offer on this item ! ****' shopping.txt

The output is as follows:

Output:

Product   Quantity  Unit_Price  Total_CostApple     2          3           6Orange    2           .8         1.6**** Buy one get one free offer on this item ! ****Papaya    2          1.5         3Chicken   3          5           15Cashew   1          10          10

The new text **** Buy one get one free offer on this item ! **** is appended after the line containing the Orange pattern.

Inserting – the i command

The i command is used for inserting text above the current pattern space line. When we use the append command, new text is inserted after the current line, which is in the pattern buffer. In this similar-to-append command, the backslash is inserted after the i command.

Here is an example:

$ cat shopping.txtProduct     Quantity  Unit_Price  Total_CostApple       2          3            6Orange            2          .8           1.6Papaya            2          1.5          3Chicken     3          5            15Cashew      1          10           10 $ sed '/Apple/i                                                New Prices will apply from Next month ! ' shopping.txt

The output is as follows:

Output:

Product     Quantity  Unit_Price  Total_Cost       New Prices will apply from Next month ! Apple          2      3            6Orange         2       .8          1.6Papaya         2      1.5          3Chicken        3      5           15Cashew         1     10           10

In this example, the new text, New Prices will be applied from next month! is inserted before the line containing the Apple pattern. Please check the i command and the backslash following it.

Changing – the c command

The c command is the change command. It allows sed to modify or change existing text with new text. The old text is overwritten with the new:

$ cat shopping.txt

The output is as follows:

Output:

Product     Quantity  Unit_Price  Total_CostApple        2         3           6Orange       2          .8         1.6Papaya       2         1.5         3Chicken      3         5          15Cashew       1        10          10

Here is an example:

$ sed '/Papaya/c  Papaya is out of stock today !' shopping.txt

The output is as follows:

Output:

Product  Quantity  Unit_Price  Total_CostApple     2         3           6Orange    2         .8          1.6  Papaya is out of stock today !Chicken   3        5           15Cashew    1       10           10

In this example, the line containing the expression Papaya is changed to the new line, Papaya is out of stock today!.

Transform – the y command

The transform command is similar to the Linux tr command. The characters are translated according to the character sequence given. For example, y/ABC/abc/ will convert lowercase abc into uppercase ABC.

Here is an example:

$ cat shopping.txt

The output will be:

Output:

Product     Quantity  Unit_Price  Total_CostApple        2         3           6Orange       2          .8         1.6Papaya       2         1.5         3Chicken      3         5          15Cashew       1        10          10$ sed '2,4y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRS
 TUVWXYZ/' shopping.txt

The output will be:

Output:

Product  Quantity  Unit_Price  Total_CostAPPLE       2        3           6ORANGE      2        .8          1.6PAPAYA      2       1.5          3Chicken     3       5           15Cashew      1      10           10

In this example, for lines 23, and 4, all the lowercase letters are converted to uppercase letters.

Quit – the q command

The q command is used for quitting the sed processing without proceeding to the next line:

$ cat shopping.txt

The output will be as follows:

Output:

Product     Quantity  Unit_Price  Total_CostApple         2        3           6Orange        2        .8          1.6Papaya        2       1.5          3Chicken       3       5           15Cashew        1      10           10

Here is an example:

$ sed '3q' shopping.txt

The output will be as follows:

Output:

Product  Quantity  Unit_Price  Total_CostApple     2         3          6Orange    2         .8         1.6

In this example, after printing the first to third lines, sed quits further processing.

Holding and getting – the h and g commands

We have already seen that sed has a pattern buffer. sed has one more type of buffer called a holding buffer. With the h command, we can inform sed to store the pattern buffer in the holding buffer. Whenever we need the line that is stored in the pattern buffer, we can get it with the g command, that is, get the buffer.

Here is an example:

$ sed -e '/Product/h' -e '$g' shopping.txt

The output is as follows:

Output:

Product  Quantity  Unit_Price  Total_CostApple     2          3           6Orange    2          .8          1.6Papaya    2         1.5          3Chicken   3         5            15Cashew    1        10            10Product  Quantity  Unit_Price  Total_Cost

In this example, the line containing the Product pattern is stored in the holding buffer by the h command. Then, the next editing command asks sed to get the line from the holding buffer when the last line of the file is reached. It then appends the line from the holding buffer after the last line of the file.

Holding and exchanging – the h and x commands

This is an exchange command. By using this command, we can exchange the holding buffer with the current line in the pattern buffer.

Here is an example:

$ sed -e '/Apple/h'  -e '/Cashew/x' shopping.txt

The output is as follows:

Output:

Product  Quantity  Unit_Price  Total_CostApple      2         3          6Orange     2        .8          1.6Papaya     2       1.5          3Chicken    3       5           15Apple      2       3            6

In this example, the line with the Apple pattern is stored in the holding buffer. When the pattern with Cashew is found, that line will be exchanged with the holding buffer.

sed scripting

The sed script file contains a list of sed commands in a file. To inform sed about our script file, we should use the -f option before the script filename. If the sed commands are not separated by a new line, then every command should be separated by a colon “:“. We should make sure that there aren’t any trailing whitespaces after any of the commands in the sed script file; otherwise, sed will give an error. sed takes each line in the pattern buffer and then it will process all commands on that line. After this line is processed, the next line will be loaded in the pattern buffer. For the continuation of any sed command that cannot be fitted on one line, we need to add one backslash at the end of the line to inform it of the continuation.

Here is an example:

$ cat shopping1.txt

The output is as follows:

Output:

Product   Quantity  Unit_PriceApple      200       3Orange     200       .8Papaya     100       1.5Chicken     65       5Cashew      50       10April, third week $ cat stock 

The output is as follows:

Output:

# This is my first sed script by : 1iStock status report/Orange/aFresh Oranges are not available in this season. Fresh Oranges will be available from next month/Chicken/c**********************************************************We will not be stocking this item for next few weeks.**********************************************************$d

Enter the following command:

$ sed -f stock shopping1.txt

The output is as follows:

Output:

Stock status reportProduct   Quantity  Unit_PriceApple      200       3Orange     200        .8Fresh Oranges are not available in this season. Fresh Oranges will be available from next monthPapaya     100       1.5**********************************************************We will not be stocking this item for next few weeks.**********************************************************Cashew      50     10

In this script, the following processing has taken place:

  • The comment line starts with the pound (#) sign.
  • The command 1i informs sed to insert the next text before line number 1.
  • The command /Orange/a informs sed to append the next text after the line containing the Orange pattern.
  • The command /Chicken/c informs sed to replace the line containing the Chicken pattern by the next line.
  • The last command, $d, tells sed to delete the last line of the input file.
Satish Kumar

Satish Kumar

I am Satish Kumar, Founder of LinuxConcept. Linux and F.O.S.S enthusiast, love to work on open source platform and technologies.

Adsense

Adsense

Adsense

Adsense

Adsense

Adsense

Adsense

Adsense

Adsense

Related Posts

Backup of files in Linux using command line

In IT or our day-to-day computer industry activities, taking backup is one of the most important activities. Previously, offices were required to keep important paper in a safe place; but if a fire breaks out, then everything is finished. In the digital world, taking...

Using awk in Linux

awk is a program that has its own programming language for performing data-processing and generating reports. The GNU version of awk is gawk. awk processes data, which can be received from a standard input, input file, or as the output of any other command or process....

0 Comments

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

4 + 7 =

News & Updates

Join Our Newsletter

Adsense

Adsense

Adsense

Adsense