Linux Sed Tutorial: Learn Text Editing with Syntax and Examples

This article was adapted from its original version on NixCraft.

Sed is an acronym for “stream editor.” A stream refers to a source or destination for bytes. In other words, sed can read its input from standard input (stdin), apply the specified edits to the stream, and automatically output the results to standard output (stdout). Sed syntax allows an input file to be specified on the command line.

However, the syntax does not directly support output file specification; this can be achieved through output redirection or editing files in place while making a backup of the original copy optionally. Sed is one of the most powerful tools on Linux and Unix-like systems. Learning it is worthwhile, so in this tutorial, we will start with the sed command syntax and examples.

READ ALSO: How to Search and Find Files for Text Strings in Linux/Ubuntu?

Using the sed editor to perform noninteractive editing

• Sed is a stream editor.
• For interactive text editing, you can use editors like vi/vim, nano, or emacs. But, sed is suitable for non-interactive file editing at the command-line interface (CLI) in your scripts or Dockerfiles.
• By default, sed operates non-destructively. You need to specify output files to save changes or use a special GNU sed option to edit the file in place.
• It provides regular expressions (regex) for powerful text manipulation.

How does sed work?

Sed works line-by-line. It will read each line into a pattern buffer, modify the line via sed commands, and then output the buffer to standard out (stdout), which can be redirected to another file. By default, the original file is not modified.

The sed maintains two data buffers

The sed command maintains two data buffers. Both are initially empty:

1. Pattern buffer (active pattern space) : When sed reads a line-by-line from the input, it places that line into the pattern space. This is where text manipulation takes place. For example, you can use sed commands like s for substitute, d for delete, p for print. By default, the pattern space is cleared at the end of each line read cycle.
2. Holding buffer (auxiliary hold space) : As the name suggests, a hold buffer acts as a hold space. It is a secondary buffer that sed uses for temporary storage. Think of it as a place to keep data you want to save and use later when processing a different line. You use this for advanced operations like a copy, append, compare, or retrieval command. Typical usage for holding buffer is finding duplicate lines in a sorted input file or concatenating multiple lines together for advanced editing. Unlike the pattern space, the hold space retains its content between cycles unless you explicitly change it. In other words, this allows you to store and recall information across multiple lines. You use specific sed commands (h, H, g, G, x) to move data between the pattern space and the hold space

In short, the pattern space is where the immediate editing happens, and the hold space provides a way to save and recall information for more complex editing tasks. Standard input (stdin) is typically the keyboard, a file, or another data stream. Standard output (stdout) is typically the screen or a file.

GNU Linux sed command syntax

Typically GNU version of sed run as follows:

sed 'commands' input_file 
sed 'commands' input_file > output_file
sed 'commands' input_file | command2

A more accurate syntax:

sed [options] ’[addresses] action [args]’ input_files [ > outfile]
sed [options] ’[addresses] action [args]’ input_files [ | command_2]

You do not need to interact with the sed editor while running; therefore, it has also been called a batch editor. This contrasts with such editors as Vim (vi), emacs, nano, and ed, which are interactive. Because sed does not require interaction, you can place sed commands in a script. You can call the script file and run it against the data file to perform repetitive editing operations:

sed SCRIPT input_file

The GNU sed editing commands

The most useful sed commands are inspired vi (vim) and ed, and 99% of users use them heavily:

Apart from that GNU/sed command has few useful CLI options:

The sed addressing

Before we see practical examples, the last thing you need to understand is sed addressing, which states how you specify which lines of input should be affected by a sed command. The sed editor processes all input file lines unless you specify an address. This address can be a range of line numbers, a regular expression, or a combination of both. If you don’t provide any addresses, the sed command will be applied to every line of input.

Types of addresses:

1. Line numbers – You can specify a specific line number (e.g., 42) to target that line. You can use $ to represent the last line of the input or $ character represents the end of line (EOL) when used in a regex.
2. Regular expressions (regex) – You can use regular expressions (e.g., /pattern/) to select lines that match a certain pattern. In short, only lines containing the pattern are edited.
3. Address ranges – You can specify a range of lines using a combination of line numbers and/or regular expressions, separated by a comma (e.g., 100,200 or /word1/,/word2/).

Examples

Consider the following data.txt file and here is the header for your information:

NAME|DOB|Location|Job Title|Salary $

Random sample data displayed using the cat command or bat command:

John Doe|1985-03-15|New York|Software Engineer|80000
Aarav Patel|1990-06-21|Mumbai|Data Analyst|88000
Jane Smith|1992-11-20|London|Data Scientist|95000
David Lee|1978-07-08|Tokyo|Project Manager|110000
Thandiwe Zulu|1993-04-03|Cape Town|Business Analyst|93000
Li Wei|2002-07-25|Shanghai|AI Researcher|86000
Priya Sharma|1987-01-14|Delhi|Software Tester|79000
Sipho Nkosi|1976-11-29|Johannesburg|IT Manager|102000
Sarah Jones|2001-05-02|Paris|Web Developer|75000
Michael Brown|1969-12-25|Sydney|System Administrator|90000
Emily Davis|1998-09-10|Berlin|UX Designer|85000
Kevin Wilson|1975-04-30|Toronto|Database Admin|100000
Jessica Garcia|2003-01-18|Rome|QA Tester|70000
Kenji Kimura|1973-02-07|Kyoto|Systems Engineer|99000
Brian Rodriguez|1982-08-22|Madrid|Network Engineer|92000
Ashley Williams|1995-06-05|Amsterdam|Frontend Developer|78000
Christopher Martinez|1972-10-12|Vienna|Security Analyst|98000
Amanda Anderson|2000-02-28|Dublin|Mobile Developer|82000
Matthew Thomas|1988-09-01|Stockholm|Cloud Architect|105000
Elizabeth Jackson|1979-11-17|Helsinki|DevOps Engineer|97000
Daniel White|2004-03-09|Copenhagen|Junior Developer|68000
Zhang Lei|1984-05-11|Beijing|Cybersecurity Expert|94000

Using sed to print (p command) text file data

The following example illustrates how to use the p (print) command, which prints a range of lines to stdout. The range is specified by a starting address followed by a comma and the ending address. For example, try to print 5 to 8 lines:
$ sed '5,8p' data.txt
The default output of sed is each line that it reads. To hide or suppress the default output, use the -n option:
$ sed -n '5,8p' data.txt

The following command prints all lines with the pattern Software, i.e., all matching lines with the word ‘Software‘ in them. Use the forward slash (/) to delimit the regular expression:
$ sed -n '/Software/p' data.txt
Outputs:Reading sed Commands From a File

John Doe|1985-03-15|New York|Software Engineer|80000
Priya Sharma|1987-01-14|Delhi|Software Tester|79000

The I flag after a regular expression makes it case-insensitive pattern:
$ sed -n '/software/Ip' data.txt

The following sed command prints the first line containing the pattern David, up to and including the next line containing the pattern Emily i.e. print line between two matching words or pattern:
$ sed -n '/David/,/Emily/p' data.txt
The following sed command display the first line containing the pattern Ashley, up through the last line of the file using $ as the last line of the input:
$ sed -n '/Ashley/,$p' data.txt
In this example, save above sed command output to a text file named ‘output.txt’ in the current directory:
$ sed -n '/Ashley/,$p' data.txt > output.txt
Verify it:
$ cat output.txt
Please note that the pattern might contain the regular expression characters used by grep command. See the following page for more info on grep regex:

• Regular expressions in grep ( regex ) with examples

Using sed to substitute text (find and replace with s command)

The sed s command allows a search and substitution operation on the text. In other words, you can find a given “word” and replace it with a “new word.” The command uses a pattern search and a literal string replacement and metacharacter expansion is done. Say, find the word vivek in /etc/passwd and replace it with a word called mr_vivek:
$ sed 's/vivek/mr_vivek/' /etc/passwd
Let us find word Software and replace with SOFTWARE_JOB:
$ sed 's/Software/SOFTWARE_JOB/' data.txt

A note about saving sed command text manipulation

There are two options. The first option to save the results of a sed command’s text manipulation to a file, you use output redirection as follows:
$ sed 'command' INPUT > OUTPUT
The > symbol redirects standard output (stdout) to a file. For, example:
$ sed 's/Software/SOFTWARE_JOB/' data.txt > output.txt
If you want to append the output to an existing file instead of overwriting it, use the >> symbol:
$ sed 's/Software/SOFTWARE_JOB/' data.txt >> output.txt
The second option for GNU/sed (the default version of sed on most Linux distros), you can use the -i option to edit the file directly in-place. This avoids the need for redirection and overwriting files. However, be cautious, as it modifies the original file. This is very useful for scripts and in your Docerkfiles:
$ sed -i 's/old_word/new_word/' file.txt
It is often a good practice to create a backup when using the -i option as follows:
$ sed -i'BAK' 's/old_word/new_word/' file.txt
For instance:
$ cp -v data.txt file.txt
$ ls -l file*
$ sed -i'.BAK' 's/Software/SOFTWARE_JOB/' file.txt
Verify it:
$ ls -l file*
$ diff file.txt file.txt.BAK

The following sed command example shows the g (global) command flag with the s (search and substitute) command, and it replaces all occurrences of the ‘old’ word/string with the ‘new’ string or word:
$ sed 's/old/new/g' input.txt
$ sed 's/Software/SOFTWARE_JOB/g' data.txt
The I flag makes it case-insensitive pattern for search and replace (s command):
$ sed 's/software/SOFTWARE_JOB/Ig' data.txt
Sometimes, when performing a search and replace, the old string may be included in the new replacement string. You can achieve this by placing an ampersand (&) in the replacement string. The position of the ampersand will determine where the old string appears within the new string. The syntax is:
$ sed 's/old/& new/g' input
In other words, the & in the replacement string of the sed s (substitute) command represents the entire matched portion of the pattern. This can be useful for adding text around or within a matched pattern without having to explicitly repeat the pattern itself. Here I’m adding * symbol around a word or pattern named ‘Vivek’:
$ echo 'Hello, Vivek'
$ echo 'Hello, Vivek' | sed 's/Vivek/*&*/'

Hello, *Vivek*

In this example, I’m prefixing $ for 1000 number:
$ echo 'The price is 1000 for new MacBook air.' | sed 's/1000/\$&/'
Outputs:

The price is $1000 for new MacBook air.

As I wrote, $ has a special meaning: “end of line.” If you want to match a literal dollar sign character within the sed, you need to escape it. The backslash (\) tells sed to treat the $ as a regular character, not as its special “end of line” metacharacter. So the sed command will find ‘old’ and replace it with ‘old new’ using ‘&’. Let us try to print salary column using egrep command:
$ grep --color -E '([0-9]+)$' data.txt
Now, I want to replace each salary number, such as 80000, as $80000:
$ sed -E 's/([0-9]+)$/\$&/g' data.txt

1. The sed command reads each line (sed ... data.txt) from the file data.txt.
2. It matches the salary (([0-9]+)$): It identifies the sequence of digits at the end of the line (which represents the salary) and stores it using extended regex.
3. Then it adds the dollar sign (\$&): It inserts a dollar sign ($) before the matched salary.
4. Outputs the modified line: It prints the modified line to screen/stdout.

A note about using shell variables within sed commands

Using shell variables within sed commands adds dynamic behavior to your text processing. You must enclose your sed command in double quotes (“) to allow the shell to expand the variable within the command. For example:

o_value=20000
sed -i'.BAK' "s/10000/$o_value/g" php.conf

This sed command will replace all occurrences of “10000” with the value of the $o_value variable which is “20000”. If you use single quotes (‘), the variable will not be expanded, and sed will try to match the literal string “$o_value“. You can use command substitution to dynamically generate the replacement text. For example:
$ sed "s/current_directory/$(pwd)/g" file1.txt
This sed command will replace “current_directory” with the output of the pwd command.

Using a different character as the delimiter

The default is / as the delimiter in substitute command. You can change the delimiter:

# Default delimiter is '/'
sed 's/OLD/NEW/' input_file
 
# Set/Change the delimiter to '_'
sed 's_OLD_NEW_' input_file

This is useful when your shell variable contains the delimiter itself. For example:

# Variable contains the delimiter used in the sed s command
bak_path="/efs/www_static_cache"
 
# This will fail as $bak_path contains '/'
sed "s/old_path/$bak_path/g" aws.nfs.config
 
# To fix this issue change the delimiter to something else
sed "s+old_path+$bak_path+g" aws.nfs.config
# OR #
sed "s_old_path_$bak_path_g" aws.nfs.config

Reading from a file for new text using the r command

The “r” command in sed stands for “read.” It allows you to read the contents of a specified file and append those contents to the current pattern space i.e. the line being processed after a matched line. In other words, instead of inserting a line of text once, you might want to repeat the procedure several times, either in the same file or across multiple files. The “r” (read) command specifies a file name, and the contents of the file are inserted into the output after the lines specified by the address. The address may be a line number or pattern combination. For example, you have foo.txt and bar.txt. You want to insert the contents of bar.txt after a line in foo.txt that contains the word “Unix.”. Here is the sed command:
$ ls -l bar.txt foo.txt
$ cat bar.txt
Outputs:

**
In Linux, every problem is solvable, and every solution is a new adventure.
**

Other file:
$ cat foo.txt
Outputs:

Unix is basically a simple operating system, but you have to be a genius to understand the simplicity.
FreeBSD is very nice.
I like macOS.
Debian is very nice for server.

Here is the sed command:
$ sed '/Unix/r bar.txt' foo.txt

Using sed to delete text

The following command deletes Lines 8 through 12 from the file:
$ sed '8,12d' data.txt

Deleting a specific line # 42:

$ sed '42d' input.txt
Deletes the 13the line and modifies input.txt using -i option:
$ sed -i '13d' input.txt
$ cat input.txt
The following command deletes any line containing the pattern ‘Mumbai’:
$ sed '/Mumbai/d' data.txt
The I flag after a regular expression makes it case-insensitive pattern, in other words match MUMBAI, Mumbai, mumbai etc all:
$ sed '/mumbai/Id' data.txt
It is possible to deletes all empty lines, too:
$ sed '/^$/d' my_file.txt
Another example using the printf command and deleting all empty lines:
$ printf "%s\n\n\n%s\n" "This is a test" "Last line"
$ printf "%s\n\n\n%s\n" "This is a test" "Last line" | sed '/^$/d'

The following command deletes any line beginning with the pattern ‘Linux’:
$ sed ’/^Linux/d’ input.txt
The following sed command deletes the range of lines beginning with the first line containing the pattern FOO, up through the next line of the file containing BAR:
$ sed '/FOO/,/BAR/d' filename.txt
Here is how to delete the first three characters of each line:
$ sed 's/^...//' input.txt
$ sed 's/^...//' input.txt > output.txt
Where sed the stream editor is used as follows. The main magic happens with 's/...//' which is the substitution command:

• s/ – Indicates the start of the substitution command.
• ^ – Matches the beginning of the line.
• ... – Matches any three characters. The . (dot) matches any single character.
• // – Replaces the matched four characters with nothing effectively deleting them.
• input.txt – Specifies the input file.
• > output.txt – Specifies the output file.

Using sed to insert and append text

The i command is used for inserting a line before a specified line. For example:
$ sed '2i\FOO line will be inserted before line 2.' data.txt
Here is how to insert multiple lines:
$ sed '5i\
FOO line\
BAR line' data.txt
The backslash escapes the newline character, allowing you to write the inserted line on the next line. You can also insert lines based on a pattern match. For instance:
$ sed '/Mumbai/i\
line will be inserted before the line containing "Mumbai".' data.txt

The a command is used for inserting a line after a specified line:

For example:
$ sed '5a\
This line will be inserted after line 5.' input.txt
$ sed '5a\
FOO line\
BAR line' input.txt
Same way, you can insert lines after a pattern match:
$ sed '/Toronto/a\
Line will be inserted after the line containing "Toronto".' data.txt

Reading sed commands from a file

Using a file to store sed commands can be very useful for complex editing tasks. The -f option in sed allows you to specify a file containing the sed commands. The syntax is:
$ sed -f script.sed input_file.txt
Multiple sed commands can be put in a file named ‘script.sed’ and executed using the -f option. When you place the commands in a file you:

1. Do not use quotes around the action and address.
2. Make sure that there is no trailing white space at the end of each line.

Let us crate a script.sed with the commands in the script that:

1. Delete the first two lines.
2. Replace any instances of Software with Software_JOB
3. Change any line that starts with Emily to Emilia

$ cat script.sed
Outputs:

1,2d
s/Software/Software_JOB/
s/^Emily/Emilia/

Run it as follows:
$ sed -f script.sed data.txt

Of course, you can save or update file in-place as follows:
$ sed -i -f sed_commands.txt input.txt
## OR ##
$ sed -f sed_commands.txt input.txt > output.txt

How to execute multiple sed commands

Try the following syntax when you want to execute multiple sed commands from the command line:
$ sed -e 'command1' -e 'command2' input_file
$ sed -e 'command1' -e 'command2' input_file > output_file
In this example, you are replacing multiple patterns using the -e CLI option:
$ sed -e 's/BSD/macOS/g' -e 's/Unix/Linux/g' input.file > output.file
This command replaces all occurrences of “BSD” with “macOS” and then all occurrences of “Unix” with “Linux” in input.file and stores output to output.file.

Using sed to write output files

The sed command itself has a w command that allows you to write specific lines or patterns to a file. For exammple:
$ sed '/pattern/w output.txt' input.txt
The w command allows a specific sed command to write the output to a given file names. Different sed commands can write to different files. For instance:
$ cat demo.sed
Outputs:

/Delhi|Mumbai/w india.office.txt
s/^Emily/Emilia/w emily.typo.txt

Run it as follows:
$ sed -E -n demo.sed data.txt
$ ls -l *.txt
$ cat india.office.txt
$ cat emily.typo.txt

How to use sed in your Dockerfile

The sed syntax is same as the CLI, say you want to edit 10000 with 20000 in /etc/env.conf while building containers and apps, you need to add the following in your Dockerfile:

RUN sed -i'.BAK' 's/^10000/20000/'  /etc/env.conf

The RUN instruction in a Dockerfile is prinary for executing commands within your Docker image during the build process. It’s used to install software, configure settings, and perform any other necessary actions to prepare your image using sed, awk and other tools. Here is another example for Dockerfile where first some variables set and then file is updated:

ENV LC_ALL=en_US.UTF-8
ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US.UTF-8
RUN sed -i "s/^# $LANG/$LANG/" /etc/locale.gen; \
    locale-gen

Using sed with shell scripts

You can simply call the sed command. The syntax is:

#!/usr/bin/evn bash
echo "Starting setup ..."
# Call sed to edit the config file
sed -i'.BAK' 'command' some_config.file
# Example:
sed -i'.factory' -e 's/;Interface ""/Interface "eth0"/g' /etc/vnstat.conf
echo "Setup done..."

In this example, I’m editing the php fpm web server file using the sed command to configure it:

#!/bin/bash
set -e
profile="$1"
 
if [ -f "$profile" ]
then
    echo "*** Using $profile file ..."
    source "$profile"
    # Config PHP
    sed -i'.factory' -e "s+listen = 127.0.0.1:9000+listen = ${php_fpm_sock_path}+" \
        -e 's/user = nobody/user = nginx/' \
        -e 's/group = nobody/group = nginx/' \
        -e 's/;listen.owner = nobody/listen.owner = nginx/' \
        -e 's/;listen.group = nobody/listen.group = nginx/' \
        -e 's/;rlimit_files = 1024/rlimit_files = 655350/' \
        -e 's/pm.max_children = 5/pm.max_children = 300/' \
        -e 's/pm.start_servers = 2/pm.start_servers = 100/' \
        -e 's/pm.min_spare_servers = 1/pm.min_spare_servers = 100/' \
        -e 's/pm.max_spare_servers = 3/pm.max_spare_servers = 200/' \
        -e 's/;pm.max_requests = 500/pm.max_requests = 500/' "$php_fpm_www_conf"
else
    echo "Error - $0 - '$profile' profile file not found. Set correct profile file."
    exit 1
fi

Conclusion

That concludes our tutorial on using sed in Linux. I strongly recommend reading the GNU sed documentation online or by typing the following info command/man command command:
$ man sed