Sed

Sed is a *nix system utility that will come with 99% of *nix systems. It's an in-place string manipulation program that can come in handy to make a whole lot of typing into a few lines of string manipulation. It can get ugly, but once you start to use it you'll wonder how you ever lived without it.

Sed introduction and tutorial: http://www.grymoire.com/Unix/Sed.html

Editing Files In-Place

Sed can be used to edit files in-place using the -i flag.

Find and Replace

You can find and replace instances of a string in a file using:

$ sed -i -e 's/peanut butter/jelly/g' file{1,2,3}.txt

This replaces peanut butter with jelly in file1.txt, file2,txt, and file3.txt. To replace more than one thing, use

$ sed -i -e 's/peanut butter/jelly/g' \
         -e 's/green eggs/ham/g'      \
         -e 's/water/wine/g'          \
         file{1,2,3}.txt

or, more succinctly:

$ sed -i -e 's/peanut butter/jelly/g;s/green eggs/ham/g' \
         file{1,2,3}.txt

Special/Escape Characters

Main article: Regular Expressions

NOTE: This section is specific to GNU sed, other versions of sed will likely behave differently.

Sometimes you want to look for generic patterns, like "four numbers in a row", rather than something specific, like "5555". This can be done using special/escape characters.

Numerical Characters

To match any number between 0 and 9, use [0-9], like this:

$ echo "5" | sed -e 's/[0-9]/replacement/'
replacement

To match a pattern of N numbers between 0 and 9, use \{N\}, like this:

$ echo "5678" | sed -e 's/[0-9]\{4\}/replacement/'
replacement

If you want to match a pattern of numbers between 0 and 9, and know there will be somewhere between M and N numbers, you can use the syntax \{M,N\}. For example, if you want to replace a number between 2 and 4 digits long:

$ echo "56" | sed -e 's/[0-9]\{2,4\}/replacement/'
replacement

$ echo "5234678" | sed -e 's/[0-9]\{2,4\}/replacement/'
replacement678

$ echo "5" | sed -e 's/[0-9]\{2,4\}/replacement/'
5

Note that in the last command executed, the replacement pattern doesn't show up because the largest pattern of numbers between 0 and 9 is 1, which does not fall in the range of 2 to 4.

Since \{M,N\} is ugly and burdensome to type, you can use the sed flag -r or --regexp-extended to eliminate the need for backslashes:

$ echo "5234678" | sed -e 's/[0-9]\{2,4\}/replacement/'
replacement678

$ echo "5234678" | sed -re 's/[0-9]{2,4}/replacement/'
replacement678

To leave the upper bound of the number size unspecified, use \{N,\}:

$ echo "52" | sed -re 's/[0-9]{2,}/replacement/'
replacement

$ echo "5234678" | sed -re 's/[0-9]{2,}/replacement/'
replacement

$ echo "5223902949082309448792387234" | sed -re 's/[0-9]{2,}/replacement/'
replacement

Sed Commands

Less Common Commands

w command

To search for a pattern, and print the resulting pattern to a file, use the w command:

$ cat list_file 
Phoenix
New York City
San Francisco
Orlando
Atlanta
Seattle
San Antonio
St. Louis

$ sed -n '/San/w search_results' list_file

$ cat search_results 
San Francisco
San Antonio

e command

To output the results of a command into a new line, the e command can be used. For example, the contents of a small file (called small_file in this example) could be inserted into a line of the file test_file:

$ cat new_item
Boston

$ sed '/New York/e cat new_item' list_file
Phoenix
Boston
New York City
San Francisco
Orlando
Atlanta
Seattle
San Antonio
St. Louis

The new line, created from the output of the command cat new_item, is inserted in a new line, above the line matching the search pattern.

References

• This page has more information on special/escape characters: http://sed.sourceforge.net/sedfaq6.html

• One-line sed commands: http://sed.sourceforge.net/grabbag/tutorials/sed1line.txt

• O'Reilly Sed/Awk book
  • Chapter 5: Basic sed commands http://docstore.mik.ua/orelly/unix/sedawk/ch05_01.htm
  • Chapter 6: Advanced sed commands http://docstore.mik.ua/orelly/unix/sedawk/ch06_01.htm

• Sed info file: http://www.gnu.org/software/sed/manual/sed.html#Common-Commands