Bash Input Output Magic

Filter and Modify

Today's Messages from /var/log/syslog

cat /var/log/syslog | grep ^"`date +%b %d`"
[$[Get Code]]

Format of date output

# date +'%Y%m%d_%H%M%S'
20081103_1759026
[$[Get Code]]

Filter Empty Lines and Comments

grep -ve ^ *#'|'^$ /etc/*.conf
[$[Get Code]]

Highlight what machted

echo "ABC123DEF456GHI" | grep --color [0-9H]
ABC'''123'''DEF'''456'''G'''H'''I
[$[Get Code]]

List each match

echo "ABC123DEF456GHI" | grep -E --only-matching [0-9H]+
123
456
H
[$[Get Code]]

List only the names of the files where hits where found

grep --files-with-match foo *
[$[Get Code]]

References in sed to machted parts

echo "ABC123DEF456GHI" | sed 's/[0-9H]+/_&_/g'

ABC_123_DEF_456_G_H_I
[$[Get Code]]
echo "ABC223DEF456GHI" | sed -E s/"([0-9]+)"/"_\\1_"/

ABC_223_DEF456GHI
[$[Get Code]]

use awk instead of grep

If you use grep to find lines and the awk to print columns

grep error /var/log/syslog | awk '{ print $3 }'
[$[Get Code]]

you can save the grep call as awk can also grep for you

awk '/error/ { print $3 }' /var/log/syslog
[$[Get Code]]

Regular Expression Lookahead

If you have the following String

EXAMMLE
[$[Get Code]]

and want to replace the second M with a P.

Positiv Lookahead

Search for an M which is followed by an L

M(?=L)
[$[Get Code]]

The difference to this one

ML
[$[Get Code]]

is that the the L in the first case would not be replaced, but in the second case it would be replaced

Negativ Lookahead

Search for an M, which is not followed by an M

M(?!M)
[$[Get Code]]

Search for a String which does not start with CA

^(?!CA)
[$[Get Code]]

See also Regular Expressions Lookaround

cut

Get the 2nd column when ; is the seperator

echo "a;bc;d" | cut -d';' -f2
[$[Get Code]]

awk

Use different field separator for awk

awk 'BEGIN {FS=" - "} {print $1 $2 $3}'
awk 'BEGIN {FS="t"}  {print $3}
[$[Get Code]]

Split a String with awk at a given separator

echo foo,bar,test | awk 'BEGIN {FS=","} {split($0,a,","); for (i=1; i<=NF; i++) print a[i]; }'
[$[Get Code]]

Calculate with awk

echo 42 11 | awk '{print $1*1000, $2}'
[$[Get Code]]

Add all values in a column

cat foo | awk '{mycol2 += $2; mycol3 += $3}; END {print mycol2":"mycol3}'
[$[Get Code]]

Only deal with lines which match a given condition

Only deal with lines where the first column is larger than 45

cat foo | awk '$1>45 {print $0}'
[$[Get Code]]

Only deal with lines beginning with line number 12

cat foo | awk 'NR>11 { print $0 }'
[$[Get Code]]

Only deal with lines where in the second column is the word "FooBar":

cat foo | awk '$2~/FooBar/ {print $0}'
[$[Get Code]]

Define output format for awk

echo 1 2 | awk '{ printf("%.3d %sn", $1, $2)  }'
[$[Get Code]]

sed

echo Cool | sed s/c/T/i
[$[Get Code]]

Change Upper and Lower Case

echo fOo | tr "[:lower:]" "[:upper:]" # FOO
echo fOo | tr "[:upper:]" "[:lower:]" # foo
[$[Get Code]]

Sort after special columns

First after the 3. one, then the 2. and finally the 6. columns (the last one numeric).

sort -k 3,3 -k 2,2 -k 6,6n
[$[Get Code]]

Do not only choose which column is used for sorting, but also define the column separator (here ":"):

# echo "1:1:1
1:1:0" | sort -t : -k 3

1:1:0
1:1:1
[$[Get Code]]

Files

Sync Two Files (Directories)

/usr/bin/rsync --rsync-path=/usr/local/bin/rsync -rlpe ssh QUELLE ZIEL
rsync --compress --progress --no-whole-file -rlpe ssh remote:/path/file file
[$[Get Code]]

List Files Sorted by Filesize

ls -lS
[$[Get Code]]

Get file attributes

Get file attributes which can be easily parsed

stat myfile.txt
[$[Get Code]]

unix2dos, dos2unix

alias unix2dos='recode lat1..ibmpc'
alias dos2unix='recode ibmpc..lat1'
[$[Get Code]]

Convert file encoding from ISO to Unicode

iconv --from-code=ISO-8859-1 --to-code=UTF-8 ./oldfile.htm > ./newfile.html
[$[Get Code]]

pdf2ps (with Password)

gs -sPDFPassword=password -sDEVICE=pswrite -dNOPAUSE -sOutputFile=out.ps in.pdf
[$[Get Code]]

Convert some files in a pretty looking file suitable for printing

a2ps --medium A4dj -E -o /tmp/pretty.ps A B C D ...
Unicode / utf-8: u2ps ...([[http://packages.debian.org/gnome-u2ps | gnome-u2ps ]])
[$[Get Code]]

Create a Postscript Book

psbook normal.ps | psnup -2 -m25 > book.ps
[$[Get Code]]

find

find / -printf "%CY%Cm%Cd%CI%CM | %d | %i | %s | %h | %fn"
[$[Get Code]]

find can spawn a new command for each file or start the command once and give it all found files

find / -exec echo  '{}' +
find / -exec echo  '{}' ';'
[$[Get Code]]

Find old files

List files which are at least 165 days old, exactly 165 days old, not older than 165 days

find /foo -type f -ctime +165
find /foo -type f -ctime  165
find /foo -type f -ctime -165
[$[Get Code]]

Find large files

List files which are at least 3MB large

find /foo -size +3000000c
find /foo -size +3M
[$[Get Code]]

Us find to find files with given file permissions

CommandDescriptionWill findWill not find
# find . -perm 400<br /># find . -perm u=rFinds files with exactly these permissions-r---------r--r-----
# find . ! -perm 400<br /># find . ! -perm u=rFinds files whose permissions are not equal to these permissions-rwxrwxrwx-r--------
# find . -perm -440# find . -perm -u=r,g=rAt least these permissions have to be there-rwxrwxrwx<br />-rw-r--r--<br />lrwxrwxrwx-r--------<br />----r-----
# find . -perm /440<br /># find . -perm /u=r,g=rAt least one of these permissions have to be there-rwxrwxrwx<br />-r--------<br />----r-------w-------

Find dangling symlinks

find /tmp/tg1/ -type l | while read link
do
 if [ -h "$link" -a ! -e "$link" ];
  then echo "$link";
 fi;
done
[$[Get Code]]
find /tmp/tg1/ -printf '%Y %p n' | grep ^N  | sed s/^N //
[$[Get Code]]

List all files which are linked to the given file

find /mnt/mnt/ -L -samefile /mnt/mnt/bin/foo
[$[Get Code]]

Pretty Print JSON

sudo apt-get install yajl-tools
cat demo.json | json_reformat
[$[Get Code]]

This runs with basic python be reorders elements in json

cat demo.json | python -mjson.tool
[$[Get Code]]

Network

Make Interactive Services Scripable with expect

#!/usr/bin/expect
set myserver "10.0.0.1"
set myport   "21"
send "Try to connect to server $myserver port $myportn"
spawn telnet $myserver $myport
expect   "220 " { send "USER ftpn" }
expect   "331 " { send "PASS foon" }
expect   "230 " { send "PASVn" }
expect   "227 " { send "HELPn" }
expect   "214 " { interact }
[$[Get Code]]

Download a File and Split It on the Fly

wget ftp://... -nv -O - | split ....
[$[Get Code]]

less

View File Without Linewarp

less -S
[$[Get Code]]

If you do a lowercase search in less the search will ignore upper and lower case

export LESS="-i"
[$[Get Code]]

Differences between two files

diff -ubB FOO BAR
[$[Get Code]]

Rename files with mmv

Rename *a* to *b*:

mmv "*a*" "#1b#2"
[$[Get Code]]

cp -r for some files

Copy recursively some files and create the folder structure, too.

Spamassassin

Bayes: How Many Mails Have Been Learned

sa-learn --dump | grep non-token data: | grep nspam$'|'nham$
[$[Get Code]]

Bayes: List with All Learned Words and if They Were Found in a Spam Message

sa-learn --dump | grep -v non-token data: | sort -n
[$[Get Code]]

Send mails from the command line

(echo "Hello world"; uuencode file1.txt ReadThis.txt; uuencode file2.txt AndThis.txt) | mailx -s subject foo@example.com

echo "Hello world" | mutt -a file1.txt foo@example.com
[$[Get Code]]

ssh

Multiple ssh connections to one host share an reuse one channel

Host *
IdentityFile ~/.ssh/id_rsa
ForwardAgent yes
ControlMaster auto
ControlPath /tmp/control-%r@%h:%p
ControlPersist 18000
ServerAliveInterval 60
[$[Get Code]]

Upload files with sftp

If you can not use scp for any reason

sftp -o "batchmode no"  -b meinebefehle.sftp user@computer
[$[Get Code]]
meinebefehle.sftp:
lcd /home/foo/4upload/
cd  /tmp/uploaded/
put Backup1.img
put Backpu2.img
quit
[$[Get Code]]

Misc

Count from ALPHA to OMEGA

seq ALPHA OMEGA
[$[Get Code]]

Console Calculation

echo "1 + 2" | bc
echo "1 / 3" | bc -l
[$[Get Code]]

Restart process and print its output for each restart

watch -n3 cat /proc/mdstat
[$[Get Code]]

Test a string vs a regular expression

expr FooBar-5.2 : ".*-5.2"
[$[Get Code]]
# shopt -s globstar
# cp -v --parents -r **/*.jpeg **/*.jpg **/*.JPG **/*.png **/*.tif **/*.gif /mnt/mnt/backup_of_pictures/
[$[Get Code]]

Bash Array

Define and initialise an array in the Bash shell. Afterwards various outputs of the array.

# fruits=(apple banana strawberry)

# echo ${fruits}
apple

# echo ${fruits[1]}
banana

# echo ${fruits[*]}
apple banana strawberry

# echo ${#fruits[*]}
3

# echo ${fruits[*]}
apple banana strawberry

# for ((idx=(${#fruits[*]}-1); idx>=0; idx--)); do echo ${fruits[$idx]}; done
strawberry
banana
apple
[$[Get Code]]

Bash Maps

(requires bash 4)

declare -A animals
animals=( ["moo"]="cow" ["woof"]="dog")
animals['snake']='hizzz'

# Show all keys
echo "${!animals[@ ]}"

# Show all values
echo "${animals[@ ]}"

# Iterate
for sound in "${!animals[@ ]}"; do
 echo "$sound - ${animals[$sound]}";
done
[$[Get Code]]

If you want to fill an array or a map via a (while) loop you can not use the ... | while read line syntax but have to use the while read line ... done < <(cat ...) syntax

Use xargs to start processes in parallel

One gzip process gets a long List of files to work with

ls *.txt | xargs gzip
[$[Get Code]]

One gzip for each file

ls *.txt | xargs -n1 gzip
[$[Get Code]]

One gzip for each file, all start in parallel

ls *.txt | xargs -n1 gzip &amp;
[$[Get Code]]

One gzip for each file, only 5 run in parallel

ls *.txt | xargs -n1 -P5 gzip
[$[Get Code]]

Get the number of CPU cores

# nproc
6
[$[Get Code]]

Use this information to start make with multiple threads

# make -j $(nproc)
[$[Get Code]]