Bash Input Output Magic
Filter and Modify
Today's Messages from /var/log/syslog
Format of date output
20081103_1759026
Filter Empty Lines and Comments
Highlight what machted
ABC'''123'''DEF'''456'''G'''H'''I
List each match
123
456
H
List only the names of the files where hits where found
References in sed to machted parts
ABC_123_DEF_456_G_H_I
ABC_223_DEF456GHI
use awk instead of grep
If you use grep to find lines and the awk to print columns
you can save the grep call as awk can also grep for you
Regular Expression Lookahead
If you have the following String
and want to replace the second M with a P.
Positiv Lookahead
Search for an M which is followed by an L
The difference to this one
is that the the L in the first case would not be replaced, but in the second case it would be replaced
Negativ Lookahead
Search for an M, which is not followed by an M
Search for a String which does not start with CA
See also Regular Expressions Lookaround
cut
Get the 2nd column when ; is the seperator
awk
Use different field separator for awk
awk 'BEGIN {FS="t"} {print $3}
Split a String with awk at a given separator
Calculate with awk
Add all values in a column
Only deal with lines which match a given condition
Only deal with lines where the first column is larger than 45
Only deal with lines beginning with line number 12
Only deal with lines where in the second column is the word "FooBar":
Define output format for awk
sed
Change Upper and Lower Case
echo fOo | tr "[:upper:]" "[:lower:]" # foo
Sort after special columns
First after the 3. one, then the 2. and finally the 6. columns (the last one numeric).
Do not only choose which column is used for sorting, but also define the column separator (here ":"):
1:1:0" | sort -t : -k 3
1:1:0
1:1:1
Files
Sync Two Files (Directories)
rsync --compress --progress --no-whole-file -rlpe ssh remote:/path/file file
List Files Sorted by Filesize
Get file attributes
Get file attributes which can be easily parsed
unix2dos, dos2unix
alias dos2unix='recode ibmpc..lat1'
Convert file encoding from ISO to Unicode
pdf2ps (with Password)
Convert some files in a pretty looking file suitable for printing
Unicode / utf-8: u2ps ...([[http://packages.debian.org/gnome-u2ps | gnome-u2ps ]])
Create a Postscript Book
find
find can spawn a new command for each file or start the command once and give it all found files
find / -exec echo '{}' ';'
Find old files
List files which are at least 165 days old, exactly 165 days old, not older than 165 days
find /foo -type f -ctime 165
find /foo -type f -ctime -165
Find large files
List files which are at least 3MB large
find /foo -size +3M
Us find to find files with given file permissions
Command | Description | Will find | Will not find |
---|
# find . -perm 400<br /># find . -perm u=r | Finds files with exactly these permissions | -r-------- | -r--r----- | |
# find . ! -perm 400<br /># find . ! -perm u=r | Finds files whose permissions are not equal to these permissions | -rwxrwxrwx | -r-------- | |
# find . -perm -440 | # find . -perm -u=r,g=r | At least these permissions have to be there | -rwxrwxrwx<br />-rw-r--r--<br />lrwxrwxrwx | -r--------<br />----r----- |
# find . -perm /440<br /># find . -perm /u=r,g=r | At least one of these permissions have to be there | -rwxrwxrwx<br />-r--------<br />----r----- | --w------- |
Find dangling symlinks
do
if [ -h "$link" -a ! -e "$link" ];
then echo "$link";
fi;
done
List all files which are linked to the given file
Pretty Print JSON
cat demo.json | json_reformat
This runs with basic python be reorders elements in json
Network
Make Interactive Services Scripable with expect
set myserver "10.0.0.1"
set myport "21"
send "Try to connect to server $myserver port $myportn"
spawn telnet $myserver $myport
expect "220 " { send "USER ftpn" }
expect "331 " { send "PASS foon" }
expect "230 " { send "PASVn" }
expect "227 " { send "HELPn" }
expect "214 " { interact }
Download a File and Split It on the Fly
less
View File Without Linewarp
If you do a lowercase search in less the search will ignore upper and lower case
Differences between two files
Rename files with mmv
Rename *a* to *b*:
cp -r for some files
Copy recursively some files and create the folder structure, too.
Spamassassin
Bayes: How Many Mails Have Been Learned
Bayes: List with All Learned Words and if They Were Found in a Spam Message
Send mails from the command line
echo "Hello world" | mutt -a file1.txt foo@example.com
ssh
Multiple ssh connections to one host share an reuse one channel
IdentityFile ~/.ssh/id_rsa
ForwardAgent yes
ControlMaster auto
ControlPath /tmp/control-%r@%h:%p
ControlPersist 18000
ServerAliveInterval 60
Upload files with sftp
If you can not use scp for any reason
lcd /home/foo/4upload/
cd /tmp/uploaded/
put Backup1.img
put Backpu2.img
quit
Misc
Count from ALPHA to OMEGA
Console Calculation
echo "1 / 3" | bc -l
Restart process and print its output for each restart
Test a string vs a regular expression
# cp -v --parents -r **/*.jpeg **/*.jpg **/*.JPG **/*.png **/*.tif **/*.gif /mnt/mnt/backup_of_pictures/
Bash Array
Define and initialise an array in the Bash shell. Afterwards various outputs of the array.
# echo ${fruits}
apple
# echo ${fruits[1]}
banana
# echo ${fruits[*]}
apple banana strawberry
# echo ${#fruits[*]}
3
# echo ${fruits[*]}
apple banana strawberry
# for ((idx=(${#fruits[*]}-1); idx>=0; idx--)); do echo ${fruits[$idx]}; done
strawberry
banana
apple
Bash Maps
(requires bash 4)
animals=( ["moo"]="cow" ["woof"]="dog")
animals['snake']='hizzz'
# Show all keys
echo "${!animals[@ ]}"
# Show all values
echo "${animals[@ ]}"
# Iterate
for sound in "${!animals[@ ]}"; do
echo "$sound - ${animals[$sound]}";
done
If you want to fill an array or a map via a (while) loop you can not use the ... | while read line syntax but have to use the while read line ... done < <(cat ...) syntax
Use xargs to start processes in parallel
One gzip process gets a long List of files to work with
One gzip for each file
One gzip for each file, all start in parallel
One gzip for each file, only 5 run in parallel
Get the number of CPU cores
6
Use this information to start make with multiple threads