Bash Input Output Magic

Filter and Modify

Today's Messages from /var/log/syslog

Format of date output

Filter Empty Lines and Comments

Highlight what machted

List each match

List only the names of the files where hits where found

References in sed to machted parts

use awk instead of grep

If you use grep to find lines and the awk to print columns

you can save the grep call as awk can also grep for you

Regular Expression Lookahead

If you have the following String

and want to replace the second M with a P.

Positiv Lookahead

Search for an M which is followed by an L

The difference to this one

is that the the L in the first case would not be replaced, but in the second case it would be replaced

Negativ Lookahead

Search for an M, which is not followed by an M

Search for a String which does not start with CA

See also Regular Expressions Lookaround

cut

Get the 2nd column when ; is the seperator

awk

Use different field separator for awk

Split a String with awk at a given separator

Calculate with awk

Add all values in a column

Only deal with lines which match a given condition

Only deal with lines where the first column is larger than 45

Only deal with lines beginning with line number 12

Only deal with lines where in the second column is the word "FooBar":

Define output format for awk

sed

Change Upper and Lower Case

Sort after special columns

First after the 3. one, then the 2. and finally the 6. columns (the last one numeric).

Do not only choose which column is used for sorting, but also define the column separator (here ":"):

Files

Sync Two Files (Directories)

List Files Sorted by Filesize

Get file attributes

Get file attributes which can be easily parsed

unix2dos, dos2unix

Convert file encoding from ISO to Unicode

pdf2ps (with Password)

Convert some files in a pretty looking file suitable for printing

Create a Postscript Book

find

find can spawn a new command for each file or start the command once and give it all found files

Find old files

List files which are at least 165 days old, exactly 165 days old, not older than 165 days

Find large files

List files which are at least 3MB large

Us find to find files with given file permissions

Command	Description	Will find	Will not find

# find . -perm 400<br /># find . -perm u=r	Finds files with exactly these permissions	-r--------	-r--r-----
# find . ! -perm 400<br /># find . ! -perm u=r	Finds files whose permissions are not equal to these permissions	-rwxrwxrwx	-r--------
# find . -perm -440	# find . -perm -u=r,g=r	At least these permissions have to be there	-rwxrwxrwx<br />-rw-r--r--<br />lrwxrwxrwx	-r--------<br />----r-----
# find . -perm /440<br /># find . -perm /u=r,g=r	At least one of these permissions have to be there	-rwxrwxrwx<br />-r--------<br />----r-----	--w-------

Find dangling symlinks

List all files which are linked to the given file

Pretty Print JSON

This runs with basic python be reorders elements in json

Network

Make Interactive Services Scripable with expect

Download a File and Split It on the Fly

less

View File Without Linewarp

If you do a lowercase search in less the search will ignore upper and lower case

Differences between two files

Rename files with mmv

Rename *a* to *b*:

cp -r for some files

Copy recursively some files and create the folder structure, too.

Spamassassin

Bayes: How Many Mails Have Been Learned

Bayes: List with All Learned Words and if They Were Found in a Spam Message

Send mails from the command line

ssh

Multiple ssh connections to one host share an reuse one channel

Upload files with sftp

If you can not use scp for any reason

Misc

Count from ALPHA to OMEGA

Console Calculation

Restart process and print its output for each restart

Test a string vs a regular expression

Bash Array

Define and initialise an array in the Bash shell. Afterwards various outputs of the array.

Bash Maps

(requires bash 4)

If you want to fill an array or a map via a (while) loop you can not use the ... | while read line syntax but have to use the while read line ... done < <(cat ...) syntax

Run things in parallel

Use xargs to start processes in parallel

I very often process lots of things (e.g. files) in bash commands. Sometimes that is not every fast. Either because IO is involved (so you often have to wait) or because it is CPU intensive and you only use one thread of your CPU. Surprisingly xargs can be very helpful to achieve this. Let's assume you process many things. You want to have always 8 threads running in parallel (maybe because your CPU can handle 8 threads) and speed things up, every thread you get a batch of 10 things to process. at once. For this example we generate the input with seq (which just counts) but I could be anything, files, a rows of file, output of other commands, etc.

That would start 8 instances of the script, and give each instance 10 numbers. The first 1-10, seconds 11-20, etc. As soon as one of the 8 scripts finishes, the next one is started. So you always have 8 threads running in parallel, no more, no less. Until you run out of input. If each thread should only handle one input, use -n1, if you only want to have one thread, use -P1.

This here for example zips all the files, one by one, but does 5 files in parallel

One gzip for each file, only 5 run in parallel

Get the number of CPU cores

make in parallel

Use this information to start make with multiple threads