Converts the symbolic permissions to octal (ie: numbers) when using 'ls -l':

$ls -l | sed -e 's/--x/1/g' -e 's/-w-/2/g' -e 's/-wx/3/g' -e 's/r--/4/g' \

-e 's/r-x/5/g' -e 's/rw-/6/g' -e 's/rwx/7/g' -e 's/---/0/g'

-755 1 jrl jrl 111943 2003-10-21 19:57 logscan

-644 1 jrl jrl 35468 2003-11-23 16:13 htfoo
-700 1 jrl jrl 3100672 2004-05-15 17:00 mutt
-644 1 jrl jrl 10162 2005-02-22 14:14 joinstep2.php
-777 1 jrl jrl 41079 2005-04-21 13:02 setistats
d755 2 jrl jrl 47 2007-10-26 14:41 rf
-700 1 jrl jrl 104 2008-02-05 11:26 getc

Add the following alias and function to your profile to be able to copy and paste
files at the command line:

ccopy(){ cp $1 /tmp/ccopy.$1; }
alias cpaste="ls /tmp/ccopy* | sed 's|[^\.]*.\.||' | xargs -I % mv /tmp/ccopy.% ./
%"

You can see below how this can be used:

blackbird:~/tst tks1$ ls
1.txt 2.txt t1.tss t2.tss t3.tss
blackbird:~/tst tks1$ ccopy 1.txt
blackbird:~/tst tks1$ ccopy 2.txt

blackbird:~/tst tks1$ cd ../tst2

blackbird:~/tst2 tks1$ ls

blackbird:~/tst2 tks1$ cpaste

blackbird:~/tst2 tks1$ ls
1.txt 2.txt

Stream YouTube videos directly to your media player

Tags: sed
Save this to del.icio.usDigg this!Reddit this!Tweet this!
YouTube is great, but the embedded flash player doesn't offer a lot in the way of
customizing your viewing experience. There are quite a few options for downloading
flv video files from YouTube, but adding the following function to your bashrc will
let you stream them directly to your choice of media player.

function mtube {
video_id=$(curl -s $1 | sed -n "/watch_fullscreen/s;.*\(video_id.\
+\)&title.*;\1;p");
mplayer -fs $(echo "http://youtube.com/get_video.php?$video_id");
}

This should work with any URL of the form http://www.youtube.com/watch?v=[video id

here], for example:

mtube http://www.youtube.com/watch?v=evQ0QNi2CzA

Add commas to all numeric strings in a file, changing "1234567" to "1,234,567"

sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta' filename

To create a set of backed up files with the current date added at the end of the
file name try the following:

ls *txt | sed "s/.*/cp & &.$(date "+%Y%d%m")/"

This will run the following commands:

cp 1.txt 1.txt.20082703

cp 2.txt 2.txt.20082703

cp 3.txt 3.txt.20082703

The following alias will print the directory structure from the current directory
in tree format.

alias dirf='find . -type d | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/"'

Selective content replace on files. For example to replace '<?' with '<?php' in all
PHP files:
find . -name '*.php' -exec sed -ie 's#<?#<?php#' {} \;

Mass-renaming files using find and sed:

find -name "*.php3" | sed 's/\(.*\).php3$/mv "&" "\1.php"/' | sh

(this example will rename all .php3 files to .php)

Prefer Perl over Sed? No problem! To use Perl as a Sed-like program:

perl -pe 's/foo/bar/; etc'

Have a Perl script ready? No problem!

perl -p foo.pl

Add the following sed commands to cal to get a calendar with the current date
marked:

cal | sed "s/^/ /;s/$/ /;s/ $(date +%e) / $(date +%e | sed 's/./#/g') /"
sed '/ *#/d; /^ *$/d' file

Remove comments and blank lines from file

The following command creates in the /usr/project directory, a copy of the current
working directory structure:

find . -type d -print|sed 's@^\.\{0,1\}@/usr/project@' | sed 's/ /\\ /' | xargs

mkdir -p

Using sed across multiple lines

Tags: sed
Save this to del.icio.usDigg this!Reddit this!Tweet this!
Sometimes when using sed you find that you need to match across line endings, this
can be achieved by getting sed to match the first line and then pulling a second
line into the buffer with the N command.

For example, if we have a file:

$ cat foo
This is a sample hello
world file.

And want to change 'hello world' to 'hello shell-fu' we need to replace across
lines. This can be done with the following command:
:~$ cat foo | sed '/hello$/N;s/hello\nworld/hello\nshell-fu/'
This is a sample hello
shell-fu file.

Here sed first looks for lines which end with 'hello' then reads the next line,
finally replacing 'hello\nworld' with 'hello\nshell-fu'.

This also has a lot of other uses, for example converting double line spaced files
to single:
cat doublespace | sed '/^$/N;s/\n$//g'

/ Print part of a file

Tags: sed
Save this to del.icio.usDigg this!Reddit this!Tweet this!
Print the contents of a file from a given regular expression to another

sed -n '/start/,/end/ p' file

This will print the contents of the file from the line that matches /start/ until
the line that matches /end/
 Number each line of a file
Tags: sed
Save this to del.icio.usDigg this!Reddit this!Tweet this!

sed = filename | sed 'N;s/\n/\t/' > filename.numbered

Convert mac addresses such as 000000abde00 into 00:00:00:ab:de:00

awk '{for(i=10;i>=2;i-=2)$0=substr($0,1,i)":"substr($0,i+1);print}' macaddress_list

sed 's/\(..\)/\1:/g;s/:$//' macaddress_list

Check low space

Tags: sed
Save this to del.icio.usDigg this!Reddit this!Tweet this!
Mail somebody about space running low in some path (ksh, bash):

PATHS="/export/home /home"

AWK=/usr/bin/awk

DU="/usr/bin/du -ks"

GREP=/usr/bin/grep

DF="/usr/bin/df -k"

TR=/usr/bin/tr

SED=/usr/bin/sed

CAT=/usr/bin/cat

MAILFILE=/tmp/mailviews$$

MAILER=/bin/mailx

mailto="all@company.com"

for path in $PATHS

do

DISK_AVAIL=`$DF $path | $GREP -v "Filesystem" | $AWK '{print $5}'|$SED 's/

%//g'`

if [ $DISK_AVAIL -gt 90 ];then

echo "Please clean up your stuff\n\n" > $MAILFILE

$CAT $MAILFILE | $MAILER -s "Clean up stuff" $mailto

fi
done

Command examples from shell-fu

Tags: sed
Save this to del.icio.usDigg this!Reddit this!Tweet this!
The following isn't particularly pretty and should be considered a work in
progress, but it's quite fun.

Get examples of ways a command can be used direct from shell-fu by adding the
following alias:
function examples { lynx -width=$COLUMNS -nonumbers -dump "http://www.shell-
fu.org/lister.php?tag=$1" | \
sed -n '/^[a-zA-Z]/,$p' | egrep -v '^http|^javas|View Comm|HIDE|] \+|to Share|^
+\*|^ +[HV][a-z]* l|^ .*efu.*ep.*!$' | \
sed -e '/^ *__*/N;s/\n$//g' | less -r; }

This pulls out the tips tagged by the given command. (Make sure you tag any tips
you submit!)

Print until no match

Tags: sed
Save this to del.icio.usDigg this!Reddit this!Tweet this!
This bit of sed will print the contents of a file until the first line which
doesn't contain the specified expression. A useful alternative to 'head' when
you're not sure how much of the file you need.

sed -n '/Hello/!q; p'

Use the following command to give a history listing without the numbers for easier
copy and pasting:

history | sed 's/^[ 0-9]* //'

Make a whole directory tree with one command

Tags: mkdir
Save this to del.icio.usDigg this!Reddit this!Tweet this!
Use the -p option to mkdir and make all parent directories along with their
children in a single command.

mkdir -p tmp/a/b/c

Use && to run a second command if and only if a first command succeeds:

cd tmp/a/b/c && tar xvf ~/archive.tar

Grepping for processes

Tags: grep ps
Save this to del.icio.usDigg this!Reddit this!Tweet this!
Grepping for a process will return the grep command, this can be avoided by adding
'| grep -v grep' to a command or easier in some cases altering the regular
expression by adding brackets around a character.

ps | grep 'ss[h]'

The regular expression 'ss[h]' matches the literal string 'ssh' when it appears in
the process list, but does not accidentally match the string 'ss[h]' when it
appears in the process list as 'grep ss[h]'.

It can be useful to not only know what has gone to stdout and stderr but also where
they occurred with respect to each other.
Allow stderr to go to err.txt, stdout to out.txt and both to mix.txt

((./program 2>&1 1>&3 | tee ~/err.txt) 3>&1 1>&2 | tee ~/out.txt) > ~/mix.txt 2>&1

Running a second command with the same arguments as the previous command, use '!*'
to repeat all arguments or '!:2' to use the second argument. '!$' uses the final
argument.

$ cd /home/user/foo

cd: /home/user/foo: No such file or directory

$ mkdir !*

mkdir /home/user/foo

Use awk to change the file extension for a group of files. For example to change
all .htm files to .php:

ls *htm | awk -F. '{print "mv "$0" "$1".php"}' | sh

This can be tested first by leaving the '| sh' off to give a list of the commands
that will be executed.

Because wireshark is big and nc is small.

Tags: redirection tee nc
Save this to del.icio.usDigg this!Reddit this!Tweet this!
mknod backpipe p; while nc -l -p 80 0<backpipe | tee -a inflow | \
nc localhost 81 | tee -a outflow 1>backpipe; do echo \"restarting\"; done

Listen on localhost:80, forward to localhost:81 and log both sides of the

conversation to outflow, automatically restarting if the connection dies.
multiple command output into a single program:

diff -u <(ls -c1 dir_1) <(ls -c1 dir_2)

Will show you a diff of files in the root of dir_1 and dir_2

network copy with ssh and tar

Tags: tar redirection
Save this to del.icio.usDigg this!Reddit this!Tweet this!
You can use ssh in conjunction with tar to pull an entire directory tree from a
remote machine into your current directory:

ssh <username@sourcehost> tar cf - -C <sourcedir> . | tar xvf -

For example, let's say you have a "bsmith" account on a host called "apple". You
want to copy those files into your "bobsmith" account on a host called "pear".
You'd log into your "bobsmith@pear" account and type the following:

ssh bsmith@apple tar cf - -C /home/bsmith . | tar xvf -

This technique is useful when you have insufficient disk space on the source
machine to make an intermediate tarball.

Save this to del.icio.usDigg this!Reddit this!Tweet this!

Strip the header line from a file:

tail +2 file

Subtract one file from another

Tags: grep
Save this to del.icio.usDigg this!Reddit this!Tweet this!
Subtract a small file from a bigger file.

grep -vf filesmall filebig

Normalize gain for all mp3 files in a directory tree

Tags: No tags
Save this to del.icio.usDigg this!Reddit this!Tweet this!
find . -type d | while read dir; do mp3gain -a "$dir/"*.mp3;done

Find function declarations in PHP or JavaScript

Tags: grep
Save this to del.icio.usDigg this!Reddit this!Tweet this!
function findfunc {
grep -R $1 * 2> /dev/null |grep function|grep -v .svn
}
Find files greater than a certain size
Tags: bash
Save this to del.icio.usDigg this!Reddit this!Tweet this!
find . -size +3000k -print

Process files whose names may contain spaces

Tags: No tags
Save this to del.icio.usDigg this!Reddit this!Tweet this!
If the names of some of the files in a directory may contain spaces, combine find's
"-print0" option with xargs' "-0" option:

# this will behave incorrectly if some files or directories have spaces:

find . -type f | xargs ls -l

# this will work correctly:

find . -type f -print0 | xargs -0 ls -l

swap files
Tags: bash
Save this to del.icio.usDigg this!Reddit this!Tweet this!
# Swap 2 files/dirs that are in the same dir
# Usage: sw f1 f2
function sw {
f1=$1
f2=$2
if [ "x$f2" = "x" ]; then
echo "Usage: sw file1 file2"
echo " swap name of 2 files"
else
d1=$(dirname $f1)
d2=$(dirname $f2)
if [ "$d1" != "." -o "$d2" != "." ]; then
echo "sw: Can swap only files in current directory"
else
if [ -e "$f1" -a -e "$f2" ]; then
mv $f1 .sw.$f1
mv $f2 $f1
mv .sw.$f1 $f2
else
echo "sw: '$f1' and '$f2' must exist"
fi
fi
fi
}
 Find and Grep
Tags: bash grep find
Save this to del.icio.usDigg this!Reddit this!Tweet this!
Find all files with given name (you can use Bash expansion if you'd like), and Grep
for a phrase:
find . -name -exec grep "phrase" {} \;

To display the filename that contained a match, use -print:

find . -name -exec grep "phrase" {} \; -print

Or, use Grep options to print the filename and line number for each match:
find . -name -exec grep -Hn "phrase" {} \;

The string `{}` is replaced by the current filename being processed everywhere it
occurs in the arguments to the command. See the `find` man page for more
information.

Finding Newer Files [OR How To Create A Patch File]

Tags: tar
Save this to del.icio.usDigg this!Reddit this!Tweet this!
You can use find with the '-newer' flag in conjunction with tar to create a patch
file:
tar -czvf patch-20070321.tar `find /path/to/project/ -newer /path/to/project/last-
archive.tgz -print`

In this example 'last-archive.tgz' is the last tarball for the given project.
-newer finds files newer that than last-archive.tgz, this way you can tar up only
the changed files.

Port forwarding that automatically disconnects

Tags: No tags
Save this to del.icio.usDigg this!Reddit this!Tweet this!
If you want to tunnel a long-lived connection over SSH such that the tunnel goes
away when the application disconnects, try something like the following example:

ssh -f -q -L 5900:localhost:5900 user@remotehost.com sleep 60

By executing "sleep 60" remotely, the tunnel stays alive for at least 60 seconds,
and assuming your application has connected by then, the tunnel will continue to
stay alive until the application disconnects.

The options given above are perfect for executing this command from a script; it is
quiet (-q) and goes to background after prompting for a password (-f). This
particular example forwards the VNC protocol so that when your VNC client connects
to localhost, it connects securely to remotehost.com over the tunnel.
 ssh proxy forwarding
Tags: bash alias
Save this to del.icio.usDigg this!Reddit this!Tweet this!
ssh -l <login> -L <port>:<destination:port> <proxymachine> <local addy>
example
ssh -l foo -L 5000:192.168.5.2:443 192.168.1.1 https://localhost:5000/

Then go to https://localhost:<port>/ to get to destination's website; through the

proxy machine.

Make an iso image of cd

Tags: dd
Save this to del.icio.usDigg this!Reddit this!Tweet this!
dd if=/dev/cdrom of=cd.iso

Output a video in ASCII format

Tags: No tags
Save this to del.icio.usDigg this!Reddit this!Tweet this!
mplayer -vo caca filename.avi

Directories and its size

Tags: bash du find
Save this to del.icio.usDigg this!Reddit this!Tweet this!
which directories and trees take up all the diskspace?
du -sm $(find /start/dir/* -type d -maxdepth 1 -xdev) | sort -g

If you want more human readable output try:

du -ha /var | sort -n -r | head -n 10

you want to see ALL directories in the tree

find $1 -type d | xargs du -sm | sort -g

To show all directories size including sub directories, type

du -h

To calculate the current directory size you are in (-s stand for summary)

du -sh

To show all the 1 level sub directories size (which you are not interested at sub
sub directories.)

du -sh *

To show the size of specific directory

du -sh /home

To show the size of all sub directories of a specific directory

du -sh /home/*

Find and replace with backup

Tags: bash perl
Save this to del.icio.usDigg this!Reddit this!Tweet this!
Find and replace recursively over several files:
perl -pi.bak -e "s/Bob/Steve/gi" *.html

The '.bak' will create copies of your original files with the .bak extension added
incase of mistakes. Be careful of running this twice though as the backups will get
overwritten.

Count files by type

Tags: awk sort find xargs
Save this to del.icio.usDigg this!Reddit this!Tweet this!
To find out the number of files of each type in your current directory try the
following:

find ${*-.} -type f | xargs file | awk -F, '{print $1}' | awk '{$1=NULL;print $0}'
| sort | uniq -c | sort -nr

5 PHP script text

2 data
2 Zip archive data
2 GIF image data
1 PNG image data

(You may want to add this as an alias rather than type it in each time!)

Using comm
Tags: diff perl comm
Save this to del.icio.usDigg this!Reddit this!Tweet this!
You can use diff to see the differences between two files, but it can be useful to
see what is the same and more clearly how they differ. This is where comm comes in
useful.

comm tells you what information is common to two lists and what information appears
uniquely in one or the other.

$ find . -type f -print -exec cat {} \;

./1.txt
a
b
c
./2.txt
a
c
e

$ comm 1.txt 2.txt

a
b
c
e

The first column shows lines only in the first file, the second column lines from
the second file and the third column lines from both.

This can be made easier still by adding a bit of perl:

$ comm 1.txt 2.txt | perl -pe 's/^/1: /g;s/1: \t/2: /g;s/2: \t/A: /g;' | sort
1: b
2: e
A: a
A: c

Remove every file but one

Tags: grep rm
Save this to del.icio.usDigg this!Reddit this!Tweet this!
It's easy to remove (or copy, move etc.) all files that match a given criteria, but
harder to move all but ones excluded by a criteria.

To do this we can combine grep's -v option with Unix command substitution:

$ ls
1.txt 2.txt 3.txt 4.txt
$ rm `ls | grep -v 4\.txt`
$ ls
4.txt

dos2unix all files in a directory

Tags: find
Save this to del.icio.usDigg this!Reddit this!Tweet this!
dos2unix requires the name of an input and output file so it can be hard to run on
a list of files. The following gets around this and will run dos2unix on all files
in a directory:

for f in `find * -type f`; do echo "dos2unix $f $f"; done | sh

Find files by modified time

Tags: find
Save this to del.icio.usDigg this!Reddit this!Tweet this!
When specifying time with find options such as -mmin (minutes) or -mtime (24 hour
periods, starting from now), you can specify a number n to mean exactly n, -n to
mean less than n, and +n to mean more than n. 2 For example:

find . -mtime -1 # find files modified within the past 24 hours

find . -mtime 1 # find files modified between 24 and 48 hours ago
find . -mtime +1 # find files modified more than 48 hours ago
find . -mmin +5 -mmin -10 # find files modifed between 5 and 10 minutes ago

/ Pushd and Popd

Tags: pushd popd
Save this to del.icio.usDigg this!Reddit this!Tweet this!
Use pushd and popd to store directories to go back to later:

Blackbird:~ flsw$ pwd

/Users/flsw
Blackbird:~ flsw$ pushd tst #current dir becomes tst
~/tst ~
Blackbird:~/tster flsw$ pushd #no argument moves back and forth
~ ~/tster

popd +n removes n entries from the stack without changing directory

pushd +n 'rotates' the stack so the nth directory moves to the top becoming the
current directory.

Store a directory name to come back to

Tags: bash csh
Save this to del.icio.usDigg this!Reddit this!Tweet this!
Put the following alias in your profile to store a directory name to use later on:
alias sd='set \!:1=$cwd'

Example usage:
$ sd mydir
...
$ cd mydir

This method works for csh but not bash. For bash try the following function in your
profile:
sd(){ export $1=$PWD; }

Then:
$ sd abc
...
$ cd $abc

Find matching programs

Tags: test
Save this to del.icio.usDigg this!Reddit this!Tweet this!
Sometimes you want to find a program without knowing the full name. This can be
done with the following one liner:

IFS=: ; for D in $PATH; do for F in $D/*PATTERN*; do [ -x $F ] && echo $F; done;

done

For example:
$ IFS=: ; for D in $PATH; do for F in $D/*text*; do [ -x $F ] && echo $F; done;
done
/usr/bin/gettext
/usr/bin/glib-gettextize
/usr/bin/gnome-text-editor
/usr/bin/xgettext

Run commands on logout

Tags: No tags
Save this to del.icio.usDigg this!Reddit this!Tweet this!
If a file named $HOME/.logout (a file named .logout in your home directory) exists,
and the following trap statement is in your .profile, .logout is executed when you
logout.

Add this to .profile:

trap "$HOME/.logout" 0

Actively Monitor a File

Tags: awk tail
Save this to del.icio.usDigg this!Reddit this!Tweet this!
This is a way to monitor "/var/log/messages" or any file for certain changes.
The example below actively monitors "stuff" for the word "now" and as soon as "now"
is added to the file, the contents of msg are sent by email

$ tail -f stuff | awk ' /now/ { system("mail -s \"Now Occured\" mail@foo.com <
msg") }'

Open port
Tags: No tags
Save this to del.icio.usDigg this!Reddit this!Tweet this!
iptables -A INPUT -p tcp -i eth0 --dport X -j ACCEPT

Deleting difficult filenames

Tags: find xargs rm
Save this to del.icio.usDigg this!Reddit this!Tweet this!
To delete a file who's file name is a pain to define (eg. ^H^H^H) find it's inode
number with the command "ls -il". Use the line below to find and delete the file.
find . -inum 12345 | xargs rm

Make a backup of existing files, afterwards copy new files from somedir:

1. Go to proddir
ls /update-200805/ |xargs -n1 -I xxx cp xxx xxx.`date +%Y%m%d` ; cp /update-
200805/* .

List non-system users

Tags: bash awk grep alias sort cut
Save this to del.icio.usDigg this!Reddit this!Tweet this!
I use the following to list non-system users. It should be portable though won't
work on systems without the getent command.

alias lsusers='getent passwd | tr ":" " " | awk "\$3 >= $(grep UID_MIN
/etc/login.defs | cut -d " " -f 2) { print \$1 }" | sort'

Get an ordered list of subdirectory sizes

Tags: awk sort du
Save this to del.icio.usDigg this!Reddit this!Tweet this!
This piece of code lists the size of every file and subdirectory of the current
directory, much like du -sch ./* except the output is sorted by size, with larger
files and directories at the end of the list. Useful to find where all that space
goes.

du -sk ./* | sort -n | awk 'BEGIN{ pref[1]="K"; pref[2]="M"; pref[3]="G";} { total

= total + $1; x = $1; y = 1; while( x > 1024 ) { x = (x + 1023)/1024; y++; }
printf("%g%s\t%s\n",int(x*10)/10,pref[y],$2); } END { y = 1; while( total > 1024 )
{ total = (total + 1023)/1024; y++; } printf("Total: %g
%s\n",int(total*10)/10,pref[y]); }'

Crawl a web page for links

Tags: awk grep tail
Save this to del.icio.usDigg this!Reddit this!Tweet this!
lynx -dump http://www.spantz.org | grep -A999 "^References$" | tail -n +3 | awk
'{print $2 }'

Portscan in one line

Tags: No tags
Save this to del.icio.usDigg this!Reddit this!Tweet this!
$HOST=127.0.0.1;for((port=1;port<=65535;++port));do echo -en "$port ";if echo -en
"open $HOST $port\nlogout\quit" | telnet 2>/dev/null | grep 'Connected to' >
/dev/null;then echo -en "\n\nport $port/tcp is open\n\n";fi;done
 list the most recent files in a directory
Tags: bash
Save this to del.icio.usDigg this!Reddit this!Tweet this!
If you just want to find out what's new in a directory:

lsnew() { ls -lt ${1+"$@"} | head -20; }

Diff Two Directories

Tags: bash diff
Save this to del.icio.usDigg this!Reddit this!Tweet this!
A quick script to compare files from two directories (for example a backup and
working directory).

#!/bin/bash

cr='*'
if [ -z $3 ]; then cr=$3; fi

for f in `find $1/$3 -type f | sed "s|$1/||"`

do
printf "===!%-76s" "$f!" | tr ' !' '= '; echo
diff $1/$f $2/$f | sed -e "s/^</$1: /" -e "s/^>/$2: /"
done

Usage: [script name] directory1 directory2 to check all files

[script name] directory1 directory2 *html to check files of type html.