More on Awk and Bash scripting

Today we are going to use the power of conditional IF within AWK (aka Conditional AWK programming).

Let's start with an exercise:

You have a file named file2.

#cat /root/file2
22110 2 even
21009 20 even
20903 2 even
24811 2 even
21703 18 even
20811 2 even
22008 2 even
29021 2 even

Where Column1 represents folder name, Column 2 to be used to compute a file name, Column 3 says that Column2 is Even number.

FileName is msg000 appended by (Colum2 -2)/2
e.g
22110 2 even

FileName is msg000 appended by (2-2)/2=0
i.e msg0000.txt


21009 20 even
FileName is msg000 appended by (20-2)/2=9
i.e msg0009.txt

Now we need to write a script, that will remove all those files.

Solution:

Step1: Write awk script that can generate commands to remove those files

#vi cleaner.awk
{
MessageNum=($2-2)/2;

#If MessageNum returns 0 then, there is only one file(may be .txt or wav) so use wildcard to delete that file
if(MessageNum == 0)
print "rm /var/spool/asterisk/voicemail/default/"$1"/INBOX/msg00*";

#If MessageNum returns greater than 0 but less than 10, then delete the bad file with name msg000
else if(MessageNum > 0 && MessageNum < 10)
print "rm /var/spool/asterisk/voicemail/default/"$1"/INBOX/msg000"MessageNum".txt";

#If MessageNum returns greater than 9 but less than 100 , then delete the bad file with name msg00
else if(MessageNum > 9 && MessageNum < 100)
print "rm /var/spool/asterisk/voicemail/default/"$1"/INBOX/msg00"MessageNum".txt";

#If MessageNum returns greater than 99 but less than 1000 , then delete the bad file with name msg0
else if(MessageNum > 99 && MessageNum < 1000)
print "rm /var/spool/asterisk/voicemail/default/"$1"/INBOX/msg0"MessageNum".txt";
}

[Here $1 returns the data on Column1]

Step2:
Run the command
awk -f cleaner.awk /root/file2 > cleanall.sh
This command runs cleaner.awk script for each line of the file file2 and prints the rm command in cleanall.sh file

To execute the script, assign execute permission to the script file.
chmod 700 cleanall.sh

Run the script that contains all the rm commands
./cleanall.sh


For more:

http://www.thegeekstuff.com/2010/02/awk-conditional-statements/
http://www.linuxfocus.org/English/September1999/article103.html

Find and remove duplicates in file | Sort data

Commands to be used
cut
sort
uniq

Step 1:
Cut command is used to select the desired data from the file. Let's say data in my file: students.txt is as follows. We need to find duplicates in second field of the data in this file.

#vi students.txt
101 101 Mike
102 102 Ryan
103 103 Dev
104 102 Steve
105 100 Bill

I can use CUT command to select second field by executing following command
#cut -d ' ' -f2 students.txt > secondField.txt

# vi secondField.txt
101
102
103
102
100

-d flag: Delimiter; here we are using space as delimiter
-f flag: Field Number

Step 2:
Now I have my desired second field. I can issue SORT command to sort the data.

#sort -n secondField.txt > sortedData.txt

#vi sortedData.txt
100
101
102
102
103

-n flag: Sort numerically

Step 3:
Finally, we can use UNIQ command to find the duplicate or unique data.

Display only unique data
#uniq -uc sortedData.txt
1 100
1 101
1 103

Display only duplicated data
#uniq -dc sortedData.txt
2 102

Display all data without repeatition
#uniq sortedData.txt
100
101
102
103


-u flag: unique data
-d flag: duplicate data
-c flag: show the count

Applications:

1. To find and remove duplicate data in voicemail.conf of asterisk

2. To sort sip peers information
Collect sip peers information
#asterisk -rx "sip show peers" >> sippeers

Collect the extension/username (Column 1) of peers
#less sippeers|cut -d' ' -f1|cut -d'/' -f1> file1

Collect the IP address(Column 2) information. We have to use awk because AWK treats multiple delimiter as one delimiter. In sippeers file we have multiple spaces separating column 1 and column 2.
#awk -F" " '{print $2}' sippeers>file2

Count the number of lines in file1 and file2. Make sure that both has same number of lines
#wc -l file*

Put the collected Column1(Username/Extension) and Column2 (IP Addresses) in one file i.e file3
#paste file1 file2 > file3

Sort the data in file3. By default sort command takes the list and sort numerically according to the first column.
#sort file3

-k switch can be used to sort by specific column.
e.g #sort -k 2 file 3 --> This will sort according to second column

Final Script:

#!/bin/bash
#Author erdevendra@gmail.com Script to sort the SIP peers registered to the server and write to file sippeers
#This script is used to find which extension is in use and which is not
#Extension in use will have IP Address attached to it

#pull sip users from asterisk
/usr/sbin/asterisk -rx "sip show peers" > sippeers
#Filter users extensions
/usr/bin/less sippeers|cut -d' ' -f1|cut -d'/' -f1>file1
#Filter IP addresses
/usr/bin/awk -F" " '{print $2}' sippeers >file2
#Put users extensions and IP addresses together in a file
/usr/bin/paste file1 file2>file3
/bin/rm sippeers
#sort the file by extensions
/usr/bin/sort file3>sippeers
#remove temporary files
/bin/rm file1 file2 file3


3. To find the total number of IP addresses leased by DHCP server. [ Note: DHCP seems to keep same IP address multiple times in dhcp.leases database. ]

#less /var/lib/dhcp/db/dhcpd.leases|grep 10.219|awk -F" " '{print $2}'|awk -F"." '{print $3 $4}'|sort -n|uniq|wc -l

In Linux, DHCP server stores dhcp leases at /var/lib/dhcp/db/dhcpd.leases
In my example, I am filtering IP addresses for 10.219.1.1 network using 'grep 10.219'
awk -F" " '{print $2}' --> This filters out IP addresses only
awk -F"." '{print $3 $4}' --> This filters out 3rd and 4th octet of IP address e.g 10.219.2.230 will return 2230 (i.e 2.230)
sort -n --> This will sort the data in ascending order
uniq --> This will remove the repetition of data
wc -l ---> This will return the total number of lines, which in turn is the total number of uniq IP addresses already being assigned by DHCP server


For more: http://www.liamdelahunty.com/tips/linux_remove_duplicate_lines_with_uniq.php

Very good explanation with examples:
http://www.techrepublic.com/article/lesser-known-linux-commands-join-paste-and-sort/5031653

Centralized LogServer in SuSE Linux

LogServer: ServerA [IP address: 192.168.1.5]
NetworkServers: ServerB, ServerC... and more

ServerB-------send log files-----> ServerA[LogServer] <-----------send log files---- ServerC

Here we want ServerB, Server C to send it's log file to ServerA for centralized access of log files.

Daemon: syslog-ng
Files:
/etc/sysconfig/syslog
/etc/syslog-ng/syslog-ng.conf

Commands:
/etc/init.d/syslog start|restart|stop

ps aux|grep syslog --> to see if syslog-ng is running or not

SuSEconfig --module syslog-ng --> to reload the change done on /etc/syslog-ng/syslog-ng.conf


Configure LogServer i.e ServerA to accept the log files from NetworkServers

Edit /etc/syslog-ng/syslog-ng.conf on ServerA(Log Server)

source src {
#
# include internal syslog-ng messages
# note: the internal() soure is required!
#
internal();

#
# the default log socket for local logging:
#
unix-dgram("/dev/log");

#
# uncomment to process log messages from network:
#
udp(ip("0.0.0.0") port(514));
#I uncommented above line telling ServerA to accept the log files from network
};


At the bottom of this file, I defined the destination and log

#
#Added by DShah 05/25/11
#
destination std { file("/var/log/HOSTS/$YEAR-$MONTH/$HOST/$FACILITY-$YEAR-$MONTH-$DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes));
};

log { source(src);
destination(std);
};

Over here I am telling ServerA to process the log files coming source src to destination std.
Destination std tells ServerA to save log messages from each host in a separate directory called /var/log/HOSTS/YEAR-MONTH/hostname/.

Now run the command
#SuSEconfig --module syslog-ng --> to reload the config changes done

#/etc/init.d/syslog restart OR
#syslog-ng restart

#ps aux|grep syslog --> to check if syslog-ng is running

If you need to kill syslog-ng process for any reason, you can simply run the command

#killall syslog-ng
or
#kill -9 [PID-of-syslog-ng]

Configure NetworkServers (Server B, ServerC... ) to send log files to LogServer(ServerA):

Edit /etc/syslog-ng/syslog-ng.conf or /etc/syslog-ng/syslog-ng.conf.in (preffered) on ServerB, ServerC

#
#Added by DShah 05/25/2011
#
destination logserver {
udp("192.168.1.5" port(514));
#Note: here 192.168.1.5 is an IP add of LogServer i.e ServerA
};

log {
source(src);
destination(logserver);
};

Now run the command
#SuSEconfig --module syslog-ng --> to reload the config changes done

#/etc/init.d/syslog restart OR
#syslog-ng restart

#ps aux|grep syslog --> to check if syslog-ng is running


ServerA should be already collecting the log files. You can go to /var/log/HOSTS directory on ServerA to see the log files from different Network Servers.

Illustration by Additional applications:
Let's say I want remote asterisk server to dump it's log file /var/log/asterisk/full in the centralized log server
Edit /etc/syslog-ng/syslog-ng.conf or /etc/syslog-ng/syslog-ng.conf.in (preffered) on remote Asterisk Server

#
# Added by DShah
#
source asterisklog { pipe("/var/log/asterisk/full");
};

destination logserver { udp("192.168.1.5" port(514));
};

log { source(asterisklog); destination(logserver); };


and run the command

#syslog-ng restart

Now please check /var/log/HOSTS , you should see log file from asterisk server.


If you need any help on Linux/Unix systems, you can email me at erdevendra@gmail.com with subject title rapidtechguide.

For more info: http://www.novell.com/coolsolutions/feature/18044.html
20 minutes video on syslog-ns : http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/overview#
Syslog-ns to collect apache logs: http://peter.blogs.balabit.com/2010/02/how-to-collect-apache-logs-by-syslog-ng/

How to rename Ethernet Device using udev? How to change ethX to ethY?

Today we replaced motherboard/system board on our R610 Dell Server. Guess what? SuSe (Linux OS) still believes that it has 8 ethernet devices ( 4 on old system board and 4 on new system board ). So, now my new ethernet devices are listed as
eth4
eth5
eth6
eth7

I wanted to change the name back to
eth0
eth1
eth2
eth3
as I know that on my new system board there are only four builtin ethernet devices

So, what did I do to fix it?

Step 1: Stop network service #rcnetwork stop

Step 2: Edit udev rules for network devices # vi /etc/udev/rules.d/70-persistent-net.rules
Change ethX to ethY. Where X is undesired name and Y is desired name
So, I changed eth4 to eth0, eth5 to eth1, eth6 to eth2 and eth7 to eth3

Step4: Reboot the server

Step5: Check if it got the right name. You can use #ifconfig command
You can also use #hwinfo --netcard (For detailed information on network hardware)

Step5: Configure the IP address. In SuSe you can use YaST setup tool

I hope this helps you too..... :)



http://www.novell.com/support/search.do?cmd=displayKC&docType=kc&externalId=3012993&sliceId=1&docTypeID=DT_TID_1_1

VI Tips

After lots of request from my friends, I finally end up with writing something on VI or VIM ( Vi IMproved ). This is one of the most powerful editor in UNIX environment.

Try this
# vi test.txt
[Press ESC]
:help

Please read through that briefly. There are various .txt files just like each chapters in text book.

You can access those .txt file using

For example
:help usr_01

I want you guys to read and learn by yourself. That really helps. You will know what you are doing.

However, I will mention some frequently used shortcuts/commands:

h j k l move left move up move down move right

:x [Enter] To save and exit file
:q To quit
:qw Same as :x
:q! Ignore the changes and quit

:set number Display line number on each line
:set nonumber Don't display line number on each line

:set ruler Display the cursor position at the bottom of the screen

CTRL + g Display the cursor position with percentage of the page

CTRL + d Scroll half window down
CTRL + u Scroll half window up

/findthis Search 'findthis' string
[ Press n or N for searching backward and forward in the file]

The characters .*[]^%/\?~$ have special meanings. If you want to use them in a search you must put a \ in front of them.

:set ignorecase To ignorecase while searching the string/pattern
:set noignorecase

G To go to the bottom of the page
gg To go to the top of the page

yy To copy/yank the line
dd To delete the line
p To paste the yanked/copied line

$ To go to the end of the line
0 To go to the front of the line

{ To go to the end of the paragraph
{ To go to the front of the paragraph

:syntax on To enable syntax highlight
:syntax off


:line_number To go to particular line e.g To go to line 22 , do the following :22 [enter]


SEARCHING FOR A WORD IN THE TEXT

Suppose you see the word "TheLongFunctionName" in the text and you want to find the next occurrence of it. You could type "/TheLongFunctionName", but that's a lot of typing. And when you make a mistake Vim won't find it. There is an easier way: Position the cursor on the word and use the "*" command. Vim will grab the word under the cursor and use it as the search
string. The "#" command does the same in the other direction. You can prepend a
count: "3*" searches for the third occurrence of the word under the cursor.

HIGHLIGHTING MATCHES

While editing a program you see a variable called "nr". You want to check where it's used. You could move the cursor to "nr" and use the "*" command and press "n" to go along all the matches.

There is another way. Type this command:

:set hlsearch

If you now search for "nr", Vim will highlight all matches. That is a verygood way to see where the variable is used, without the need to type commands.

To switch this off:
:set nohlsearch


REPLACE THE WORD

:%s/old_word/new_word/g

This will substitute old_word with new_word globally.

TUNING SEARCHES

There are a few options that change how searching works. These are the essential ones:

:set incsearch

This makes Vim display the match for the string while you are still typing it. Use this to check if the right match will be found. Then press to really jump to that location. Or type more to change the search string.

:set nowrapscan

This stops the search at the end of the file. Or, when you are searching backwards, at the start of the file. The 'wrapscan' option is on by default, thus searching wraps around the end of the file.


Bonus Tips:
If you like one of the options mentioned before, and set it each time you use Vim, you can put the command in your Vim startup file.
Edit the file, as mentioned at |not-compatible|. Or use this command to find out where it is:

:scriptnames

Edit the file, for example with:

:edit ~/.vimrc

Then add a line with the command to set the option, just like you typed it in
Vim. Example:

Go:set hlsearch