WordFinder script: The script that crawls thru' every files to find what you are looking for

Have you ever wanted to find the 'word' in files located under directory containing sub-directories and tons of files. It would be nightmare to search in individual file. You can make your life easy by writing a script that crawls thru' every files under that directory and return the file name along with the line that contains the word you are looking for.

Ok let's get started with BASH script that will do the job for us.

$vim WordFinder.sh
#!/bin/bash
#Author: erdevendra@gmail.com
#
#Usage: This scripts finds a word in each and every files located under the specified location
#
#Syntax:
# .\WordFinder.sh
#

for x in `find $1 -type f`
do
#find files under the specified location and use for loop to go thru each files
         grep -i $2 $x
#search for the pattern in the file
          if [ $? -eq 0 ]
#Check if grep cmd executed successfully
             then echo $x
#if grep cmd executed successfully (i.e if pattern found) display the file name
          fi
#end IF loop
done
#end FOR loop

Give an executable permission to the file

#chmod 777 WordFinder.sh

(Note: 777 gives full permission to everyone in the system)

You are good to go. Now you just need to run that script to find what you are looking for.

Example:

#WordFinder.sh  <location>  <word_you_are_looking_for>

Let's say I want to find word 'listen' in '/etc' directory, I can run following command:
#WordFinder.sh  /etc  listen

Find and delete

Let's say there are thousands of WAV files in my Recordings folder and I want to clean up WAV files older than 5 days, I don't have to write fancy script. I just need to run this command

#find  .  -name  "*.wav"  -mtime +5 -exec rm {} \;

Explanation:

#find : Find command

. : Dot stands for current location

-name: Look for the name pattern
"*.wav"   : Find wave files

+5 : older than 5 days

-exec: Execute

rm : remove command

{} \;  : Terminate the command line [Note: there is space between rm, {} and \; ]



Let's say I want to remove every files older than 5 days in current directory, I can simply run the following command

#find . -mtime +5 -exec rm {} \;


dot (.)  represents the current directory


Let's say I want to setup interactive way (ask use before deleting file) to remove the files plder than 5 days

#find . -mtime +5 -exec rm -i  {} \;


rm - i : remove interactively


Let's say I want to forcefully remove files older than 5 days,


#find . -mtime +5 -exec rm -f  {} \;

rm - f : remove forecefully

More on Awk and Bash scripting

Today we are going to use the power of conditional IF within AWK (aka Conditional AWK programming).

Let's start with an exercise:

You have a file named file2.

#cat /root/file2
22110 2 even
21009 20 even
20903 2 even
24811 2 even
21703 18 even
20811 2 even
22008 2 even
29021 2 even

Where Column1 represents folder name, Column 2 to be used to compute a file name, Column 3 says that Column2 is Even number.

FileName is msg000 appended by (Colum2 -2)/2
e.g
22110 2 even

FileName is msg000 appended by (2-2)/2=0
i.e msg0000.txt


21009 20 even
FileName is msg000 appended by (20-2)/2=9
i.e msg0009.txt

Now we need to write a script, that will remove all those files.

Solution:

Step1: Write awk script that can generate commands to remove those files

#vi cleaner.awk
{
MessageNum=($2-2)/2;

#If MessageNum returns 0 then, there is only one file(may be .txt or wav) so use wildcard to delete that file
if(MessageNum == 0)
print "rm /var/spool/asterisk/voicemail/default/"$1"/INBOX/msg00*";

#If MessageNum returns greater than 0 but less than 10, then delete the bad file with name msg000
else if(MessageNum > 0 && MessageNum < 10)
print "rm /var/spool/asterisk/voicemail/default/"$1"/INBOX/msg000"MessageNum".txt";

#If MessageNum returns greater than 9 but less than 100 , then delete the bad file with name msg00
else if(MessageNum > 9 && MessageNum < 100)
print "rm /var/spool/asterisk/voicemail/default/"$1"/INBOX/msg00"MessageNum".txt";

#If MessageNum returns greater than 99 but less than 1000 , then delete the bad file with name msg0
else if(MessageNum > 99 && MessageNum < 1000)
print "rm /var/spool/asterisk/voicemail/default/"$1"/INBOX/msg0"MessageNum".txt";
}

[Here $1 returns the data on Column1]

Step2:
Run the command
awk -f cleaner.awk /root/file2 > cleanall.sh
This command runs cleaner.awk script for each line of the file file2 and prints the rm command in cleanall.sh file

To execute the script, assign execute permission to the script file.
chmod 700 cleanall.sh

Run the script that contains all the rm commands
./cleanall.sh


For more:

http://www.thegeekstuff.com/2010/02/awk-conditional-statements/
http://www.linuxfocus.org/English/September1999/article103.html

Find and remove duplicates in file | Sort data

Commands to be used
cut
sort
uniq

Step 1:
Cut command is used to select the desired data from the file. Let's say data in my file: students.txt is as follows. We need to find duplicates in second field of the data in this file.

#vi students.txt
101 101 Mike
102 102 Ryan
103 103 Dev
104 102 Steve
105 100 Bill

I can use CUT command to select second field by executing following command
#cut -d ' ' -f2 students.txt > secondField.txt

# vi secondField.txt
101
102
103
102
100

-d flag: Delimiter; here we are using space as delimiter
-f flag: Field Number

Step 2:
Now I have my desired second field. I can issue SORT command to sort the data.

#sort -n secondField.txt > sortedData.txt

#vi sortedData.txt
100
101
102
102
103

-n flag: Sort numerically

Step 3:
Finally, we can use UNIQ command to find the duplicate or unique data.

Display only unique data
#uniq -uc sortedData.txt
1 100
1 101
1 103

Display only duplicated data
#uniq -dc sortedData.txt
2 102

Display all data without repeatition
#uniq sortedData.txt
100
101
102
103


-u flag: unique data
-d flag: duplicate data
-c flag: show the count

Applications:

1. To find and remove duplicate data in voicemail.conf of asterisk

2. To sort sip peers information
Collect sip peers information
#asterisk -rx "sip show peers" >> sippeers

Collect the extension/username (Column 1) of peers
#less sippeers|cut -d' ' -f1|cut -d'/' -f1> file1

Collect the IP address(Column 2) information. We have to use awk because AWK treats multiple delimiter as one delimiter. In sippeers file we have multiple spaces separating column 1 and column 2.
#awk -F" " '{print $2}' sippeers>file2

Count the number of lines in file1 and file2. Make sure that both has same number of lines
#wc -l file*

Put the collected Column1(Username/Extension) and Column2 (IP Addresses) in one file i.e file3
#paste file1 file2 > file3

Sort the data in file3. By default sort command takes the list and sort numerically according to the first column.
#sort file3

-k switch can be used to sort by specific column.
e.g #sort -k 2 file 3 --> This will sort according to second column

Final Script:

#!/bin/bash
#Author erdevendra@gmail.com Script to sort the SIP peers registered to the server and write to file sippeers
#This script is used to find which extension is in use and which is not
#Extension in use will have IP Address attached to it

#pull sip users from asterisk
/usr/sbin/asterisk -rx "sip show peers" > sippeers
#Filter users extensions
/usr/bin/less sippeers|cut -d' ' -f1|cut -d'/' -f1>file1
#Filter IP addresses
/usr/bin/awk -F" " '{print $2}' sippeers >file2
#Put users extensions and IP addresses together in a file
/usr/bin/paste file1 file2>file3
/bin/rm sippeers
#sort the file by extensions
/usr/bin/sort file3>sippeers
#remove temporary files
/bin/rm file1 file2 file3


3. To find the total number of IP addresses leased by DHCP server. [ Note: DHCP seems to keep same IP address multiple times in dhcp.leases database. ]

#less /var/lib/dhcp/db/dhcpd.leases|grep 10.219|awk -F" " '{print $2}'|awk -F"." '{print $3 $4}'|sort -n|uniq|wc -l

In Linux, DHCP server stores dhcp leases at /var/lib/dhcp/db/dhcpd.leases
In my example, I am filtering IP addresses for 10.219.1.1 network using 'grep 10.219'
awk -F" " '{print $2}' --> This filters out IP addresses only
awk -F"." '{print $3 $4}' --> This filters out 3rd and 4th octet of IP address e.g 10.219.2.230 will return 2230 (i.e 2.230)
sort -n --> This will sort the data in ascending order
uniq --> This will remove the repetition of data
wc -l ---> This will return the total number of lines, which in turn is the total number of uniq IP addresses already being assigned by DHCP server


For more: http://www.liamdelahunty.com/tips/linux_remove_duplicate_lines_with_uniq.php

Very good explanation with examples:
http://www.techrepublic.com/article/lesser-known-linux-commands-join-paste-and-sort/5031653

Sending email using Command line + Scripting with example

Trial 1:

mail -s "This is a subject" john.doe@test.net < /root/myBodyFile

This simple command will send the email to john.doe@test.net with the subject "This is a subject" and body with the content of the file /root/myBodyFile

Trial 2:

You could check if your linux machine delievered the message to john doe or not.

# mail

If you see any MAILER-DAEMON@ , check if it is for john.doe@test.net (i.e email recepient) or not.

There could be various possible issues.

In my case I am using PostFix as my MTA.
I have configured mailrelayhost for my MTA.

For postfix, you can go to

#vi /etc/postfix/main.ca
------
-----
mailrelayhost= < specify ur mailrelay host or IP address of your mail server >
-----
-----

And also make sure that your linux machine's IP address is allowed to send email using email server (contact your email server administrator). He will add IP address of your linux machine as trusted host in his email server

Trail 3:

I would like to create alias so that I can send email to the group of people. It's easy.

Login as a root in your linux machine

#vi /etc/aliases
-----
----
#myEmailGroup: List of all emails separated by comma
-----
-----
ServerAlert: user1@gmail.com,user2@hotmail.com,214000000@txt.att.net
-----
-----

Next step is to load all the aliases

#newaliases

#echo $?
If it returns 0, then above command was executed succesfully


Finally you can send email to the group of users by

#mail -s "Hi.. all of you" ServerAlert < /root/myBodyFile


Application of email alert for the Asterisk PRI monitoring:

1. Create the script file

#vi astPRIcontrol.sh
#!/bin/bash
#author DShah erdevendra@gmail.com
#created on 10/22/10
#This script checks the PRI status; If PRI is down, it pages the admins

pristatus=echo /usr/sbin/asterisk -rx "pri show spans" |grep -i down

if [ -z $pristatus ]
then
#echo "PRI up"
notify=0
else
#echo "PRI down"
notify=1
fi

if [ $notify -eq 1 ]
then
mail -s " PRI down" Server_Alert < /root/pristatus

fi


2. Change the permission (make the script executable)

#chmod 700 /root/astPRIstatus.sh

3. Create a file called /root/pristatus

#vi /root/pristatus
PRI at asterisk server is down

4. Add the script file in crontab: it runs the script every minute

#crontab -e

*/1 * * * * bash /root/astPRIstatus.sh >/dev/null