Step by Step guide: Resizing File System on the Linux Virtual Machine

Assumptions made:
You must have basic knowledge of LVM to follow the instruction.
Linux Environment : I have done this on SLES 11; However the LVM commands used are the generic Linux commands. Thus, this instruction should work on wide flavors of Linux.
Volume Group (VG) name: testvg
Logical Volume(LV) name: testlv
Root file system is on /dev/testvg/testlv file system  and I want to increase the file system size by 5 GB.


Step1: Provision an extra disk space to the Virtual Machine [ i.e Increase the VMDK size of the VM via vSphere client if you are on VMWare environment]

[Note: If you provision an additional disk (some VMs doesn't allow to increase the VMDK size), check if you can see that additional disk. Say an additional disk is /dev/sdb. Then, run the command  #fdisk -l   . It should show  /dev/sdb. If you cannot see /dev/sdb, then reboot the system and run #fdisk -l  command again]

Step2: Partition the additional disk space you just added [ Say your system sees disk as /dev/sda ]
           # fdisk   /dev/sda

• Create a new partition (primary if possible) Let's say  /dev/sda4
• Define size
• Change the partition type from 83 (EXT3) to 8e (LVM)
• Write the change

Step3: Create a Physical Volume 
            # pvcreate <partition_name>

                    Example: # pvcreate  /dev/sda4

If you get an error reporting that system can't see the partition, reboot the Virtual Machine.
And re-run the pvcreate command once the system is back up. 

Note: You can run #vgdisplay command to check what Volume Groups are available

Step4: Extend the Volume Group [ i.e Add the physical volume we just created to the existing volume group]

           # vgextend    <vgname>   <partition_name>

                 Example: # vgextend   testvg   /dev/sda4

Note: You can run #lvdisplay command to check what logical volumes  are available
Step5: Extend the Logical Volume  [ Say increase by 5GB ]
           #lvextend   -L     +5G    /dev/<vgname>/<lvname>   <partition_name>

                   Example: #lvextend   -L   +5G   /dev/testvg/testlv     /dev/sda4

Warning!!! Check if a space is free in the physical volume or not. You can run the command #pvdisplay  and check how much free space is available in the physical partition. 

Step6: Resize the file system
          #resize2fs     /dev/<vgname>/<lvname>

                Example: #resize2fs    /dev/testvg/testlv

Note: If resizze2fs doesn't work, reboot the system. For example: SLES 10 doesn't support online resize of the mounted file-system. Thus, SLES10 requires reboot after 'lvextend' of root file-system. 

Validate if the size of the file system has been increased or not. 

         #df  -h

Best of Luck! 

Is your ssh login slow? It could be DNS issue

There could be  various reasons that could result in slow ssh login. Usually it's DNS configuration issue on your server.

In my SUSE environment,

#ping www.google.com

would wait 15 seconds before it displayed something like this

PING www.l.google.com (74.125.227.19) 56(84) bytes of data

I had similar 15 seconds delay for ssh login. Thus, I used following option in  sshd_config file (located under /etc/ssh/sshd_config)

UseDNS  no

Restart SSH Daemon

#service sshd restart

Now there is no delay in SSH login.

However delay in pining google.com was still bothering me. Finally I figured out that you shouldn't be changing DNS configuration from /etc/resolv.conf if you are using netconfig or YaST tool. Thus to clean this mess, I ran following commmand

#netconfig update -f

This command updated  resolv.conf as per netconfig/YaST configuration for DNS and my pinging issue was resolved.
I know you might be thinking of enabling UseDNS for ssh. Yes, you can do it as DNS issue is resolved.

Good Luck

WordFinder script: The script that crawls thru' every files to find what you are looking for

Have you ever wanted to find the 'word' in files located under directory containing sub-directories and tons of files. It would be nightmare to search in individual file. You can make your life easy by writing a script that crawls thru' every files under that directory and return the file name along with the line that contains the word you are looking for.

Ok let's get started with BASH script that will do the job for us.

$vim WordFinder.sh
#!/bin/bash
#Author: erdevendra@gmail.com
#
#Usage: This scripts finds a word in each and every files located under the specified location
#
#Syntax:
# .\WordFinder.sh
#

for x in `find $1 -type f`
do
#find files under the specified location and use for loop to go thru each files
         grep -i $2 $x
#search for the pattern in the file
          if [ $? -eq 0 ]
#Check if grep cmd executed successfully
             then echo $x
#if grep cmd executed successfully (i.e if pattern found) display the file name
          fi
#end IF loop
done
#end FOR loop

Give an executable permission to the file

#chmod 777 WordFinder.sh

(Note: 777 gives full permission to everyone in the system)

You are good to go. Now you just need to run that script to find what you are looking for.

Example:

#WordFinder.sh  <location>  <word_you_are_looking_for>

Let's say I want to find word 'listen' in '/etc' directory, I can run following command:
#WordFinder.sh  /etc  listen

Setting PATH variable in SUSE linux

Usually for your local settings, favorite global aliases, VISUAL and EDITOR variable, PATH environment variable and more, you can create a file /etc/profile.local in SuSE Linux.  SuSE Linux doesn't recommend you to change /etc/profile as there are the chances that your changes will be lost during system upgrades.


#vim  /etc/profile.local
PATH=$PATH:/path/to/wherever
export PATH

Log-out and log-back in.

#echo $PATH

You will see that your PATH variable has been updated.


Practical Application:

I write various scripts for system administration. I like my scripts to be in one place. I don't want to go that that folder each time to run the script. I want to execute those scripts no matter where I am.

Say, my scripts are in /usr/local/bin/myscripts

#vim  /etc/profile.local
PATH=$PATH:/usr/local/bin/myscripts
export PATH

Now I logged-out and logged-back in. I can run my scripts within myscripts folder anywhere I like.

Find and delete

Let's say there are thousands of WAV files in my Recordings folder and I want to clean up WAV files older than 5 days, I don't have to write fancy script. I just need to run this command

#find  .  -name  "*.wav"  -mtime +5 -exec rm {} \;

Explanation:

#find : Find command

. : Dot stands for current location

-name: Look for the name pattern
"*.wav"   : Find wave files

+5 : older than 5 days

-exec: Execute

rm : remove command

{} \;  : Terminate the command line [Note: there is space between rm, {} and \; ]



Let's say I want to remove every files older than 5 days in current directory, I can simply run the following command

#find . -mtime +5 -exec rm {} \;


dot (.)  represents the current directory


Let's say I want to setup interactive way (ask use before deleting file) to remove the files plder than 5 days

#find . -mtime +5 -exec rm -i  {} \;


rm - i : remove interactively


Let's say I want to forcefully remove files older than 5 days,


#find . -mtime +5 -exec rm -f  {} \;

rm - f : remove forecefully

More on Awk and Bash scripting

Today we are going to use the power of conditional IF within AWK (aka Conditional AWK programming).

Let's start with an exercise:

You have a file named file2.

#cat /root/file2
22110 2 even
21009 20 even
20903 2 even
24811 2 even
21703 18 even
20811 2 even
22008 2 even
29021 2 even

Where Column1 represents folder name, Column 2 to be used to compute a file name, Column 3 says that Column2 is Even number.

FileName is msg000 appended by (Colum2 -2)/2
e.g
22110 2 even

FileName is msg000 appended by (2-2)/2=0
i.e msg0000.txt


21009 20 even
FileName is msg000 appended by (20-2)/2=9
i.e msg0009.txt

Now we need to write a script, that will remove all those files.

Solution:

Step1: Write awk script that can generate commands to remove those files

#vi cleaner.awk
{
MessageNum=($2-2)/2;

#If MessageNum returns 0 then, there is only one file(may be .txt or wav) so use wildcard to delete that file
if(MessageNum == 0)
print "rm /var/spool/asterisk/voicemail/default/"$1"/INBOX/msg00*";

#If MessageNum returns greater than 0 but less than 10, then delete the bad file with name msg000
else if(MessageNum > 0 && MessageNum < 10)
print "rm /var/spool/asterisk/voicemail/default/"$1"/INBOX/msg000"MessageNum".txt";

#If MessageNum returns greater than 9 but less than 100 , then delete the bad file with name msg00
else if(MessageNum > 9 && MessageNum < 100)
print "rm /var/spool/asterisk/voicemail/default/"$1"/INBOX/msg00"MessageNum".txt";

#If MessageNum returns greater than 99 but less than 1000 , then delete the bad file with name msg0
else if(MessageNum > 99 && MessageNum < 1000)
print "rm /var/spool/asterisk/voicemail/default/"$1"/INBOX/msg0"MessageNum".txt";
}

[Here $1 returns the data on Column1]

Step2:
Run the command
awk -f cleaner.awk /root/file2 > cleanall.sh
This command runs cleaner.awk script for each line of the file file2 and prints the rm command in cleanall.sh file

To execute the script, assign execute permission to the script file.
chmod 700 cleanall.sh

Run the script that contains all the rm commands
./cleanall.sh


For more:

http://www.thegeekstuff.com/2010/02/awk-conditional-statements/
http://www.linuxfocus.org/English/September1999/article103.html

Find and remove duplicates in file | Sort data

Commands to be used
cut
sort
uniq

Step 1:
Cut command is used to select the desired data from the file. Let's say data in my file: students.txt is as follows. We need to find duplicates in second field of the data in this file.

#vi students.txt
101 101 Mike
102 102 Ryan
103 103 Dev
104 102 Steve
105 100 Bill

I can use CUT command to select second field by executing following command
#cut -d ' ' -f2 students.txt > secondField.txt

# vi secondField.txt
101
102
103
102
100

-d flag: Delimiter; here we are using space as delimiter
-f flag: Field Number

Step 2:
Now I have my desired second field. I can issue SORT command to sort the data.

#sort -n secondField.txt > sortedData.txt

#vi sortedData.txt
100
101
102
102
103

-n flag: Sort numerically

Step 3:
Finally, we can use UNIQ command to find the duplicate or unique data.

Display only unique data
#uniq -uc sortedData.txt
1 100
1 101
1 103

Display only duplicated data
#uniq -dc sortedData.txt
2 102

Display all data without repeatition
#uniq sortedData.txt
100
101
102
103


-u flag: unique data
-d flag: duplicate data
-c flag: show the count

Applications:

1. To find and remove duplicate data in voicemail.conf of asterisk

2. To sort sip peers information
Collect sip peers information
#asterisk -rx "sip show peers" >> sippeers

Collect the extension/username (Column 1) of peers
#less sippeers|cut -d' ' -f1|cut -d'/' -f1> file1

Collect the IP address(Column 2) information. We have to use awk because AWK treats multiple delimiter as one delimiter. In sippeers file we have multiple spaces separating column 1 and column 2.
#awk -F" " '{print $2}' sippeers>file2

Count the number of lines in file1 and file2. Make sure that both has same number of lines
#wc -l file*

Put the collected Column1(Username/Extension) and Column2 (IP Addresses) in one file i.e file3
#paste file1 file2 > file3

Sort the data in file3. By default sort command takes the list and sort numerically according to the first column.
#sort file3

-k switch can be used to sort by specific column.
e.g #sort -k 2 file 3 --> This will sort according to second column

Final Script:

#!/bin/bash
#Author erdevendra@gmail.com Script to sort the SIP peers registered to the server and write to file sippeers
#This script is used to find which extension is in use and which is not
#Extension in use will have IP Address attached to it

#pull sip users from asterisk
/usr/sbin/asterisk -rx "sip show peers" > sippeers
#Filter users extensions
/usr/bin/less sippeers|cut -d' ' -f1|cut -d'/' -f1>file1
#Filter IP addresses
/usr/bin/awk -F" " '{print $2}' sippeers >file2
#Put users extensions and IP addresses together in a file
/usr/bin/paste file1 file2>file3
/bin/rm sippeers
#sort the file by extensions
/usr/bin/sort file3>sippeers
#remove temporary files
/bin/rm file1 file2 file3


3. To find the total number of IP addresses leased by DHCP server. [ Note: DHCP seems to keep same IP address multiple times in dhcp.leases database. ]

#less /var/lib/dhcp/db/dhcpd.leases|grep 10.219|awk -F" " '{print $2}'|awk -F"." '{print $3 $4}'|sort -n|uniq|wc -l

In Linux, DHCP server stores dhcp leases at /var/lib/dhcp/db/dhcpd.leases
In my example, I am filtering IP addresses for 10.219.1.1 network using 'grep 10.219'
awk -F" " '{print $2}' --> This filters out IP addresses only
awk -F"." '{print $3 $4}' --> This filters out 3rd and 4th octet of IP address e.g 10.219.2.230 will return 2230 (i.e 2.230)
sort -n --> This will sort the data in ascending order
uniq --> This will remove the repetition of data
wc -l ---> This will return the total number of lines, which in turn is the total number of uniq IP addresses already being assigned by DHCP server


For more: http://www.liamdelahunty.com/tips/linux_remove_duplicate_lines_with_uniq.php

Very good explanation with examples:
http://www.techrepublic.com/article/lesser-known-linux-commands-join-paste-and-sort/5031653