Archive for the ‘UNIX’ Category

Awk One Liners

Print column1, column5 and column7 of a data file or output of any columns list

awk '{print $1, $5, $7}' data_file

cat file_name |awk ‘{print $1 $5 $7}’

ls –al |awk '{print $1, $5, $7}' -- Prints file_permissions,size and date

List all files names whose file size greater than zero.

ls –al |awk '$5 > 0 {print $9}'

List all files whose file size equal to 512bytes.

ls –al |awk '$5 == 0 {print $9}'

print all lines

awk '{print }' file_name

awk '{print 0}' file_name

Number of lines in a file


awk ' END {print NR}' file_name

Number of columns in each row of a file

awk '{print NF}' file_name

Sort the output of file and eliminate duplicate rows

awk '{print $1, $5, $7}' |sort –u

List all file names whose file size is greater than 512bytes and owner is “oracle”

ls –al |awk '$3 == "oracle" && $5 > 512 {print $9}'

List all file names whose owner could be either “oracle” or “root”

ls –al |awk '$3 == "oracle" || $3 == "root" {print $9}'

list all the files whose owner is not “oracle

ls –al |awk '$3 != "oracle" {print $9}'

List all lines which has at least one or more characters

awk 'NF > 0 {print }' file_name

List all lines longer that 50 characters

awk 'length($0) > 50 {print }' file_name

List first two columns

awk '{print $1, $2}' file_name

Swap first two columns of a file and print

awk '{temp = $1; $1 = $2; $2 = temp; print }' file_name

Replace first column as “ORACLE” in a data file

awk '{$1 = "ORACLE"; print }' data_file

Remove first column values in a data file

awk '{$1 =""; print }' data_file

Calculate total size of a directory in Mb

ls –al |awk '{total +=$5};END {print "Total size: " total/1024/1024 " Mb"}'

Calculate total size of a directory including sub directories in Mb

ls –lR |awk '{total +=$5};END {print "Total size: " total/1024/1024 " Mb"}'


Find largest file in a directory including sub directories

ls –lR |awk '{print $5 "\t" $9}' |sort –n |tail -1

Advertisements

SMP Symmetric Multiprocessing

SMP (symmetric multiprocessing) is the processing of programs by multiple processors that share a common operating system andmemory. In symmetric (or "tightly coupled") multiprocessing, the processors share memory and the I/O bus or data path. A single copy of the operating system is in charge of all the processors. SMP, also known as a "shared everything" system, does not usually exceed 16 processors.

SMP systems are considered better than MPP systems for online transaction processing (OTP) in which many users access the same database in a relatively simple set of transactions. An advantage of SMP for this purpose is the ability to dynamically balance the workload among computers (and as a result serve more users faster).

 

An SMP gives consistent performance because all processors have equal access to disk resources, and when communication between processes in the DBMS are required, it is accomplished at memory speeds which are two orders of magnitude faster than MPP message-based interconnects. With the consequences of disk layout being so minor compared to database partitioning on an MPP, the choice of SMP architectures is one that brings consistency in performance, as well as scalability.

A skeptical database administrator, however, will not accept these qualitative arguments without real performance data to back them up, and fortunately the TPC-D benchmarks provide a means for all vendors to put forth their best performance measurements.

Crontab

Setting up cronjobs in Unix and Solaris

cron is a unix, solaris utility that allows tasks to be automatically run in the background at regular intervals by the cron daemon. These tasks are often termed as cron jobs in unix , solaris.
Crontab (CRON TABle) is a file which contains the schedule of cron entries to be run and at specified times.

Crontab Restrictions

You can execute crontab if your name appears in the file /usr/lib/cron/cron.allow. If that file does not exist, you can use crontab if your name does not appear in the file /usr/lib/cron/cron.deny. 
If only cron.deny exists and is empty, all users can use crontab. If neither file exists, only root user can use crontab.The allow/deny files consist of one user name per line.

Crontab Commands
crontab -e     Edit your crontab file, or create one if it doesn’t already exist.
crontab -l      Display your crontab file.
crontab -r      Remove your crontab file.
crontab -v      Display the last time you edited your crontab file. (This option is only available on a few systems.)

Crontab file
A crontab file has five fields for specifying day , date and time  followed by the command to be run at that interval.

min hr dom month dow cmd

*     *   *   *    *  command to be executed

-     -    -    -    –

|     |     |     |     |

|     |     |     |     +—– day of week (0 – 6) (Sunday=0)

|     |     |     +——- month (1 – 12)

|     |     +——— day of month (1 – 31)

|     +———– hour (0 – 23)

+————- min (0 – 59)

* in the value field above means all legal values as in braces for that column.

The value column can have a * or a list of elements separated by commas. An element is either a number in the ranges shown above or two numbers in the range separated by a hyphen (meaning an inclusive range).

     * 12 16 * Mon root cmd

Will run cmd at midday every Monday and every 16th, and will produce the

same result as both of these entries put together would:

     59 11 * * 1,2,3,4,5 root backup.sh

Will run backup.sh at 11:59 Monday, Tuesday, Wednesday, Thursday

and Friday,as will:

59 11 * * 1-5 root backup.sh

Cron also supports ‘step’ values.A value of */2 in the dom field would mean the command runs every two daysand likewise, */5 in the hours field would mean the command runs every 5 hours.e.g.

* 12 10-16/2 * * root backup.sh

is the same as:

* 12 10,12,14,16 * * root backup.sh

*/15 9-17 * * * root connection.test

Will run connection.test every 15 mins between the hours or 9am and 5pm

To collect the cron execution execution log in a file :

30 18  *    *   *    rm /home/someuser/tmp/* > /home/someuser/cronlogs/clean_tmp_dir.log

LAST DAY of EVERY MONTH

cron job executing ‘myJob.sh’ to be ran on the LAST day of every month at 23:58 :

58 23 * * * [ `date +%d` -eq `echo \`cal\` | awk ‘{print $NF}’` ] && myJob.sh

 

AWK Tutorial

COINS.txt

gold     1      1986  USA                            American Eagle
gold     1      1908  Austria-Hungary     Franz Josef 100 Korona
silver  10     1981  USA                           ingot
gold     1      1984  Switzerland             ingot
gold     1      1979  RSA                           Krugerrand
gold     0.5   1981  RSA                           Krugerrand
gold     0.1   1986  PRC                           Panda
silver   1      1986  USA                           Liberty dollar
gold     0.25 1986  USA                          Liberty 5-dollar piece
silver   0.5   1986  USA                          Liberty 50-cent piece
silver   1      1987  USA                          Constitution dollar
gold     0.25 1987  USA                         Constitution 5-dollar piece
gold     1       1988  Canada                   Maple Leaf

 

List all the gold pieces:  

awk ‘/gold/ {print $5,$6,$7,$8}’ coins.txt

List all the coins that were minted before 1980.

awk ‘{if ($3 < 1980) print $3, "    ",$5,$6,$7,$8}’ coins.txt

1908      Franz Josef 100 Korona
1979      Krugerrand

 

List number of lines in coins.txt

awk ‘END {print NR,"coins"}’ coins.txt

 

awk ‘/gold/ {ounces += $2} END {print "value = $" 425*ounces}’  coins.txt

 

Working with Patterns

emp_names
46012   DULANEY     EVAN        MOBILE   AL
46013   DURHAM      JEFF        MOBILE   AL
46015   STEEN       BILL        MOBILE   AL
46017   FELDMAN     EVAN        MOBILE   AL
46018   SWIM        STEVE       UNKNOWN  AL
46019   BOGUE       ROBERT      PHOENIX  AR
46021   JUNE        MICAH       PHOENIX  AR
46022   KANE        SHERYL      UNKNOWN  AR
46024   WOOD        WILLIAM     MUNCIE   IN
46026   FERGUS      SARAH       MUNCIE   IN
46027   BUCK        SARAH       MUNCIE   IN
46029   TUTTLE      BOB         MUNCIE   IN
Displaying lines having pattern AL in emp_names file.
awk '/AL/ {print $3,$2}' emp_names
EVAN DULANEY
JEFF DURHAM
BILL STEEN
EVAN FELDMAN
STEVE SWIM
awk '$5 ˜ /AR/' emp_names
46019   BOGUE       ROBERT   PHOENIX   AR
46021   JUNE        MICAH    PHOENIX   AR
46022   KANE        SHERYL   UNKNOWN   AR
 
cat emp_names
46012:DULANEY:EVAN:MOBILE:AL
46013:DURHAM:JEFF:MOBILE:AL
46015:STEEN:BILL:MOBILE:AL
46017:FELDMAN:EVAN:MOBILE:AL
46018:SWIM:STEVE:UNKNOWN:AL
46019:BOGUE:ROBERT:PHOENIX:AR
46021:JUNE:MICAH:PHOENIX:AR
46022:KANE:SHERYL:UNKNOWN:AR
46024:WOOD:WILLIAM:MUNCIE:IN
46026:FERGUS:SARAH:MUNCIE:IN
46027:BUCK:SARAH:MUNCIE:IN
46029:TUTTLE:BOB:MUNCIE:IN
awk '{FS=":"}{print $2}' emp_names
awk -F: '{print $2}' emp_names
 

SSH and SCP

Setting Up Passwordless Connection

Secure Copy or SCP is a means of securely transferring computer files between a local and a remote host or between two remote hosts, using the Secure Shell (SSH) protocol.

The term SCP can refer to one of two related things, the SCP protocol or the SCP program.

*** use password less only if needed , making life easier then it needs to be has adverse consequences …
hosta/hostb

$ /usr/bin/ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/export/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /export/home/oracle/.ssh/id_rsa.
Your public key has been saved in /export/home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
3e:89:0c:44:5c:09:e9:07:98:cc:6a:35:35:17:3d:dd oracle@wdlilnxsusrs01

cat id_rsa.pub >> user@hostb:~/.ssh/authorized_keys

Now you can ssh/scp from host A to host B do the same on host B to allow both servers only if needed that host B connects to host A without password.

 

Using SCP to copy Files

If user dvader is on a computer called empire.gov, and wants to copy a file called file1.txt to a directory called somedir in his account on a computer called deathstar.com, he would enter:

scp file1.txt dvader@deathstar.com:somedir

Likewise, if he wanted to copy the entire contents of the somedir directory on deathstar.com back to his empire.gov account, he would enter:

scp -r dvader@deathstar.com:somedir somedir

Similarly, if he is working on another computer, but wanted to copy a file called file1.txt from his home directory on empire.gov to a directory called somedir in his account on deathstar.com, he would enter:

scp dvader@empire.gov:file1.txt dvader@deathstar.com:somedir

 

FIND Command

To look across the /bin and /var/html directory trees for filenames that contain the pattern *.htm*, you can use the following command:

find /usr /var/html -name “*.htm*” -print

Finding udump directory starting from ORACLE_BASE

find $ORACLE_BASE -name “udump” -print

Finding Directories

To find a list of the directories use can use the -type specifier.

find . -type d -print

 

Access,Modified,Changed Time

“-atime/-ctime/-mtime” [+|-]n

Specify selection of the files based on  three Unix timestamps: the last time a files’s “access time”, “file status” and “modification time”.  
n is time interval — an integer with optional sign. It is measured in 24-hour periods (days) or minutes counted from the current moment.

  • +n: if it has plus sing, then it means “more than n 24-hour periods (days) ago”, or older then n,
  • -n: if it has the minus sign, then it means less than n 24-hour periods (days) ago (-n), or younger than n. It’s evident that -1,  and 0 are the same and both means “today”.
  • n: If the integer n does not have sign this means exactly n 24-hour periods (days) ago, 0 means today.

find everything in your home directory modified in the last 24 hours:
                        find $HOME -mtime -1

Find everything in your home directory modified in the last seven 24-hour periods (days):
                       find $HOME -mtime -7

Find everything in your home directory that have NOT been modified in the last year:
                      find $HOME -mtime +365

To find html files that have been modified in the last seven 24-hour periods (days), I can use -mtime with the argument -7 (include the hyphen): 
                     find . -mtime -7 -name “*.html” -print

If you use the number 7 (without a hyphen), find will match only html files that were modified exactly seven 24-hour periods (days) ago:
                      find . -mtime 7 -name “*.html” -print

To find those html files that I haven’t touched for at least seven 24-hour periods (days), I use +7:
                     find . -mtime +7 -name “*.html” -print

 

-newer/-anewer/-cnewer baseline_file

The time of modification, access time or creation time are compared with the same timestamp in the baseline file. If file is a symbolic link and the -H option or the -L option is in effect, the modification time of the file it points to is always used.

  • -newer Modification time is compared with modification time of the basline_file True if file was modified more recently than baseline file.
  • -anewer Access time is compared with access time of basline_file . True if file was last accessed more recently than baseline file.
  • -cnewer Creation file is compared. For example: find everything in your home that has been modified more recently than “~joeuser/lastbatch.txt “:

find $HOME -newer ~joeuser/lastbatch.txt

 

-size

Locates files with specified size. -size attribute lets you specify how big the files should be to match. You can specify your size in kilobytes and optionally also use + or – to specify size greater than or less than specified argument. For example:

find /home -name “*.txt” -size 100k
find /home -name “*.txt” -size +100k
find /home -name “*.txt” -size -100k

 

Using Find with OR options

$ find . \( -name a.out -o -name ‘*.o’ -o -name ‘core’ \) -exec rm {} \;

File masks identifying the file types to be removed are located between the parentheses; each file mask is preceded by -name. This list can be extended to include any temporary file types you can come up with that need to be cleaned off the system. In the course of compiling and linking code, programmers and their tools generate file types like those shown in the example: a.out, *.o, and core. Other users have similar commonly generated temporary files and can edit the command accordingly, using file masks like *.tmp, *.junk, and so on. You might also find it useful to put the command into a script called clean, which you can execute whenever you need to clean a directory.

 

$ find $LOGDIR -type d -mtime +0 -exec compress -r {} \;

$ find $LOGDIR -type d -mtime +5 -exec rm -f {} \;

The first command runs all the directories (-type d) found in the $LOGDIR directory wherein a file’s data has been modified within the last 24 hours (-mtime +0) and compresses them (compress -r {}) to save disk space. The second command deletes them (rm -f {}) if they are more than a work-week old (-mtime +5), to increase the free space on the disk.

 

To list all zero-length files, use this command:

find . -empty -exec ls {} \;

FIND Logical Expressions

To negate a condition (logical NOT) . NOT should be specified with a backslash before exclamation point ( \! ). For example

find . \! -name “*.gz” -exec gzip {} \;

\( expression \)  —  “escaped parentheses” to specify any composite condition. For example

find / -type f \( -perm -4000 -o -perm -2000 \) -exec ls -l {} \;

If you wanted to find a list of files that have been modified in the last 24 hours and which has a permission of 777, you would execute the following command:

find . -perm 777  -mtime 0 -print
Which is the same as:
find . -perm 777 -a -mtime 0 -a -print

 

Link of Checking Age of Files with FIND command

http://www.softpanorama.org/Tools/Find/selecting_files_by_age.shtml

 

Touch Command

Touch command change only mtime. For example if you need to change the date to Jun 21, 2008 9AM to example.txt, then you can go (-t parameter in touch has format [[CC]YY]MMDDhhmm[.SS]):

    touch -t 200907210900 example.txt

 

To list files in /u/bill that were last read from 2 to 6 minutes ago:
     

    find /u/bill -amin +2 -amin -6

 

Comparing TimeStamps

There are two ways to list files in /usr modified after February 1 of the current year. One uses ‘-newermt’:

     find /usr -newermt “Feb 1”

The other way of doing this works on the versions of find before 4.3.3:

     touch -t 02010000 /tmp/stamp$$
     find /usr -newer /tmp/stamp$$
     rm -f /tmp/stamp$$

 

Using -exec and XARGS with FIND

The find command is commonly used to remove core files that are more than a few 24-hour periods (days) old. These core files are copies of the actual memory image of a running program when the program dies unexpectedly. They can be huge, so occasionally trimming them is wise:

find . -name core -ctime +4 -exec /bin/rm -f {} \;

 

For example often one needs to find files containing a specific pattern in multiple directories one can use an exec option in find (please note that you should use the -l flag for grep so that grep specifies the matched filenames):

find . -type f -exec grep -li ‘/bin/ksh’ {} \;

Find how many directories are in a path (counts current directory)
# find . -type d -exec basename {} \; wc -l

Find how many files are in a path
# find . -type f -exec basename {} \; wc -l

Find files that were modified less than 7 days ago and archive
# find . -type f -mtime -7 xargs tar -cvf `date ‘+%d%m%Y’_archive.tar`

More examples on this link

 

FIND EXAMPLES

 

Korn Shell conditional expressions

A conditional expression is an expression placed between double brackets (‘[[‘ and ‘]]’). These expressions are used in conjuction with the test command to test the truth of a comparison. Their use is most beneficial in shell scripts. Word splitting and file name substitution are not performed in the expressions. The table below contains the unary and binary expressions allowed in these statements.

-a File
True, if the specified file is a symbolic link that points to another file that does exist.

-b File
True, if the specified file exists and is a block special file.

-c File
True, if the specified file exists and is a character special file.

-d File
True, if the specified file exists and is a directory.

-e File
True, if the specified file exists.

-f File
True, if the specified file exists and is an ordinary file.

-g File
True, if the specified file exists and its setgid bit is set.

-h File
True, if the specified file exists and is a symbolic link.

-k File
True, if the specified file exists and its sticky bit is set.

-n String
True, if the length of the specified string is nonzero.

-o Option
True, if the specified option is on.

-p File
True, if the specified file exists and is a FIFO special file or a pipe.

-r File
True, if the specified file exists and is readable by the current process.

-s File
True, if the specified file exists and has a size greater than 0.

-t FileDescriptor
True, if specified file descriptor number is open and associated with a terminal device.

-u File
True, if the specified file exists and its setuid bit is set.

-w File
True, if the specified file exists and the write bit is on. However, the file will not be writable on a read-only file system even if this test indicates true.

-x File
True, if the specified file exists and the execute flag is on. If the specified file exists and is a directory, then the current process has permission to search in the directory.

-z String
True, if length of the specified string is 0.

-L File
True, if the specified file exists and is a symbolic link.

-O File
True, if the specified file exists and is owned by the effective user ID of this process.

-G File
True, if the specified file exists and its group matches the effective group ID of this process.

-S File
True, if the specified file exists and is a socket.

File1 -nt File2
True, if File1 exists and is newer than File2.

File1 -ot File2
True, if File1 exists and is older than File2.

File1 -ef File2
True, if File1 and File2 exist and refer to the same file.

String1 = String2
True, if String1 is equal to String2.

String1 != String2
True, if String1 is not equal to String2.

String = Pattern
True, if the specified string matches the specified pattern.

String != Pattern
True, if the specified string does not match the specified pattern.

String1 < String2
True, if String1 comes before String2 based on the ASCII value of their characters.

String1 > String2
True, if String1 comes after String2 based on the ASCII value of their characters.

Expression1 -eq Expression2
True, if Expression1 is equal to Expression2.

Expression1 -ne Expression2
True, if Expression1 is not equal to Expression2.

Expression1 -lt Expression2
True, if Expression1 is less than Expression2.

Expression1 -gt Expression2
True, if Expression1 is greater than Expression2.

Expression1 -le Expression2
True, if Expression1 is less than or equal to Expression2.

Expression1 -ge Expression2
True, if Expression1 is greater than or equal to Expression2.