Category: Linux

Backup rsync

1. Initial source folder structure:

me@MacBook:~/tmp$ find source
source
source/.DS_Store
source/a
source/b
source/dev
source/dev/.DS_Store
source/dev/x
source/dev/y.orig.vhd
source/dev/y.vhd

2. Lets backup source folder to destination folder using rsync

me@MacBook:~/tmp$ rsync -abvP  --backup-dir=backup_`date +%Y-%m-%d--%H-%M-%S` --include=*.orig.vhd --exclude={/backup_*,.DS_Store,*.vhd} --delete source/ destination/
building file list ... 
6 files to consider
created directory destination
./
a
           1 100%    0.00kB/s    0:00:00 (xfer#1, to-check=4/6)
b
           1 100%    0.98kB/s    0:00:00 (xfer#2, to-check=3/6)
dev/
dev/x
           1 100%    0.98kB/s    0:00:00 (xfer#3, to-check=1/6)
dev/y.orig.vhd
           1 100%    0.98kB/s    0:00:00 (xfer#4, to-check=0/6)

sent 343 bytes  received 120 bytes  926.00 bytes/sec
total size is 4  speedup is 0.01

explanation:

--include=*.orig.vhd

do not exclude files *.orig.vhd for any source (sub)directories (since include is before exclude then inclusion will first match rather then exclusion from *.vhd)

--exclude={/backup_*,.DS_Store,*.vhd}

copy all files excluding ~/tmp/backup_* files, excluding all .DS_Store and *.vhd files in any (sub)directories

3. The destination directory content:

me@MacBook:~/tmp$ find destination
destination
destination/a
destination/b
destination/dev
destination/dev/x
destination/dev/y.orig.vhd

4. Lets modify one file and delete another and then run the same rsync command:

me@MacBook:~/tmp$ echo "2" > source/a 
me@MacBook:~/tmp$ rm source/dev/x
me@MacBook:~/tmp$ rsync -abvP  --backup-dir=backup_`date +%Y-%m-%d--%H-%M-%S` --include=*.orig.vhd --exclude={/backup_*,.DS_Store,*.vhd} --delete source/ destination/
building file list ... 
5 files to consider
deleting dev/x
./
a
           2 100%    0.00kB/s    0:00:00 (xfer#1, to-check=3/5)
dev/

sent 195 bytes  received 54 bytes  498.00 bytes/sec
total size is 4  speedup is 0.02
--backup-dir=backup_`date +%Y-%m-%d--%H-%M-%S`

creates a backup folder in destination dir with previous versions of modified or deleted files

--delete

deletes the file in destination folder if they were deleted in the source dir

5. Then the destination folder will contain backup_2016-10-29–12-26-06 with previous version of a and dev/x before deletion:

me@MacBook:~/tmp$ find destination
destination
destination/a
destination/b
destination/backup_2016-10-29--12-26-06
destination/backup_2016-10-29--12-26-06/a
destination/backup_2016-10-29--12-26-06/dev
destination/backup_2016-10-29--12-26-06/dev/x
destination/dev
destination/dev/y.orig.vhd
Advertisements

Understanding unix load for java processes

Given: Unix has 16 cores.

1.State before the load:
top – 15:52:37 up 128 days, 11:58, 2 users, load average: 0.20, 0.97, 0.83
Tasks: 343 total, 1 running, 340 sleeping, 0 stopped, 2 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.8%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 49453936k total, 36820724k used, 12633212k free, 1120572k buffers
Swap: 8388600k total, 0k used, 8388600k free, 33165052k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18198 tktdev 15 0 11016 1264 784 R 0.3 0.0 0:00.07 top

2. State during load:
$ top
top – 16:00:53 up 128 days, 12:06, 2 users, load average: 18.05, 12.28, 6.18
Tasks: 359 total, 7 running, 349 sleeping, 0 stopped, 3 zombie
Cpu(s): 99.6%us, 0.3%sy, 0.0%ni, 0.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 49453936k total, 36900812k used, 12553124k free, 1120572k buffers
Swap: 8388600k total, 0k used, 8388600k free, 33165072k cached

load of 18.05 from last minute means that work load is getting bigger than it can consume (usually system admins tend to keep load as 0.75 * number of cores, so here 12)

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31111 tktdev 16 0 14.0g 33m 11m S 1576.2 0.1 30:19.38 java

Cpu0 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 99.7%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 99.7%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu12 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu13 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu14 : 99.0%us, 0.3%sy, 0.0%ni, 0.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu15 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

ssh port forwarding

Problem:

You are on box A and you want to connect to box C (to some port),but there is not direct connectivity from box A to box C on to a particular port: e.g.

A $ telnet C_IP_OR_NAME C_PORT ( does not work)

However, there is a box B which has connectivity to box C to that port

B $ telnet C_IP_OR_NAME C_PORT ( does work)

Solution:

Assuming you have ssh connectivity from box A to box B, then start port forwarding on box A to be routed to box C via box B:

ssh -f B_USER@B_IP_OR_NAME -L A_PORT:C_IP_OR_NAME:C_PORT -N

that will make every call to the A_PORT on box A to be routed to box C port (C_PORT) via box B

grep pattern inluding new line character

bawi@hp:~$ cat a.txt
aaabbb
cccddd

dddbbb
xxxxxx
cccfff
bawi@hp:~$ pcregrep -M 'b\nc' a.txt
aaabbb
cccddd
bawi@hp:~$ pcregrep -M 'b(\n|.)*c' a.txt
aaabbb
cccddd

dddbbb
xxxxxx
cccfff
bawi@hp:~$ cat b.txt 
1.ERROR test1
known-pattern1
other1.1
2.DEBUG test2
some data2.1
some data2.2
3.ERROR test3
undefined
other3.1
4.ERROR test4
known-pattern2

bawi@hp:~$ pcregrep -M 'ERROR .*\n' b.txt | pcregrep -M -v '\nknown-pattern1' | pcregrep -M -v '\nknown-pattern2'
3.ERROR test3
undefined