Category: Linux

Backup rsync

1. Initial source folder structure:

me@MacBook:~/tmp$ find source
source
source/.DS_Store
source/a
source/b
source/dev
source/dev/.DS_Store
source/dev/x
source/dev/y.orig.vhd
source/dev/y.vhd

2. Lets backup source folder to destination folder using rsync

me@MacBook:~/tmp$ rsync -abvP  --backup-dir=backup_`date +%Y-%m-%d--%H-%M-%S` --include=*.orig.vhd --exclude={/backup_*,.DS_Store,*.vhd} --delete source/ destination/
building file list ... 
6 files to consider
created directory destination
./
a
           1 100%    0.00kB/s    0:00:00 (xfer#1, to-check=4/6)
b
           1 100%    0.98kB/s    0:00:00 (xfer#2, to-check=3/6)
dev/
dev/x
           1 100%    0.98kB/s    0:00:00 (xfer#3, to-check=1/6)
dev/y.orig.vhd
           1 100%    0.98kB/s    0:00:00 (xfer#4, to-check=0/6)

sent 343 bytes  received 120 bytes  926.00 bytes/sec
total size is 4  speedup is 0.01

explanation:

--include=*.orig.vhd

do not exclude files *.orig.vhd for any source (sub)directories (since include is before exclude then inclusion will first match rather then exclusion from *.vhd)

--exclude={/backup_*,.DS_Store,*.vhd}

copy all files excluding ~/tmp/backup_* files, excluding all .DS_Store and *.vhd files in any (sub)directories

3. The destination directory content:

me@MacBook:~/tmp$ find destination
destination
destination/a
destination/b
destination/dev
destination/dev/x
destination/dev/y.orig.vhd

4. Lets modify one file and delete another and then run the same rsync command:

me@MacBook:~/tmp$ echo "2" > source/a 
me@MacBook:~/tmp$ rm source/dev/x
me@MacBook:~/tmp$ rsync -abvP  --backup-dir=backup_`date +%Y-%m-%d--%H-%M-%S` --include=*.orig.vhd --exclude={/backup_*,.DS_Store,*.vhd} --delete source/ destination/
building file list ... 
5 files to consider
deleting dev/x
./
a
           2 100%    0.00kB/s    0:00:00 (xfer#1, to-check=3/5)
dev/

sent 195 bytes  received 54 bytes  498.00 bytes/sec
total size is 4  speedup is 0.02
--backup-dir=backup_`date +%Y-%m-%d--%H-%M-%S`

creates a backup folder in destination dir with previous versions of modified or deleted files

--delete

deletes the file in destination folder if they were deleted in the source dir

5. Then the destination folder will contain backup_2016-10-29–12-26-06 with previous version of a and dev/x before deletion:

me@MacBook:~/tmp$ find destination
destination
destination/a
destination/b
destination/backup_2016-10-29--12-26-06
destination/backup_2016-10-29--12-26-06/a
destination/backup_2016-10-29--12-26-06/dev
destination/backup_2016-10-29--12-26-06/dev/x
destination/dev
destination/dev/y.orig.vhd

Understanding unix load for java processes

Given: Unix has 16 cores.

1.State before the load:
top – 15:52:37 up 128 days, 11:58, 2 users, load average: 0.20, 0.97, 0.83
Tasks: 343 total, 1 running, 340 sleeping, 0 stopped, 2 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.8%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 49453936k total, 36820724k used, 12633212k free, 1120572k buffers
Swap: 8388600k total, 0k used, 8388600k free, 33165052k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18198 tktdev 15 0 11016 1264 784 R 0.3 0.0 0:00.07 top

2. State during load:
$ top
top – 16:00:53 up 128 days, 12:06, 2 users, load average: 18.05, 12.28, 6.18
Tasks: 359 total, 7 running, 349 sleeping, 0 stopped, 3 zombie
Cpu(s): 99.6%us, 0.3%sy, 0.0%ni, 0.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 49453936k total, 36900812k used, 12553124k free, 1120572k buffers
Swap: 8388600k total, 0k used, 8388600k free, 33165072k cached

load of 18.05 from last minute means that work load is getting bigger than it can consume (usually system admins tend to keep load as 0.75 * number of cores, so here 12)

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31111 tktdev 16 0 14.0g 33m 11m S 1576.2 0.1 30:19.38 java

Cpu0 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 99.7%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 99.7%us, 0.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu12 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu13 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu14 : 99.0%us, 0.3%sy, 0.0%ni, 0.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu15 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

ssh port forwarding

Problem:

You are on box A and you want to connect to box C (to some port),but there is not direct connectivity from box A to box C on to a particular port: e.g.

A $ telnet C_IP_OR_NAME C_PORT ( does not work)

However, there is a box B which has connectivity to box C to that port

B $ telnet C_IP_OR_NAME C_PORT ( does work)

Solution:

Assuming you have ssh connectivity from box A to box B, then start port forwarding on box A to be routed to box C via box B:

ssh -f B_USER@B_IP_OR_NAME -L A_PORT:C_IP_OR_NAME:C_PORT -N

that will make every call to the A_PORT on box A to be routed to box C port (C_PORT) via box B

grep pattern inluding new line character

bawi@hp:~$ cat a.txt
aaabbb
cccddd

dddbbb
xxxxxx
cccfff
bawi@hp:~$ pcregrep -M 'b\nc' a.txt
aaabbb
cccddd
bawi@hp:~$ pcregrep -M 'b(\n|.)*c' a.txt
aaabbb
cccddd

dddbbb
xxxxxx
cccfff
bawi@hp:~$ cat b.txt 
1.ERROR test1
known-pattern1
other1.1
2.DEBUG test2
some data2.1
some data2.2
3.ERROR test3
undefined
other3.1
4.ERROR test4
known-pattern2

bawi@hp:~$ pcregrep -M 'ERROR .*\n' b.txt | pcregrep -M -v '\nknown-pattern1' | pcregrep -M -v '\nknown-pattern2'
3.ERROR test3
undefined

Linux ubuntu lucid apt sources.list

My list:
deb http://pl.archive.ubuntu.com/ubuntu/ lucid main restricted universe multiverse
deb-src http://pl.archive.ubuntu.com/ubuntu/ lucid main restricted universe multiverse
deb http://pl.archive.ubuntu.com/ubuntu/ lucid-updates main restricted universe multiverse
deb-src http://pl.archive.ubuntu.com/ubuntu/ lucid-updates main restricted universe multiverse
# deb http://pl.archive.ubuntu.com/ubuntu/ lucid-proposed main restricted universe multiverse
# deb-src http://pl.archive.ubuntu.com/ubuntu/ lucid-proposed main restricted universe multiverse
# deb http://pl.archive.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse
# deb-src http://pl.archive.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu lucid-security main restricted universe multiverse
deb-src http://security.ubuntu.com/ubuntu lucid-security main restricted universe multiverse
deb http://archive.canonical.com/ubuntu lucid partner
deb-src http://archive.canonical.com/ubuntu lucid partner
deb http://packages.medibuntu.org/ lucid free non-free
deb-src http://packages.medibuntu.org/ lucid free non-free
deb http://archive.getdeb.net/ubuntu lucid-getdeb apps
deb-src http://archive.getdeb.net/ubuntu lucid-getdeb apps

# deb http://ppa.launchpad.net/kkszysiu/telepathy/ubuntu lucid main
# deb-src http://ppa.launchpad.net/kkszysiu/telepathy/ubuntu lucid main

deb http://download.virtualbox.org/virtualbox/debian lucid non-free

#sipe pidgin
# deb http://ppa.launchpad.net/aavelar/ppa/ubuntu lucid main
# deb-src http://ppa.launchpad.net/aavelar/ppa/ubuntu lucid main

#W: GPG error: http://ppa.launchpad.net lucid Release: The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY 5A9A06AEF9CB8DB0
# to solve (type for the last 8 chars of the missing key)
gpg –keyserver subkeys.pgp.net –recv-keys F9CB8DB0
gpg –export –armor F9CB8DB0 | sudo apt-key add –

Log4j syslog appender

Ubuntu 10.04 Lucid rsyslogd

1) Uncomment
/etc/rsyslog.conf:
# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

2) sudo service rsyslog restart

3)

    
        
        
        
        
            
        
    

4)
public class TestSyslog {

  static Logger logger = Logger.getLogger("monitoring");

  @Test
  public void shouldLogToSyslog() throws Exception {
    // given
    String msg = "My message!!!";


    // when
    logger.error(message);


    // then
    if (isLinux()) {
      assertTrue(logContains("/var/log/syslog", msg));
      assertTrue(logContains("/var/log/user.log", msg));
    }
  }


  private boolean isLinux() {
    String os = System.getProperty("os.name");
    return "Linux".equals(os);
  }


  boolean logContains(String userLogFile, String msg) throws IOException {
    String fileAsString = FileUtils.readFileToString(new File(userLogFile));
    return fileAsString.contains(msg);
  }
}

linux installation

Pre installation
/etc/apt/sources.list:
# java
deb http://archive.canonical.com/ubuntu lucid partner
deb-src http://archive.canonical.com/ubuntu lucid partner
# virtualbox
deb http://download.virtualbox.org/virtualbox/debian lucid non-free
#
deb http://packages.medibuntu.org/ lucid free non-free
deb-src http://packages.medibuntu.org/ lucid free non-free
# sipe for pidgin-sipe
deb http://ppa.launchpad.net/aavelar/ppa/ubuntu jaunty main
deb-src http://ppa.launchpad.net/aavelar/ppa/ubuntu jaunty main

Installation
sudo apt-get install sun-java6-bin sun-java6-jdk sun-java6-jre sun-java6-plugin sun-java6-source ssh vim w32codecs mplayer mplayer-gui vlc virtualbox-3.2 adobe-flashplugin pidgin pidgin-sipe acroread

Post installation
virtualbox usb: add user to vboxusers (menu: users/groups -> manage groups -> check box user)
*.sh scripts in /etc/init.d/: sudo update-rc.d script.sh defaults 90
alt+F2: gconf-editor -> apps/metacity/general/button_layout

~/.bashrc:
alias psj=’ps -ef | grep java | grep -v grep’
export JAVA_HOME=/usr/lib/jvm/java-6-sun
export M2_HOME=$HOME/dev/environment/apache-maven-2.2.1
export ANT_HOME=$HOME/dev/environment/apache-ant-1.8.1
export PATH=$JAVA_HOME/bin:$M2_HOME/bin:$ANT_HOME/bin:$HOME/dev/environment/flash_player_10_linux_dev/standalone/release:$HOME/bin:$PATH
export MAVEN_OPTS=” -Dsocket.base=80 ”