rsync + sersync data synchronization

rsync + sersync

rsync is a data mirroring backup tool under linux system. Use Remote Sync, a fast incremental backup tool, to synchronize remotely, support local replication, or synchronize with other SSH or rsync hosts.

Sersync is developed based on inotify and is similar to inotify-tools. It monitors the specified directory files (records the addition, deletion, and modification of specific file directories), and performs incremental data synchronization through rsync, and only synchronizes changed files or directories. .

** rsync+sersync data synchronization**

sersync monitors the changes of the specified directory file event;
Sersync calls the rsync client command to synchronize the updated directories and files incrementally to the rsync server to achieve data synchronization;

system version

cat /etc/redhat-release 
CentOS Linux release 8.1.1911(Core)

Install rsync

dnf install  rsync

View rsync version

rpm -qa rsync
rsync --version

rsync server

rsync configuration file

cat /etc/rsyncd.conf 
uid=root
gid=root
max connections=100
use chroot=true
log file=/var/log/rsyncd.log
motd file=/etc/rsyncd.motd
transfer logging=true
hosts allow=192.168.99.0/24[data]
path=/data
comment=data
read only =no
list=yes
auth users=root
secrets file=/etc/rsyncd.pwd

rsync configuration file parameters

uid=root        #When transferring files on the server side, which user and user group should be sent for execution
gid=root        #When transferring files on the server side, which user and user group should be sent for execution
max connections=100      #Maximum number of client connections
use chroot=true          
# With chroot, before transferring files, the server daemon is chrooting to a directory in the file system, so
# The advantage is that it may protect the system from being attacked by installation loopholes. The disadvantage is that it requires super user permissions. Symbolic link
# Files will be excluded. In other words, you are on the rsync server, if there is a symbolic link, you are on the backup server
# When you run the synchronization data of the client on the client, only the symbolic link name will be synchronized, and the content of the symbolic link will not be synchronized
log file=/var/log/rsyncd.log        #rsync server log
motd file=/etc/rsyncd.motd         
# Define motd file path rsyncd.The motd content defines the server information, and the user will see this information when logging in
transfer logging=true             #Transfer file log
hosts allow=192.168.99.0/24       #Allowed ip address
[ data]                  #Module name
path=/data              #Synced directory
exclude=web             #The directory removed from the synchronization directory, that is, the web is not synchronized
comment=data            #Note, the content of the note can be defined by yourself, which serves as a reminder
read only=no            #Read-only option, if yes, the client is not allowed to upload files to the server
list=yes                #Whether the directory that provides synchronization data on the server is displayed
auth users=root         #Authenticated user is root
secrets file=/etc/rsyncd.pwd       
# Password file save path, note that the password of this file must be the same as rsyncd on 223.Same password in pwd2 file,Authority
# Must be set to 600, the key file format used to authenticate the client USERNAME:PASSWD
# Note: There can be no comment information in the configuration file

rsync password file

echo "root:123456">/etc/rsyncd.pwd       
echo "123456">/etc/rsyncd.pwd2

Modify permissions

chmod 600/etc/rsyncd.pwd                   
chmod 600/etc/rsyncd.pwd2

systemctl manages rsync service

/etc/sysconfig/rsyncd

vim /etc/sysconfig/rsyncd 
OPTIONS=""

rsyncd.service

cat  /lib/systemd/system/rsyncd.service
[ Unit]
Description=fast remote file copy program daemon
ConditionPathExists=/etc/rsyncd.conf

[ Service]
EnvironmentFile=/etc/sysconfig/rsyncd
ExecStart=/usr/bin/rsync --daemon --no-detach "$OPTIONS"[Install]
WantedBy=multi-user.target

rsyncd start

systemctl   start  rsyncd
systemctl   enable  rsyncd

rsync port process

netstat  -auntlp |grep 873
ps -ef |grep rsync

firewall

firewall-cmd    --add-port=873/tcp --permanent
firewall-cmd    --reload

rsync client

rsync local disk synchronization data

rsync -vlzrtogp /mnt/data /backups
rsync -vlzrtogp /mnt/data//backups

/mnt/data means copy the entire /data directory to the target directory

/mnt/data/ means to copy all the contents in the data directory to the target directory

rsync remote synchronization

rsync -vlzrtopg --progress --delete/mnt/data/ [email protected]::data --password-file=/etc/rsyncd.pwd2

rsync data synchronization current limit

rsync -vlzrtogp --progress --delete--bwlimit=1000 remote files local files//Limited to 1000kBytes/s
rsync -vlzrtogp --progress --delete--bwlimit=2048 local file remote file//Limited to 2M/s

rsync client parameters

- v,--verbose verbose output mode
- l,--links keep symbolic link files
- z,--compress compresses files when transferring
- r,--recursive processes subdirectories in recursive mode
- t,--times keep file time information
- o,--owner keeps file owner information(super-user only)-p,--perms keep file permissions
- g,--group keep file group information

- - progress displays the transfer process while transferring
- - delete Delete the redundant file directory of the target end than the local end to maintain data consistency
- - exclude=web specifies to exclude a file matching pattern that does not need to be transferred
- - exclude-from=FILE reads exclusion rules from FILE
- - include=PATTERN specifies the file matching mode to be transferred
- - include-from=FILE reads inclusion rules from FILE
- - password-file=FILE reads the password from FILE to avoid entering the password on the terminal
- - bwlimit=RATE limit transfer I/O bandwidth

rsync + sersync

Install rsync

dnf install  rsync

rsync password file

echo "123456">/etc/rsyncd.pwd2

Modify permissions

chmod 600/etc/rsyncd.pwd2

Download sersync

cd /opt
wget https://sersync.googlecode.com/files/sersync2.5.4_64bit_binary_stable_final.tar.gz

Unzip sersync

tar -zxvf sersync2.5.4_64bit_binary_stable_final.tar.gz  
mv GNU-Linux-x86 /usr/local/sersync

** sersync configuration file**

vim /usr/local/sersync/confxml.xml

Start sersync

/usr/local/sersync/sersync2  -d -r -o  /usr/local/sersync/confxml.xml

View sersync process

ps  -ef |grep sersync2

sersync variables

[ root@kafka-node2 /]# echo "export PATH=$PATH:/usr/local/sersync">>/etc/profile
[ root@kafka-node2 /]# source /etc/profile

Data synchronization test

Source server creation file

Destination server view

rsync data synchronization log

sersync multi-instance

When monitoring multiple directories for data synchronization, configure different sersync configuration files separately

/usr/local/sersync/sersync2  -d -r -o  /usr/local/sersync/nginx_confxml.xml
/usr/local/sersync/sersync2  -d -r -o  /usr/local/sersync/zabbix_confxml.xml

rsync + sersync data synchronization - CentOS 8