使用inotify-tools与rsync构建实时备份系统

使用inotifywait监控文件变动

inotifywait是 inotify-tools 包中提供的一个工具,它使用 inotify API 来监控文件/目录中的变动情况。

在archlinux上,我们可以使用下面命令来安装

sudo pacman -S --noconfirm inotify-tools

平时 inotifywait 会挂起在那里,直到文件/目录发生了要引起关注的事件后,它会退出并输出事件发生的场所、事件的名称以及引起事件的文件(当事件发生在目录上时才会输出).

inotifywait 最常用的选项有两个,一个是 -r 一个是 -e ,其中:

-r
表示递归监控子目录中文件发生的事件

-e
指定要监控的事件列表。对于备份系统来说,只需要监控 modify、create和delete三种事件就行了。

比如,我们运行

inotifywait -r -e modify,create,delete /tmp

表示监控 /tmp 目录及其子目录中文件修改、文件创建和文件删除三种事件。

这时程序一直在挂起状态

[lujun9972@X61 ~]$ inotifywait -r -e modify,create,delete /tmp
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.

这时在 /tmp 目录下新建一个文件

touch /tmp/newFile

则 inotifywait 进程退出,并输出如下信息

/tmp/ CREATE newFile

使用rsync同步变动

rsync是一款快速增量备份工具。它的具有以下几个特点使得它很适合用作做备份的工具:

  • 增量备份,只会传输修改过的内容
  • 可以在传输过程中实时解压缩,减少带宽消耗
  • 可以保持原来文件的权限、事件、软硬链接
  • 即支持本机复制,也支持远程复制

rsync常用法为:

rsync -avz --delete  src/ foo:/data

其中

-a
表示archive mode,即备份目录下的所有内容(包括子目录中的内容),并且保持软链接、文件属性、文件修改事件、文件的所有者和宿主信息不变,并且同步字符/块设备以及命名socket和fifo等特殊文件。

-v
表示输出备份的详细信息

-z
表示传输时进行压缩

--delete
删除备份目的地里src中没有的文件

src/
表示要备份的是src目录下的所有内容,注意这里最后的 / 不能去掉,否则会把src目录本身备份过去

foo:/data
表示备份的目的地是foo主机下的 /data/ 目录

整合起来

接下来我们只需要用个 while 死循环把两个工具整合起来就行了,非常简单

#!/bin/bash

if [[ $# -ne 2 ]];then
    cat<<EOF
Usage $(basename $0) source_dir [host:]dest_dir
EOF
    exit 0
fi

source_dir=$1
dest_dir=$2
while :
do
    inotifywait -r -e modify,create,delete ${source_dir} && rsync -avz ${source_dir}/ ${dest_dir} --delete
done

这里有必要说明的是,虽然用 inotifywait 能探测出文件具体做了什么改动,但实际上我们根本不需要知道具体的改变是什么。

我们只需要知道有所改变了,然后具体改变了什么由 rsync 来自己处理就行了。

CentOS 6.9上inotify-tools 安装及使用方法

Linux内核从2.6.13开始,引入了inotify机制。通过intofity机制,能够对文件系统的变化进行监控,如对文件进行创建、删除、修改等操作,可以及时通知应用程序进行相关事件的处理。这种响应处理机制,避免了频繁的文件轮询任务,提高了任务的处理效率。

一、检查系统内核版本

[root@localhost tan]# uname -r
2.6.32-696.el6.x86_64

二、检查系统是否支持inotify

[root@localhost tan]# ls -lsart /proc/sys/fs/inotify  
total 0
0 dr-xr-xr-x 0 root root 0 Jan 17 08:47 ..
0 dr-xr-xr-x 0 root root 0 Jan 17 08:53 .
0 -rw-r--r-- 1 root root 0 Jan 19 13:33 max_user_watches
0 -rw-r--r-- 1 root root 0 Jan 19 13:33 max_queued_events
0 -rw-r--r-- 1 root root 0 Jan 19 13:40 max_user_instances
[root@localhost tan]#

如果出现上面结果说明系统支持inotify。

三、下载安装(下载有点慢)

[root@localhost tan]#wget http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz  

[root@localhost tan]# tar -zvxf inotify-tools-3.14.tar.gz  
[root@localhost tan]# cd inotify-tools-3.14  

[root@localhost inotify-tools-3.14]# ./configure --prefix=/usr/local/inotify  
[root@localhost inotify-tools-3.14]# make  
[root@localhost inotify-tools-3.14]# make install

四、查看inotify默认参数

[root@localhost bin]# sysctl -a | grep max_queued_events  
fs.inotify.max_queued_events = 16384  

[root@localhost bin]# sysctl -a | grep max_user_watches  
fs.inotify.max_user_watches = 8192  
fs.epoll.max_user_watches = 798863  

[root@localhost bin]# sysctl -a | grep max_user_instances  
fs.inotify.max_user_instances = 128

五、修改inotify参数

1、命令修改

[root@localhost bin]# sysctl -w fs.inotify.max_user_instances=130  
fs.inotify.max_user_instances = 130

2、文件修改

[root@localhost]# vi /etc/sysctl.conf  
#添加如下代码  
fs.inotify.max_user_instances=130

3、参数说明

  • max_user_instances:每个用户创建inotify实例最大值
  • max_queued_events:inotify队列最大长度,如果值太小,会出现错误,导致监控文件不准确
  • max_user_watches:要知道同步的文件包含的目录数,可以用:
    [root@localhost]# find /home/rain -type d|wc -l 统计,必须保证参数值大于统计结果(/home/tan/uploadFile/为同步文件目录)。

六、创建实时监控脚本 (file 里面放的需要监听的目录)

[root@localhost shell]# vi inotify.sh
/usr/local/inotify/bin/inotifywait -mrq -e modify,create,move,delete --fromfile '/root/shell/file' --timefmt '%y-%m-%d %H:%M' --format '%T %w%f %e' --outfile '/home/tan/inotify.log'
[root@localhost shell]# vi file 

/home/tan
@/home/tan/uploadFile

inotifywait常用参数:

  • –timefmt 时间格式
  • %y年 %m月 %d日 %H小时 %M分钟
  • –format 输出格式
  • %T时间 %w路径 %f文件名 %e状态
  • -m 始终保持监听状态,默认触发事件即退出。
  • -r 递归查询目录
  • -q 打印出监控事件
  • -e 定义监控的事件,可用参数:
  • open 打开文件
  • access 访问文件
  • modify 修改文件
  • delete 删除文件
  • create 新建文件
  • attrb 属性变更
事件  描述

access  访问,读取文件。
modify  修改,文件内容被修改。
attrib  属性,文件元数据被修改。
move    移动,对文件进行移动操作。
create  创建,生成新文件
open    打开,对文件进行打开操作。
close   关闭,对文件进行关闭操作。
delete  删除,文件被删除。

七:实例操作

1、首先启动监听脚本,权限问题的话先:chmod 755 inotify.sh

[root@localhost shell]# ./inotify.sh

我在监听的目录中上传文件,没截图,自己想象吧

[root@localhost tan]# cat inotify.log 
18-01-19 15:07 /home/tan/uploadfile/test//ssh.txt CREATE
18-01-19 15:13 /home/tan/uploadfile/test/QQ20180119143826.png CREATE

这里可以看到打印的日志了。

八、附

1、inotifywait

使用方法和参数说明: 使用命令help就行

[root@localhost bin]# ./inotifywait -h
inotifywait 3.14
Wait for a particular event on a file or set of files.
Usage: inotifywait [ options ] file1 [ file2 ] [ file3 ] [ ... ]
Options:
        -h|--help       Show this help text.
        @<file>         Exclude the specified file from being watched.
        --exclude <pattern>
                        Exclude all events on files matching the
                        extended regular expression <pattern>.
        --excludei <pattern>
                        Like --exclude but case insensitive.
        -m|--monitor    Keep listening for events forever.  Without
                        this option, inotifywait will exit after one
                        event is received.
        -d|--daemon     Same as --monitor, except run in the background
                        logging events to a file specified by --outfile.
                        Implies --syslog.
        -r|--recursive  Watch directories recursively.
        --fromfile <file>
                        Read files to watch from <file> or `-' for stdin.
        -o|--outfile <file>
                        Print events to <file> rather than stdout.
        -s|--syslog     Send errors to syslog rather than stderr.
        -q|--quiet      Print less (only print events).
        -qq             Print nothing (not even events).
        --format <fmt>  Print using a specified printf-like format
                        string; read the man page for more details.
        --timefmt <fmt> strftime-compatible format string for use with
                        %T in --format string.
        -c|--csv        Print events in CSV format.
        -t|--timeout <seconds>
                        When listening for a single event, time out after
                        waiting for an event for <seconds> seconds.
                        If <seconds> is 0, inotifywait will never time out.
        -e|--event <event1> [ -e|--event <event2> ... ]
                Listen for specific event(s).  If omitted, all events are 
                listened for.

Exit status:
        0  -  An event you asked to watch for was received.
        1  -  An event you did not ask to watch for was received
              (usually delete_self or unmount), or some error occurred.
        2  -  The --timeout option was given and no events occurred
              in the specified interval of time.

Events:
        access          file or directory contents were read
        modify          file or directory contents were written
        attrib          file or directory attributes changed
        close_write     file or directory closed, after being opened in
                        writeable mode
        close_nowrite   file or directory closed, after being opened in
                        read-only mode
        close           file or directory closed, regardless of read/write mode
        open            file or directory opened
        moved_to        file or directory moved to watched directory
        moved_from      file or directory moved from watched directory
        move            file or directory moved to or from watched directory
        create          file or directory created within watched directory
        delete          file or directory deleted within watched directory
        delete_self     file or directory was deleted
        unmount         file system containing file or directory unmounted

2、inotifywatch

使用方法和参数说明:

[root@localhost bin]# ./inotifywatch -h
inotifywatch 3.14
Gather filesystem usage statistics using inotify.
Usage: inotifywatch [ options ] file1 [ file2 ] [ ... ]
Options:
        -h|--help       Show this help text.
        -v|--verbose    Be verbose.
        @<file>         Exclude the specified file from being watched.
        --fromfile <file>
                Read files to watch from <file> or `-' for stdin.
        --exclude <pattern>
                Exclude all events on files matching the extended regular
                expression <pattern>.
        --excludei <pattern>
                Like --exclude but case insensitive.
        -z|--zero
                In the final table of results, output rows and columns even
                if they consist only of zeros (the default is to not output
                these rows and columns).
        -r|--recursive  Watch directories recursively.
        -t|--timeout <seconds>
                Listen only for specified amount of time in seconds; if
                omitted or 0, inotifywatch will execute until receiving an
                interrupt signal.
        -e|--event <event1> [ -e|--event <event2> ... ]
                Listen for specific event(s).  If omitted, all events are 
                listened for.
        -a|--ascending <event>
                Sort ascending by a particular event, or `total'.
        -d|--descending <event>
                Sort descending by a particular event, or `total'.

Exit status:
        0  -  Exited normally.
        1  -  Some error occurred.

Events:
        access          file or directory contents were read
        modify          file or directory contents were written
        attrib          file or directory attributes changed
        close_write     file or directory closed, after being opened in
                        writeable mode
        close_nowrite   file or directory closed, after being opened in
                        read-only mode
        close           file or directory closed, regardless of read/write mode
        open            file or directory opened
        moved_to        file or directory moved to watched directory
        moved_from      file or directory moved from watched directory
        move            file or directory moved to or from watched directory
        create          file or directory created within watched directory
        delete          file or directory deleted within watched directory
        delete_self     file or directory was deleted
        unmount         file system containing file or directory unmounted