标签：keepalived

本篇文章记录通过Ansible自动化部署nginx的负载均衡高可用，前端代理使用nginx+keepalived,后端web server使用3台nginx用于负载效果的体现，结构图如下：

1. 部署前准备工作

主机规划

• Ansible ： 192.168.214.144
• Keepalived-node-1 : 192.168.214.148
• Keepalived-node-2 : 192.168.214.143
• web1 ： 192.168.214.133
• web2 ： 192.168.214.135
• web3 ： 192.168.214.139

2. Ansible主机与远程主机秘钥认证

#!/bin/bash

keypath=/root/.ssh
[ -d ${keypath} ] || mkdir -p ${keypath}
rpm -q expect &> /dev/null || yum install expect -y
ssh-keygen -t rsa -f /root/.ssh/id_rsa  -P ""
password=centos
while read ip;do
expect <<EOF
set timeout 5
spawn ssh-copy-id $ip
expect {
"yes/no" { send "yesn";exp_continue }
"password" { send "$passwordn"  }
}
expect eof
EOF
done < /home/iplist.txt

iplist.txt

192.168.214.148
192.168.214.143
192.168.214.133
192.168.214.135
192.168.214.139
192.168.214.134

执行脚本

[root@Ansible script]# ./autokey.sh

测试验证

[root@Ansible script]# ssh 192.168.214.148 'date'
Address 192.168.214.148 maps to localhost, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Sat Jul 14 11:35:21 CST 2018

配置Ansible基于主机名认证，方便单独管理远程主机

vim  /etc/hosts
#
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.214.148 node-1
192.168.214.143 node-2
192.168.214.133 web-1
192.168.214.135 web-2
192.168.214.139 web-3

3. 安装配置Ansible

#安装ansible
[root@Ansible ~]# yum install ansible -y

#配置ansible主机清单
[root@Ansible ~]# vim /etc/ansible/hosts 
[all]

192.168.214.148
192.168.214.143
192.168.214.133
192.168.214.135
192.168.214.139

[node]

192.168.214.148
192.168.214.143

[web]
192.168.214.133 
192.168.214.135
192.168.214.139

#Ansible执行ping测试 
 [root@Ansible ~]# ansible all -m ping

4. 编写roles，实现web的部署

先看一下web的目录结构

[root@Ansible ~]# tree /opt/roles/web
/opt/roles/web
.
├── tasks
│   ├── install_nginx.yml
│   ├── main.yml
│   ├── start.yml
│   ├── temps.yml
│   └── user.yml
└── templates
    ├── index.html.j2
    └── nginx.conf.j2

2 directories, 7 files

按照角色执行的顺序编写

编写user.yml

- name: create group nginx
  group: name=nginx
- name: create user nginx
  user: name=nginx group=nginx system=yes shell=/sbin/nologin

编写install_nginx.yml

- name: install nginx webserver
  yum: name=nginx

创建nginx配置文件的template模板

由于是测试，后端web服务的nginx.conf配置文件基本保持默认，只只更具后端主机情况设置worker进程数，使用ansible的setup模块中的变量获取远程主机的cpu的数量值

#将配置文件转换成template文件
[root@Ansible conf]# cp nginx.conf /opt/roles/web/templates/nginx.conf.j2
#做出修改的内容如下
worker_processes {{ansible_proccessor_vcpus}};

#在templates目录写一个测试页内如下
vim index.html.j2
{{ ansible_hostname }} test page.

编写temps.yml

- name: cp nginx.conf.j2 to nginx web server rename nginx.conf
  template: src=/opt/roles/web/templates/nginx.conf.j2 dest=/etc/nginx/nginx.conf
- name: cp index test page to nginx server
  template: src=/opt/roles/web/templates/index.html.j2 dest=/usr/share/nginx/html/index.html

编写start.yml

- name: restart nginx
  service: name=nginx state=started

编写main.yml

- import_tasks: user.yml
- import_tasks: install_nginx.yml
- import_tasks: temps.yml
- import_tasks: start.yml

编写执行主文件web_install.yml，执行文件不能与web角色放在同一目录，通常放在roles目录

[root@Ansible ~]# vim /opt/roles/web_install.yml 


---
- hosts: web
  remote_user: root
  roles:
    - web

安装前测试: -C选项为测试

[root@Ansible ~]# ansible-playbook -C /opt/roles/web_install.yml 

如没有问题则执行安装

[root@Ansible ~]# ansible-playbook /opt/roles/web_install.yml 

测试访问

[root@Ansible ~]# ansible web -m shell -a 'iptables -F'
192.168.214.139 | SUCCESS | rc=0 >>


192.168.214.135 | SUCCESS | rc=0 >>


192.168.214.133 | SUCCESS | rc=0 >>


[root@Ansible ~]# curl 192.168.214.133
web-1 test page.

5. 编写roles角色部署nginx+keepalived

部署高可用集群需要注意各节点包括后端主机的时间问题，保证各主机时间一致。

[root@Ansible ~]# ansible all -m shell -a 'yum install ntpdate -y'

[root@Ansible ~]# ansible all -m shell -a 'ntpdate gudaoyufu.com'

6. 编写roles角色

编写user.yml

- name: create nginx group
  group: name=nginx
- name: create nginx user
  user: name=nginx group=nginx system=yes shell=/sbin/nologin

编写install_server.yml

- name: install nginx and keepalived
  yum: name={{ item }} state=latest
  with_items:
    - nginx
    - keepalived

编写temps.yml

- name: copy nginx proxy conf and rename
  template: src=/opt/roles/ha_proxy/templates/nginx.conf.j2  dest=/etc/nginx/nginx.conf

- name: copy master_keepalived.conf.j2 to MASTER node
  when: ansible_hostname == "node-1"
  template: src=/opt/roles/ha_proxy/templates/master_keepalived.conf.j2 dest=/etc/keepalived/keepalived.conf

- name: copy backup_keepalived.conf.j2 to BACKUP node
  when: ansible_hostname == "node-2"
  template: src=/opt/roles/ha_proxy/templates/backup_keepalived.conf.j2 dest=/etc/keepalived/keepalived.conf

配置nginx proxy配置文件模板

[root@Ansible ~]# cp /opt/conf/nginx.conf /opt/roles/ngx_proxy/templates/nginx.conf.j2

[root@Ansible ~]# vim /opt/roles/ngx_proxy/templates/nginx.conf.j2

user nginx;
worker_processes {{ ansible_processor_vcpus }};
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.

events {
    worker_connections  1024;
}


http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;


    include /etc/nginx/conf.d/*.conf;

    upstream web {

        server 192.168.214.133:80 max_fails=3 fail_timeout=30s;
        server 192.168.214.135:80 max_fails=3 fail_timeout=30s;
        server 192.168.214.139:80 max_fails=3 fail_timeout=30s;


    }



    server {

    listen       80 default_server;
    server_name  {{ ansible_hostname }};
    root         /usr/share/nginx/html;
    index index.html index.php;

         location / {
                proxy_pass http://web;
             }

         error_page 404 /404.html;

          }


}

配置keepalived配置文件模板

[root@Ansible ~]# cp /opt/conf/keepalived.conf /opt/roles/ha_proxy/templates/master_keepalived.conf.j2

[root@Ansible templates]# vim master_keepalived.conf.j2

#

! Configuration File for keepalived

global_defs {
   notification_email {
     [email protected]
     [email protected]
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 192.168.214.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
   vrrp_iptables
   vrrp_mcast_group4 224.17.17.17
}


vrrp_script chk_nginx {
                script "killall -0 nginx"
                interval 1
                weight -20
                fall 2
                rise 1
            }

vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 55
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 12345678
    }
    virtual_ipaddress {
        192.168.214.100
    }

    track_script {
        chk_nginx
      }

   }

同样，在master_keepalived.conf.j2基础修改另存为backup_keepalived.conf.j2，只修改角色与优先级即可。注意：master_keepalived.conf.j2文件中的检测故障降低优先级的值要确保降低后MASTER优先级小于BACKUP的优先级
编写start.yml

- name: start nginx proxy server
  service: name=nginx state=started

编写main.yml

- import_tasks: user.yml
- import_tasks: install_server.yml
- import_tasks: temps.yml
- import_tasks: start.yml

编写执行主文件

[root@Ansible ~]# vim /opt/roles/ha_proxy_install.yml


---
- hosts: node
  remote_user: root
  roles:
    - ha_proxy

执行检测roles

[root@Ansible ~]# ansible-playbook -C /opt/roles/ha_proxy_install.yml 

执行测试没问题即可执行自动部署

执行过程如下：

[root@Ansible ~]# ansible-playbook  /opt/roles/ha_proxy_install.yml 



PLAY [node] **********************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************
ok: [192.168.214.148]
ok: [192.168.214.143]

TASK [ha_proxy : create nginx group] *********************************************************************************************
changed: [192.168.214.148]
ok: [192.168.214.143]

TASK [ha_proxy : create nginx user] **********************************************************************************************
changed: [192.168.214.148]
ok: [192.168.214.143]

TASK [ha_proxy : install nginx and keepalived] ***********************************************************************************
changed: [192.168.214.143] => (item=[u'nginx', u'keepalived'])
changed: [192.168.214.148] => (item=[u'nginx', u'keepalived'])

TASK [ha_proxy : copy nginx proxy conf and rename] *******************************************************************************
changed: [192.168.214.148]
changed: [192.168.214.143]

TASK [ha_proxy : copy master_keepalived.conf.j2 to MASTER node] ******************************************************************
skipping: [192.168.214.143]
changed: [192.168.214.148]

TASK [ha_proxy : copy backup_keepalived.conf.j2 to BACKUP node] ******************************************************************
skipping: [192.168.214.148]
changed: [192.168.214.143]

TASK [ha_proxy : start nginx proxy server] ***************************************************************************************
changed: [192.168.214.143]
changed: [192.168.214.148]

PLAY RECAP ***********************************************************************************************************************
192.168.214.143            : ok=7    changed=4    unreachable=0    failed=0   
192.168.214.148            : ok=7    changed=6    unreachable=0    failed=0   

至此,自动部署nginx+keepalived高可用负载均衡完成了

最后看一下roles目录的结构

[root@Ansible ~]# tree /opt/roles/
/opt/roles/
├── ha_proxy
│   ├── tasks
│   │   ├── install_server.yml
│   │   ├── main.yml
│   │   ├── start.yml
│   │   ├── temps.yml
│   │   └── user.yml
│   └── templates
│       ├── backup_keepalived.conf.j2
│       ├── master_keepalived.conf.j2
│       └── nginx.conf.j2
├── ha_proxy_install.retry
├── ha_proxy_install.yml
├── web
│   ├── tasks
│   │   ├── install_nginx.yml
│   │   ├── main.yml
│   │   ├── start.yml
│   │   ├── temps.yml
│   │   └── user.yml
│   └── templates
│       ├── index.html.j2
│       └── nginx.conf.j2
├── web_install.retry
└── web_install.yml

6 directories, 19 files

下面测试服务：keepalived的服务没有在ansible中设置自动启动，到keepalived节点启动即可。

测试node节点

[root@Ansible ~]# for i in {1..10};do curl 192.168.214.148;done
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.

将node-1 的MASTER服务停掉测试故障转移,同时查看node-2状态变化

执行： nginx -s stop

查看vrrp通知，可以看到主备切换正常：

[root@node-2 ~]# tcpdump -i ens33 -nn host 224.17.17.17

listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes

16:55:20.804327 IP 192.168.214.148 > 224.17.17.17: VRRPv2, Advertisement, vrid 55, prio 100, authtype simple, intvl 1s, length 20
16:55:25.476397 IP 192.168.214.148 > 224.17.17.17: VRRPv2, Advertisement, vrid 55, prio 0, authtype simple, intvl 1s, length 20
16:55:26.128474 IP 192.168.214.143 > 224.17.17.17: VRRPv2, Advertisement, vrid 55, prio 90, authtype simple, intvl 1s, length 20
16:55:27.133349 IP 192.168.214.143 > 224.17.17.17: VRRPv2, Advertisement, vrid 55, prio 90, authtype simple, intvl 1s, length 20

再测试访问：

[root@Ansible ~]# for i in {1..10};do curl 192.168.214.148;done
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.

node-1恢复主节点，抢回MASTER角色

node-1节点执行nginx指令,可以看到VIP漂移回到node-1节点，测试访问

[root@Ansible ~]# for i in {1..10};do curl 192.168.214.148;done
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.
web-2 test page.
web-3 test page.
web-1 test page.

7. 其他问题

上面的自动部署方式还有可以改进的地方，比如，可以将配置keepalived的配置文件中的许多参数在roles中以统一变量的方式定义，然后在template模板文件中引用参数就可以了

此外还有一个需要注意的地方是：keepalived的配置文件中使用了killall指令检测本地的nginx服务状态，如果检测结果状态为非0就会执行vrrp_script中定义的降级操作，要确保系统这个指令可以执行，有时该指令没有被安装，如果该指令没有存在，即使MASTER节点发生故障也不会发生变化。

前言：

ansible作为一款灵活、高效、功能丰富的自动化部署工具在企业运维管理中备受推崇。本文演示使用ansible部署小型企业服务框架，实现高可用、负载均衡的目标。如有错误敬请赐教。
目标环境拓扑：

环境介绍：

前端代理层由两台nginx实现，并安装keepalived实现地址滑动达成高可用。
web层由两套Apache+PHP+WordPress 构建应用环境。数据层由一台mariadb组成，篇幅限制这里并没有做数据库主从复制、读写分离(实际环境数据库一定要实现这两项功能)。

IP一览：

环境准备：

1.管理端安装ansible，配置ssh秘钥使主机间实现基于秘钥的认证

ssh-keygen  -t rsa  #三次回车，中途的问题是问秘钥存放位置（默认/root/.ssh）,是否加密秘钥。实验方便这里不加密
ssh-copy-id -i .ssh/id_rsa.pub [email protected] #将公钥发送给目标主机
ssh-copy-id -i .ssh/id_rsa.pub [email protected]
ssh-copy-id -i .ssh/id_rsa.pub [email protected]
ssh-copy-id -i .ssh/id_rsa.pub [email protected]
ssh-copy-id -i .ssh/id_rsa.pub [email protected]

2.编辑ansible的hosts文件，定义所有的主机
vim /etc/ansible/hosts

3.为所有主机同步时间

 ansible all -a 'ntpdate 172.18.0.1' #我这里是同步自己局域网的ntp服务器，实验的话选取同一台主机保证时间相同即可

4.创建ansible相关角色的目录

mkdir -pv /etc/ansible/roles/{mysql,web,nginx}/{files,tasks,templates,vars,handlers,meta}

配置web的playbook:

1.创建tasks文件

vim /etc/ansible/roles/web/task/main.yml
- name: install web pakgs
  yum: name={{ item }}
  with_items:
  - httpd
  - php
  - php-mysql
- name: config  web
  copy: src=httpd.conf dest=/etc/httpd/conf/httpd.conf
  notify: restart the service # 注意这里要与handlers里定义的name相同
- name: copy wordpress
  synchronize: src=wordpress dest=/var/www/html/wordpress/
- name: restart the service
  service: name=httpd state=started

2.创建handles

vim /etc/ansible/roles/web/handlers/main.yml
- name: restart the service  #就这
  service: name=httpd state=restarted

3.添加要复制过去的配置文件
放在/etc/ansible/roles/web/files/下 ① WordPress目录 ② httpd.conf #从别的地方考过来
4.修改WordPress连接数据库的配置文件

 cd wordpress
 cp wp-sample-config.php  wp-config.php
 vim wp-config.php

5.添加web主剧本

vim /etc/ansible/web.yml
- hosts: web
  remote_user: root
  roles:
  - web

6.测试，没问题的话就下一步

ansible-playbook -C /etc/ansible/web.yml

配置代理层：

1.添加task任务

vim /etc/ansible/roles/nginx/tasks/main.yml
- name: install package
  yum: name={{ item }}
  with_items:
  - nginx
  - keepalived
- name: config keepalived
  template: src=keepalived.conf.j2 dest=/etc/keepalived/keepalived.conf
  notify: restart keepalived
- name: config nginx
  template: src=nginx.conf.j2 dest=/etc/nginx/nginx.conf
  notify: restart nginx
- name: start service
  service: name={{ item }} state=started enabled=true
  with_items:
  - keepalived
  - nginx

2.添加handlers

vim /etc/ansible/roles/nginx/handlers
- name: restart keepalived
  service: name=keepalived state=restarted
- name: restart nginx
  service: name=nginx state=restarted

3.准备template文件 ①keepalived.conf.j2 ②nginx.conf.j2
4.修改keepalived模板文件

global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id {{ansible_hostname}} #自带变量，通过ansible 主机IP -m setup 查询
   vrrp_mcast_group4 224.0.0.43
}

vrrp_instance VI_1 {
    state {{ state }} #已通过hosts文件定义变量
    interface ens33 #网卡名
    virtual_router_id 51
    priority {{ priority }}
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass lovelinux #设置密码
    }
    virtual_ipaddress {
        172.18.43.88 #虚拟IP
    }
}

5.修改nginx模板文件(定义在http段)

upstream web {                        #新增段
        server 172.18.43.61;
        server 172.18.43.62;
    }
    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

    location / {                     #新增段
        proxy_pass    
        }
    }

6.添加nginx主剧本

vim /etc/ansible/nginx.yml
- hosts: nginx
  remote_user: root
  roles:
  - nginx

7.测试，没问题的话就下一步

ansible-playbook -C /etc/ansible/nginx.yml

配置mariadb:

1.配置mariadb的任务清单

roles/mysql/tasks/main.yml
- name: install mariadb
  yum: name=mariadb-server
- name: copy sql file
  copy: src=mysql.sql dest=/tmp/mysql.sql
- name: start mysql service
  service: name=mariadb state=started
- name: config mysql
  shell: "mysql < /tmp/mysql.sql"

2.设置files文件

vim roles/mysql/files/mysql.sql 
CREATE DATABASE wp;
GRANT ALL ON wp.* TO 'wpuser'@'%' IDENTIFIED BY 'lovelinux';

3.添加mysql主剧本

vim /etc/ansible/mysql.yml
- hosts: mysql
  remote_user: root
  roles:
  - mysql

4.测试，没问题的话就下一步

ansible-playbook -C /etc/ansible/mysql.yml

开始表演（执行剧本）：

1.目录结构

2.分别执行

ansible-playbook  web.yml
ansible-playbook  nginx.yml
ansible-playbook  mysql.yml

3.访问页面http://172.18.43.88/wordpress

项目总结：

1.在定义web的playbook时复制wordpress时开始用的是copy模块执行总是不成功，报错ERROR! A worker was found in a dead state。在确认自己没有语法错误后，百度查找原因无果最后在Google上找到了答案(英文不好不要心虚，技术问题语法都很简单很容易看懂，个别单词查查有道词典就好了)，所以有在IT技术的问题问Google准没错。用synchronize模块要比copy模块高效安全的多，synchronize采用rsync复制文件，所以系统必须安装rsync 包否则无法使用这个模块。使用该模块的优点有①增量复制(只复制与目标主机有差异的文件) ② 复制时采用压缩，对复制大文件支持优秀（用copy复制大文件会出错），以下整理了一些synchronize参数：
archive # 是否采用归档模式同步，即以源文件相同属性同步到目标地址
copy_links # 同步的时候是否复制连接
links # Copy symlinks as symlinks
delete # 删除源中没有而目标存在的文件（即以推送方为主）
dest= # 目标地址
dest_port # 目标接受的端口，ansible配置文件中的 ansible_ssh_port 变量优先级高于该 dest_port 变量
dirs # 以非递归的方式传输目录
2.如mysql主机曾经安装过mariadb可能会出现导入SQL命令失败的情况，这时要将mysql的数据库删掉，默认位置在/var/lib/mysql/下
3.编辑nginx代理时注意语句的位置不要写错
4.出现错误仔细看看错误日志，耐心点问题肯没想的那么难。