2017年12月 – 第7页 – Linux系统运维日志

MariaDB Galera Cluster集群优缺点

一、MariaDB Galera Cluster概要

1.简述：

MariaDB Galera Cluster 是一套在mysql innodb存储引擎上面实现multi-master及数据实时同步的系统架构，业务层面无需做读写分离工作，数据库读写压力都能按照既定的规则分发到各个节点上去。在数据方面完全兼容 MariaDB 和 MySQL。

2.特性：

(1).同步复制 Synchronous replication
(2).Active-active multi-master 拓扑逻辑
(3).可对集群中任一节点进行数据读写
(4).自动成员控制，故障节点自动从集群中移除
(5).自动节点加入
(6).真正并行的复制，基于行级
(7).直接客户端连接，原生的 MySQL 接口
(8).每个节点都包含完整的数据副本
(9).多台数据库中数据同步由 wsrep 接口实现

3.局限性：

(1).目前的复制仅仅支持InnoDB存储引擎,任何写入其他引擎的表，包括mysql.*表将不会复制,但是DDL语句会被复制的,因此创建用户将会被复制,但是insert into mysql.user…将不会被复制的.
(2).DELETE操作不支持没有主键的表,没有主键的表在不同的节点顺序将不同,如果执行SELECT…LIMIT… 将出现不同的结果集.
(3).在多主环境下LOCK/UNLOCK TABLES不支持,以及锁函数GET_LOCK(), RELEASE_LOCK()…
(4).查询日志不能保存在表中。如果开启查询日志，只能保存到文件中。
(5).允许最大的事务大小由wsrep_max_ws_rows和wsrep_max_ws_size定义。任何大型操作将被拒绝。如大型的LOAD DATA操作。
(6).由于集群是乐观的并发控制，事务commit可能在该阶段中止。如果有两个事务向在集群中不同的节点向同一行写入并提交，失败的节点将中止。对于集群级别的中止，集群返回死锁错误代码(Error: 1213 SQLSTATE: 40001 (ER_LOCK_DEADLOCK)).
(7).XA事务不支持，由于在提交上可能回滚。
(8).整个集群的写入吞吐量是由最弱的节点限制，如果有一个节点变得缓慢，那么整个集群将是缓慢的。为了稳定的高性能要求，所有的节点应使用统一的硬件。
(9).集群节点建议最少3个。
(10).如果DDL语句有问题将破坏集群。

安装mariadb速度慢的解决方法-使用国内源

在centos7上安装mariadb，按照官网的方法设置安装源，但是国外源慢得几乎崩溃。

搜索国内源，发现清华的源的确快，每秒2M，快如闪电。

国外源的配置（慢）：

[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.1/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

修改为国内源的配置（快）：

[mariadb]
name = MariaDB
baseurl = https://mirrors.ustc.edu.cn/mariadb/yum/10.1/centos7-amd64
gpgkey=https://mirrors.ustc.edu.cn/mariadb/yum/RPM-GPG-KEY-MariaDB
gpgcheck=1

linux磁盘管理系列三：LVM的使用

LVM是什么

LVM是Linux操作系统的逻辑卷管理器。现在有两个Linux版本的LVM,分别是 LVM1,LVM2。LVM1是一种已经被认为稳定了几年的成熟产品，LVM2 是最新最好的LVM版本。 LVM2几乎完全向后兼容使用LVM1创建的卷。此例外是快照（在升级到LVM 2之前，必须删除快照卷）。

LVM简介

逻辑卷管理提供了比传统的磁盘和分区视图更高级别的计算机系统上磁盘存储的视图。这使得系统管理员可以更灵活地将存储分配给应用程序和用户。

在逻辑卷管理器的控制下创建的存储卷可以随意调整大小和移动，尽管这可能需要对文件系统工具进行一些升级。

逻辑卷管理器还允许管理用户定义组中的存储卷，允许系统管理员处理明显命名的卷组（如“development”和“sales”），而不是物理磁盘名称，例如“sda”和“sdb”

LVM基本术语

volume group (VG)

卷组是LVM中使用的最高级别的抽象。它将逻辑卷和物理卷集合在一起成为一个管理单元。

physical volume (PV)

一个物理卷通常是一个硬盘，虽然它可能只是一个看起来像一个硬盘（如软件RAID设备）的设备。

logical volume (LV)

相当于非LVM系统中的磁盘分区。 LV作为标准块设备可见; 因此LV可以包含文件系统（例如/ home）。

physical extent (PE)

每个物理卷都被划分为数据块（称为物理扩展盘区），这些扩展盘区与卷组的逻辑盘区具有相同的大小。

logical extent (LE)

每个逻辑卷都被分割成数据块，称为逻辑盘区。卷组中的所有逻辑卷的区段大小都是相同的。

几个概念的关系

一个具体的例子将有助于：

假设我们有一个名为VG1的卷组，这个卷组的物理盘区大小为4MB。在这个卷组中，我们引入了2个硬盘分区/dev/hda1和/dev/hdb1。
这些分区将成为物理卷PV1和PV2（管理员可以自行决定是否提供更有意义的名称）。
PV被分成4MB块，因为这是卷组的大小。磁盘大小不同，我们在PV1中获得99个扩展盘区，在PV2中获得248个扩展盘区。
我们现在可以创建自己的逻辑卷，它可以是1到347（248 + 99）范围之间的任何大小。
当创建逻辑卷时，在逻辑盘区和物理盘区之间定义映射，例如，逻辑盘区1可以映射到PV1的物理盘区51上，写入逻辑盘的第一个4MB的数据实际上被写入到PV1的第51盘区。

映射模式

线性映射 : 将分配一定范围的PE到LV的一个区域，例如LE 1 – 99映射到PV1，LE 100 – 347映射到PV2。

分条映射 : 将跨多个物理卷交织逻辑盘区的块，这个方式一定程度上提高了性能，具体如下

　　1st chunk of LE[1] -> PV1[1], #第一个逻辑块在PV1设备上， 第二个在PV2设备上，等等。
　　2nd chunk of LE[1] -> PV2[1],
　　3rd chunk of LE[1] -> PV3[1],
　　4th chunk of LE[1] -> PV1[2],

Snapshots(快照）：

这是LVM提供的一个非常棒的功能就是快照了，他允许管理员创建一个新的块设备来呈现一个逻辑卷的的精确副本，在某个时间冻结，管理员可以及时备份指定一致状态下的数据，备份完毕就可以删除设备了。

快照最初包含有关快照的一些元数据，但没有来自源逻辑卷的实际数据。快照使用写时复制（copy on write）技术来检测数据在原始数据块中何时更改。它将拍摄快照时保存的值复制到快照卷中的块中，然后允许将新数据存储在源块中。

随着更多块从源逻辑卷上的原始值发生更改，快照大小也会增加。如果快照卷变满，它将被丢弃，所以分配足够的空间是非常重要的，具体取决于你数据的变换情况，如果快照大小和原始数据一样大，它将永远不会溢出的。

安装LVM

[root@centos7 ~]$ rpm -q lvm2                           # 查看lvm2是否安装了， 我使用的centos7，这是安装的了
lvm2-2.02.171-8.el7.x86_64
[root@centos7 ~]$ # yum -y install lvm2                 # 没有安装的话使用yum安装

常用命令简介

[root@centos7 ~]$ pv                                                   # 使用tab按键获取pv相关命令
pvchange   pvck       pvcreate   pvdisplay  pvmove     pvremove   pvresize   pvs        pvscan
[root@centos7 ~]$ vg                                                   # 使用tabl按键获取vg相关的命令
vgcfgbackup    vgck           vgdisplay      vgimport       vgmknodes      vgrename       vgsplit
vgcfgrestore   vgconvert      vgexport       vgimportclone  vgreduce       vgs            
vgchange       vgcreate       vgextend       vgmerge        vgremove       vgscan         
[root@centos7 ~]$ lv                                                   # 使用tabl按键获取lv先关的命令
lvchange     lvdisplay    lvmconf      lvmdump      lvmsadc      lvremove     lvs          
lvconvert    lvextend     lvmconfig    lvmetad      lvmsar       lvrename     lvscan       
lvcreate     lvm          lvmdiskscan  lvmpolld     lvreduce     lvresize

上面的命令比较多，简单介绍几个常用的命令。

pv打头的：代表pv相关的命令
vg带头的：代表vg相关的命令
lv带头的：代表lv相关的命令
create:创建相关
remove:移除相关
display:显示相关
import:导入相关
export:导出相关
rename:重命名
vgchange:改变状态相关
extend:扩展相关
reduce:缩进相关

使用LVM

pv创建

pv作用：初始化磁盘和磁盘分区

在将磁盘或者磁盘分区用作物理卷（PV）之前，你需要对其进行初始化工作。

[root@centos7 ~]$ pvcreate /dev/sdb1

注意：

不建议对整个磁盘做PV初始化工作。因为其他操作系统不能识别LVM元数据，导致我们认为磁盘是空闲的，重新设置分区覆盖原有数据
如果是分区需要调整分区的类型为LVM(代码为8e)
分区并设置分区类型

[root@centos7 ~]$ fdisk -l

Disk /dev/sda: 214.7 GB, 214748364800 bytes, 419430400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b0b8a

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   270534655   134217728   83  Linux
/dev/sda3       270534656   372934655    51200000   83  Linux
/dev/sda4       372934656   419430399    23247872    5  Extended
/dev/sda5       372938752   413898751    20480000   83  Linux

Disk /dev/sdb: 107.4 GB, 107374182400 bytes, 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x93d380cf

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048     2099199     1048576   8e  Linux LVM
/dev/sdb2         2099200     4196351     1048576   8e  Linux LVM

Disk /dev/sde: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x73afb36f

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1            2048     2099199     1048576   8e  Linux LVM
/dev/sde2         2099200     6293503     2097152   8e  Linux LVM

Disk /dev/sdd: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x7e0900d8

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1            2048     2099199     1048576   8e  Linux LVM
/dev/sdd2         2099200    12584959     5242880   8e  Linux LVM

Disk /dev/sdc: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xc56b90d8

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048     2099199     1048576   8e  Linux LVM
/dev/sdc2         2099200     4196351     1048576   8e  Linux LVM

Disk /dev/sdf: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

[root@centos7 ~]$ partprobe 
Warning: Unable to open /dev/sr0 read-write (Read-only file system).  /dev/sr0 has been opened read-only.
[root@centos7 ~]$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0  200G  0 disk 
├─sda1   8:1    0    1G  0 part /boot
├─sda2   8:2    0  128G  0 part 
├─sda3   8:3    0 48.8G  0 part /
├─sda4   8:4    0  512B  0 part 
└─sda5   8:5    0 19.5G  0 part /app
sdb      8:16   0  100G  0 disk 
├─sdb1   8:17   0    1G  0 part 
└─sdb2   8:18   0    1G  0 part 
sdc      8:32   0   20G  0 disk 
├─sdc1   8:33   0    1G  0 part 
└─sdc2   8:34   0    1G  0 part 
sdd      8:48   0   20G  0 disk 
├─sdd1   8:49   0    1G  0 part 
└─sdd2   8:50   0    5G  0 part 
sde      8:64   0   20G  0 disk 
├─sde1   8:65   0    1G  0 part 
└─sde2   8:66   0    2G  0 part 
sdf      8:80   0   20G  0 disk 
sr0     11:0    1  8.1G  0 rom  /run/media/root/CentOS 7 x86_64

上面我们在/dev/sd{b,c,d,e}这4个盘每个都有2个类型为lvm的盘。接下来使用pvcreat初始化为物理卷

[root@centos7 ~]$ pvcreate /dev/sd{b1,b2,c1,c2,d1,d2,e1,e2} -y        # 这里加入了-y选项，之前的盘有文件系统的。 我这里强制创建了
  Wiping xfs_external_log signature on /dev/sdb2.
  Wiping xfs_external_log signature on /dev/sdb2.
  Wiping xfs_external_log signature on /dev/sdb2.
  Wiping xfs_external_log signature on /dev/sdb2.
  Wiping xfs_external_log signature on /dev/sdb2.
  Wiping xfs_external_log signature on /dev/sdb2.
  Wiping xfs_external_log signature on /dev/sdb2.
  Wiping xfs_external_log signature on /dev/sdb2.
  Physical volume "/dev/sdb1" successfully created.
  Physical volume "/dev/sdb2" successfully created.
  Physical volume "/dev/sdc1" successfully created.
  Physical volume "/dev/sdc2" successfully created.
  Physical volume "/dev/sdd1" successfully created.
  Physical volume "/dev/sdd2" successfully created.
  Physical volume "/dev/sde1" successfully created.
  Physical volume "/dev/sde2" successfully created.

pv信息查看

[root@centos7 ~]$ pvs                                           # 详细信息可以使用pvdisplay查看
  PV         VG         Fmt  Attr PSize    PFree   
  /dev/sdb1  vg_test_01 lvm2 a--  1020.00m 1020.00m
  /dev/sdb2             lvm2 ---     1.00g    1.00g
  /dev/sdc1  vg_test_01 lvm2 a--  1020.00m 1020.00m
  /dev/sdc2             lvm2 ---     1.00g    1.00g
  /dev/sdd1  vg_test_01 lvm2 a--  1020.00m 1020.00m
  /dev/sdd2             lvm2 ---     5.00g    5.00g
  /dev/sde1  vg_test_01 lvm2 a--  1020.00m 1020.00m
  /dev/sde2             lvm2 ---     2.00g    2.00g

vg创建

vg的创建是比较简单的。

[root@centos7 ~]$ vgcreate  vg_test_01 /dev/sd{b1,c1,d1,e1}    # 上面我们创建了8个pv，这里我们使用4个1的设备去创建vg,创建后默认vg是激活状态的
  Volume group "vg_test_01" successfully created

vg信息的查看

[root@centos7 ~]$ vgs                                 # 查看简短的信息
  VG         #PV #LV #SN Attr   VSize VFree
  vg_test_01   4   0   0 wz--n- 3.98g 3.98g
[root@centos7 ~]$ vgdisplay                           # 查看详细的信息
  --- Volume group ---
  VG Name               vg_test_01                    # vg的名字
  System ID             
  Format                lvm2                          # lvm格式， 这里是lvm2的
  Metadata Areas        4
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                4
  Act PV                4
  VG Size               3.98 GiB                       # 这是我们4个pv的总大小
  PE Size               4.00 MiB                       # pe的大小
  Total PE              1020
  Alloc PE / Size       0 / 0                          # 分配的大小，这里我们还没有创建lv，所以为0
  Free  PE / Size       1020 / 3.98 GiB                # 分配后剩余的空间，第一个数是剩余的个数，第二个数是剩余的空间大小，2个都是剩余的只是单位表示不一样而已
  VG UUID               Y5NjjP-e3Aq-3u6E-gw6P-fLyy-Y5pN-ryyN9R  # vg设备的uuid

vg的激活

[root@centos7 ~]$ vgchange -a y vg_test_01                               # 我们上面就是激活状态的，如果我们重启系统，或者vgchange -y n命令关闭了，就需要这个命令启动下
  0 logical volume(s) in volume group "vg_test_01" now active

vg的移除

[root@centos7 ~]$ vgchange -a n vg_test_01                       # 要想移除vg，需要先关闭vg才能移除，这里先关闭
  0 logical volume(s) in volume group "vg_test_01" now active
[root@centos7 ~]$ vgremove  vg_test_01                           # 移除vg,

vg的添加成员

[root@centos7 ~]$ vgcreate  vg_test_01 /dev/sd{b1,c1,d1,e1}       # 上面我把vg删了，这里重新添加进来
  Volume group "vg_test_01" successfully created
[root@centos7 ~]$ vgextend vg_test_01  /dev/sdb2                  # 添加一个pv2到vg中来
  Volume group "vg_test_01" successfully extended

vg的删除成员

这个操作需要使用vgdisplay查看vg的使用情况和删除的pv上没有数据。如果有使用pvmove迁移完毕数据才能移除。

[root@centos7 ~]$ vgreduce  vg_test_01  /dev/sdb2
  Removed "/dev/sdb2" from volume group "vg_test_01"

lv的创建

[root@centos7 ~]$ vgdisplay                        # lv逻辑卷创建之前， 需要查看下vg信息
  --- Volume group ---
  VG Name               vg_test_01
  System ID             
  Format                lvm2
  Metadata Areas        4
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                4
  Act PV                4
  VG Size               3.98 GiB
  PE Size               4.00 MiB
  Total PE              1020
  Alloc PE / Size       0 / 0   
  Free  PE / Size       1020 / 3.98 GiB
  VG UUID               sZwHea-LabA-SJaV-Wz40-qFp3-Um2w-pvmJAY

[root@centos7 ~]$ lvcreate -L 2G  -n lv_test_01 vg_test_01                 # 我们的vg是近4G的，我这里创建一个2G的lv,之前磁盘使用了文件系统慈爱有如下警告。输入y即可。
WARNING: linux_raid_member signature detected on /dev/vg_test_01/lv_test_01 at offset 4096. Wipe it? [y/n]: y
  Wiping linux_raid_member signature on /dev/vg_test_01/lv_test_01.
WARNING: ext4 signature detected on /dev/vg_test_01/lv_test_01 at offset 1080. Wipe it? [y/n]: y
  Wiping ext4 signature on /dev/vg_test_01/lv_test_01.
  Logical volume "lv_test_01" created.

lv的移除

[root@centos7 ~]$ lvremove  /dev/vg_test_01/lv_test_01     
# 这个移除我们上面创建lv_test_01这个逻辑卷， 下面提示这个是活动的逻辑卷，输入y就可以了
Do you really want to remove active logical volume vg_test_01/lv_test_01? [y/n]: y
  Logical volume "lv_test_01" successfully removed

注意：如果这个lv被挂载了。请先umount

挂载文件系统

[root@centos7 ~]$ lvcreate -L 2G  -n lv_test_01 vg_test_01                             # 上面我们删除了lv,这里需要创建出来
  Logical volume "lv_test_01" created.
[root@centos7 ~]$ mkfs.ext4  /dev/vg_test_01/lv_test_01                                  # 这里我使用ext4 文件系统了。centos7默认使用的文件系统是xfs,但是xfs不支持缩减的

[root@centos7 ~]$ mkdir /mnt/test_01                                                    # 创建挂载点目录
[root@centos7 ~]$ mount /dev/vg_test_01/lv_test_01  /mnt/test_01/                       # 挂载逻辑卷

扩展LVM

上面我们的测试是vg是4G，我们的创建的lv是2G的，我们想给创建的lv扩展1G空间。

[root@centos7 ~]$ lvextend -L +1G /dev/vg_test_01/lv_test_01                 # 我们扩大1G， +1G表示在原有基础上扩大1G,我们这里是有vg是有空间的，没有的话就需要创建pv，扩大vg在做
  Size of logical volume vg_test_01/lv_test_01 changed from 2.00 GiB (512 extents) to 3.00 GiB (768 extents).
  Logical volume vg_test_01/lv_test_01 successfully resized.
[root@centos7 ~]$ df -h                                                      # 上面的命令提示lv已经到3G了， 但是使用df-h 查看还是2g
Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda3                           48G  4.9G   41G  11% /
devtmpfs                           979M  4.0K  979M   1% /dev
tmpfs                              993M     0  993M   0% /dev/shm
tmpfs                              993M   22M  971M   3% /run
tmpfs                              993M     0  993M   0% /sys/fs/cgroup
/dev/sda1                          976M  130M  780M  15% /boot
/dev/sda5                           20G  307M   18G   2% /app
tmpfs                              199M   32K  199M   1% /run/user/0
/dev/sr0                           8.1G  8.1G     0 100% /run/media/root/CentOS 7 x86_64
/dev/mapper/vg_test_01-lv_test_01  2.0G   33M  2.0G   2% /mnt/test_01
[root@centos7 ~]$ resize2fs /dev/mapper/vg_test_01-lv_test_01              # 需要增长下文件系统的，让文件系统和lv大小匹配下
meta-data=/dev/mapper/vg_test_01-lv_test_01 isize=512    agcount=4, agsize=131072 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=524288, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 524288 to 786432
[root@centos7 ~]$ df -h                                                    # 再次查看就是一致的了
Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda3                           48G  4.9G   41G  11% /
devtmpfs                           979M  4.0K  979M   1% /dev
tmpfs                              993M     0  993M   0% /dev/shm
tmpfs                              993M   22M  971M   3% /run
tmpfs                              993M     0  993M   0% /sys/fs/cgroup
/dev/sda1                          976M  130M  780M  15% /boot
/dev/sda5                           20G  307M   18G   2% /app
tmpfs                              199M   32K  199M   1% /run/user/0
/dev/sr0                           8.1G  8.1G     0 100% /run/media/root/CentOS 7 x86_64
/dev/mapper/vg_test_01-lv_test_01  3.0G   33M  3.0G   2% /mnt/test_01

注意：

ext系列可以使用resieze2fs去调整。其他的文件系统可以参考http://tldp.org/HOWTO/LVM-HOWTO/extendlv.html
　　
lvextentd 有个-r选项，自动去调整大小，匹配lv和文件系统大小的。

缩减LVM

逻辑卷的大小可以减小，也可以增加。但是，在缩小卷本身之前，请记住缩小文件系统的大小或存在于卷中的任何文件是非常重要的，否则可能会丢失数据。

注意： xfs,jsf文件系统是没法缩减lvm的，因为这2个文件系统没法缩减文件系统大小的。参考http://tldp.org/HOWTO/LVM-HOWTO/reducelv.html

主要步骤：

检查空间使用情况和分析
　　
umount设备
　　
缩减文件系统大小
　　
缩减lv大小
　　
恢复挂载设备

上面我们的逻辑卷是3G的大小，在缩减之前我们需要确保缩减的空间不能影响数据的存放的。

检查工作

[root@centos7 ~]$ cp -a /etc/ /mnt/test_01/                              # 我这里模拟下，只能目录有文件的情况 
[root@centos7 ~]$ du -sh /mnt/test_01/                                   # 查看下大小
42M    /mnt/test_01/
[root@centos7 ~]$ df -h                                                   # 查看逻辑卷分区利用情况 ，数据才存放76M,我们的逻辑卷都3G呢， 如果我们想缩减1G,完全没有问题的。
Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda3                           48G  4.9G   41G  11% /
devtmpfs                           979M  4.0K  979M   1% /dev
tmpfs                              993M     0  993M   0% /dev/shm
tmpfs                              993M   22M  971M   3% /run
tmpfs                              993M     0  993M   0% /sys/fs/cgroup
/dev/sda1                          976M  130M  780M  15% /boot
/dev/sda5                           20G  307M   18G   2% /app
tmpfs                              199M   32K  199M   1% /run/user/0
/dev/sr0                           8.1G  8.1G     0 100% /run/media/root/CentOS 7 x86_64
/dev/mapper/vg_test_01-lv_test_01  3.0G   76M  3.0G   3% /mnt/test_01

缩减空间

[root@centos7 ~]$ umount /dev/vg_test_01/lv_test_01                           # 取消挂载
[root@centos7 ~]$ resize2fs  /dev/vg_test_01/lv_test_01  1G                   # resieze2fs定义下文件系统大小
resize2fs 1.42.9 (28-Dec-2013)
Please run 'e2fsck -f /dev/vg_test_01/lv_test_01' first.

[root@centos7 ~]$ e2fsck -f /dev/vg_test_01/lv_test_01                        #上面的命令要检查，那就检查下
e2fsck 1.42.9 (28-Dec-2013)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vg_test_01/lv_test_01: 3614/196608 files (0.1% non-contiguous), 42361/786432 blocks
[root@centos7 ~]$ resize2fs  /dev/vg_test_01/lv_test_01  1G                   # 检查完，那就执行缩减文件系统大小
resize2fs 1.42.9 (28-Dec-2013)
Resizing the filesystem on /dev/vg_test_01/lv_test_01 to 262144 (4k) blocks.
The filesystem on /dev/vg_test_01/lv_test_01 is now 262144 blocks long.
[root@centos7 ~]$ lvreduce  -L 1G /dev/vg_test_01/lv_test_01                  # 缩减下逻辑卷到1G，这里有提示，缩减可能会丢失数据，输入y即可
  WARNING: Reducing active logical volume to 1.00 GiB.
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce vg_test_01/lv_test_01? [y/n]: yG                 
  Size of logical volume vg_test_01/lv_test_01 changed from 3.00 GiB (768 extents) to 1.00 GiB (256 extents).
  Logical volume vg_test_01/lv_test_01 successfully resized.
[root@centos7 ~]$ mount /dev/vg_test_01/lv_test_01  /mnt/test_01/            # 恢复挂载
[root@centos7 ~]$ ll /mnt/test_01/
total 28
drwxr-xr-x. 140 root root 12288 Dec  7 13:54 etc
drwx------.   2 root root 16384 Dec  7 15:47 lost+found
[root@centos7 ~]$ df -h                                                     # 检查下
Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda3                           48G  4.9G   41G  11% /
devtmpfs                           979M  4.0K  979M   1% /dev
tmpfs                              993M     0  993M   0% /dev/shm
tmpfs                              993M   22M  971M   3% /run
tmpfs                              993M     0  993M   0% /sys/fs/cgroup
/dev/sda1                          976M  130M  780M  15% /boot
/dev/sda5                           20G  307M   18G   2% /app
tmpfs                              199M   32K  199M   1% /run/user/0
/dev/sr0                           8.1G  8.1G     0 100% /run/media/root/CentOS 7 x86_64
/dev/mapper/vg_test_01-lv_test_01  944M   52M  825M   6% /mnt/test_01

迁移LVM的vg

移动vg到另外一个机器上

主要步骤：

检查和分析
一些预处理工作
umount文件系统
　　
设置卷组非活动
　　
导出卷组
　　
取出设备，插入到新机器
　　
导入卷组
　　
激活卷组
　　
挂载文件系统

检查分析

[root@centos7 ~]$ vgs
  VG         #PV #LV #SN Attr   VSize VFree
  vg_test_01   4   1   0 wz--n- 3.98g 2.98g
[root@centos7 ~]$ pvs
l  PV         VG         Fmt  Attr PSize    PFree   
  /dev/sdb1  vg_test_01 lvm2 a--  1020.00m       0 
  /dev/sdb2             lvm2 ---     1.00g    1.00g
  /dev/sdc1  vg_test_01 lvm2 a--  1020.00m 1016.00m
  /dev/sdc2             lvm2 ---     1.00g    1.00g
  /dev/sdd1  vg_test_01 lvm2 a--  1020.00m 1020.00m
  /dev/sdd2             lvm2 ---     5.00g    5.00g
  /dev/sde1  vg_test_01 lvm2 a--  1020.00m 1020.00m
  /dev/sde2             lvm2 ---     2.00g    2.00g
[root@centos7 ~]$ lvs
  LV         VG         Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_test_01 vg_test_01 -wi-ao---- 1.00g

我们使用lvs,vgs,pvs分别查看信息。发现我们vg是3G的大小。lv是1G的大小。但是vg的pv来自不同的磁盘的。我们要迁移vg。不能把包含pv都迁移过去把。那这一下子需要4个盘的。

我们把pv数据迁移下，到时候摘盘的时候只那一个就可以多好。发现/dev/sdd2是一个5G的盘。打算使用/dev/sdd这个盘作为迁移盘来迁移数据。

一些预处理工作

[root@centos7 ~]$ umount /mnt/test_01/                                    # 去除挂载
[root@centos7 ~]$ vgchange  -an vg_test_01                                # 设置vg不激活
  0 logical volume(s) in volume group "vg_test_01" now active
[root@centos7 ~]$ pvmove /dev/sd{b1,c1,e1}                                 # 迁移pv数据
  /dev/sdb1: Moved: 0.00%
  /dev/sdb1: Moved: 100.00%
[root@centos7 ~]$ pvs                                                      # 确保pv都是空的，没有数据存放了。
  PV         VG         Fmt  Attr PSize    PFree   
  /dev/sdb1  vg_test_01 lvm2 a--  1020.00m 1020.00m
  /dev/sdb2             lvm2 ---     1.00g    1.00g
  /dev/sdc1  vg_test_01 lvm2 a--  1020.00m 1016.00m
  /dev/sdc2             lvm2 ---     1.00g    1.00g
  /dev/sdd1  vg_test_01 lvm2 a--  1020.00m       0 
  /dev/sdd2  vg_test_01 lvm2 a--    <5.00g   <5.00g
  /dev/sde1  vg_test_01 lvm2 a--  1020.00m 1020.00m
  /dev/sde2             lvm2 ---     2.00g    2.00g

[root@centos7 ~]$ vgreduce vg_test_01  /dev/sd{b1,c1,e1}                   # 缩减vg 去除b1,c1,e1
  Removed "/dev/sdb1" from volume group "vg_test_01"
  Removed "/dev/sdc1" from volume group "vg_test_01"
  Removed "/dev/sde1" from volume group "vg_test_01"
[root@centos7 ~]$ pvremove /dev/sd{b1,c1,e1}                               # 移除pv，这里提示，
  Lables on physical volume "/dev/sdb1" sucdessfully wiped.
  Labels on physical volume "/dev/sdc1" successfully wiped.
  Labels on physical volume "/dev/sde1" successfully wiped.

导出vg

[root@centos7 ~]$ vgdisplay                                       # 导出前的确认
  --- Volume group ---
  VG Name               vg_test_01
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  24
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               5.99 GiB
  PE Size               4.00 MiB
  Total PE              1534
  Alloc PE / Size       256 / 1.00 GiB
  Free  PE / Size       1278 / 4.99 GiB
  VG UUID               sZwHea-LabA-SJaV-Wz40-qFp3-Um2w-pvmJAY

[root@centos7 ~]$ vgexport vg_test_01                                    # 导出vg发现有激活的设备，需要关闭
  Volume group "vg_test_01" has active logical volumes
[root@centos7 ~]$ vgchange -an vg_test_01                                # 设置为非激活
　　0 logical volume(s) in volume group "vg_test_01" now active
[root@centos7 ~]$ vgexport vg_test_01                                    # 导出， 要看到成功导出才可以，这里注意了， 如果这个vg名字和目标机器有重名的话， 这里使用rename改下在导出
　　Volume group "vg_test_01" successfully exported

[root@centos7 ~]$ sync                                                   # 多执行几次sync。 防止内存和磁盘没有同步的问题。
[root@centos7 ~]$ sync
[root@centos7 ~]$ sync
[root@centos7 ~]$ sync

我这里把这个磁盘取出来安装到另外一个机器上面。下面的一部分操作在新机器上执行。注意啦。

识别硬件

[root@centos6 ~]$ /git/Bash/other/scan_scsi_device.sh                             # 执行下扫描硬件脚本，我自己写的，后面把这个bash脚本贴出来
start scan
/sys/devices/pci0000:00/0000:00:07.1/host0/scsi_host/host0/scan scan finished.
/sys/devices/pci0000:00/0000:00:07.1/host1/scsi_host/host1/scan scan finished.
/sys/devices/pci0000:00/0000:00:10.0/host2/scsi_host/host2/scan scan finished.
end  scan
please use lsblk check
[root@centos6 ~]$ lsblk                                                           # 使用lsblk查看，可以发现，识别出来就是那个/dev/sdc
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0     11:0    1  3.7G  0 rom  /media/CentOS_6.9_Final
sr1     11:1    1  8.1G  0 rom  /media/CentOS 7 x86_64
sda      8:0    0  200G  0 disk 
├─sda1   8:1    0    1G  0 part /boot
├─sda2   8:2    0 48.8G  0 part /
├─sda3   8:3    0 19.5G  0 part /app
├─sda5   8:5    0    2G  0 part [SWAP]
├─sda4   8:4    0    2M  0 part 
└─sda6   8:6    0    1G  0 part 
sdb      8:16   0  100G  0 disk 
├─sdb1   8:17   0    1G  0 part 
├─sdb2   8:18   0   20G  0 part 
├─sdb3   8:19   0    1G  0 part 
├─sdb4   8:20   0    1K  0 part 
├─sdb5   8:21   0   20G  0 part 
├─sdb6   8:22   0    1G  0 part 
└─sdb7   8:23   0    1G  0 part 
sr2     11:2    1  3.7G  0 rom  /media/CentOS_6.9_Final_
sr3     11:3    1    2G  0 rom  /media/CentOS_6.9_Final__
sdc      8:32   0   20G  0 disk 
├─sdc1   8:33   0    1G  0 part 
└─sdc2   8:34   0    5G  0 part

脚本内容如下

[root@centos6 ~]$ cat /git/Bash/other/scan_scsi_device.sh 
#!/bin/bash
#================================================
#FileName   :scan_scsi_device
#Author     :zhaojiedi
#Description:如果你添加了一个新的硬盘设备， 但是没有被系统识别，使用如下脚本可以识别出来， 
#DateTime   :2017-11-30 18:03:27
#Version    :V1.0
#Other      :最新版本建议从这里获取https://github.com/zhaojiedi1992/Bash
#================================================
#!/bin/bash
echo "start scan"
for i in `find /sys/devices/ -name scan`; do 
    echo "- - -" > $i
    echo  $i scan finished.
done
echo "end  scan"
echo -e "33[32mplease use lsblk check33[0m"

导入并挂载

[root@centos6 ~]$ vgscan                                               # vg扫描
  Reading all physical volumes.  This may take a while...
  Found exported volume group "vg_test_01" using metadata type lvm2
[root@centos6 ~]$ pvscan                                               # pv扫描
  PV /dev/sdc1    is in exported VG vg_test_01 [1020.00 MiB / 0    free]
  PV /dev/sdc2    is in exported VG vg_test_01 [5.00 GiB / 4.99 GiB free]
  Total: 2 [5.99 GiB] / in use: 2 [5.99 GiB] / in no VG: 0 [0   ]
[root@centos6 ~]$ vgimport vg_test_01                                  # 确认无错误的就导入。
  Volume group "vg_test_01" successfully imported
[root@centos6 ~]$ vgs                                                  # 查看下vg信息
  VG         #PV #LV #SN Attr   VSize VFree
  vg_test_01   2   1   0 wz--n- 5.99g 4.99g
[root@centos6 ~]$ vgchange -ay vg_test_01                              # 激活vg
  1 logical volume(s) in volume group "vg_test_01" now active
[root@centos6 ~]$ mkdir /mnt/new_test                                  # 创建挂载点
[root@centos6 ~]$ mount /dev/vg_test_01/lv_test_01  /mnt/new_test/     # 挂载
[root@centos6 ~]$ du -sh /mnt/new_test/                                # 确认文件没有丢失
45M    /mnt/new_test/

快照LVM

分析下当前环境，我把vg移动到centos6上了。一下操作在centos6上执行的。

[root@centos6 ~]$ vgs                                        # 查看下vg信息， 一共6G,还有5G呢。
  VG         #PV #LV #SN Attr   VSize VFree
  vg_test_01   2   1   0 wz--n- 5.99g 4.99g
[root@centos6 ~]$ pvs                                       # 有2个pv
  PV         VG         Fmt  Attr PSize    PFree
  /dev/sdc1  vg_test_01 lvm2 a--u 1020.00m    0 
  /dev/sdc2  vg_test_01 lvm2 a--u    5.00g 4.99g

我们刚才给/mnt/new_test目录挂载逻辑卷vg_test_01。假如我们的/mnt/new_test的内容一直在变化。想去备份数据就需要快照卷了。

创建快照

[root@centos6 etc]$ lvcreate  -L 500M -s -p r -n vg_test_01_snapshot /dev/vg_test_01/lv_test_01       # -L指定大小， -s 表示创建快照而不是普通逻辑卷，-p r 指定权限为只读的， -n指定名字
  Logical volume "vg_test_01_snapshot" created.

注意：如果使用的xfs文件系统，挂载的时候需要加入nouuid,ro选项，且创建快照的时候不能指定-p r选项。具体参考： http://tldp.org/HOWTO/LVM-HOWTO/snapshots_backup.html

模拟一些数据修改操作

[root@centos6 etc]$ echo "zhaojiedi1992" >>/mnt/new_test/zhaojiedi1992.txt                            # 制作一些修改情况
[root@centos6 etc]$ vim /mnt/new_test/etc/motd

监控快照情况

[root@centos6 ~]$ lvdisplay  /dev/vg_test_01/vg_test_01_snapshot 
  --- Logical volume ---
  LV Path                /dev/vg_test_01/vg_test_01_snapshot
  LV Name                vg_test_01_snapshot
  VG Name                vg_test_01
  LV UUID                pZ8x2u-lqPA-uLef-P816-dS8V-wZ0y-PKWWwL
  LV Write Access        read only                                         #只读的
  LV Creation host, time centos6.magedu.com, 2017-12-02 06:11:56 +0800
  LV snapshot status     active destination for lv_test_01                 # 这是lv_test_01的快照
  LV Status              available
  # open                 0
  LV Size                1.00 GiB                                          # 快照的总大小
  Current LE             256
  COW-table size         500.00 MiB
  COW-table LE           125
  Allocated to snapshot  0.01%                                             # 这个地方可以看当前快照占用情况。 
  Snapshot chunk size    4.00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3

挂载快照并备份

[root@centos6 etc]$ mkdir /mnt/test_bak                                            # 创建一个快照挂载目录
[root@centos6 etc]$ mount /dev/vg_test_01/vg_test_01_snapshot  /mnt/test_bak/      # 挂载，因为我们上面创建快照的时候指定-pr只读的
mount: block device /dev/mapper/vg_test_01-vg_test_01_snapshot is write-protected, mounting read-only
[root@centos6 etc]$ cat /mnt/test_bak/                                             # 快照后的zhaojiedi1992.txt没有的。
etc/        lost+found/ 
[root@centos6 etc]$ cat /mnt/test_bak/etc/motd                                     # 这个地方的文件也是快照前的样子。没有问题
Welcom you this system
[root@centos6 etc]$ tar -jcvf /root/test.tar.bz --acl --selinux  --xattrs /mnt/test_bak/   # 备份数据赶紧的

移除快照

[root@centos6 etc]$ umount /mnt/test_bak/                                    # 取消挂载
[root@centos6 etc]$ lvremove /dev/vg_test_01/vg_test_01_snapshot             # 移除了快照卷
Do you really want to remove active logical volume vg_test_01_snapshot? [y/n]: y
  Logical volume "vg_test_01_snapshot" successfully removed

linux磁盘管理系列二：软RAID的实现

1、什么是RAID

RAID全称是独立磁盘冗余阵列（Redundant Array of Independent Disks），基本思想是把多个磁盘组合起来，组合一个磁盘阵列组，使得性能大幅提高。

RAID分为几个不同的等级，各个不同的等级均在数据可靠性及读写性能做了不同的权衡。实际工作中根据自己的业务需求选择不同的RAID方案。

2、RAID的实现方式

外接式磁盘阵列：通过扩展卡提供适配能力
内接式RAID：主板集成RAID控制器安装OS前在BIOS里配置
软件RAID：通过OS实现

3、标准的RAID

3.1 RAID0

RAID0称为条带化存储，将数据分段存储在各个磁盘中，读写均可以并行处理，因此读写速率为单个磁盘的N倍，没有冗余功能，任何一个磁盘的损坏就会导致的数据不可用。

未分类

3.2 RAID1

RADI1是镜像存储，没有数据校验，数据被同等的写入到2个或者多个磁盘中，写入速度相对慢，但是读取速度比较快。

未分类

3.3 RAID 4

RADI4在RAID1的基础上，N个盘用于数据存储，另外加入了1个磁盘作为校验盘。一共N+1个盘，任何一个盘坏掉也不影响数据的访问

3.4 RAID 5

RAID5在RAID4的基础上，由原来的一个盘来存储校验数据，改为每个盘都有数据和校验信息的。

未分类

4、混合RAID

4.1 RAID01

先组成RAID0,然后组成RAID1.

未分类

4.2 RAID10

先组成RAID1,然后组成RAID0

未分类

5、软RAID的实现

5.1 RAID5的实现

创建由三块硬盘组成的可用空间为2G的RAID5设备，要求其chunk大小为256k，文件系统为ext4，开机可自动挂载至/mydata目录

5.1.1 先看看我们的磁盘情况

[root@centos7 Bash]$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0  200G  0 disk
├─sda1   8:1    0    1G  0 part /boot
├─sda2   8:2    0  128G  0 part
├─sda3   8:3    0 48.8G  0 part /
├─sda4   8:4    0  512B  0 part
└─sda5   8:5    0 19.5G  0 part /app
sdb      8:16   0  100G  0 disk
sdc      8:32   0   20G  0 disk
sdd      8:48   0   20G  0 disk
sde      8:64   0   20G  0 disk
sdf      8:80   0   20G  0 disk
sr0     11:0    1  8.1G  0 rom  /run/media/root/CentOS 7 x86_64

这里我们使用sdb,sdc,sdd,每个盘创建一个主分区1G,构建RADI5.

5.1.2 根据实际情况分区

[root@centos7 Bash]$ fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x93d380cf.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-209715199, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-209715199, default 209715199): +1G
Partition 1 of type Linux and of size 1 GiB is set

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'

Command (m for help): p

Disk /dev/sdb: 107.4 GB, 107374182400 bytes, 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x93d380cf

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048     2099199     1048576   fd  Linux raid autodetect

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@centos7 Bash]$ fdisk /dev/sdc
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xc56b90d8.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-41943039, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039): +1G
Partition 1 of type Linux and of size 1 GiB is set

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'

Command (m for help): p

Disk /dev/sdc: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xc56b90d8

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048     2099199     1048576   fd  Linux raid autodetect

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@centos7 Bash]$ fdisk /dev/sdd
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x7e0900d8.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-41943039, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039): +1G
Partition 1 of type Linux and of size 1 GiB is set

Command (m for help): p

Disk /dev/sdd: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x7e0900d8

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1            2048     2099199     1048576   83  Linux

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

5.1.3 创建raid

[root@centos7 Bash]$ mdadm -C /dev/md5 -a yes -l 5 -n 3 /dev/sd{b1,c1,d1} -c 256       # -C指定创建， -a yes 自动创建设备 ， -l 设定level , -n 设定磁盘个数， -c chunk大小
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md5 started.
[root@centos7 Bash]$ mdadm -Ds                                                          # 查看信息
ARRAY /dev/md5 metadata=1.2 name=centos7.magedu.com:5 UUID=2c8ae60d:a799fcb7:9008a046:ae6ea430
[root@centos7 Bash]$ mdadm -Ds >/etc/mdadm.conf                                         # 将软raid信息写入到配置文件中去
[root@centos7 Bash]$ mkdir /mnt/md5                                                     # 创建挂载点目录 
[root@centos7 Bash]$ mkfs.ext4 /dev/md5                                                 # 创建文件系统
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=64 blocks, Stripe width=128 blocks
131072 inodes, 523776 blocks
26188 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536870912
16 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done 

[root@centos7 Bash]$ mount /dev/md5 /mnt/md5                                          # 挂载设备 
[root@centos7 Bash]$ tail -n 1 /etc/mtab
/dev/md5 /mnt/md5 ext4 rw,seclabel,relatime,stripe=128,data=ordered 0 0               # 查看挂载信息
[root@centos7 Bash]$ tail -n 1 /etc/mtab >>/etc/fstab                                 #添加到fstab文件中，确保开机启动，这里建议使用uuid

5.1.4 验证raid

[root@centos7 md5]$ mdadm -D /dev/md5                                                 #查看详细raid5详细信息，可以发现有3个都是working状态的
/dev/md5:
           Version : 1.2
     Creation Time : Wed Dec  6 19:28:22 2017
        Raid Level : raid5
        Array Size : 2095104 (2046.00 MiB 2145.39 MB)
     Used Dev Size : 1047552 (1023.00 MiB 1072.69 MB)
      Raid Devices : 3
     Total Devices : 3
       Persistence : Superblock is persistent

       Update Time : Wed Dec  6 19:39:06 2017
             State : clean 
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 256K

Consistency Policy : resync

              Name : centos7.magedu.com:5  (local to host centos7.magedu.com)
              UUID : 2c8ae60d:a799fcb7:9008a046:ae6ea430
            Events : 18

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       3       8       49        2      active sync   /dev/sdd1
[root@centos7 md5]$ man mdadm
[root@centos7 md5]$ mdadm /dev/md5 -f /dev/sdc1                                                 # -f 设定指定设备故障， 将/dev/sdc1 这个盘标记失败， 看是否数据能访问，我这里使用-f标记失败，工作中可以根据硬盘指示灯判断磁盘状态
mdadm: set /dev/sdc1 faulty in /dev/md5            
[root@centos7 md5]$ mdadm -D /dev/md5                                                           #在次查看信息，发现工作的是2个， 一个失败的设备 
/dev/md5:
           Version : 1.2
     Creation Time : Wed Dec  6 19:28:22 2017
        Raid Level : raid5
        Array Size : 2095104 (2046.00 MiB 2145.39 MB)
     Used Dev Size : 1047552 (1023.00 MiB 1072.69 MB)
      Raid Devices : 3
     Total Devices : 3
       Persistence : Superblock is persistent

       Update Time : Wed Dec  6 19:41:08 2017
             State : clean, degraded                                                             # 这里注意了。 我们的一个盘坏掉了。 raid5状态为降级使用了。
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 1
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 256K

Consistency Policy : resync

              Name : centos7.magedu.com:5  (local to host centos7.magedu.com)
              UUID : 2c8ae60d:a799fcb7:9008a046:ae6ea430
            Events : 20

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       -       0        0        1      removed
       3       8       49        2      active sync   /dev/sdd1

       1       8       33        -      faulty   /dev/sdc1
[root@centos7 md5]$ cat a.txt                                          # 发现我们的数据还是能访问的。没有问题。

5.1.5 替换设备

我这里是磁盘坏掉后的执行替换的，完全可以多一个备用盘，坏掉自动替换的。

[root@centos7 md5]$ mdadm /dev/md5 -a /dev/sde1                        # 上面我们的sdc1数据损坏，我们需要更换新的磁盘来顶替他的位置。这里添加一个sde1的磁盘， fdisk操作这里省去了。
mdadm: added /dev/sde1
[root@centos7 md5]$ mdadm -Ds                                          # 查看详细信息
ARRAY /dev/md5 metadata=1.2 name=centos7.magedu.com:5 UUID=2c8ae60d:a799fcb7:9008a046:ae6ea430
[root@centos7 md5]$ mdadm -D /dev/md5                                  # 查看详细信息
/dev/md5:
           Version : 1.2
     Creation Time : Wed Dec  6 19:28:22 2017
        Raid Level : raid5
        Array Size : 2095104 (2046.00 MiB 2145.39 MB)
     Used Dev Size : 1047552 (1023.00 MiB 1072.69 MB)
      Raid Devices : 3
     Total Devices : 4
       Persistence : Superblock is persistent

       Update Time : Wed Dec  6 19:50:01 2017
             State : clean                                                  # 状态恢复正常了。没有问题
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 1
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 256K

Consistency Policy : resync

              Name : centos7.magedu.com:5  (local to host centos7.magedu.com)
              UUID : 2c8ae60d:a799fcb7:9008a046:ae6ea430
            Events : 43

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       4       8       65        1      active sync   /dev/sde1
       3       8       49        2      active sync   /dev/sdd1

       1       8       33        -      faulty   /dev/sdc1             # 这个盘是坏掉的，我们已经加入了新的磁盘， 这个盘可以干掉了
[root@centos7 md5]$ man mdadm
[root@centos7 md5]$ mdadm /dev/md5 --remove /dev/sdc1                  # 这个盘我们从raid5中移除去。        
mdadm: hot removed /dev/sdc1 from /dev/md5

5.1.6 扩展raid

我们上面使用的是2+1构成的raid5,磁盘利用率为66%，如果我们想改成3+1 可以执行类似如下命令

[root@centos7 mnt]$ mkadm -G -r  /dev/md5 -n 4 -a /dev/sdxx                  # 这里我就不测试了。使用/dev/sdxx代替一个设备。-G 是Grown增长的意思，-r 是resizefs的意思，

5.1.7 清空raid信息

[root@centos7 mnt]$ umount /dev/md5                                   # 卸载设备
[root@centos7 mnt]$ mdadm -S /dev/md5                                 # 停止raid5 
mdadm: stopped /dev/md5
[root@centos7 mnt]$ sed -i '$d' /etc/fstab                            # 删除fstab中关于raid5挂载的行
[root@centos7 mnt]$ cat /etc/fstab                                    # 确保fstab没有大问题

#
# /etc/fstab
# Created by anaconda on Tue Nov  7 16:07:01 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#

UUID=59ccea87-3c4e-4bbc-9e2f-3fadb1dcf2e6 /                       ext4    defaults        1 1
UUID=f4e867e8-bcde-43a2-adc7-c80b0948e85f /app                    ext4    noatime,usrquota,grpquota        1 2
UUID=1d6cbe88-ffb4-4adf-bacf-76be1fa75708 /boot                   ext4    defaults        1 2
#UUID=b2c064f5-1ee5-4b5c-9e75-ed41cb99c5aa swap                    swap    defaults        0 0
#UUID=a0516c4f-40e6-4919-905a-8b44db12ff7b swap               swap    defaults,pri=0        0 0 
#/dev/sdb2 /test ext4 rw,seclabel,relatime,data=ordered 0 0
#/dev/sdb1 /home xfs rw,seclabel,relatime,attr2,inode64,usrquota,grpquota  0 0
[root@centos7 mnt]$ rm -rf /etc/mdadm.conf                                   # 删除raid默认配置文件
[root@centos7 mnt]$ mdadm --zero-superblock /dev/sd{b1,e1,d1,c1}             # 清空设置上的超级块信息

5.2 RAID10的实现

raid10 ，6个分区，2个一组raid1,3组raid0

5.2.1 案例分析

分析下，我们创建一个raid10设置，2个设备组成一个raid1,6个设备2个一组可以组成3个raid1, 然后把3个raid1组成一个raid0即可

5.2.2 先创建6个设备

[root@centos7 mnt]$ lsblk                                # 就是使用fdisk 创建的设备， 具体这里就不写了。 最终使用lsblk显示，我们可以看到sdb1,sdb2,sdd1,sde1一共6个磁盘
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0  200G  0 disk 
├─sda1   8:1    0    1G  0 part /boot
├─sda2   8:2    0  128G  0 part 
├─sda3   8:3    0 48.8G  0 part /
├─sda4   8:4    0  512B  0 part 
└─sda5   8:5    0 19.5G  0 part /app
sdb      8:16   0  100G  0 disk 
├─sdb1   8:17   0    1G  0 part 
└─sdb2   8:18   0    1G  0 part 
sdc      8:32   0   20G  0 disk 
├─sdc1   8:33   0    1G  0 part 
└─sdc2   8:34   0    1G  0 part 
sdd      8:48   0   20G  0 disk 
└─sdd1   8:49   0    1G  0 part 
sde      8:64   0   20G  0 disk 
└─sde1   8:65   0    1G  0 part 
sdf      8:80   0   20G  0 disk 
sr0     11:0    1  8.1G  0 rom  /run/media/root/CentOS 7 x86_64

5.2.3 创建raid

[root@centos7 mnt]$ mdadm -C /dev/md11 -a yes -l 1 -n 2 /dev/sd{b1,c1}                               # 创建第一个raid1
mdadm: /dev/sdb1 appears to be part of a raid array:
       level=raid5 devices=3 ctime=Wed Dec  6 19:28:22 2017
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: /dev/sdc1 appears to be part of a raid array:
       level=raid5 devices=3 ctime=Wed Dec  6 19:28:22 2017
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md11 started.
[root@centos7 mnt]$ mdadm -C /dev/md12 -a yes -l 1 -n 2 /dev/sd{b2,c2}                                #创建第二个raid1
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md12 started.
[root@centos7 mnt]$ mdadm -C /dev/md13 -a yes -l 1 -n 2 /dev/sd{d1,e1}                                 # 创建第三个raid1
mdadm: /dev/sdd1 appears to be part of a raid array:
       level=raid5 devices=3 ctime=Wed Dec  6 19:28:22 2017
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: /dev/sde1 appears to be part of a raid array:
       level=raid5 devices=3 ctime=Wed Dec  6 19:28:22 2017
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md13 started.
[root@centos7 mnt]$ mdadm -C /dev/md10 -a yes -l 0 -n 3 /dev/md{11,12,13}                             # 将3个raid1 合并为一个raid0 
mdadm: /dev/md11 appears to contain an ext2fs file system
       size=2095104K  mtime=Wed Dec  6 19:29:45 2017
mdadm: /dev/md13 appears to contain an ext2fs file system
       size=2095104K  mtime=Wed Dec  6 19:29:45 2017
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md10 started.
[root@centos7 mnt]$ mkfs.ext
mkfs.ext2  mkfs.ext3  mkfs.ext4            
[root@centos7 mnt]$ mkfs.ext4 /dev/md10                                                                 # 创建文件系统                                                                                         
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=384 blocks
196224 inodes, 784896 blocks
39244 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=805306368
24 block groups
32768 blocks per group, 32768 fragments per group
8176 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

[root@centos7 mnt]$ mdadm -Ds                                                                             # 查看配置信息
ARRAY /dev/md11 metadata=1.2 name=centos7.magedu.com:11 UUID=0ce2cd6c:cd21fab6:3e65cfb5:64bd86f3
ARRAY /dev/md12 metadata=1.2 name=centos7.magedu.com:12 UUID=8af31dff:efab06ed:48e2613b:a599c774
ARRAY /dev/md13 metadata=1.2 name=centos7.magedu.com:13 UUID=a8c99d60:2d0c61e7:97a76809:9396c020
ARRAY /dev/md10 metadata=1.2 name=centos7.magedu.com:10 UUID=50b2fa58:4ce65d67:8c50c853:fa175a28
[root@centos7 mnt]$ mdadm -Ds  >> /etc/mdadm.conf                                                         # 写配置文件到mdadm的配置文件中
[root@centos7 mnt]$ mkdir /mnt/md10                                                                       # 创建挂载目录
[root@centos7 mnt]$ mount /dev/md10 /mnt/md10                                                             # 挂载文件系统
[root@centos7 mnt]$ tail -n 1 /etc/mtab                                                                   # 查看mtab文件中的最后一行， 也就是我们的md10挂载信息
/dev/md10 /mnt/md10 ext4 rw,seclabel,relatime,stripe=384,data=ordered 0 0
[root@centos7 mnt]$ tail -n 1 /etc/mtab >> /etc/fstab                                                     #添加到开机启动

5.2.4 raid 清除工作

[root@centos7 mnt]$ umount /dev/md10                                                                      # 取消挂载
[root@centos7 mnt]$ rm -rf /etc/mdadm.conf                                                                # 删除mdadm的默认配置
[root@centos7 mnt]$ mdadm -S /dev/md10                                                                    # 停止raid0设置
mdadm: stopped /dev/md10
[root@centos7 mnt]$ mdadm -S /dev/md11                                                                    # 停止raid1设置
mdadm: stopped /dev/md11
[root@centos7 mnt]$ mdadm -S /dev/md12                                                                    # 停止radi1 设置
mdadm: stopped /dev/md12 
[root@centos7 mnt]$ mdadm -S /dev/md13                                                                    # 停止raid 1 设置
mdadm: stopped /dev/md13
[root@centos7 mnt]$ sed -i '$d' /etc/fstab                                                                # 删除fstab的挂载  
[root@centos7 mnt]$ cat /etc/fstab                                                                        # 确保正确

#
# /etc/fstab
# Created by anaconda on Tue Nov  7 16:07:01 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#

UUID=59ccea87-3c4e-4bbc-9e2f-3fadb1dcf2e6 /                       ext4    defaults        1 1
UUID=f4e867e8-bcde-43a2-adc7-c80b0948e85f /app                    ext4    noatime,usrquota,grpquota        1 2
UUID=1d6cbe88-ffb4-4adf-bacf-76be1fa75708 /boot                   ext4    defaults        1 2
#UUID=b2c064f5-1ee5-4b5c-9e75-ed41cb99c5aa swap                    swap    defaults        0 0
#UUID=a0516c4f-40e6-4919-905a-8b44db12ff7b swap               swap    defaults,pri=0        0 0 
#/dev/sdb2 /test ext4 rw,seclabel,relatime,data=ordered 0 0
#/dev/sdb1 /home xfs rw,seclabel,relatime,attr2,inode64,usrquota,grpquota  0 0
[root@centos7 mnt]$ mdadm -D                                                                                  # 再次查看下mdadm信息，确保没有了
mdadm: No devices given.
[root@centos7 mnt]$ mdadm --zero-superblock /dev/sd{b1,b2,c1,c2,d1,e1}                                        # 请求md的元数据信息

Mariadb二进制安装和配置说明

这几天搞智能DNS，用到了mariadb，在yum install mariadb以后，bind编译安装设置mysql目录时会报出一系列的错误，所以解决办法有3种：1.想都不要想跟定是库lib的问题，只能建立链接库 2.编译安装mariadb 3.二进制安装mariadb，编译很好时，所以选择了二进制安装mariadb。mariadb官网下载：https://downloads.mariadb.org/mariadb/10.2.11/ 你会发现有很多包，这里说明一下这些包都是干什么用的：

未分类

Mariadb安装步骤

#创建mysql用户和用户组，并禁止用户登陆
[root@toydns local]# groupadd mysql
[root@toydns local]# useradd -r -g mysql -s /sbin/nologin mysql

#下载mariadb包：
wget https://downloads.mariadb.org/interstitial/mariadb-10.2.11/bintar-linux-systemd-x86_64/mariadb-10.2.11-linux-systemd-x86_64.tar.gz

#解压：
tar xf mariadb-10.2.11-linux-systemd-x86_64.tar.gz 

#更改安装目录：注意不要提前建立/usr/local/mysql目录，直接放过去即可
mv mariadb-10.2.11-linux-systemd-x86_64 /usr/local/mysql/

#进入/usr/local/mysql目录
[root@toydns mysql]#cd  /usr/local/mysql
[root@toydns mysql]# ll
total 204
drwxr-xr-x.  2 1021 1004  4096 Oct 26 14:04 bin
-rw-r--r--.  1 1021 1004 17987 Nov 27 18:09 COPYING
-rw-r--r--.  1 1021 1004 86263 Nov 27 18:09 COPYING.thirdparty
-rw-r--r--.  1 1021 1004  2275 Nov 27 18:09 CREDITS
drwxrwxr-x.  3 1021 1004  4096 Nov 28 06:16 data
drwxr-xr-x.  2 root root  4096 Dec  4 23:06 etc
-rw-r--r--.  1 1021 1004  8245 Nov 27 18:09 EXCEPTIONS-CLIENT
drwxrwxr-x.  3 1021 1004  4096 Nov 28 06:15 include
-rw-r--r--.  1 1021 1004  8694 Nov 27 18:09 INSTALL-BINARY
drwxr-xr-x.  4 1021 1004  4096 Nov 14 22:35 lib
drwxrwxr-x.  4 1021 1004  4096 Nov 28 06:16 man
drwxrwxr-x. 11 1021 1004  4096 Nov 28 06:15 mysql-test
-rw-r--r--.  1 1021 1004  2374 Nov 27 18:09 README.md
-rw-r--r--.  1 1021 1004 19510 Nov 27 18:09 README-wsrep
drwxrwxr-x.  2 1021 1004  4096 Nov 28 06:15 scripts
drwxrwxr-x. 32 1021 1004  4096 Nov 28 06:15 share
drwxrwxr-x.  4 1021 1004  4096 Nov 28 06:15 sql-bench
drwxrwxr-x.  4 1021 1004  4096 Nov 28 06:15 support-files


相关目录介绍：
bin: 可执行的二进制程序的存放目录，客户端程序mysql就位于这个目录下。
COPYING：版权以及开源信息
COPYING.thirdparty： 版权信息
CREDITS：关于MariaDB软件基金会的一些相关信息，里面还有国内的互联网巨头阿里
data:默认的数据库存放目录，如果我们一开始没有指定数据库存放目录的话，那就会被存储到这个位置。
EXCEPTIONS-CLIENT：例外情况
include：MariaDB 所需要的一些程序文件
INSTALL-BINARY： 安装帮助文档，可以详细阅读，对安装数据库有很大的帮助
lib： 软件运行所需要的库文件
man：软件的帮助文档
mysql-test： 数据库的测试组件
scipts：mysql初始化初始化时要用到的脚本文件，通读一下脚本，可以了解Mysql 的安装过程
share： 共享的文件内容
support-files： mysql 正常运行所需要的配置文件或者文档，这一点很重要，如果我们要自定义配置文件的话，就需要参考这里面的配置文件来进行定义。

这里有一点需要注意：data目录是数据库的存放路径，我们在之前已经手动指定。在实际生产中，企业数据增长很快，数据库文件有可能会很大，因此最好将该目录指定到一个单独的磁盘上，或者大分区，或者使用逻辑卷都可以，避免因物理空间不足，导致出现故障。

#创建mysql配置文件目录
mkdir  /usr/local/mysql/etc

#在/usr/local/mysql/support-files/下的配置文件模板，已经配置好的部分参数，分别用于不同的环境，这里说明一下：
my-small.cnf 这个是为小型数据库或者个人测试使用的，不能用于生产环境
my-medium.cnf 这个适用于中等规模的数据库，比如个人项目或者小型企业项目中，
my-large.cnf 一般用于专门提供SQL服务的服务器中，即专门运行数据库服务的主机，配置要求要更高一些，适用于生产环境
my-huge.cnf 用于企业级服务器中的数据库服务，一般更多用于生产环境使用
所以根据以上几个文件，如果个人使用或者测试，那么可以使用前两个模板；企业服务器或者64G以上的高配置服务器可以使用后面两个模板，另外也可以根据自己的需求来加大参数和扩充配置获得更好的性能。

[root@toydns mysql]# ll support-files/
-rw-r--r--. 1 1021 1004  4914 Nov 27 18:32 my-huge.cnf
-rw-r--r--. 1 1021 1004 20421 Nov 27 18:32 my-innodb-heavy-4G.cnf
-rw-r--r--. 1 1021 1004  4901 Nov 27 18:32 my-large.cnf
-rw-r--r--. 1 1021 1004  4914 Nov 27 18:32 my-medium.cnf
-rw-r--r--. 1 1021 1004  2840 Nov 27 18:32 my-small.cnf

#复制my-medium.cnf到etc下并改名为my.cnf，修改数据库文件存放的目录
cp support-files/my-medium.cnf  etc/my.cnf
vi etc/my.cnf

# The MariaDB server
[mysqld]
port            = 3306   
socket          = /tmp/mysql.sock
basedir = /usr/local/mysql   //mysql目录
datadir= /usr/local/mysql/data  //数据存放位置
innodb_file_per_table = on  //每张表一个单独文件，便于管理
skip_name_resolve = on  //忽略反向解析，加快访问速度
skip-external-locking
key_buffer_size = 16M
max_allowed_packet = 1M
table_open_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M

：wq保存后给权限
[root@toydns local]# chown -R mysql.mysql /usr/local/mysql/
[root@toydns local]# chmod -R 755 /usr/local/mysql/

#添加MySQL到PATH环境变量里面去，省的以后敲命令麻烦
[root@toydns mysql]# vi /etc/profile.d/mysql.sh
export  PATH=/usr/local/mysql/bin:$PATH

source /etc/profile.d/mysql.sh　　使这个配置文件及时生效

#初始化mysql
/usr/local/mysql/scripts/mysql_install_db --user=mysql

#创建mariadb自启动脚本，并加入开机启动
cp /usr/local/mysql/support-files/systemd/mariadb.service  /usr/lib/systemd/system/

systemctl enable mariadb
systemctl start mariadb
systemctl stop mariadb

MySQL的安全设置

[root@toydns mysql]# ./bin/mysql_secure_installation 

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):   //直接回车，默认为空
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] y     //是否设置MySQL管理员root的密码，y设置，输入2次
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] y    //是否删除匿名账户 y删除
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n   //是否不允许root用户远程登陆，n不禁用
 ... skipping.

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] y   //是否删除test测试数据库，y删除
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y   //重新加载可用的数据库表  y 是
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

linux磁盘管理系列一：磁盘配额管理

一、前言

在linux系统中，由于是多用户、多任务的环境，如果有少数几个用户大量使用磁盘空间，导致其他用户的正常使用，因此需要对各个用户的磁盘空间进行管理和限定。

二、quota的用途

限制某一个用户的最大磁盘配额

三、quota的使用限制

仅能针对整个文件系统
内核必须支持
只对一般用户生效
这里提供一个样例，针对样例对quota的配置管理做个描述

四、案例讲解

4.1 案例描述

创建5个用户user1,user2,user3,user4,user5，密码和用户名相同，初始组为usergrp组。
5个用户都可以取得300M的磁盘使用空间，文件数量不限。超过250M,给于提示。
usergrp这个组内成员最大使用空间1GB。
如果有用户超过soft限制，给14天的宽限时间。

4.2 准备磁盘

[root@mail ~]# fdisk -l      #查看磁盘情况

Disk /dev/sda: 42.9 GB, 42949672960 bytes, 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000bd275

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200    83886079    40893440   8e  Linux LVM

Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/cl-root: 39.7 GB, 39720058880 bytes, 77578240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/cl-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

[root@mail ~]# fdisk /dev/sdb     #对sdb这个盘进行分区，这里就分一个区
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xbcd17d69.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-20971519, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519): 
Using default value 20971519
Partition 1 of type Linux and of size 10 GiB is set

Command (m for help): p

Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xbcd17d69

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048    20971519    10484736   83  Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@mail ~]# mkfs.ext4 /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
655360 inodes, 2621184 blocks
131059 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2151677952
80 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done 

[root@mail ~]# mkdir /mnt/home          #创建一个目录
[root@mail ~]# mount /dev/sdb1 /mnt/home   #测试挂载下

4.4 创建用户

[root@mail ~]# vim adduserbat.sh  #创建一个添加用户的脚本
[root@mail ~]# cat adduserbat.sh  #确认下脚本
#!/bin/bash

groupadd usergrp
for user in user1 user2 user3 user4 user5
do 
    useradd -g usergrp -b /mnt/home $user
    echo $user |passwd --stdin $user
done
[root@mail ~]# sh adduserbat.sh    #运行脚本去创建用户
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
Creating mailbox file: File exists
Changing password for user user1.
passwd: all authentication tokens updated successfully.
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
Creating mailbox file: File exists
Changing password for user user2.
passwd: all authentication tokens updated successfully.
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
Creating mailbox file: File exists
Changing password for user user3.
passwd: all authentication tokens updated successfully.
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
Creating mailbox file: File exists
Changing password for user user4.
passwd: all authentication tokens updated successfully.
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
Creating mailbox file: File exists
Changing password for user user5.
passwd: all authentication tokens updated successfully.
[root@mail ~]# finger user1             #查看用户信息，确保家目录在/dev/sdb1的挂载目录上。
Login: user1                      Name: 
Directory: /mnt/home/user1              Shell: /bin/bash
Never logged in.
No mail.
No Plan.
[root@mail ~]# id user1              #查看用户信息
uid=2531(user1) gid=2532(usergrp) groups=2532(usergrp)

4.5 检查操作系统支持

前面提到了quota仅仅针对整个文件系统来进行规划的。需要确认我们为各个用户提供存储的位置是独立的文件系统。

[root@mail ~]# df -h /mnt/home      #查看我们的挂载点是否是独立文件系统
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       9.8G   37M  9.2G   1% /mnt/home
[root@mail ~]# mount |grep /mnt/home     #查看我们的文件系统
/dev/sdb1 on /mnt/home type ext4 (rw,relatime,data=ordered)

4.6 让文件系统支持quota设置

[root@mail ~]# mount -o remount,usrquota,grpquota /mnt/home   #重新挂载/mnt/home 支持usrquota,grpquota
[root@mail ~]# mount |grep /mnt/home                           #确认下
/dev/sdb1 on /mnt/home type ext4 (rw,relatime,quota,usrquota,grpquota,data=ordered)
[root@mail ~]# tail -n 1 /etc/mtab >> /etc/fstab         #追加到/etc/fstab中去，确保开机启用quota
[root@mail ~]# cat /etc/fstab                            #确保fstab文件正确性

#
# /etc/fstab
# Created by anaconda on Fri Feb 10 03:56:55 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/cl-root     /                       xfs     defaults        0 0
UUID=dd4c6743-bdf5-4899-a43b-814cbe75c618 /boot                   xfs     defaults        0 0
/dev/mapper/cl-swap     swap                    swap    defaults        0 0
/dev/sr0 /mnt/cdrom iso9660 ro,relatime,uid=0,gid=0,iocharset=utf8,mode=0400,dmode=0500 0 0
/dev/sdb1 /mnt/home ext4 rw,relatime,quota,usrquota,grpquota,data=ordered 0 0

4.7 扫描文件系统并新建quota的配置文件

[root@mail ~]# quotacheck -avug
quotacheck: Your kernel probably supports journaled quota but you are not using it. Consider switching to journaled quota to avoid running quotacheck after an unclean shutdown.
quotacheck: Scanning /dev/sdb1 [/mnt/home] done
quotacheck: Cannot stat old user quota file /mnt/home/aquota.user: No such file or directory. Usage will not be subtracted.
quotacheck: Cannot stat old group quota file /mnt/home/aquota.group: No such file or directory. Usage will not be subtracted.
quotacheck: Cannot stat old user quota file /mnt/home/aquota.user: No such file or directory. Usage will not be subtracted.
quotacheck: Cannot stat old group quota file /mnt/home/aquota.group: No such file or directory. Usage will not be subtracted.
quotacheck: Checked 30 directories and 20 files
quotacheck: Old file not found.
quotacheck: Old file not found.

主要参数

-a:　　扫描所有在/etc/mtab内含有quota参数的文件系统
-u:　　针对用户扫描文件与目录的使用情况，会新建一个aquota.user文件
-g:　　针对用户组扫描文件与目录的使用情况，会新增一个aquota.group文件
-v:　　显示扫描过程的信息

4.8 启用quota

[root@mail ~]# quotaon  -avug    #启用quota
/dev/sdb1 [/mnt/home]: group quotas turned on
/dev/sdb1 [/mnt/home]: user quotas turned on

这个命令（quotaon）几乎只需要在第一次启动quota时才需要进行，因为下次等你重新启动时，系统的/etc/rc.d/rc.sysinit这个初始化脚本就会自动执行这个命令。

如果想关闭可以使用quotaoff -avug

4.9 编辑账户的的限值

[root@mail ~]# edquota -u user1

会打开一个vi编辑器，修改我们的设置如下图。

未分类

软限制：这个值超过了基本上没事，还是可以创建文件继续使用文件，但是在指定grace天过后就不能在创建文件了。
硬限值：这个值不能超过。

执行如下命令将user1的设置应用到其他用户上

[root@mail ~]# edquota -p user1 -u user2    #-p 指定参考用户，这句话的意思就是将user1的quota信息赋值给user2
[root@mail ~]# edquota -p user1 -u user3
[root@mail ~]# edquota -p user1 -u user4
[root@mail ~]# edquota -p user1 -u user5

4.10 编辑组的设置

[root@mail ~]# edquota -g usergrp

4.11 修改宽限时间

[root@mail ~]# edquota -t

4.12 对用户和组合quota限制查看

[root@mail ~]# quota -uvs user1       #查看user1的限制信息
Disk quotas for user user1 (uid 2531): 
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
      /dev/sdb1     28K    245M    293M               7       0       0        
[root@mail ~]# quota -gvs usergrp
Disk quotas for group usergrp (gid 2532): #查看usergrp的限制信息
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
      /dev/sdb1      0K    879M    977M               0       0       0

参数说明

-u:　　指定用户
-g:　　指定用户组
-s:　　以1024为倍数来指定单位，显示M之类的单位
-v:　　显示用户在文件系统的quota值

4.13 对文件系统quota限制查看

[root@mail ~]# repquota -as
*** Report for user quotas on device /dev/sdb1                   #这里看到是针对/dev/sdb1的文件系统的
Block grace time: 14days; Inode grace time: 7days
                        Space limits                File limits
User            used    soft    hard  grace    used  soft  hard  grace
----------------------------------------------------------------------
root      --     20K      0K      0K              2     0     0       
zhao      --     52K      0K      0K             13     0     0       
user1     --     28K    245M    293M              7     0     0       
user2     --     28K    245M    293M              7     0     0       
user3     --     28K    245M    293M              7     0     0       
user4     --     28K    245M    293M              7     0     0       
user5     --     28K    245M    293M              7     0     0

4.14 quota测试

[user1@mail ~]$ dd if=/dev/zero of=bigfile bs=1M count=270  #先创建一个270M的文件看看 
sdb1: warning, user block quota exceeded.                   #这里提示警告了。 也就是我们超过了软限制的值250了。
270+0 records in
270+0 records out
283115520 bytes (283 MB) copied, 0.715086 s, 396 MB/s
[user1@mail ~]$ dd if=/dev/zero of=bigfile2 bs=1M count=40  #这里我们创建一个40M的文件
sdb1: write failed, user block limit reached.               #提示错误了。超出限制了。
dd: error writing ‘bigfile2’: Disk quota exceeded
23+0 records in
22+0 records out
24035328 bytes (24 MB) copied, 0.1165 s, 206 MB/s
[user1@mail ~]$ du -sk                                      #查看两个文件占用情况
300000    .

4.15 脚本设置quota信息

上面我们对用户和组的设置，它会启动一个vi编辑器，修改保存才生效。需要交互。如果我们想使用script方式快速设置，那就需要使用setquota命令了。

命令使用 setquota [ -u | -g ] 用户名或者组名块大小软限制块大小硬限制文件数量软限制文件数量大小硬限制文件系统

[root@mail ~]# quota -usv user1     #查看user1的quota信息
Disk quotas for user user1 (uid 2531): 
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
      /dev/sdb1    293M*   245M    293M  13days      14       0       0        
[root@mail ~]# setquota -u user1 400000 500000 100 200 /dev/sdb1    #使用setquota修改
[root@mail ~]# quota -usv user1                                     #再次查看quota信息
Disk quotas for user user1 (uid 2531): 
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
      /dev/sdb1    293M    391M    489M              14     100     200

Virsh管理kvm虚拟机认证和加密

前提：使用virsh管理kvm虚拟机的时候出现需要使用密码的情况

[root@a-kvm-t-1 ~]# virsh list --all
Please enter your authentication name: root
Please enter your password: 
error: Failed to reconnect to the hypervisor
error: no valid connection
error: authentication failed: authentication failed

原因分析使用了sasl认证

# 解决
[root@a-kvm-t-1 ~]# saslpasswd2 -a libvirt admin
Password:  # 输入密码
Again (for verification): # 确认密码
[root@a-kvm-t-1 ~]# virsh list --all            
Please enter your authentication name: admin
Please enter your password: #密码
 Id    Name                           State
----------------------------------------------------
 1     kvm-1                          running
 2     kvm-2                          running
 -     kvm-3                          shut off
# 可以看出创建密码后可以通过virsh正常管理kvm虚拟机

下面详细介绍摘自：IBM

使用 SASL 认证和加密进行远程管理

Simple Authentication and Security Layer (SASL) 提供了安全认证和数据加密，但是允许与传统或外部认证和授权服务集成。

在其最简单的形式下，SASL 可用来定义用于进行授权的凭证的数据库。在较复杂的场景中，它可与外部认证服务（例如 Kerberos 或轻量级目录访问协议 (LDAP)）一起工作以对用户进行认证。在这两种场景中，如果远程管理请求并非在受保护的传输层安全性 (TLS) 连接上面运行，那么 libvirtd 守护程序通过需要“类属安全性服务”应用程序编程接口 (GSSAPI) 作为 SASL 方法来提供机密性。libvirtd 守护程序可将 DIGEST-MD5 而不是 GSSAPI 用作 SASL 方法。但是，MD5 散列被视为不安全，不应该使用。SASL 的这些变体支持对推送的数据进行加密。为简便起见，此示例将 DIGEST-MD5 用作 SASL 方法。
要在最简单的场景（没有外部认证或 TLS 安全性）中使用 SASL 配置远程管理，请完成下列步骤：
1. 登录到 KVM 主机。
2. 保存 /etc/libvirt/libvirtd.conf 文件和 /etc/sysconfig/libvirtd 文件的副本。
3. 编辑 /etc/libvirt/libvirtd.conf 文件，并进行下列更改：

1)通过将 listen_tls 配置伪指令设置为 0 来禁用该伪指令（因为没有配置任何 TLS 证书）。否则，libvirtd 守护程序启动会失败。

2)确保通过将配置伪指令 listen_tcp 设置为 1 来启用该伪指令。

3)将 auth_tcp 配置伪指令设置为 sasl，以启用通过 TCP 进行 SASL 认证。

以下示例显示了这些参数（与股票 libvirtd.conf 文件对比），突出显示了要删除 (-) 和增加 (+) 的更改：

--- libvirtd.conf.orig  2012-01-04 11:28:32.000000000 -0600
+++ libvirtd.conf       2012-01-04 11:34:02.000000000 -0600
@@ -19,7 +19,7 @@
 # using this capability.
 #
 # This is enabled by default, uncomment this to disable it
-#listen_tls = 0
+listen_tls = 0
 # Listen for unencrypted TCP connections on the public TCP/IP port.
 # NB, must pass the --listen flag to the libvirtd daemon process for this to
@@ -30,7 +30,7 @@
 # DIGEST_MD5 and GSSAPI (Kerberos5)
 #
 # This is disabled by default, uncomment this to enable it.
-#listen_tcp = 1
+listen_tcp = 1


@@ -143,7 +143,7 @@
 # Don't do this outside of a dev/test scenario. For real world
 # use, always enable SASL and use the GSSAPI or DIGEST-MD5
 # mechanism in /etc/sasl2/libvirt.conf
-#auth_tcp = "sasl"
+auth_tcp = "sasl"
 # Change the authentication scheme for TLS sockets.
 #

编辑 /etc/sysconfig/libvirtd 文件并启用 –listen 参数，以便 libvirtd 守护程序侦听 TCP/IP 连接：

--- libvirtd.orig       2012-01-04 11:41:37.000000000 -0600
+++ libvirtd    2012-01-04 11:31:33.000000000 -0600
@@ -3,7 +3,7 @@

 # Listen for TCP/IP connections
 # NB. must setup TLS/SSL keys prior to using this
-#LIBVIRTD_ARGS="--listen"
+LIBVIRTD_ARGS="--listen"

 # Override Kerberos service keytab for SASL/GSSAPI
 #KRB5_KTNAME=/etc/libvirt/krb5.tab

重新启动 libvirtd 守护程序，以使更改生效：

# /etc/init.d/libvirtd restart
Stopping libvirtd daemon:                                  [  OK  ]
Starting libvirtd daemon:                                  [  OK  ]

既然 3libvirtd 守护程序接受 TCP 连接，请将一些用户添加至 SASL 数据库。以下示例使用 saslpasswd2 命令将 admin 用户添加至 libvirt 凭证数据库。

# saslpasswd2 -a libvirt admin
Password:
Again (for verification):

注：
libvirt 是 libvirtd 守护程序认证域的 SASL 数据库的预期名称。请勿将任何其他名称用于此数据库。
请使这些凭证保持安全状态，因为此数据库中的每个用户都有权登录并运行远程虚拟机管理。

如果 KVM 主机正在运行防火墙，请确保该防火墙允许传入的流量通过 libvirtd 守护程序 TCP 侦听端口。在缺省情况下，该侦听端口为 16509。
通过使用 SASL 认证（以指示启用了 libvirt 的应用程序与 TCP 传输进行连接）来验证设置是否已成功。以下示例从远程管理站运行 virsh 命令，并以新创建的用户 admin 的身份登录，来在 kvmhost.company.org 系统中启动 guest02 实例：

# virsh -c qemu+tcp://kvmhost.company.org/system start guest02
Please enter your authentication name:admin
Please enter your password:
Domain guest02 started

kubernetes 1.8 高可用安装（六）

6 、安装kube-dns

下载kube-dns.yaml

#获取文件
wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/kube-dns.yaml.sed
mv kube-dns.yaml.sed kube-dns.yaml

#修改配置
sed -i 's/$DNS_SERVER_IP/10.96.0.12/g' kube-dns.yaml 
sed -i 's/$DNS_DOMAIN/cluster.local/g' kube-dns.yaml

# 创建
kubectl create -f kube-dns.yaml

kube-dns.yaml

# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Should keep target in cluster/addons/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml
# in sync with this file.

# Warning: This is a file generated from the base underscore template file: kube-dns.yaml.base

apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "KubeDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: $DNS_SERVER_IP
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  # replicas: not specified here:
  # 1. In order to make Addon Manager do not reconcile this replicas parameter.
  # 2. Default is 1.
  # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  strategy:
    rollingUpdate:
      maxSurge: 10%
      maxUnavailable: 0
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      volumes:
      - name: kube-dns-config
        configMap:
          name: kube-dns
          optional: true
      containers:
      - name: kubedns
        image: foxchan/k8s-dns-kube-dns-amd64:1.14.7
        resources:
          # TODO: Set memory limits when we've profiled the container for large
          # clusters, then set request = limit to keep this container in
          # guaranteed class. Currently, this container falls into the
          # "burstable" category so the kubelet doesn't backoff from restarting it.
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        livenessProbe:
          httpGet:
            path: /healthcheck/kubedns
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /readiness
            port: 8081
            scheme: HTTP
          # we poll on pod startup for the Kubernetes master service and
          # only setup the /readiness HTTP server once that's available.
          initialDelaySeconds: 3
          timeoutSeconds: 5
        args:
        - --domain=$DNS_DOMAIN.
        - --dns-port=10053
        - --config-dir=/kube-dns-config
        - --v=2
        env:
        - name: PROMETHEUS_PORT
          value: "10055"
        ports:
        - containerPort: 10053
          name: dns-local
          protocol: UDP
        - containerPort: 10053
          name: dns-tcp-local
          protocol: TCP
        - containerPort: 10055
          name: metrics
          protocol: TCP
        volumeMounts:
        - name: kube-dns-config
          mountPath: /kube-dns-config
      - name: dnsmasq
        image: foxchan/k8s-dns-dnsmasq-nanny-amd64:1.14.7
        livenessProbe:
          httpGet:
            path: /healthcheck/dnsmasq
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - -v=2
        - -logtostderr
        - -configDir=/etc/k8s/dns/dnsmasq-nanny
        - -restartDnsmasq=true
        - --
        - -k
        - --cache-size=1000
        - --no-negcache
        - --log-facility=-
        - --server=/$DNS_DOMAIN/127.0.0.1#10053
        - --server=/in-addr.arpa/127.0.0.1#10053
        - --server=/ip6.arpa/127.0.0.1#10053
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        # see: https://github.com/kubernetes/kubernetes/issues/29055 for details
        resources:
          requests:
            cpu: 150m
            memory: 20Mi
        volumeMounts:
        - name: kube-dns-config
          mountPath: /etc/k8s/dns/dnsmasq-nanny
      - name: sidecar
        image: foxchan/k8s-dns-sidecar-amd64:1.14.7
        livenessProbe:
          httpGet:
            path: /metrics
            port: 10054
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        args:
        - --v=2
        - --logtostderr
        - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.$DNS_DOMAIN,5,SRV
        - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.$DNS_DOMAIN,5,SRV
        ports:
        - containerPort: 10054
          name: metrics
          protocol: TCP
        resources:
          requests:
            memory: 20Mi
            cpu: 10m
      dnsPolicy: Default  # Don't use cluster DNS.
      serviceAccountName: kube-dns

kubernetes 1.8 高可用安装（五）

5 安装网络组件calico

安装前需要确认kubelet配置是否已经增加–network-plugin=cni
如果没有配置就加到kubelet配置文件里

Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin

5.1先装rbac

官方URL
https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/rbac.yaml

calico-rbac.yaml

# Calico Version v2.6.1
# https://docs.projectcalico.org/v2.6/releases#v2.6.1

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: calico-kube-controllers
  namespace: kube-system
rules:
  - apiGroups:
    - ""
    - extensions
    resources:
      - pods
      - namespaces
      - networkpolicies
    verbs:
      - watch
      - list
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: calico-kube-controllers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-kube-controllers
subjects:
- kind: ServiceAccount
  name: calico-kube-controllers
  namespace: kube-system

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: calico-node
  namespace: kube-system
rules:
  - apiGroups: [""]
    resources:
      - pods
      - nodes
    verbs:
      - get

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: calico-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-node
subjects:
- kind: ServiceAccount
  name: calico-node
  namespace: kube-system

5.2 创建calico.yaml

官方URL
https://docs.projectcalico.org/v2.6/getting-started/kubernetes/installation/hosted/calico.yaml

配置修改请看下面的参数说明

# Calico Version v2.6.1
# https://docs.projectcalico.org/v2.6/releases#v2.6.1
# This manifest includes the following component versions:
#   calico/node:v2.6.1
#   calico/cni:v1.11.0
#   calico/kube-controllers:v1.0.0

# This ConfigMap is used to configure a self-hosted Calico installation.
kind: ConfigMap
apiVersion: v1
metadata:
  name: calico-config
  namespace: kube-system
data:
  # Configure this with the location of your etcd cluster.
  etcd_endpoints: " :2379"

  # Configure the Calico backend to use.
  calico_backend: "bird"

  # The CNI network configuration to install on each node.
  cni_network_config: |-
    {
        "name": "k8s-pod-network",
        "cniVersion": "0.1.0",
        "type": "calico",
        "etcd_endpoints": "__ETCD_ENDPOINTS__",
        "etcd_key_file": "__ETCD_KEY_FILE__",
        "etcd_cert_file": "__ETCD_CERT_FILE__",
        "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",
        "log_level": "info",
        "mtu": 1500,
        "ipam": {
            "type": "calico-ipam"
        },
        "policy": {
            "type": "k8s",
            "k8s_api_root": "https://__KUBERNETES_SERVICE_HOST__:__KUBERNETES_SERVICE_PORT__",
            "k8s_auth_token": "__SERVICEACCOUNT_TOKEN__"
        },
        "kubernetes": {
            "kubeconfig": "__KUBECONFIG_FILEPATH__"
        }
    }

  # If you're using TLS enabled etcd uncomment the following.
  # You must also populate the Secret below with these files.
  etcd_ca: ""   # "/calico-secrets/etcd-ca"
  etcd_cert: "" # "/calico-secrets/etcd-cert"
  etcd_key: ""  # "/calico-secrets/etcd-key"

---

# The following contains k8s Secrets for use with a TLS enabled etcd cluster.
# For information on populating Secrets, see http://kubernetes.io/docs/user-guide/secrets/
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: calico-etcd-secrets
  namespace: kube-system
data:
  # Populate the following files with etcd TLS configuration if desired, but leave blank if
  # not using TLS for etcd.
  # This self-hosted install expects three files with the following names.  The values
  # should be base64 encoded strings of the entire contents of each file.
  # etcd-key: null
  # etcd-cert: null
  # etcd-ca: null

---

# This manifest installs the calico/node container, as well
# as the Calico CNI plugins and network config on
# each master and worker node in a Kubernetes cluster.
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: calico-node
  namespace: kube-system
  labels:
    k8s-app: calico-node
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  template:
    metadata:
      labels:
        k8s-app: calico-node
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
        scheduler.alpha.kubernetes.io/tolerations: |
          [{"key": "dedicated", "value": "master", "effect": "NoSchedule" },
           {"key":"CriticalAddonsOnly", "operator":"Exists"}]
    spec:
      hostNetwork: true
      serviceAccountName: calico-node
      containers:
        # Runs calico/node container on each Kubernetes node.  This
        # container programs network policy and routes on each
        # host.
        - name: calico-node
          image: quay.io/calico/node:v2.6.1
          env:
            # The location of the Calico etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # Choose the backend to use.
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
            # Cluster type to identify the deployment type
            - name: CLUSTER_TYPE
              value: "k8s,bgp"
            # Disable file logging so `kubectl logs` works.
            - name: CALICO_DISABLE_FILE_LOGGING
              value: "true"
            # Set Felix endpoint to host default action to ACCEPT.
            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
              value: "ACCEPT"
            # Configure the IP Pool from which Pod IPs will be chosen.
            - name: CALICO_IPV4POOL_CIDR
              value: "192.168.0.0/16"
            - name: CALICO_IPV4POOL_IPIP
              value: "always"
            # Disable IPv6 on Kubernetes.
            - name: FELIX_IPV6SUPPORT
              value: "false"
            # Set Felix logging to "info"
            - name: FELIX_LOGSEVERITYSCREEN
              value: "info"
            # Set MTU for tunnel device used if ipip is enabled
            - name: FELIX_IPINIPMTU
              value: "1440"
            # Auto-detect the BGP IP address.
            - name: IP
              value: "autodetect"
            - name: IP_AUTODETECTION_METHOD
              value: "can-reach=www.baidu.com"
            - name: FELIX_HEALTHENABLED
              value: "true"
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 250m
          livenessProbe:
            httpGet:
              path: /liveness
              port: 9099
            periodSeconds: 10
            initialDelaySeconds: 10
            failureThreshold: 6
          readinessProbe:
            httpGet:
              path: /readiness
              port: 9099
            periodSeconds: 10
          volumeMounts:
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /var/run/calico
              name: var-run-calico
              readOnly: false
            - mountPath: /calico-secrets
              name: etcd-certs
        # This container installs the Calico CNI binaries
        # and CNI network config file on each node.
        - name: install-cni
          image: quay.io/calico/cni:v1.11.0
          command: ["/install-cni.sh"]
          env:
            # The location of the Calico etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            # The CNI network config to install on each node.
            - name: CNI_NETWORK_CONFIG
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: cni_network_config
          volumeMounts:
            - mountPath: /host/opt/cni/bin
              name: cni-bin-dir
            - mountPath: /host/etc/cni/net.d
              name: cni-net-dir
            - mountPath: /calico-secrets
              name: etcd-certs
      volumes:
        # Used by calico/node.
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: var-run-calico
          hostPath:
            path: /var/run/calico
        # Used to install CNI.
        - name: cni-bin-dir
          hostPath:
            path: /opt/cni/bin
        - name: cni-net-dir
          hostPath:
            path: /etc/cni/net.d
        # Mount in the etcd TLS secrets.
        - name: etcd-certs
          secret:
            secretName: calico-etcd-secrets

---

# This manifest deploys the Calico Kubernetes controllers.
# See https://github.com/projectcalico/kube-controllers
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  labels:
    k8s-app: calico-kube-controllers
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ''
    scheduler.alpha.kubernetes.io/tolerations: |
      [{"key": "dedicated", "value": "master", "effect": "NoSchedule" },
       {"key":"CriticalAddonsOnly", "operator":"Exists"}]
spec:
  # The controllers can only have a single active instance.
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      name: calico-kube-controllers
      namespace: kube-system
      labels:
        k8s-app: calico-kube-controllers
    spec:
      # The controllers must run in the host network namespace so that
      # it isn't governed by policy that would prevent it from working.
      hostNetwork: true
      serviceAccountName: calico-kube-controllers
      containers:
        - name: calico-kube-controllers
          image: quay.io/calico/kube-controllers:v1.0.0
          env:
            # The location of the Calico etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints

---

# This deployment turns off the old "policy-controller". It should remain at 0 replicas, and then
# be removed entirely once the new kube-controllers deployment has been deployed above.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: calico-policy-controller
  namespace: kube-system
  labels:
    k8s-app: calico-policy
spec:
  # Turn this deployment off in favor of the kube-controllers deployment above.
  replicas: 0
  strategy:
    type: Recreate
  template:
    metadata:
      name: calico-policy-controller
      namespace: kube-system
      labels:
        k8s-app: calico-policy
    spec:
      hostNetwork: true
      serviceAccountName: calico-kube-controllers
      containers:
        - name: calico-policy-controller
          image: quay.io/calico/kube-controllers:v1.0.0
          env:
            # The location of the Calico etcd cluster.
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-kube-controllers
  namespace: kube-system

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-node
  namespace: kube-system

参数说明：

etcd_endpoints
改为你自己的etcd集群
CALICO_IPV4POOL_CIDR
calico的IP池，不要和集群的cidr，以及机器的其他IP段冲突，比如用：10.10.0.0/16
IP Autodetection methods
机器多网卡的时候，安装calico-node会报错，因为calico默认IP的获取方式是first-found，这个ip可能不是你需要的那个。导致网络不成功，导致注册失败

#calico报错日志
Skipping datastore connection test
IPv4 address 10.96.0.1 discovered on interface kube-ipvs0
No AS number configured on node resource, using global value

需要修改calico.yaml,修改IP的获取方式为autodetect,注意顺序,修改如下

- name: IP
  value: "autodetect"
- name: IP_AUTODETECTION_METHOD
  value: "can-reach=www.baidu.com"

IP_AUTODETECTION_METHOD 参数说明
官方文档URL:https://docs.projectcalico.org/v2.6/reference/node/configuration

使用通过ip访问的interface
can-reach=61.135.169.121
使用通过域名访问的interface
can-reach=www.baidu.com
使用指定的interface
interface=ethx

此时node都应该处于Ready状态

[root@kvm-master network]# kubectl get nodes
NAME         STATUS    ROLES     AGE       VERSION
node2        Ready     <none>    23h       v1.8.0
node1        Ready     <none>    1d        v1.8.0

5.3 安装calicoctl管理calico网络

calicoctl.yaml

# Calico Version v2.6.1
# https://docs.projectcalico.org/v2.6/releases#v2.6.1
# This manifest includes the following component versions:
#   calico/ctl:v1.6.1

apiVersion: v1
kind: Pod
metadata:
  name: calicoctl
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - name: calicoctl
    image: quay.io/calico/ctl:v1.6.1
    command: ["/bin/sh", "-c", "while true; do sleep 3600; done"]
    env:
    - name: ETCD_ENDPOINTS
      valueFrom:
        configMapKeyRef:
          name: calico-config
          key: etcd_endpoints

注意：calicoctl以pod形式运行时，无法使用calicoctl node 命令

kubernetes 1.8 高可用安装（四）

4、安装kubernetes node

Kubernetes的一个Node节点上需要运行如下组件：

Docker，目前安装的是docker-1.12.6
kubelet
kube-proxy 使用daemonset安装

4.1 安装kubelet和cni

安装rpm包

yum localinstall -y kubelet-1.8.0-1.x86_64.rpm kubernetes-cni-0.5.1-1.x86_64.rpm

在任一master节点创建ClusterRoleBinding

kubectl create clusterrolebinding kubelet-bootstrap 
  --clusterrole=system:node-bootstrapper 
  --user=kubelet-bootstrap

4.2 将证书和配置文件同步到本机

rsync -avSH rsync://master_ip/k8s/pki /etc/kubernetes/
rsync -avSH rsync://master_ip/k8s/bootstrap.kubeconfig /etc/kubernetes/

4.3 配置kubelet

/etc/systemd/system/kubelet.service.d/kubelet.conf

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.12 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.pem"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
Environment="KUBELET_EXTRA_ARGS=--v=2 --pod-infra-container-image=foxchan/pause-amd64:3.0 --fail-swap-on=false"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $K
UBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

4.4 配置kube-proxy

修改后启动kubelet

systemctl daemon-reload
systemctl start kubelet

由于采用了 TLS Bootstrapping，所以 kubelet 启动后不会立即加入集群，而是进行证书申请，
看日志

Oct 24 16:45:43  kubelet[240975]: I1024 16:45:43.566069  240975 bootstrap.go:57] Using bootstrap kubeconfig to generate TLS client cert, key and kubeconfig file

看csr,仍然是pending状态

[root@kvm-master manifests]# kubectl get csr
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr-VJFRWBpJqhe3lpLKPULmJ9wfYeF0xoMQF8VzfcvYyqw   2h        kubelet-bootstrap   Approved,Issued
node-csr-yCn3MIUz-luhqwEVva1haugCmoz48ykxU7x4er3pfQs   44s       kubelet-bootstrap   Pending

需要在 master 允许其证书申请

kubectl get csr | grep Pending | awk '{print $1}' | xargs kubectl certificate approve

此时看node已经加入集群

[root@kvm-master manifests]# kubectl get nodes
NAME            STATUS     ROLES     AGE       VERSION
node2   NotReady   <none>    5m        v1.8.0
node1    Ready      <none>    1h        v1.8.0

因为kubelet配置了network-plugin=cni，但是还没安装，所以状态会是NotReady,不想看这个报错或者不需要网络，就可以修改kubelet配置文件，去掉network-plugin=cni 就可以了。

Oct 25 15:48:15 localhost kubelet: W1025 15:48:15.584765  240975 cni.go:196] Unable to update cni config: No networks found in /etc/cni/net.d
Oct 25 15:48:15 localhost kubelet: E1025 15:48:15.585057  240975 kubelet.go:2095] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

创建kube-proxy 相关文件

在master操作

kubectl apply -f kube-proxy-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-proxy
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: system:kube-proxy
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
  - kind: ServiceAccount
    name: kube-proxy
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:node-proxier
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f kubeproxy-ds.yaml

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    k8s-app: kube-proxy
  name: kube-proxy
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kube-proxy
  template:
    metadata:
      labels:
        k8s-app: kube-proxy
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - /usr/local/bin/kube-proxy
          --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
          --cluster-cidr=10.96.0.0/12
          --conntrack-max-per-core=655360
          --conntrack-min=655360
          --conntrack-tcp-timeout-established=1h
          --conntrack-tcp-timeout-close-wait=60s
          --v=2 1>>/var/log/kube-proxy.log 2>&1
        name: kube-proxy
        image: foxchan/kube-proxy-amd64:v1.8.1
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /etc/kubernetes/
          name: k8s
        - mountPath: /var/log/kube-proxy.log
          name: logfile
        - mountPath: /run/xtables.lock
          name: xtables-lock
        - mountPath: /lib/modules
          name: modprobe
      hostNetwork: true
      serviceAccountName: kube-proxy
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      volumes:
      - hostPath:
          path: /etc/kubernetes
        name: k8s
      - hostPath:
          path: /var/log/kube-proxy.log
        name: logfile
      - hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
        name: xtables-lock
      - hostPath:
          path: /lib/modules
          type: ""
        name: modprobe
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate

查看 proxy 是否正常

[root@kvm-master kubeproxy]# kubectl get pods -n kube-system
NAME               READY     STATUS    RESTARTS   AGE
kube-proxy-rw2bt   1/1       Running   0          1m
kube-proxy-sct84   1/1       Running   0          1m