用openresty实现动态upstream反向代理

前言

此文的读者定义为对openresty有一定了解的读者。

openresty:
https://github.com/openresty/lua-nginx-module

此文要讲什么

大家都知道openresty可以用ngx.location.capture和ngx.exec来实现内部跳转,
下面要讲怎么将ngx.location.capture和ngx.exec与upstream模块结合起来,实现一个动态的upstream。

下面的演示中:

80端口表示首次请求入口
8080端口表示upstream的出口

直接上配置和源码

配置: conf/nginx.conf

    worker_processes  1;  
    error_log logs/error.log;  
    events {  
        worker_connections 1024;  
    }  

    http {  

        log_format  main  '$msec $status $request $request_time '  
                          '$http_referer $remote_addr [ $time_local ] '  
                          '$upstream_response_time $host $bytes_sent '  
                          '$request_length $upstream_addr';  

        access_log  logs/access.log main buffer=32k flush=1s;  


        upstream remote_hello {  
            server 127.0.0.1:8080;  
        }  

        upstream remote_world {  
            server 127.0.0.1:8080;  
        }  

        server {  
            listen 80;  

            location /capture {  
                content_by_lua '  
                    local test = require "lua.test"  
                    test.capture_test()  
                ';  
            }  

            location /exec {  
                content_by_lua '  
                    local test = require "lua.test"  
                    test.exec_test()  
                ';  
            }  

            location /upstream {  
                internal;  

                set $my_upstream $my_upstream;  
                set $my_uri $my_uri;  
                proxy_pass http://$my_upstream$my_uri;  
            }  
        }  


        server {  
            listen 8080;  
            location /hello {  
                echo "hello";  
            }  

            location /world {  
                echo "world";  
            }  
        }  
    }  

源码: lua/test.lua

    local _M = { _VERSION = '1.0' }  

    function _M:capture_test()  
        local res = ngx.location.capture("/upstream",  
            {  
                 method = ngx.HTTP_GET,  
                 vars = {  
                     my_upstream = "remote_hello",  
                     my_uri = "/hello",  
                 },  
            }  
        )  
        if res == nil or res.status ~= ngx.HTTP_OK then  
            ngx.say("capture failed")  
            return  
        end  
        ngx.print(res.body)  
    end  

    function _M:exec_test()  
        ngx.var.my_upstream = "remote_world"  
        ngx.var.my_uri = "/world"  
        ngx.exec("/upstream")  
    end  

    return _M  

运行效果

未分类

OpenResty json 删除转义符

OpenResty 中删除 json 中的转义符

cjson 在 encode 时 “/” 会自动添加转义符 “”; 在 decode 时也会自动将转义符去掉。工作中有个特殊需求,需要手工删除转义符。记录备忘,代码如下:

#! /usr/bin/env lua
json = require "cjson"

result = {}
result["stream"] = "lufei"
result["app"] = "live/cartoon"
oldStr = json.encode(result)
local from, to, err = ngx.re.find(oldStr, [[\]])
ngx.say(from, to)
newStr, n, err = ngx.re.gsub(oldStr, [[\/]], [[/]])
ngx.say("oldStr: "..oldStr)
ngx.say("newStr: "..newStr )
t = json.decode(newStr)
ngx.say(t["app"])
dill@bunbun:~/openresty-test/locations$ curl -i localhost:6699/test
HTTP/1.1 200 OK
Server: openresty/1.11.2.2
Date: Wed, 19 Jul 2017 04:38:07 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive

1313
oldStr: {"app":"live/cartoon","stream":"lufei"}
newStr: {"app":"live/cartoon","stream":"lufei"}
live/cartoon

openresty设置用于access_log的自定义变量

期望:在access_log打印自定义变量define_error_code

nginx配置:

    worker_processes  1;  

    events {  
        worker_connections  1024;  
    }  


    http {  

        log_format main '[$time_local] $request $status $remote_addr $define_error_code';  

        server {  
            listen       80;  

            location / {  
                set $define_error_code '';  
                content_by_lua_block {  
                    ngx.var.define_error_code = 9527  
                    ngx.say("hello world")  
                }  
            }  

            access_log logs/access.log main;  
            error_log logs/error.log;  
        }  

    }  

测试结果:

未分类

CentOS Nginx安装配置Let’s Encrypt CA证书

  • 创建存放整数的目录
mkdir /data/www/ssl
  • 创建 CSR 文件

接着就可以生成 CSR(Certificate Signing Request,证书签名请求)文件了。在这之前,还需要创建域名私钥(一定不要使用上面的账户私钥),根据证书不同类型,域名私钥也可以选择 RSA 和 ECC 两种不同类型。以下两种方式请根据实际情况二选一。

  1. 创建 RSA 私钥(兼容性好):
openssl genrsa 4096 > domain.key
  1. 创建 ECC 私钥(部分老旧操作系统、浏览器不支持。优点是证书体积小):
#secp256r1
openssl ecparam -genkey -name secp256r1 | openssl ec -out domain.key
#secp384r1
openssl ecparam -genkey -name secp384r1 | openssl ec -out domain.key

有了私钥文件,就可以生成 CSR 文件了。在 CSR 中推荐至少把域名带 www 和不带 www 的两种情况都加进去,其它子域可以根据需要添加(目前一张证书最多可以包含 100 个域名):

openssl req -new -sha256 -key domain.key -subj "/" -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]nsubjectAltName=DNS:yoursite.com,DNS:www.yoursite.com")) > domain.csr
执行这一步时,如果提示找不到 /etc/ssl/openssl.cnf 文件,请看看 /usr/local/openssl/ssl/openssl.cnf 是否存在。如果还是不行,也可以使用交互方式创建 CSR(需要注意 Common Name 必须为你的域名):
openssl req -new -sha256 -key domain.key -out domain.csr
  • 添加nginx配置,支持验证服务

我们知道,CA 在签发 DV(Domain Validation)证书时,需要验证域名所有权。传统 CA 的验证方式一般是往 [email protected] 发验证邮件,而 Let’s Encrypt 是在你的服务器上生成一个随机验证文件,再通过创建 CSR 时指定的域名访问,如果可以访问则表明你对这个域名有控制权。首先创建用于存放验证文件的目录,例如:

mkdir /data/www/challenges/

然后配置一个 HTTP 服务监听80端口,以 Nginx 为例:

server {
    listen 80;
    # listen [::]:80 default_server;

    server_name www.yoursite.com yoursite.com;
    location ^~ /.well-known/acme-challenge/ {
        alias /home/xxx/www/challenges/;
        try_files $uri =404;
    }

    location / {
        return 301 https://$server_name$request_uri;
    }
}

以上配置优先查找 ~/www/challenges/ 目录下的文件,如果找不到就重定向到 HTTPS 地址。这个验证服务以后更新证书还要用到,建议一直保留。

  • 获取网站证书

先把 acme-tiny 脚本保存到之前的 ssl 目录:

wget https://raw.githubusercontent.com/diafygi/acme-tiny/master/acme_tiny.py

指定账户私钥、CSR 以及验证目录,执行脚本:

python acme_tiny.py --account-key ./account.key --csr ./domain.csr --acme-dir ~/www/challenges/ &gt; ./signed.crt

如果一切正常,当前目录下就会生成一个 signed.crt,这就是申请好的证书文件。
如果你把域名 DNS 解析放在国内,这一步很可能会遇到类似这样的错误:

ValueError: Wrote file to /home/xxx/www/challenges/oJbvpIhkwkBGBAQUklWJXyC8VbWAdQqlgpwUJkgC1Vg, but couldn't download http://www.yoursite.com/.well-known/acme-challenge/oJbvpIhkwkBGBAQUklWJXyC8VbWAdQqlgpwUJkgC1Vg

尝试更改到国外的dns

搞定网站证书后,还要下载 Let’s Encrypt 的中间证书。配置 HTTPS 证书时既不要漏掉中间证书,也不要包含根证书。在 Nginx 配置中,需要把中间证书和网站证书合在一起:

wget -O - https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem &gt; intermediate.pem
cat signed.crt intermediate.pem &gt; chained.pem

为了后续能顺利启用 OCSP Stapling,我们再把根证书和中间证书合在一起:

wget -O - https://letsencrypt.org/certs/isrgrootx1.pem &gt; root.pem
cat intermediate.pem root.pem &gt; full_chained.pem
  • 配置nginx监听443端口
server {
    listen 443;
    client_max_body_size 4G; 
    server_name chinachenshun.com www.chinachenshun.com sdchenshun.com www.sdchenshun.com; 

    ssl on; 
    ssl_session_timeout 5m; 
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; 
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA; 
    ssl_session_cache shared:SSL:50m; 
    ssl_prefer_server_ciphers on; 
    ssl_certificate /data/www/ssl/chained.pem; 
    ssl_certificate_key /data/www/ssl/domain.key;
    ........
  • 配置自动更新

Let’s Encrypt 签发的证书只有 90 天有效期,需要写个自动更新的脚本叫做update_ca.sh,放到crontab执行定时任务,脚本如下:

#!/bin/bash
         cd /home/xxx/www/ssl/
         python acme_tiny.py --account-key account.key --csr domain.csr --acme-dir /home/xxx/www/challenges/ &gt; signed.crt || exit
         wget -O - https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem &gt; intermediate.pem
         cat signed.crt intermediate.pem &gt; chained.pem
         service nginx reload

         vim /etc/crontab 加入以下内容,一个月执行一次:
         0 0 1 * * nobody /home/xxx/update_ca.sh &gt;/dev/null 2&gt;&amp;1

注意,为了安全,crond执行的用户角色是nobody,所以相关联的文件属组需要改成nobody,这样才有权限执行相应的读写。

saltstack快速批量安装nginx

本文讲介绍

使用sls安装nginx,并管理nginx的配置文件,当nginx配置文件被修改时,自动更新配置文件,并重启nginx

在master端上配置nginx.sls文件

 mkdir -p /srv/salt/nginx
 cd /srv/salt/nginx/
vim init.sls
nginx:
  pkg:
    - installed
  service:
    - running
    - enable: True
    - reload: True
    - watch:
      - pkg: nginx
      - file: /etc/nginx/nginx.conf
      - file: /etc/nginx/conf.d/default.conf
/etc/nginx/nginx.conf:
  file.managed:
    - source: salt://etc/nginx/nginx.conf
    - user: root
    - group: root
    - mode: 644
/etc/nginx/conf.d/default.conf:
  file.managed:
    - source: salt://etc/nginx/conf.d/default.conf
    - user: root
    - group: root
    - mode: 644

文件讲解

  • nginx: 这是要安装的包名,也是sls文件的id,不能重复
  • pkg: pkg是包管理模块,对应/usr/lib/python2.6/site-packages/salt/states下的模块pkg.py
  • installed installed是pkg模块下的函数,id(nginx)作为installed的参数进行调用
  • service: service是服务模块,对应/usr/lib/python2.6/site-packages/salt/states下的模块service.py, 由于service是一个key,其下的running, require, watch是列表形式的值,因此service之后有冒号
  • running running是service.py模块下的函数,id(nginx)作为running的参数进行调用
  • enable: True
  • reload: True
  • watch: watch: 表示对文件$file的监控,当master 向minion传递$file时,新的$file与minion上原有文件不一致时,会重启nginx服务
  • pkg: nginx
  • file: /etc/nginx/nginx.conf
  • file: /etc/nginx/conf.d/default.conf
  • /etc/nginx/nginx.conf: 这一行同样是id不能重复:表示传递到minion时所处的位置,同时也作为file.managed函数的参数
  • file.managed:
  • file.py模块的managed函数,下面的source,user, group, mode都是managed函数的参数
  • source: salt://etc/nginx/nginx.conf
  • source 是managed函数的参数,指定要传递到minion端的源文件. –
  • salt://etc/nginx/nginx.conf 表示/etc/nginx/nginx.conf在/srv/salt之下,/srv/salt是saltstack的根目录
  • user: root
    表示文件的属主
  • group: root
    表示文件的属组
  • mode: 644
    表示文件的权限

开始配置

1:在master端上安装nginx,方便生成nginx的配置文件

 yum -y install nginx

2:创建nginx同步目录

mkdir /srv/salt/etc/nginx/conf.d -p

3:拷贝nginx的配置文件到/srv/salt/etc/nginx/目录下

cp /etc/nginx/nginx.conf /srv/salt/etc/nginx/

4:拷贝default.conf配置文件到/srv/salt/nginx/conf.d/目录下

cp /etc/nginx/conf.d/default.conf /srv/salt/etc/nginx/conf.d/

5:开始安装

 salt '*' state.sls nginx

6:测试是否安装成功

salt '*' cmd.run 'rpm -qa | grep nginx'

接下来实现配置更新

手动更新配置文件

在master端将默认端口更改为8080
vim /srv/salt/etc/nginx/conf.d/default.conf

 listen       8080 default_server;

在minion端执行指令,观察
salt-call state.sls nginx

自动更新配置文件

定义pillar的主目录,同时创建pillar目录(master端)

vim /etc/salt/master   #找到以下内容取消注释
pillar_roots:
  base:
    - /srv/pillar
pillar_opts: True
mkdir -p /srv/pillar

定义入口文件top.sls

入口文件的作用一般是定义pillar的数据覆盖被控主机的有效范围,’*’代表任意主机,默认从 base 标签开始解析执行,下一级是操作的目标

cat /srv/pillar/top.sls
base:
  '*':
    - nginx    #指代的是nginx.sls文件

定义nginx文件,每分钟更新一次

install -d /srv/pillar/nginx
cd nginx/
cat init.sls
schedule:
 nginx:
    function: state.sls
    minutes: 1
    args:
        - 'nginx'
刷新被控主机的pillar信息
salt '*' saltutil.refresh_pillar
查看上面定义的nginx.sls数据项,出现以下内容表示成功
salt '*' pillar.data
192.168.31.166:
    ----------
    schedule:
        ----------
        nginx:
            ----------
            args:
                - nginx
            function:
                state.sls
            minutes:
                1
192.168.31.188:
    ----------
    schedule:
        ----------
        nginx:
            ----------
            args:
                - nginx
            function:
                state.sls
            minutes:
                1

测试

在master端将默认端口更改为666
vim /srv/salt/etc/nginx/conf.d/default.conf
 listen       666 default_server;
一分钟后在minion端查看端口:
netstat -tnl

Kubernetes(k8s)部署并测试nginx service

创建2个pod的nginx service

[root@node1 data]#  kubectl run nginx –replicas=2 –labels="run=load-balancer-example" –image=nginx:1.9  –port=80
deployment "nginx" created
[root@node1 yaml]# kubectl get pod –all-namespaces -o wide|grep nginx
default       nginx-3431010723-6kv1z                  1/1       Running   2          1h        10.244.5.6      node5
default       nginx-3431010723-bw22q                  1/1       Running   2          1h        10.244.3.14     node4
[root@node1 kube-config]# kubectl expose deployment nginx –type=NodePort –name=example-service
service "example-service" exposed
[root@node1 kube-config]# kubectl describe svc example-service
Name:            example-service
Namespace:        default
Labels:            run=load-balancer-example
Annotations:        <none>
Selector:        run=load-balancer-example
Type:            NodePort
IP:            10.105.170.116
Port:            <unset>    80/TCP
NodePort:        <unset>    30457/TCP
Endpoints:        10.244.3.14:80,10.244.5.6:80
Session Affinity:    None
Events:            <none>

测试nginx服务

[root@node1 yaml]# curl 10.105.170.116:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

浏览器访问都能显示nginx welcome界面

http://172.172.20.14:30457
http://172.172.20.15:30457

http://nodes:30457

Nginx

Ubuntu 16.04使用Nginx安装HTTP Git服务器

现在使用ISPProtect扫描Web服务器的恶意软件。 免费试用

Git是一个免费的开源版本控制系统,可用于跟踪代码的更改。 Git允许您为同一应用程序创建许多存储库,并在多个人员之间协调这些文件的工作。 它主要用于软件开发中的源代码管理。

在本文中,我们将学习如何在Ubuntu 16.04上安装带有Nginx的HTTP Git服务器。

要求

  • 新的Ubuntu 16.04服务器安装在您的系统上。
  • 具有root权限的Sudo用户。
  • 在您的服务器上配置静态IP地址192.168.15.189

1. 入门指南

开始之前,您将需要使用最新的稳定版本来更新系统。

您可以通过运行以下命令来执行此操作:

sudo apt-get update -y
sudo apt-get upgrade -y

更新系统后,重新启动系统并使用sudo用户登录。

2. 安装所需的软件包

首先,您将需要安装一些所需的软件包,包括nginx,git,nano和fcgiwrap到您的系统。 您可以通过运行以下命令来安装它们:

sudo apt-get install nginx git nano fcgiwrap apache2-utils -y

一旦安装了所有必需的软件包,您将需要为Git存储库创建一个目录。 您可以通过运行以下命令来执行此操作:

sudo mkdir /var/www/html/git

接下来,给予Git目录的正确许可:

sudo chown -R www-data:www-data /var/www/html/git

完成后,您可以继续配置Nginx Web服务器。

3. 配置Nginx

首先,您需要配置Nginx将Git流量传递给Git。 您可以通过编辑Nginx默认配置文件来执行此操作:

sudo nano /etc/nginx/sites-available/default

更改文件如下所示:

# Default server configuration
#
server {
        listen 80 default_server;
        listen [::]:80 default_server;


        root /var/www/html/git;

        # Add index.php to the list if you are using PHP
        index index.html index.htm index.nginx-debian.html;

        server_name _;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }

location ~ (/.*) {
    client_max_body_size 0; # Git pushes can be massive, just to make sure nginx doesn't suddenly cut the connection add this.
    auth_basic "Git Login"; # Whatever text will do.
    auth_basic_user_file "/var/www/html/git/htpasswd";
    include /etc/nginx/fastcgi_params; # Include the default fastcgi configs
    fastcgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend; # Tells fastcgi to pass the request to the git http backend executable
    fastcgi_param GIT_HTTP_EXPORT_ALL "";
    fastcgi_param GIT_PROJECT_ROOT /var/www/html/git; # /var/www/git is the location of all of your git repositories.
    fastcgi_param REMOTE_USER $remote_user;
    fastcgi_param PATH_INFO $1; # Takes the capture group from our location directive and gives git that.
    fastcgi_pass  unix:/var/run/fcgiwrap.socket; # Pass the request to fastcgi
}

}

完成后保存并关闭文件。 然后使用以下命令测试Nginx的任何配置错误:

sudo nginx -t

如果一切正常,您应该看到以下输出:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

接下来,您将需要创建一个用户帐户,您需要使用它来浏览提交到存储库。 您可以使用htpasswd实用程序创建名称为hitesh的用户:

sudo htpasswd -c /var/www/html/git/htpasswd hitesh

最后,重新启动Nginx以使用以下命令应用所有更改:

sudo systemctl restart nginx

您可以使用以下命令检查Nginx服务器的状态:

sudo systemctl status nginx

您应该看到以下输出:

?? nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2017-06-20 23:00:11 IST; 51min ago
  Process: 12415 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
  Process: 7616 ExecReload=/usr/sbin/nginx -g daemon on; master_process on; -s reload (code=exited, status=0/SUCCESS)
  Process: 12423 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 12419 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 12427 (nginx)
   CGroup: /system.slice/nginx.service
           ??????12427 nginx: master process /usr/sbin/nginx -g daemon on; master_process on
           ??????12431 nginx: worker process                           

Jun 20 23:00:11 localhost systemd[1]: Stopped A high performance web server and a reverse proxy server.
Jun 20 23:00:11 localhost systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 20 23:00:11 localhost systemd[1]: nginx.service: Failed to read PID from file /run/nginx.pid: Invalid argument
Jun 20 23:00:11 localhost systemd[1]: Started A high performance web server and a reverse proxy server.

4. 创建Git存储库

一旦配置正确,就可以建立Git仓库了。

您可以使用以下命令创建名称为repo.git的存储库:

cd /var/www/html/git
sudo mkdir hitesh.gitt
sudo git –bare initt
sudo git update-server-info
sudo chown -R www-data.www-data .
sudo chmod -R 777 .

接下来,您将需要通过UFW防火墙允许http服务。 默认情况下,UFW在系统中被禁用,因此您需要先启用它。 您可以使用以下命令启用它:

sudo ufw enable

一旦UFW防火墙启用,您可以通过运行以下命令来允许HTTP服务:

sudo ufw allow http

您现在可以通过运行以下命令检查UFW防火墙的状态:

sudo ufw status

好的,这是服务器端配置。 您现在可以转到客户端来测试Git。

5. 客户机上的测试Git

在启动之前,您将需要在客户端系统上安装git。 您可以使用以下命令安装它:

sudo apt-get install git -y

首先,使用以下命令创建本地存储库:

sudo mkdir ~/testproject

接下来,将目录更改为testproject并使用以下命令启动新的远程存储库:

cd ~/testproject
git init
git remote add origin http://[email protected]/hitesh.git

接下来,使用以下命令创建一些文件和目录:

mkdir test1 test2 test3
echo “This is my first repository” &gt; test1/repo1
echo “This is my second repository” &gt; test2/repo2
echo “This is my third repository” &gt; test3/repo3

接下来,运行以下命令将所有文件和目录添加到存储库中:

git add .
git commit -a -m “Add files and directoires”

您应该看到以下输出:

[master 002fac9] Add files and directoires
 3 files changed, 3 insertions(+)
 create mode 100644 repo1
 create mode 100644 repo2
 create mode 100644 repo3

接下来,使用以下命令将所有文件和目录推送到Git服务器:

git push origin master

您应该看到以下输出:

Password for 'http://[email protected]': 
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (5/5), 422 bytes | 0 bytes/s, done.
Total 5 (delta 0), reused 0 (delta 0)
To http://[email protected]/hitesh.git
   68f1270..002fac9  master -&gt; master

现在,您的所有文件和目录已经提交到您的Git服务器。

您的Git存储库创建过程现已完成。 您将来可以轻松克隆您的存储库。 您可以使用远程系统上的以下命令克隆您的存储库:

git clone [email protected]:/var/www/html/git/hitesh.git

您应该看到以下输出:

Cloning into 'hitesh'...
[email protected]'s password: 
remote: Counting objects: 8, done.
remote: Compressing objects: 100% (3/3), done.
Receiving objects: 100% (8/8), 598 bytes | 0 bytes/s, done.
remote: Total 8 (delta 0), reused 0 (delta 0)
Checking connectivity... done.

现在,使用以下命令将目录更改为克隆的存储库:

cd hitesh
tree

您应该看到以下输出:

.
|-- test1
|   `-- repo1
|-- test2
|   `-- repo2
`-- test3
    `-- repo3

3 directories, 3 files

centos7安装配置gitlab(使用外部nginx)

1、安装依赖:

sudo yum install curl policycoreutils openssh-server openssh-clients
sudo systemctl enable sshd
sudo systemctl start sshd
sudo yum install postfix
sudo systemctl enable postfix
sudo systemctl start postfix
sudo firewall-cmd --permanent --add-service=http
sudo systemctl reload firewalld

2、添加gitlab源:

curl -sS https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.rpm.sh | sudo bash

3、安装gitlab-ce

sudo yum install gitlab-ce

如果您不喜欢通过管道脚本安装存储库,您可以在这里找到整个脚本并手动选择并下载包并使用:

curl -LJO https://packages.gitlab.com/gitlab/gitlab-ce/packages/el/7/gitlab-ce-XXX.rpm/download
rpm -i gitlab-ce-XXX.rpm

4、配置gitlab:

sudo gitlab-ctl reconfigure
至此gitlab安装成功,默认用户名为root。

5、停止gitlab自带的nginx

打开文件$sudo vi /etc/gitlab/gitlab.rb。
将nginx['enable'] = ture改为nginx['enable'] = false
重启gitlab:sudo gitlab-ctl reconfigure。

6、修改gitlab域名:

打开/etc/gitlab/gitlab.rb文件,将external_url参数修改为自己的域名。

7、添加外部nginx的gitlab配置文件:

vim /etc/nginx/conf.d/gitlab.conf

添加以下内容:

upstream gitlab {
# 7.x 版本在此位置
# server unix:/var/opt/gitlab/gitlab-rails/tmp/sockets/gitlab.socket;
# 8.0 位置
server unix://var/opt/gitlab/gitlab-rails/sockets/gitlab.socket;
}

server {
 listen *:80;

 server_name gitlab.xuwanqiu.com; # 请修改为你的域名

 server_tokens off; # don't show the version number, a security best practice
 root /opt/gitlab/embedded/service/gitlab-rails/public;

 # Increase this if you want to upload large attachments
 # Or if you want to accept large git objects over http
 client_max_body_size 250m;

 # individual nginx logs for this gitlab vhost
 access_log /var/log/gitlab/nginx/gitlab_access.log;
 error_log /var/log/gitlab/nginx/gitlab_error.log;

 location / {
 # serve static files from defined root folder;.
 # @gitlab is a named location for the upstream fallback, see below
 try_files $uri $uri/index.html $uri.html @gitlab;
 }

 # if a file, which is not found in the root folder is requested,
 # then the proxy pass the request to the upsteam (gitlab unicorn)
 location @gitlab {
 # If you use https make sure you disable gzip compression
 # to be safe against BREACH attack

 proxy_read_timeout 300; # Some requests take more than 30 seconds.
 proxy_connect_timeout 300; # Some requests take more than 30 seconds.
 proxy_redirect off;

 proxy_set_header X-Forwarded-Proto $scheme;
 proxy_set_header Host $http_host;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Frame-Options SAMEORIGIN;

 proxy_pass http://gitlab;
 }

 # Enable gzip compression as per rails guide: http://guides.rubyonrails.org/asset_pipeline.html#gzip-compression
 # WARNING: If you are using relative urls do remove the block below
 # See config/application.rb under "Relative url support" for the list of
 # other files that need to be changed for relative url support
 location ~ ^/(assets)/ {
 root /opt/gitlab/embedded/service/gitlab-rails/public;
 # gzip_static on; # to serve pre-gzipped version
 expires max;
 add_header Cache-Control public;
 }

 error_page 502 /502.html;
}

参考资料:

http://blog.csdn.net/peterxiaoq/article/details/73330302
http://www.cnblogs.com/lixiuran/p/6761299.html
https://segmentfault.com/q/1010000003695935?_ea=337139
https://laravel-china.org/topics/2829/centos-7-install-gitlab-ce-community-edition-and-modify-the-default-nginx
https://about.gitlab.com/installation/#centos-7
https://github.com/gitlabhq/gitlabhq/blob/master/doc/install/installation.md
http://jiankg.github.io/2015/06/12/%E5%9C%A8centos7%E4%B8%8A%E6%90%AD%E5%BB%BAgitlab%E7%9A%84%E6%AD%A3%E7%A1%AE%E5%A7%BF%E5%8A%BF/

awk查看统计Nginx访问日志

nginx日志最好实现每天定时切割下,特别是在访问量比较大的时候,方便查看与处理,如果没切割,可以用sed直接切割,

切割日志

查找7月17日访问log导出到17.log文件中:

cat gelin_web_access.log | egrep "17/Jul/2017" | sed  -n '/00:00:00/,/23:59:59/p' > /tmp/17.log

查看访问量前10的IP

awk '{print $1}' 17.log | sort | uniq -c | sort -nr | head -n 10 

查看访问前10的URL

awk '{print $11}' gelin_web_access.log | sort | uniq -c | sort -nr | head -n 10

查询访问最频繁的URL

awk '{print $7}' gelin_web_access.log | sort | uniq -c | sort -n -k 1 -r | more

查询访问最频繁的IP

awk '{print $1}' gelin_web_access.log | sort | uniq -c | sort -n -k 1 -r | more

根据访问IP统计UV

awk '{print $1}' gelin_web_access.log | sort | uniq -c | wc -l

统计访问URL统计PV

awk '{print $7}' gelin_web_access.log | wc -l

根据时间段统计查看日志

cat gelin_web_access.log | sed -n '/17/Jul/2017:12/,/17/Jul/2017:13/p' | more

Nginx配置basic_auth密码验证

为Nginx添加basic_auth,意思就是访问页面的时候需要弹出来一个用户和密码验证的东西,本文基于CentOS 6

1. 安装密码生成工具htpasswd并生成用户密码文件

yum install httpd-tools               #适用centos
sudo apt-get install apache2-utils    #适用ubuntu

生成用户密码文件

$ htpasswd -c /var/www/html/.htpasswd user1  #回车会要求输入两遍密码,会清除所有用户!
$ htpasswd -bc /var/www/html/.htpasswd user1 password  #不用回车,直接指定user1的密码为password
$ htpasswd -b /var/www/html/.htpasswd user2 password   #添加一个用户,如果用户已存在,则是修改密码
$ htpasswd -D /var/www/html/.htpasswd user2  #删除用户

2. 为Nginx添加basic_auth配置

server {
    listen        80;
#    root        /tmp;
#    index        index.html index.htm;
    server_name    zhukun.net www.zhukun.net;

    location / {
        auth_basic        "input you user name and password";
        auth_basic_user_file    /export/servers/.htpasswd;
        proxy_pass http://127.0.0.1:9000;
    }
}

然后再次访问zhukun.net时便会弹出验证框要求输入用户名和密码。

3. 可能遇到的问题

访问zhukun.net没有弹出验证框怎么办?
首先修改nginx.conf,将日志级别调为info,如下

$ cat /export/servers/nginx-1.12.1/conf/nginx.conf
.......
user  admin;
worker_processes  8;

error_log  logs/error.log info;
......

然后再次访问让其产error_log
看到error_log时会发现有如下错误产生

*69 no user/password was provided for basic authentication, client: 10.12.138.126, server: www.zhukun.net, request: "GET /date_lateral HTTP/1.1", host: "www.zhukun.net"

原因在于

The HTTP Basic authentication works as following:
*) A browser requests a page without user/password.
*) A server response with 401 page, sending realm as well.
   At this stage the 401 code appears in access_log and the message
   “no user/password …” appears in error_log.
*) The browser shows a realm/login/password prompt.
*) If a user will press cancel, then the browser will show the received
   401 page.
*) If the user enters login/password, then the browser repeats the request
   with login/password.

Then until you will exit the browser, it will send these login/password
with all requests in protected hierarchy.

error_page配置的401页面不存在或者指向问题导致的,可以注释掉401配置或者保证401配置指向的文件可用,然后basic_auth便会生效。