nginx、apache反向代理用户请求至后端Tomcat 集群以及session绑定，session共享-linux交流-黑帽联盟

admin 发表于 2019-10-13 19:50:15

nginx、apache反向代理用户请求至后端Tomcat 集群以及session绑定，session共享

环境：
CentOS-7.3 ： 172.31.225.243  tomcat
CentOS-7.3 ： 172.31.225.244  tomcat
CentOS-6.7 ： 172.31.225.246  nginx或者apache（2.2.15）
物理机  ：  172.31.225.76  主要用来测试访问，无无其它用途

在三台主机上进行本地host解析：/etc/hosts（添加如下内容）
172.31.225.243 node1.cnblackhat.com
172.31.225.244 node2.cnblackhat.com
172.31.225.246 node6.cnblackhat.com

1、nginx（反向代理负载均衡）+tomcat
CentOS-6.7(172.31.225.246) 安装nginx
yum install nginx -y

配置nginx
在http区段内添加如下内容：
upstream tcsrvs {
ip_hash;
      server node2.cnblackhat.com:8080;
      server node1.cnblackhat.com:8080;
}

为了实现会话绑定，这里使用了ip_hash算法，基于ip地址进行hash，使同一个ip地址始终发往同一台后端server

在server区段内添加如下内容，以实现后端后端分发，进行分在均衡
      location / {
proxypass http://tcsrvs;
      }

CentOS-7.3(172.31.225.243)和CentOS-7.3(172.31.225.244)上安装tomcat
前提需要安装oracle官方的JDK或者开源的openjdk，centos7自带openjdk，如果你的是centos6，请自行安装
下载tomcat：wget http://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-8/v8.5.46/bin/apache-tomcat-8.5.46.tar.gz

tar xf apache-tomcat-8.5.46.tar.gz -C /usr/local/
ln -sv apache-tomcat-8.5.46 tomcat
vim /etc/profile.d/tomcat.sh #内容如下
export CATALINA_HOME=/usr/local/tomcat
export PATH=$CATALINA_HOME/bin:$PATH

如果你安装的是oracle官方的jdk，需要你自行添加java环境变量，如果是开源的openjdk，就不要做如下步骤了，跳过即可
vim /etc/profile.d/tomcat.sh #内容如下
export JAVA_HOME=/usr/java/latest
export PATH=$JAVA_HOME/bin:$PATH

最后加载环境变量，使其生效：
. /etc/profile.d/tomcat.sh
. /etc/profile.d/java.sh

最后编辑配置文件/usr/local/tomcat/conf/server.xml，直接贴出配置文件
172.31.225.243的配置文件：
<?xml version="1.0" encoding="UTF-8"?>


<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.startup.VersionLoggerListener" />


<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />

<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />


<GlobalNamingResources>

<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>


<Service name="Catalina">





<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />








<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />




<Engine name="Catalina" defaultHost="node1.cnblackhat.com" jvmRoute="TomcatA">





<Realm className="org.apache.catalina.realm.LockOutRealm">

<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>

<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">





<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log" suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />

</Host>
<Host name="node1.cnblackhat.com" appBase="/data/webapps" unpackWARS="true" autoDeploy="false">
<Context path="" docBase="ROOT" reloadable="true" />
<Context path="/shop" docBase="shopxx" reloadable="true" />
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="/data/logs"
prefix="web1_access_log" suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
</Host>
</Engine>
</Service>
</Server>

172.31.225.244的配置文件：
<?xml version="1.0" encoding="UTF-8"?>


<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.startup.VersionLoggerListener" />


<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />

<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />


<GlobalNamingResources>

<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>


<Service name="Catalina">





<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />








<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />




<Engine name="Catalina" defaultHost="node2.cnblackhat.com" jvmRoute="TomcatB">





<Realm className="org.apache.catalina.realm.LockOutRealm">

<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
</Realm>

<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">





<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log" suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />

</Host>
<Host name="node2.cnblackhat.com" appBase="/data/webapps"
unpackWARs="true" autoDeploy="false">
<Context path="" docBase="ROOT" reloadable="true">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="/data/logs"
prefix="web1_access_log" suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
</Context>
</Host>
</Engine>
</Service>
</Server>

最后启动

两台主机网站根目录创建
mkdir -pv /data/webapps/ROOT
172.31.225.243主页内容：
vim /data/webapps/ROOT/index.jsp
<%@ page language="java" %>
<html>
  <head><title>TomcatA_172.31.225.243</title></head>
  <body>
<h1><font color="red">TomcatA 172.31.225.243 </font></h1>
<table align="centre" border="1">
   <tr>
      <td>Session ID</td>
<% session.setAttribute("abc","abc"); %>
      <td><%= session.getId() %></td>
   </tr>
   <tr>
      <td>Created on</td>
      <td><%= session.getCreationTime() %></td>
   </tr>
</table>
  </body>
</html>

172.31.225.244主页内容：
vim /data/webapps/ROOT/index.jsp
<%@ page language="java" %>
<html>
<head><title>TomcatB_172.31.225.244</title></head>
  <body>
<h1><font color="red">TomcatB 172.31.225.244 </font></h1>
<table align="centre" border="1">
   <tr>
      <td>Session ID</td>
<% session.setAttribute("abc","abc"); %>
      <td><%= session.getId() %></td>
   </tr>
   <tr>
      <td>Created on</td>
      <td><%= session.getCreationTime() %></td>
   </tr>
</table>
  </body>
</html>

最后启动三台主机的web服务
service nginx start
catalina.sh start (两台tomcat主机上执行的操作)

测试：（部署成功，会话始终指向172.31.225.243这台机器）

为了突出负载均衡效果，这里我们把nginx里面的ip_hash给注释掉，基于轮询的方式调度，测试效果图如下：

以上就是基于nginx反向代理用户至后端tomcat集群以及session绑定

2、apache（反向代理负载均衡）+tomcat
这里apache反向代理至后端有三种方式，下面一一介绍：
2.1、基于http协议反向代理至后端tomcat集群
安装apache
yum install httpd -y

编辑配置文件：/etc/httpd/conf.d/vhost.conf

Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
<proxy balancer://lbcluster>
      BalancerMember http://172.31.225.243:8080 loadfactor=10 route=TomcatA
      BalancerMember http://172.31.225.244:8080 loadfactor=10 route=TomcatB
      ProxySet stickysession=ROUTEID
</proxy>

<VirtualHost *:80>
      ServerName web1.cnblackhat.com
      ProxyVia on
      ProxyRequests Off
      ProxyPreserveHost On
      <Proxy>
#             Require all granted
            Order allow,deny
            Allow from all
      </Proxy>
      ProxyPass / balancer://lbcluster/
      ProxyPassReverse / balancer://lbcluster/
      <Location />
#             Require all granted
            Order allow,deny
            Allow from all
      </Location>
</VirtualHost>

这里我定义的虚拟主机是web1.cnblackhat.com，所以我需要在物理机器上解析web1.cnblackhat.com，编辑C:\Windows\System32\drivers\etc\hosts文件
172.31.225.246 web1.cnblackhat.com

最后我们启动httpd服务
service httpd start

在物理机器上测试：（session会话绑定部署成功）

同样为了能够更直观的看到负载均衡效果，我们把配置文件里面的这两行注释掉，把session会话绑定功能注释
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
ProxySet stickysession=ROUTEID

测试效果图如下：

2.2、基于ajp协议反向代理至后端tomcat集群
直接修改配置文件/etc/httpd/conf.d/vhost.conf
<proxy balancer://lbcluster>
      BalancerMember ajp://172.31.225.243:8009 loadfactor=10 route=TomcatA
      BalancerMember ajp://172.31.225.244:8009 loadfactor=10 route=TomcatB
      ProxySet stickysession=ROUTEID
</proxy>

<VirtualHost *:80>
      ServerName web1.cnblackhat.com
      ProxyVia on
      ProxyRequests Off
      ProxyPreserveHost On
      <Proxy>
#             Require all granted
            Order allow,deny
            Allow from all
      </Proxy>
      ProxyPass / balancer://lbcluster/
      ProxyPassReverse / balancer://lbcluster/
      <Location />
#             Require all granted
            Order allow,deny
            Allow from all
      </Location>
</VirtualHost>

保存配置文件，以上只是删除了http协议配置文件里面的第一行，和对应的协议（即把http换成ajp）

测试（session会话绑定部署成功）

老样子，我们把session会话绑定功能所涉及的代码注释掉，以实现可以更直观的看到负载均衡效果：
注释的内容如下：
ProxySet stickysession=ROUTEID

2.3、基于第三方模块mod_jk以实现反向代理至后端tomcat集群
mod_jk模块需要我们自己下下载编译安装：（mod_jk其实也是基于ajp协议的）
wget http://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-connectors/jk/tomcat-connectors-1.2.46-src.tar.gz

tar xf tomcat-connectors-1.2.46-src.tar.gz
cd tomcat-connectors-1.2.46-src/native/

安装的时候我们需要添加一个参数--with-apxs=/usr/sbin/apxs
默认我们是没有apxs，所以我们需要安装，直接安装httpd-devel即可
yum install httpd-devel -y

然后我们接着编译安装tomcat的连接器：
./configure --with-apxs=/usr/sbin/apxs
make && make install

编辑vhost.conf文件：
LoadModule jk_module /usr/lib64/httpd/modules/mod_jk.so
JkWorkersFile /etc/httpd/conf.d/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel    debug
JkMount /*    lbcluster1
JkMount /status/       stat1

紧接着编辑/etc/httpd/conf.d/workers.properties
worker.list=lbcluster1,stat1
worker.TomcatA.port=8009
worker.TomcatA.host=172.31.225.243
worker.TomcatA.type=ajp13
worker.TomcatA.lbfactor=1
worker.TomcatB.port=8009
worker.TomcatB.host=172.31.225.244
worker.TomcatB.type=ajp13
worker.TomcatB.lbfactor=1
worker.lbcluster1.type=lb
worker.lbcluster1.sticky_session=1
worker.lbcluster1.balance_workers=TomcatA,TomcatB
worker.stat1.type = status

最后重启httpd服务
service httpd restart

效果如下（session会话绑定部署成功）

实现负载均衡效果，需要修改如下参数的值
worker.lbcluster1.sticky_session=1  修改成  worker.lbcluster1.sticky_session=0

修改完，记得重启httpd服务，效果图如下：

以上就是基于apache反向代理用户至后端tomcat集群以及session绑定的三种方式

这里我们已经实现了session的会话绑定，session共享还没有实现，不知道你们有没有发现，上面我们所作的操作都是在反向代理端进行session会话绑定的；这里我们也可以在后端tomcat主机上进行设置，以实现能够session共享，共享的同时，还能实现看到更直观的负载均衡，因为有了session共享，每次请求的内容都不一样，但是session都一样的，没有变化，这里的session指的是如下图所示：

下面我们就在tomcat的两台主机进行如下操作：
这里我就直接给出两台主机的配置文件了：
172.31.225.243的配置文件：（其实就是在原来的基础上添加cluster标签段）
<?xml version="1.0" encoding="UTF-8"?>


<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  
  
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  
  <GlobalNamingResources>

<Resource name="UserDatabase" auth="Container"
            type="org.apache.catalina.UserDatabase"
            description="User database that can be updated and saved"
            factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
            pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  
  <Service name="Catalina">





<Connector port="8080" protocol="HTTP/1.1"
            connectionTimeout="20000"
            redirectPort="8443" />








<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />




<Engine name="Catalina" defaultHost="node1.cnblackhat.com" jvmRoute="TomcatA">

   
   

   
   <Realm className="org.apache.catalina.realm.LockOutRealm">
      
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
            resourceName="UserDatabase"/>
   </Realm>

   <Host name="localhost"  appBase="webapps"
         unpackWARs="true" autoDeploy="true">

      
      

      
      <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
            prefix="localhost_access_log" suffix=".txt"
            pattern="%h %l %u %t "%r" %s %b" />

   </Host>
   <Host name="node1.cnblackhat.com" appBase="/data/webapps" unpackWARS="true" autoDeploy="false">

      <Context path="" docBase="ROOT" reloadable="true" />
      <Context path="/shop" docBase="shopxx" reloadable="true" />
      <Valve className="org.apache.catalina.valves.AccessLogValve" directory="/data/logs"
            prefix="web1_access_log" suffix=".txt"
            pattern="%h %l %u %t "%r" %s %b" />
   </Host>
</Engine>
  </Service>
</Server>

172.31.225.244的配置文件：
<?xml version="1.0" encoding="UTF-8"?>


<Server port="8005" shutdown="SHUTDOWN">
  <Listener className="org.apache.catalina.startup.VersionLoggerListener" />
  
  
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
  <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />

  
  <GlobalNamingResources>

<Resource name="UserDatabase" auth="Container"
            type="org.apache.catalina.UserDatabase"
            description="User database that can be updated and saved"
            factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
            pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  
  <Service name="Catalina">





<Connector port="8080" protocol="HTTP/1.1"
            connectionTimeout="20000"
            redirectPort="8443" />








<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />




<Engine name="Catalina" defaultHost="node2.cnblackhat.com" jvmRoute="TomcatB">

   
   

   
   <Realm className="org.apache.catalina.realm.LockOutRealm">
      
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
            resourceName="UserDatabase"/>
   </Realm>

   <Host name="localhost"  appBase="webapps"
         unpackWARs="true" autoDeploy="true">

      
      

      
      <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
            prefix="localhost_access_log" suffix=".txt"
            pattern="%h %l %u %t "%r" %s %b" />

   </Host>
   <Host name="node2.cnblackhat.com" appBase="/data/webapps"
         unpackWARs="true" autoDeploy="false">

      <Context path="" docBase="ROOT" reloadable="true">
      <Valve className="org.apache.catalina.valves.AccessLogValve" directory="/data/logs"
            prefix="web1_access_log" suffix=".txt"
            pattern="%h %l %u %t "%r" %s %b" />
      </Context>
   </Host>
</Engine>
  </Service>
</Server>

详细参数介绍点击这里：https://bbs.cnblackhat.com/thread-2777-1-1.html

我们如果想要使其sesion能够共享，还要做最后一步，这一步必须要做，不然session实现不了共享
在172.31.225.243和172.31.225.244做如下步骤：
mkdir /data/webapps/ROOT/WEB-INF/
cp /usr/local/tomcat/conf/web.xml /data/webapps/ROOT/WEB-INF/

编辑/data/webapps/ROOT/WEB-INF/web.xml，在<web-app>标签内部添加如下标签：
<distributable/>

保存web.xml配置文件

最后在两台主机上重启tomcat
catalina stop
catalina start

在/usr/local/tomcat/logs/catalina.out日志里面看到如下记录：
13-Oct-2019 18:07:25.548 INFO org.apache.catalina.ha.tcp.SimpleTcpCluster.memberAdded Replication member added:]

13-Oct-2019 18:07:24.717 INFO org.apache.catalina.ha.tcp.SimpleTcpCluster.memberAdded Replication member added:]

说明集群添加成功，并可以实现高可用，因为session是共享的，即使一台主机挂了，另一台也能够顶上

下面我们就在172.31.225.246这台主机上就可以把nginx和apache配置文件里面实现session会话绑定功能的相关参数给注释掉，注释哪些内容，上面内容都提到过，自行回忆。

最后通过nginx和apache的三种反向代理都一一测试，成功部署session共享，效果图如下：（Session ID是没有变化的，后面的TomcatA和TomcatB会变化）

页: [1]

黑帽联盟's Archiver

nginx、apache反向代理用户请求至后端Tomcat 集群以及session绑定，session共享