Kubernetes 1.10.0 multi-master installation with 3 Master and 3 Slaves configuration with SSL on CentOS7 - Configuring standalone HAPROXY for kube-apiserver services on masters

Configuring simple external HAPROXY

Configuration of HAPROXY on the haproxy server.

In this instance the haproxy server is kube-haproxy.sujitnet11.net(172.16.254.199) and this will be running keepalived and haproxy. This is where keepalived will bring up the KUBE API Server IP 172.16.254.201 and will route the traffics to the kube-apiserver services running on the master nodes.

On this server install keepalived and haproxy

yum -y install haproxy keepalived

Configure the keepalived configuration file.

The file /etc/keepalived/keepalived.conf looks like this. Please note the IP 172.16.254.201/16 is the IP address that will be used as the KUBE API server IP that will be accepting the requests coming to this IP and will send the traffic to the kube-apiserver services running on the master nodes. Also see the haproxy configuration related to this.

[root@kube-haproxy ~]# cat /etc/keepalived/keepalived.conf

global_defs {

  router_id centos51.virtual.local

}

vrrp_script haproxy {

  script "killall -0 haproxy"

  interval 2

  weight 2

}

vrrp_instance 1 {

  virtual_router_id 1

  advert_int 1

  priority 100

  state MASTER

  interface eth0

  virtual_ipaddress {

    172.16.254.201/16 dev eth0

  }

  track_script {

    haproxy

  }

}

Enable and Start keepalived

systemctl enable keepalived

systemctl statrt keepalived

Install and configure the HAPROXY

On the same server install haproxy and configure.

yum -y install haproxy

The haproxy configuration file /etc/haproxy/haproxy.cfg looks like this.

global

  log /dev/log  local0

  log /dev/log  local1 notice

  chroot /var/lib/haproxy

  pidfile /var/run/haproxy.pid

  stats socket /var/lib/haproxy/stats mode 660 level admin

  stats timeout 30s

  user root

  group root

  daemon

  # Default SSL material locations

  ca-base /srv/kubernetes

  crt-base /srv/kubernetes

  ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

  ssl-default-bind-options no-sslv3

defaults

  log global

  mode  http

  option  httplog

  option  dontlognull

  timeout connect 5000

  timeout client  50000

  timeout server  50000

  timeout http-request 15s

  timeout http-keep-alive 15s

frontend monitor-in

  bind *:33305

  mode http

  option httplog

  monitor-uri /monitor

listen stats

  bind    *:9000

  mode    http

  stats   enable

  stats   hide-version

  stats   uri       /stats

  stats   refresh   30s

  stats   realm     Haproxy\ Statistics

  stats   auth      Admin:Password

frontend k8s-api

  bind 172.16.254.201:6443

  mode tcp

  option tcplog

  tcp-request inspect-delay 5s

  tcp-request content accept if { req.ssl_hello_type 1 }

  default_backend k8s-api

backend k8s-api

  mode tcp

  option tcplog

  option tcp-check

  balance roundrobin

  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100

  server kubem1.sujitnet11.net 172.16.254.221:6443 check

  server kubem2.sujitnet11.net 172.16.254.222:6443 check

  server kubem3.sujitnet11.net 172.16.254.223:6443 check

frontend k8s-api-nonhttps

  bind 172.16.254.201:8080

  #bind 127.0.0.1:6443

  mode tcp

  option tcplog

  tcp-request inspect-delay 5s

  tcp-request content accept if { req.ssl_hello_type 1 }

  default_backend k8s-api-nonhttps

backend k8s-api-nonhttps

  mode tcp

  option tcplog

  option tcp-check

  balance roundrobin

  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100

  server kubem1.sujitnet11.net 172.16.254.221:8080 check

  server kubem2.sujitnet11.net 172.16.254.222:8080 check

  server kubem3.sujitnet11.net 172.16.254.223:8080 check

Start and Enable Haproxy service.

systemctl enable haproxy

systemctl restart haproxy

Other relevant links for this documentation.
You can click on any of the link to view them.

The Main Document - Kubernetes 1.10.0 with 3 Master and Slave nodes and SSL on CentOS7

KVM Host and Guest Preprations

SSL Certificate Generations

Configure simple external HAPROXY

Configuring ETCD with SSL on the Master servers

Creation of the POD Network information in ETCD for flanneld

Install and Configure the Master Service on the Kubernetes Master servers

Installation and Configuration of the Kubernetes Slaves

Installation and testing of kube-dns

Configure the masters not to run kubernetes scheduled PODs