1.K8S是什么?
K8S全称是Kubernetes,是一个全新的基于容器技术的分布式架构领先方案,基于容器技术,目的是实现资源管理的自动化,以及跨多个数据中心的资源利用率的最大化。
如果我们的系统设计遵循了kubernetes的设计思想,那么传统系统架构中那些和业务没有多大关系的底层代码或功能模块,都可以使用K8S来管理,我们不必再费心于负载均衡的选型和部署实施问题,不必再考虑引入或自己开发一个复杂的服务治理框架,不必再头疼与服务监控和故障处理模块的开发。总之,使用kubernetes提供的解决方案,会大大减少开发成本,同时可以将精力更加集中于业务本身,而且由于kubernetes提供了强大的自动化机制,所以系统后期的运维难度和运维成本大幅降低。
2.为什么要用K8S?
Docker 这个新兴的容器化技术当前已经被很多公司所采用,其从单机走向集群已成必然,而云计算的蓬勃发展正在加速这一进程。Kubernetes 作为当前唯一被业界广泛认可和看好的 Docker 分布式系统解决方案。可以预见,在未来几年内,会有大量的新系统选择它,不管是运行在企业本地服务器上还是被托管到公有云上。
3.使用K8S有哪些好处?
使用Kubernetes就是在全面部署微服务架构。微服务架构的核心就是将一个巨大的单体应用分解为很多小的互相连接的微服务,一个微服务背后可能有多个实例副本在支撑,副本的数量可能会随着系统的负荷变化而进行调整,内嵌的负载均衡器在 k8s 平台中有多个实例副本在支撑,副本的数量可能会随着系统的负荷变化而进行调整,内嵌的负载均衡器 在k8s 平台中发挥了重要的作用。微服务架构使得每个服务都可以由专门的开发团队来开发,开发者可以自由选择开发技术,这对于大规模团队来说很有价值。另外,每个微服务独立开发、升级、扩展,使得系统具备很高的稳定性和快速迭代进化能力。
4.环境构成
整套环境的搭建包含:Docker环境的搭建、docker-compose环境的搭建、K8S集群的搭建、GitLab代码仓库的搭建、SVN仓库的搭建、Jenkins自动化部署环境的搭建、Harbor私有仓库的搭建。
本文档中,整套环境的搭建包括:
- 安装Docker环境
- 安装docker-compose
- 安装K8S集群环境
- 重启K8S集群引起的问题
- K8S安装ingress-nginx
- K8S安装gitlab代码仓库
- 安装Harbor私有仓库
- 安装Jenkins
- 物理机安装SVN(推荐)
- 物理机安装Jenkins(推荐)
- 配置Jenkins运行环境
- Jenkins发布Docker项目到K8S
服务器规划
IP | 主机名 | 节点 | 操作系统 |
---|---|---|---|
192.168.0.10 | test10 | K8S Master | CentOS 8.0.1905 |
192.168.0.11 | test11 | K8S Worker | CentOS 8.0.1905 |
192.168.0.12 | test12 | K8S Worker | CentOS 8.0.1905 |
安装环境版本
软件名称 | 软件版本 | 说明 |
---|---|---|
Docker | 19.03.8 | 提供容器环境 |
docker-compose | 1.25.5 | 定义和运行由多个容器组成的应用 |
K8S | 1.18.2 | 是一个开源的,用于管理云平台中多个主机上的容器化的应用,Kubernetes的目标是让部署容器化的应用简单并且高效(powerful),Kubernetes提供了应用部署,规划,更新,维护的一种机制。 |
GitLab | 12.1.6 | 代码仓库 |
Harbor | 1.10.2 | 私有镜像仓库 |
Jenkins | 2.222.3 | 持续集成交付 |
安装Docker环境
Docker 是一个开源的应用容器引擎,基于 Go 语言 并遵从 Apache2.0 协议开源。
Docker 可以让开发者打包他们的应用以及依赖包到一个轻量级、可移植的容器中,然后发布到任何流行的 Linux 机器上,也可以实现虚拟化。
本文档基于Docker 19.03.8 版本搭建Docker环境。
在所有服务器上创建install_docker.sh脚本,脚本内容如下所示。
#使用阿里云镜像中心
export REGISTRY_MIRROR=https://registry.cn-hangzhou.aliyuncs.com
#安装yum工具
dnf install yum*
#安装docker环境
yum install -y yum-utils device-mapper-persistent-data lvm2
#配置Docker的yum源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
#安装容器插件
dnf install https://mirrors.aliyun.com/docker-ce/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.13-3.1.el7.x86_64.rpm
#指定安装docker 19.03.8版本
yum install -y docker-ce-19.03.8 docker-ce-cli-19.03.8
#设置Docker开机启动
systemctl enable docker.service
#启动Docker
systemctl start docker.service
#查看Docker版本
docker version
在每台服务器上为install_docker.sh脚本赋予可执行权限,并执行脚本,如下所示。
# 赋予install_docker.sh脚本可执行权限
chmod a+x ./install_docker.sh
# 执行install_docker.sh脚本
./install_docker.sh
安装docker-compose
Compose 是用于定义和运行多容器 Docker 应用程序的工具。通过 Compose,您可以使用 YML 文件来配置应用程序需要的所有服务。然后,使用一个命令,就可以从 YML 文件配置中创建并启动所有服务。
注意:在每台服务器上安装docker-compose
1.下载docker-compose文件
#下载并安装docker-compose
curl -L https://github.com/docker/compose/releases/download/1.25.5/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
2.为docker-compose文件赋予可执行权限
#赋予docker-compose可执行权限
chmod a+x /usr/local/bin/docker-compose
3.查看docker-compose版本
#查看docker-compose版本
[root@binghe ~]# docker-compose version
docker-compose version 1.25.5, build 8a1c60f6
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
安装K8S集群环境
Kubernetes是一个开源的,用于管理云平台中多个主机上的容器化的应用,Kubernetes的目标是让部署容器化的应用简单并且高效(powerful),Kubernetes提供了应用部署,规划,更新,维护的一种机制。
本文档基于K8S 1.8.12版本来搭建K8S集群
安装K8S基础环境
在所有服务器上创建install_k8s.sh脚本文件,脚本文件的内容如下所示。
#################配置阿里云镜像加速器开始########################
mkdir -p /etc/docker
tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://zz3sblpi.mirror.aliyuncs.com"]
}
EOF
systemctl daemon-reload
systemctl restart docker
######################配置阿里云镜像加速器结束#########################
#安装nfs-utils
yum install -y nfs-utils
#安装wget软件下载命令
yum install -y wget
#启动nfs-server
systemctl start nfs-server
#配置nfs-server开机自启动
systemctl enable nfs-server
#关闭防火墙
systemctl stop firewalld
#取消防火墙开机自启动
systemctl disable firewalld
#关闭SeLinux
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
# 关闭 swap
swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab
############################修改 /etc/sysctl.conf开始###########################
# 如果有配置,则修改
sed -i "s#^net.ipv4.ip_forward.*#net.ipv4.ip_forward=1#g" /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-ip6tables.*#net.bridge.bridge-nf-call-ip6tables=1#g" /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-iptables.*#net.bridge.bridge-nf-call-iptables=1#g" /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.disable_ipv6.*#net.ipv6.conf.all.disable_ipv6=1#g" /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.default.disable_ipv6.*#net.ipv6.conf.default.disable_ipv6=1#g" /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.lo.disable_ipv6.*#net.ipv6.conf.lo.disable_ipv6=1#g" /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.forwarding.*#net.ipv6.conf.all.forwarding=1#g" /etc/sysctl.conf
# 可能没有,追加
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.lo.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.forwarding = 1" >> /etc/sysctl.conf
############################修改 /etc/sysctl.conf结束###########################
# 执行命令使修改后的/etc/sysctl.conf文件生效
sysctl -p
################# 配置K8S的yum源开始#############################
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF ################# 配置K8S的yum源结束############################# # 卸载旧版本K8S yum remove -y kubelet kubeadm kubectl # 安装kubelet、kubeadm、kubectl,这里我安装的是1.18.2版本,你也可以安装1.17.2版本 yum install -y kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2 # 修改docker Cgroup Driver为systemd # # 将/usr/lib/systemd/system/docker.service文件中的这一行 ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock # # 修改为 ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock –exec-opt native.cgroupdriver=systemd # 如果不修改,在添加 worker 节点时可能会碰到如下错误 # [WARNING IsDockerSystemdCheck]: detected “cgroupfs” as the Docker cgroup driver. The recommended driver is “systemd”. # Please follow the guide at https://kubernetes.io/docs/setup/cri/ sed -i “s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock –exec-opt native.cgroupdriver=systemd#g” /usr/lib/systemd/system/docker.service # 设置 docker 镜像,提高 docker 镜像下载速度和稳定性 # 如果访问 https://hub.docker.io 速度非常稳定,也可以跳过这个步骤,一般不需配置 # curl -sSL https://kuboard.cn/install-script/set_mirror.sh | sh -s ${REGISTRY_MIRROR} # 重新加载配置文件 systemctl daemon-reload #重启 docker systemctl restart docker # 将kubelet设置为开机启动并启动kubelet systemctl enable kubelet && systemctl start kubelet # 查看docker版本 docker version
在每台服务器上为install_k8s.sh脚本赋予可执行权限,并执行脚本
# 赋予install_k8s.sh脚本可执行权限
chmod a+x ./install_k8s.sh
# 运行install_k8s.sh脚本
./install_k8s.sh
初始化Master节点
只在test10服务器上执行的操作。
1.初始化Master节点的网络环境
注意:下面的命令需要在命令行手动执行。
# 只在 master 节点执行
# export 命令只在当前 shell 会话中有效,开启新的 shell 窗口后,如果要继续安装过程,请重新执行此处的 export 命令
export MASTER_IP=192.168.0.10
# 替换 k8s.master 为 您想要的 dnsName
export APISERVER_NAME=k8s.master
# Kubernetes 容器组所在的网段,该网段安装完成后,由 kubernetes 创建,事先并不存在于物理网络中
export POD_SUBNET=172.18.0.1/16
echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts
2.初始化Master节点
在test10服务器上创建init_master.sh脚本文件,文件内容如下所示。
#!/bin/bash
# 脚本出错时终止执行
set -e
if [ ${#POD_SUBNET} -eq 0 ] || [ ${#APISERVER_NAME} -eq 0 ]; then
echo -e "\033[31;1m请确保您已经设置了环境变量 POD_SUBNET 和 APISERVER_NAME \033[0m"
echo 当前POD_SUBNET=$POD_SUBNET
echo 当前APISERVER_NAME=$APISERVER_NAME
exit 1
fi
# 查看完整配置选项 https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2
rm -f ./kubeadm-config.yaml
cat <<EOF > ./kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.18.2
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
controlPlaneEndpoint: "${APISERVER_NAME}:6443"
networking:
serviceSubnet: "10.96.0.0/16"
podSubnet: "${POD_SUBNET}"
dnsDomain: "cluster.local"
EOF
# kubeadm init
# 初始化kebeadm
kubeadm init --config=kubeadm-config.yaml --upload-certs
# 配置 kubectl
rm -rf /root/.kube/
mkdir /root/.kube/
cp -i /etc/kubernetes/admin.conf /root/.kube/config
# 安装 calico 网络插件
# 参考文档 https://docs.projectcalico.org/v3.13/getting-started/kubernetes/self-managed-onprem/onpremises
echo "安装calico-3.13.1"
rm -f calico-3.13.1.yaml
wget https://kuboard.cn/install-script/calico/calico-3.13.1.yaml
kubectl apply -f calico-3.13.1.yaml
赋予init_master.sh脚本文件可执行权限并执行脚本。
# 赋予init_master.sh文件可执行权限
chmod a+x ./init_master.sh
# 运行init_master.sh脚本
./init_master.sh
3.查看Master节点的初始化结果
(1)确保所有容器组处于Running状态
# 执行如下命令,等待 3-10 分钟,直到所有的容器组处于 Running 状态
watch kubectl get pod -n kube-system -o wide
具体执行如下所示。
[root@test10 ~]# watch kubectl get pod -n kube-system -o wide
Every 2.0s: kubectl get pod -n kube-system -o wide test10: Sun May 10 11:01:32 2020
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-5b8b769fcd-5dtlp 1/1 Running 0 118s 172.18.203.66 test10 <none> <none>
calico-node-fnv8g 1/1 Running 0 118s 192.168.0.10 test10 <none> <none>
coredns-546565776c-27t7h 1/1 Running 0 2m1s 172.18.203.67 test10 <none> <none>
coredns-546565776c-hjb8z 1/1 Running 0 2m1s 172.18.203.65 test10 <none> <none>
etcd-test10 1/1 Running 0 2m7s 192.168.0.10 test10 <none> <none>
kube-apiserver-test10 1/1 Running 0 2m7s 192.168.0.10 test10 <none> <none>
kube-controller-manager-test10 1/1 Running 0 2m7s 192.168.0.10 test10 <none> <none>
kube-proxy-dvgsr 1/1 Running 0 2m1s 192.168.0.10 test10 <none> <none>
kube-scheduler-test10 1/1 Running 0 2m7s 192.168.0.10 test10 <none> <none>
(2) 查看 Master 节点初始化结果
# 查看Master节点的初始化结果
kubectl get nodes -o wide
具体执行如下所示。
[root@test10 ~]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
test10 Ready master 3m28s v1.18.2 192.168.0.10 <none> CentOS Linux 8 (Core) 4.18.0-80.el8.x86_64 docker://19.3.8
初始化Worker节点
1.获取join命令参数
在Master节点(test10服务器)上执行如下命令获取join命令参数。
kubeadm token create --print-join-command
具体执行如下所示。
[root@test10 ~]# kubeadm token create --print-join-command
W0510 11:04:34.828126 56132 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join k8s.master:6443 --token 8nblts.62xytoqufwsqzko2 --discovery-token-ca-cert-hash sha256:1717cc3e34f6a56b642b5751796530e367aa73f4113d09994ac3455e33047c0d
其中,有如下一行输出。
kubeadm join k8s.master:6443 --token 8nblts.62xytoqufwsqzko2 --discovery-token-ca-cert-hash sha256:1717cc3e34f6a56b642b5751796530e367aa73f4113d09994ac3455e33047c0d
这行代码就是获取到的join命令。
注意:join命令中的token的有效时间为 2 个小时,2小时内,可以使用此 token 初始化任意数量的 worker 节点。
2.初始化Worker节点
针对所有的 worker 节点执行,在这里,就是在test11服务器和test12服务器上执行。
在命令分别手动执行如下命令。
# 只在 worker 节点执行
# 192.168.0.10 为 master 节点的内网 IP
export MASTER_IP=192.168.0.10
# 替换 k8s.master 为初始化 master 节点时所使用的 APISERVER_NAME
export APISERVER_NAME=k8s.master
echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts
# 替换为 master 节点上 kubeadm token create 命令输出的join
kubeadm join k8s.master:6443 --token 8nblts.62xytoqufwsqzko2 --discovery-token-ca-cert-hash sha256:1717cc3e34f6a56b642b5751796530e367aa73f4113d09994ac3455e33047c0d
具体执行如下所示。
[root@test11 ~]# export MASTER_IP=192.168.0.10
[root@test11 ~]# export APISERVER_NAME=k8s.master
[root@test11 ~]# echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts
[root@test11 ~]# kubeadm join k8s.master:6443 --token 8nblts.62xytoqufwsqzko2 --discovery-token-ca-cert-hash sha256:1717cc3e34f6a56b642b5751796530e367aa73f4113d09994ac3455e33047c0d
W0510 11:08:27.709263 42795 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING FileExisting-tc]: tc not found in system path
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
根据输出结果可以看出,Worker节点加入了K8S集群。
注意:kubeadm join…就是master 节点上 kubeadm token create 命令输出的join。
3.查看初始化结果
在Master节点(test10服务器)执行如下命令查看初始化结果。
kubectl get nodes -o wide
具体执行如下所示。
[root@test10 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
test10 Ready master 20m v1.18.2
test11 Ready <none> 2m46s v1.18.2
test12 Ready <none> 2m46s v1.18.2
注意:kubectl get nodes命令后面加上-o wide参数可以输出更多的信息。
重启K8S集群引起的问题
1.Worker节点故障不能启动
Master 节点的 IP 地址发生变化,导致 worker 节点不能启动。需要重新安装K8S集群,并确保所有节点都有固定的内网 IP 地址。
2.Pod崩溃或不能正常访问
重启服务器后使用如下命令查看Pod的运行状态。
#查看所有pod的运行情况
kubectl get pods --all-namespaces
发现很多 Pod 不在 Running 状态,此时,需要使用如下命令删除运行不正常的Pod。
kubectl delete pod <pod-name> -n <pod-namespece>
注意:如果Pod 是使用 Deployment、StatefulSet 等控制器创建的,K8S 将创建新的 Pod 作为替代,重新启动的 Pod 通常能够正常工作。
其中,pod-name表示运行在K8S中的pod的名称,pod-namespece表示命名空间。例如,需要删除pod名称为pod-test,命名空间为pod-test-namespace的pod,可以使用下面的命令。
kubectl delete pod pod-test -n pod-test-namespace
K8S安装ingress-nginx
作为反向代理将外部流量导入集群内部,将 Kubernetes 内部的 Service 暴露给外部,在 Ingress 对象中通过域名匹配 Service,这样就可以直接通过域名访问到集群内部的服务了。相对于 traefik 来说,nginx-ingress 性能更加优秀。
注意:在Master节点(test10服务器上执行)
1.创建ingress-nginx命名空间
创建ingress-nginx-namespace.yaml文件,主要的作用是创建ingress-nginx命名空间,文件内容如下所示。
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
name: ingress-nginx
执行如下命令创建ingress-nginx命名空间。
kubectl apply -f ingress-nginx-namespace.yaml
2.安装ingress controller
创建ingress-nginx-mandatory.yaml文件,主要的作用是安装ingress-nginx。文件内容如下所示。
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: default-http-backend
labels:
app.kubernetes.io/name: default-http-backend
app.kubernetes.io/part-of: ingress-nginx
namespace: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: default-http-backend
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: default-http-backend
app.kubernetes.io/part-of: ingress-nginx
spec:
terminationGracePeriodSeconds: 60
containers:
- name: default-http-backend
# Any image is permissible as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/defaultbackend-amd64:1.5
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: ingress-nginx
labels:
app.kubernetes.io/name: default-http-backend
app.kubernetes.io/part-of: ingress-nginx
spec:
ports:
- port: 80
targetPort: 8080
selector:
app.kubernetes.io/name: default-http-backend
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: registry.cn-qingdao.aliyuncs.com/kubernetes_xingej/nginx-ingress-controller:0.20.0
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
---
执行如下命令安装ingress controller。
kubectl apply -f ingress-nginx-mandatory.yaml
3.安装K8S SVC:ingress-nginx
主要是用来用于暴露pod:nginx-ingress-controller。
创建service-nodeport.yaml文件,文件内容如下所示。
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
nodePort: 30080
- name: https
port: 443
targetPort: 443
protocol: TCP
nodePort: 30443
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
执行如下命令安装。
kubectl apply -f service-nodeport.yaml
4.访问K8S SVC:ingress-nginx
查看ingress-nginx命名空间的部署情况,如下所示。
[root@test10 k8s]# kubectl get pod -n ingress-nginx
NAME READY STATUS RESTARTS AGE
default-http-backend-796ddcd9b-vfmgn 1/1 Running 1 10h
nginx-ingress-controller-58985cc996-87754 1/1 Running 2 10h
在命令行服务器命令行输入如下命令查看ingress-nginx的端口映射情况。
kubectl get svc -n ingress-nginx
具体如下所示。
[root@test10 k8s]# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default-http-backend ClusterIP 10.96.247.2 <none> 80/TCP 7m3s
ingress-nginx NodePort 10.96.40.6 <none> 80:30080/TCP,443:30443/TCP 4m35s
所以,可以通过Master节点(test10服务器)的IP地址和30080端口号来访问ingress-nginx,如下所示。
[root@test10 k8s]# curl 192.168.0.10:30080
default backend - 404
也可以在浏览器打开http://192.168.0.10:30080 来访问ingress-nginx,如下所示。
K8S安装gitlab代码仓库
GitLab是由GitLabInc.开发,使用MIT许可证的基于网络的Git仓库管理工具,且具有Wiki和issue跟踪功能。使用Git作为代码管理工具,并在此基础上搭建起来的web服务。
注意:在Master节点(test10服务器上执行)
1.创建k8s-ops命名空间
创建k8s-ops-namespace.yaml文件,主要作用是创建k8s-ops命名空间。文件内容如下所示。
apiVersion: v1
kind: Namespace
metadata:
name: k8s-ops
labels:
name: k8s-ops
执行如下命令创建命名空间。
kubectl apply -f k8s-ops-namespace.yaml
2.安装gitlab-redis
创建gitlab-redis.yaml文件,文件的内容如下所示。
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
namespace: k8s-ops
labels:
name: redis
spec:
selector:
matchLabels:
name: redis
template:
metadata:
name: redis
labels:
name: redis
spec:
containers:
- name: redis
image: sameersbn/redis
imagePullPolicy: IfNotPresent
ports:
- name: redis
containerPort: 6379
volumeMounts:
- mountPath: /var/lib/redis
name: data
livenessProbe:
exec:
command:
- redis-cli
- ping
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
exec:
command:
- redis-cli
- ping
initialDelaySeconds: 10
timeoutSeconds: 5
volumes:
- name: data
hostPath:
path: /data1/docker/xinsrv/redis
---
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: k8s-ops
labels:
name: redis
spec:
ports:
- name: redis
port: 6379
targetPort: redis
selector:
name: redis
首先,在命令行执行如下命令创建/data1/docker/xinsrv/redis目录。
mkdir -p /data1/docker/xinsrv/redis
执行如下命令安装gitlab-redis。
kubectl apply -f gitlab-redis.yaml
3.安装gitlab-postgresql
创建gitlab-postgresql.yaml,文件内容如下所示。
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgresql
namespace: k8s-ops
labels:
name: postgresql
spec:
selector:
matchLabels:
name: postgresql
template:
metadata:
name: postgresql
labels:
name: postgresql
spec:
containers:
- name: postgresql
image: sameersbn/postgresql
imagePullPolicy: IfNotPresent
env:
- name: DB_USER
value: gitlab
- name: DB_PASS
value: passw0rd
- name: DB_NAME
value: gitlab_production
- name: DB_EXTENSION
value: pg_trgm
ports:
- name: postgres
containerPort: 5432
volumeMounts:
- mountPath: /var/lib/postgresql
name: data
livenessProbe:
exec:
command:
- pg_isready
- -h
- localhost
- -U
- postgres
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
exec:
command:
- pg_isready
- -h
- localhost
- -U
- postgres
initialDelaySeconds: 5
timeoutSeconds: 1
volumes:
- name: data
hostPath:
path: /data1/docker/xinsrv/postgresql
---
apiVersion: v1
kind: Service
metadata:
name: postgresql
namespace: k8s-ops
labels:
name: postgresql
spec:
ports:
- name: postgres
port: 5432
targetPort: postgres
selector:
name: postgresql
首先,执行如下命令创建/data1/docker/xinsrv/postgresql目录。
mkdir -p /data1/docker/xinsrv/postgresql
接下来,安装gitlab-postgresql,如下所示。
kubectl apply -f gitlab-postgresql.yaml
4.安装gitlab
(1)配置用户名和密码
首先,在命令行使用base64编码为用户名和密码进行转码,本示例中,使用的用户名为admin,密码为admin.1231
转码情况如下所示。
[root@test10 k8s]# echo -n 'admin' | base64
YWRtaW4=
[root@test10 k8s]# echo -n 'admin.1231' | base64
YWRtaW4uMTIzMQ==
转码后的用户名为:YWRtaW4= 密码为:YWRtaW4uMTIzMQ==
也可以对base64编码后的字符串解码,例如,对密码字符串解码,如下所示。
[root@test10 k8s]# echo 'YWRtaW4uMTIzMQ==' | base64 --decode
admin.1231
接下来,创建secret-gitlab.yaml文件,主要是用户来配置GitLab的用户名和密码,文件内容如下所示。
apiVersion: v1
kind: Secret
metadata:
namespace: k8s-ops
name: git-user-pass
type: Opaque
data:
username: YWRtaW4=
password: YWRtaW4uMTIzMQ==
执行配置文件的内容,如下所示。
kubectl create -f ./secret-gitlab.yaml
(2)安装GitLab
创建gitlab.yaml文件,文件的内容如下所示。
apiVersion: apps/v1
kind: Deployment
metadata:
name: gitlab
namespace: k8s-ops
labels:
name: gitlab
spec:
selector:
matchLabels:
name: gitlab
template:
metadata:
name: gitlab
labels:
name: gitlab
spec:
containers:
- name: gitlab
image: sameersbn/gitlab:12.1.6
imagePullPolicy: IfNotPresent
env:
- name: TZ
value: Asia/Shanghai
- name: GITLAB_TIMEZONE
value: Beijing
- name: GITLAB_SECRETS_DB_KEY_BASE
value: long-and-random-alpha-numeric-string
- name: GITLAB_SECRETS_SECRET_KEY_BASE
value: long-and-random-alpha-numeric-string
- name: GITLAB_SECRETS_OTP_KEY_BASE
value: long-and-random-alpha-numeric-string
- name: GITLAB_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: git-user-pass
key: password
- name: GITLAB_ROOT_EMAIL
value: 12345678@qq.com
- name: GITLAB_HOST
value: gitlab.binghe.com
- name: GITLAB_PORT
value: "80"
- name: GITLAB_SSH_PORT
value: "30022"
- name: GITLAB_NOTIFY_ON_BROKEN_BUILDS
value: "true"
- name: GITLAB_NOTIFY_PUSHER
value: "false"
- name: GITLAB_BACKUP_SCHEDULE
value: daily
- name: GITLAB_BACKUP_TIME
value: 01:00
- name: DB_TYPE
value: postgres
- name: DB_HOST
value: postgresql
- name: DB_PORT
value: "5432"
- name: DB_USER
value: gitlab
- name: DB_PASS
value: passw0rd
- name: DB_NAME
value: gitlab_production
- name: REDIS_HOST
value: redis
- name: REDIS_PORT
value: "6379"
ports:
- name: http
containerPort: 80
- name: ssh
containerPort: 22
volumeMounts:
- mountPath: /home/git/data
name: data
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 180
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 5
timeoutSeconds: 1
volumes:
- name: data
hostPath:
path: /data1/docker/xinsrv/gitlab
---
apiVersion: v1
kind: Service
metadata:
name: gitlab
namespace: k8s-ops
labels:
name: gitlab
spec:
ports:
- name: http
port: 80
nodePort: 30088
- name: ssh
port: 22
targetPort: ssh
nodePort: 30022
type: NodePort
selector:
name: gitlab
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gitlab
namespace: k8s-ops
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: gitlab.binghe.com
http:
paths:
- backend:
serviceName: gitlab
servicePort: http
注意:在配置GitLab时,监听主机时,不能使用IP地址,需要使用主机名或者域名,上述配置中,我使用的是gitlab.binghe.com主机名。
在命令行执行如下命令创建/data1/docker/xinsrv/gitlab目录。
mkdir -p /data1/docker/xinsrv/gitlab
安装GitLab,如下所示。
kubectl apply -f gitlab.yaml
5.安装完成
查看k8s-ops命名空间部署情况,如下所示。
[root@test10 k8s]# kubectl get pod -n k8s-ops
NAME READY STATUS RESTARTS AGE
gitlab-7b459db47c-5vk6t 0/1 Running 0 11s
postgresql-79567459d7-x52vx 1/1 Running 0 30m
redis-67f4cdc96c-h5ckz 1/1 Running 1 10h
也可以使用如下命令查看。
[root@test10 k8s]# kubectl get pod --namespace=k8s-ops
NAME READY STATUS RESTARTS AGE
gitlab-7b459db47c-5vk6t 0/1 Running 0 36s
postgresql-79567459d7-x52vx 1/1 Running 0 30m
redis-67f4cdc96c-h5ckz 1/1 Running 1 10h
二者效果一样。
接下来,查看GitLab的端口映射,如下所示。
[root@test10 k8s]# kubectl get svc -n k8s-ops
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
gitlab NodePort 10.96.153.100 <none> 80:30088/TCP,22:30022/TCP 2m42s
postgresql ClusterIP 10.96.203.119 <none> 5432/TCP 32m
redis ClusterIP 10.96.107.150 <none> 6379/TCP 10h
此时,可以看到,可以通过Master节点(test10)的主机名gitlab.binghe.com和端口30088就能够访问GitLab。由于我这里使用的是虚拟机来搭建相关的环境,在本机访问虚拟机映射的gitlab.binghe.com时,需要配置本机的hosts文件,在本机的hosts文件中加入如下配置项。
192.168.0.10 gitlab.binghe.com
注意:在Windows操作系统中,hosts文件所在的目录如下。
C:\Windows\System32\drivers\etc
接下来,就可以在浏览器中通过链接:http://gitlab.binghe.com:30088 来访问GitLab了,如下所示。
此时,可以通过用户名root和密码admin.1231来登录GitLab了。
注意:这里的用户名是root而不是admin,因为root是GitLab默认的超级用户。
到此,K8S安装gitlab完成。
安装Harbor私有仓库
Habor是由VMWare公司开源的容器镜像仓库。事实上,Habor是在Docker Registry上进行了相应的企业级扩展,从而获得了更加广泛的应用,这些新的企业级特性包括:管理用户界面,基于角色的访问控制 ,AD/LDAP集成以及审计日志等,足以满足基本企业需求。
注意:这里将Harbor私有仓库安装在Master节点(test10服务器)上,实际生产环境中建议安装在其他服务器。
1.下载Harbor的离线安装版本
wget https://github.com/goharbor/harbor/releases/download/v1.10.2/harbor-offline-installer-v1.10.2.tgz
2.解压Harbor的安装包
tar -zxvf harbor-offline-installer-v1.10.2.tgz
解压成功后,会在服务器当前目录生成一个harbor目录。
3.配置Harbor
注意:这里,我将Harbor的端口修改成了1180,如果不修改Harbor的端口,默认的端口是80。
(1)修改harbor.yml文件
cd harbor
vim harbor.yml
修改的配置项如下所示。
hostname: 192.168.0.10
http:
port: 1180
harbor_admin_password: binghe123
###并把https注释掉,不然在安装的时候会报错:ERROR:root:Error: The protocol is https but attribute ssl_cert is not set
#https:
#port: 443
#certificate: /your/certificate/path
#private_key: /your/private/key/path
(2)修改daemon.json文件
修改/etc/docker/daemon.json文件,没有的话就创建,在/etc/docker/daemon.json文件中添加如下内容。
[root@binghe~]# cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://zz3sblpi.mirror.aliyuncs.com"],
"insecure-registries":["192.168.0.10:1180"]
}
也可以在服务器上使用 ip addr 命令查看本机所有的IP地址段,将其配置到/etc/docker/daemon.json文件中。这里,我配置后的文件内容如下所示。
{
"registry-mirrors": ["https://zz3sblpi.mirror.aliyuncs.com"],
"insecure-registries":["192.168.175.0/16","172.17.0.0/16", "172.18.0.0/16", "172.16.29.0/16", "192.168.0.10:1180"]
}
4.安装并启动harbor
配置完成后,输入如下命令即可安装并启动Harbor
[root@binghe harbor]# ./install.sh
5.登录Harbor并添加账户
安装成功后,在浏览器地址栏输入http://192.168.0.10:1180打开链接,输入用户名admin和密码binghe123,登录系统。
接下来,我们选择用户管理,添加一个管理员账户,为后续打包Docker镜像和上传Docker镜像做准备。
密码为Binghe123。点击确,此时,账户binghe还不是管理员,此时选中binghe账户,点击“设置为管理员”。
此时,binghe账户就被设置为管理员了。到此,Harbor的安装就完成了。
6.修改Harbor端口
如果安装Harbor后,大家需要修改Harbor的端口,可以按照如下步骤修改Harbor的端口,这里,我以将80端口修改为1180端口为例
(1)修改harbor.yml文件
cd harbor
vim harbor.yml
修改的配置项如下所示。
hostname: 192.168.0.10
http:
port: 1180
harbor_admin_password: binghe123
###并把https注释掉,不然在安装的时候会报错:ERROR:root:Error: The protocol is https but attribute ssl_cert is not set
#https:
#port: 443
#certificate: /your/certificate/path
#private_key: /your/private/key/path
(2)修改docker-compose.yml文件
vim docker-compose.yml
修改的配置项如下所示。
ports:
- 1180:80
(3)修改config.yml文件
cd common/config/registry
vim config.yml
修改的配置项如下所示。
realm: http://192.168.0.10:1180/service/token
(4)重启Docker
systemctl daemon-reload
systemctl restart docker.service
(5)重启Harbor
[root@binghe harbor]# docker-compose down
Stopping harbor-log ... done
Removing nginx ... done
Removing harbor-portal ... done
Removing harbor-jobservice ... done
Removing harbor-core ... done
Removing redis ... done
Removing registry ... done
Removing registryctl ... done
Removing harbor-db ... done
Removing harbor-log ... done
Removing network harbor_harbor
[root@binghe harbor]# ./prepare
prepare base dir is set to /mnt/harbor
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/registry/root.crt
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
[root@binghe harbor]# docker-compose up -d
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-db ... done
Creating redis ... done
Creating registry ... done
Creating registryctl ... done
Creating harbor-core ... done
Creating harbor-jobservice ... done
Creating harbor-portal ... done
Creating nginx ... done
[root@binghe harbor]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
安装Jenkins(一般的做法)
Jenkins是一个开源的、提供友好操作界面的持续集成(CI)工具,起源于Hudson(Hudson是商用的),主要用于持续、自动的构建/测试软件项目、监控外部任务的运行(这个比较抽象,暂且写上,不做解释)。Jenkins用Java语言编写,可在Tomcat等流行的servlet容器中运行,也可独立运行。通常与版本管理工具(SCM)、构建工具结合使用。常用的版本控制工具有SVN、GIT,构建工具有Maven、Ant、Gradle。
1.安装nfs(之前安装过的话,可以省略此步)
使用 nfs 最大的问题就是写权限,可以使用 kubernetes 的 securityContext/runAsUser 指定 jenkins 容器中运行 jenkins 的用户 uid,以此来指定 nfs 目录的权限,让 jenkins 容器可写;也可以不限制,让所有用户都可以写。这里为了简单,就让所有用户可写了。
如果之前已经安装过nfs,则这一步可以省略。找一台主机,安装 nfs,这里,我以在Master节点(test10服务器)上安装nfs为例。
在命令行输入如下命令安装并启动nfs。
yum install nfs-utils -y
systemctl start nfs-server
systemctl enable nfs-server
2.创建nfs共享目录
在Master节点(test10服务器)上创建 /opt/nfs/jenkins-data
目录作为nfs的共享目录,如下所示。
mkdir -p /opt/nfs/jenkins-data
接下来,编辑/etc/exports文件,如下所示。
vim /etc/exports
在/etc/exports文件文件中添加如下一行配置。
/opt/nfs/jenkins-data 192.168.175.0/24(rw,all_squash)
这里的 ip 使用 kubernetes node 节点的 ip 范围,后面的 all_squash
选项会将所有访问的用户都映射成 nfsnobody 用户,不管你是什么用户访问,最终都会压缩成 nfsnobody,所以只要将 /opt/nfs/jenkins-data
的属主改为 nfsnobody,那么无论什么用户来访问都具有写权限。
这个选项在很多机器上由于用户 uid 不规范导致启动进程的用户不同,但是同时要对一个共享目录具有写权限时很有效。
接下来,为 /opt/nfs/jenkins-data
目录授权,并重新加载nfs,如下所示。
#为/opt/nfs/jenkins-data/目录授权
chown -R 1000 /opt/nfs/jenkins-data/
#重新加载nfs-server
systemctl reload nfs-server
在K8S集群中任意一个节点上使用如下命令进行验证:
#查看nfs系统的目录权限
showmount -e NFS_IP
如果能够看到 /opt/nfs/jenkins-data 就表示 ok 了。
具体如下所示。
[root@test10 ~]# showmount -e 192.168.0.10
Export list for 192.168.0.10:
/opt/nfs/jenkins-data 192.168.175.0/24
[root@test11 ~]# showmount -e 192.168.0.10
Export list for 192.168.0.10:
/opt/nfs/jenkins-data 192.168.175.0/24
3.创建PV
Jenkins 其实只要加载对应的目录就可以读取之前的数据,但是由于 deployment 无法定义存储卷,因此我们只能使用 StatefulSet。
首先创建 pv,pv 是给 StatefulSet 使用的,每次 StatefulSet 启动都会通过 volumeClaimTemplates 这个模板去创建 pvc,因此必须得有 pv,才能供 pvc 绑定。
创建jenkins-pv.yaml文件,文件内容如下所示。
apiVersion: v1
kind: PersistentVolume
metadata:
name: jenkins
spec:
nfs:
path: /opt/nfs/jenkins-data
server: 192.168.0.10
accessModes: ["ReadWriteOnce"]
capacity:
storage: 1Ti
我这里给了 1T存储空间,可以根据实际配置。
执行如下命令创建pv。
kubectl apply -f jenkins-pv.yaml
4.创建serviceAccount
创建service account,因为 jenkins 后面需要能够动态创建 slave,因此它必须具备一些权限。
创建jenkins-service-account.yaml文件,文件内容如下所示。
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: jenkins
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: jenkins
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins
上述配置中,创建了一个 RoleBinding 和一个 ServiceAccount,并且将 RoleBinding 的权限绑定到这个用户上。所以,jenkins 容器必须使用这个 ServiceAccount 运行才行,不然 RoleBinding 的权限它将不具备。
RoleBinding 的权限很容易就看懂了,因为 jenkins 需要创建和删除 slave,所以才需要上面这些权限。至于 secrets 权限,则是 https 证书。
执行如下命令创建serviceAccount。
kubectl apply -f jenkins-service-account.yaml
5.安装Jenkins
创建jenkins-statefulset.yaml文件,文件内容如下所示。
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: jenkins
labels:
name: jenkins
spec:
selector:
matchLabels:
name: jenkins
serviceName: jenkins
replicas: 1
updateStrategy:
type: RollingUpdate
template:
metadata:
name: jenkins
labels:
name: jenkins
spec:
terminationGracePeriodSeconds: 10
serviceAccountName: jenkins
containers:
- name: jenkins
image: docker.io/jenkins/jenkins:lts
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
- containerPort: 32100
resources:
limits:
cpu: 4
memory: 4Gi
requests:
cpu: 4
memory: 4Gi
env:
- name: LIMITS_MEMORY
valueFrom:
resourceFieldRef:
resource: limits.memory
divisor: 1Mi
- name: JAVA_OPTS
# value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home
livenessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12 # ~2 minutes
readinessProbe:
httpGet:
path: /login
port: 8080
initialDelaySeconds: 60
timeoutSeconds: 5
failureThreshold: 12 # ~2 minutes
# pvc 模板,对应之前的 pv
volumeClaimTemplates:
- metadata:
name: jenkins-home
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Ti
jenkins 部署时需要注意它的副本数,你的副本数有多少就要有多少个 pv,同样,存储会有多倍消耗。这里我只使用了一个副本,因此前面也只创建了一个 pv。
使用如下命令安装Jenkins。
kubectl apply -f jenkins-statefulset.yaml
6.创建Service
创建jenkins-service.yaml文件,主要用于后台运行Jenkins,文件内容如下所示。
apiVersion: v1
kind: Service
metadata:
name: jenkins
spec:
# type: LoadBalancer
selector:
name: jenkins
# ensure the client ip is propagated to avoid the invalid crumb issue when using LoadBalancer (k8s >=1.7)
#externalTrafficPolicy: Local
ports:
- name: http
port: 80
nodePort: 31888
targetPort: 8080
protocol: TCP
- name: jenkins-agent
port: 32100
nodePort: 32100
targetPort: 32100
protocol: TCP
type: NodePort
使用如下命令安装Service。
kubectl apply -f jenkins-service.yaml
7.安装 ingress
jenkins 的 web 界面需要从集群外访问,这里我们选择的是使用 ingress。创建jenkins-ingress.yaml文件,文件内容如下所示。
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: jenkins
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: jenkins
servicePort: 31888
host: jekins.binghe.com
这里,需要注意的是host必须配置为域名或者主机名,否则会报错,如下所示。
The Ingress "jenkins" is invalid: spec.rules[0].host: Invalid value: "192.168.0.10": must be a DNS name, not an IP address
使用如下命令安装ingress。
kubectl apply -f jenkins-ingress.yaml
最后,由于我这里使用的是虚拟机来搭建相关的环境,在本机访问虚拟机映射的jekins.binghe.com时,需要配置本机的hosts文件,在本机的hosts文件中加入如下配置项。
192.168.0.10 jekins.binghe.com
注意:在Windows操作系统中,hosts文件所在的目录如下。
C:\Windows\System32\drivers\etc
接下来,就可以在浏览器中通过链接:http://jekins.binghe.com:31888 来访问Jekins了。
物理机安装SVN
Apache Subversion 通常被缩写成 SVN,是一个开放源代码的版本控制系统,Subversion 在 2000 年由 CollabNet Inc 开发,现在发展成为 Apache 软件基金会的一个项目,同样是一个丰富的开发者和用户社区的一部分。
SVN相对于的RCS、CVS,采用了分支管理系统,它的设计目标就是取代CVS。互联网上免费的版本控制服务多基于Subversion。
这里,以在Master节点(binghe101服务器)上安装SVN为例。
1.使用yum安装SVN
在命令行执行如下命令安装SVN。
yum -y install subversion
2.创建SVN库
依次执行如下命令。
#创建/data/svn
mkdir -p /data/svn
#初始化svn
svnserve -d -r /data/svn
#创建代码仓库
svnadmin create /data/svn/test
3.配置SVN
mkdir /data/svn/conf
cp /data/svn/test/conf/* /data/svn/conf/
cd /data/svn/conf/
[root@binghe101 conf]# ll
总用量 20
-rw-r--r-- 1 root root 1080 5月 12 02:17 authz
-rw-r--r-- 1 root root 885 5月 12 02:17 hooks-env.tmpl
-rw-r--r-- 1 root root 309 5月 12 02:17 passwd
-rw-r--r-- 1 root root 4375 5月 12 02:17 svnserve.conf
- 配置authz文件,
vim authz
配置后的内容如下所示。
[aliases]
# joe = /C=XZ/ST=Dessert/L=Snake City/O=Snake Oil, Ltd./OU=Research Institute/CN=Joe Average
[groups]
# harry_and_sally = harry,sally
# harry_sally_and_joe = harry,sally,&joe
SuperAdmin = admin
binghe = admin,binghe
# [/foo/bar]
# harry = rw
# &joe = r
# * =
# [repository:/baz/fuz]
# @harry_and_sally = rw
# * = r
[test:/]
@SuperAdmin=rw
@binghe=rw
- 配置passwd文件
vim passwd
配置后的内容如下所示。
[users]
# harry = harryssecret
# sally = sallyssecret
admin = admin123
binghe = binghe123
- 配置 svnserve.conf
vim svnserve.conf
配置后的文件如下所示。
### This file controls the configuration of the svnserve daemon, if you
### use it to allow access to this repository. (If you only allow
### access through http: and/or file: URLs, then this file is
### irrelevant.)
### Visit http://subversion.apache.org/ for more information.
[general]
### The anon-access and auth-access options control access to the ### repository for unauthenticated (a.k.a. anonymous) users and ### authenticated users, respectively. ### Valid values are “write”, “read”, and “none”. ### Setting the value to “none” prohibits both reading and writing; ### “read” allows read-only access, and “write” allows complete ### read/write access to the repository. ### The sample settings below are the defaults and specify that anonymous ### users have read-only access to the repository, while authenticated ### users have read and write access to the repository. anon-access = none auth-access = write ### The password-db option controls the location of the password ### database file. Unless you specify a path starting with a /, ### the file’s location is relative to the directory containing ### this configuration file. ### If SASL is enabled (see below), this file will NOT be used. ### Uncomment the line below to use the default password file. password-db = /data/svn/conf/passwd ### The authz-db option controls the location of the authorization ### rules for path-based access control. Unless you specify a path ### starting with a /, the file’s location is relative to the ### directory containing this file. The specified path may be a ### repository relative URL (^/) or an absolute file:// URL to a text ### file in a Subversion repository. If you don’t specify an authz-db, ### no path-based access control is done. ### Uncomment the line below to use the default authorization file. authz-db = /data/svn/conf/authz ### The groups-db option controls the location of the file with the ### group definitions and allows maintaining groups separately from the ### authorization rules. The groups-db file is of the same format as the ### authz-db file and should contain a single [groups] section with the ### group definitions. If the option is enabled, the authz-db file cannot ### contain a [groups] section. Unless you specify a path starting with ### a /, the file’s location is relative to the directory containing this ### file. The specified path may be a repository relative URL (^/) or an ### absolute file:// URL to a text file in a Subversion repository. ### This option is not being used by default. # groups-db = groups ### This option specifies the authentication realm of the repository. ### If two repositories have the same authentication realm, they should ### have the same password database, and vice versa. The default realm ### is repository’s uuid. realm = svn ### The force-username-case option causes svnserve to case-normalize ### usernames before comparing them against the authorization rules in the ### authz-db file configured above. Valid values are “upper” (to upper- ### case the usernames), “lower” (to lowercase the usernames), and ### “none” (to compare usernames as-is without case conversion, which ### is the default behavior). # force-username-case = none ### The hooks-env options specifies a path to the hook script environment ### configuration file. This option overrides the per-repository default ### and can be used to configure the hook script environment for multiple ### repositories in a single file, if an absolute path is specified. ### Unless you specify an absolute path, the file’s location is relative ### to the directory containing this file. # hooks-env = hooks-env
[sasl]
### This option specifies whether you want to use the Cyrus SASL ### library for authentication. Default is false. ### Enabling this option requires svnserve to have been built with Cyrus ### SASL support; to check, run ‘svnserve –version’ and look for a line ### reading ‘Cyrus SASL authentication is available.’ # use-sasl = true ### These options specify the desired strength of the security layer ### that you want SASL to provide. 0 means no encryption, 1 means ### integrity-checking only, values larger than 1 are correlated ### to the effective key length for encryption (e.g. 128 means 128-bit ### encryption). The values below are the defaults. # min-encryption = 0 # max-encryption = 256
接下来,将/data/svn/conf目录下的svnserve.conf文件复制到/data/svn/test/conf/目录下。如下所示。
[root@binghe101 conf]# cp /data/svn/conf/svnserve.conf /data/svn/test/conf/
cp:是否覆盖'/data/svn/test/conf/svnserve.conf'? y
4.启动SVN服务
(1)创建svnserve.service服务
创建svnserve.service文件
vim /usr/lib/systemd/system/svnserve.service
文件的内容如下所示。
[Unit]
Description=Subversion protocol daemon
After=syslog.target network.target
Documentation=man:svnserve(8)
[Service]
Type=forking
EnvironmentFile=/etc/sysconfig/svnserve
#ExecStart=/usr/bin/svnserve --daemon --pid-file=/run/svnserve/svnserve.pid $OPTIONS
ExecStart=/usr/bin/svnserve --daemon $OPTIONS
PrivateTmp=yes
[Install]
WantedBy=multi-user.target
接下来执行如下命令使配置生效。
systemctl daemon-reload
命令执行成功后,修改 /etc/sysconfig/svnserve 文件。
vim /etc/sysconfig/svnserve
修改后的文件内容如下所示。
# OPTIONS is used to pass command-line arguments to svnserve.
#
# Specify the repository location in -r parameter:
OPTIONS="-r /data/svn"
(2)启动SVN
首先查看SVN状态,如下所示。
[root@test10 conf]# systemctl status svnserve.service
● svnserve.service - Subversion protocol daemon
Loaded: loaded (/usr/lib/systemd/system/svnserve.service; disabled; vendor preset: disabled)
Active: inactive (dead)
Docs: man:svnserve(8)
可以看到,此时SVN并没有启动,接下来,需要启动SVN。
systemctl start svnserve.service
设置SVN服务开机自启动。
systemctl enable svnserve.service
接下来,就可以下载安装TortoiseSVN,输入链接svn://192.168.0.10/test 并输入用户名binghe,密码binghe123来连接SVN了。
Docker安装SVN
拉取SVN镜像
docker pull docker.io/elleflorio/svn-server
启动SVN容器
docker run -v /usr/local/svn:/home/svn -v /usr/local/svn/passwd:/etc/subversion/passwd -v /usr/local/apache2:/run/apache2 --name svn_server -p 3380:80 -p 3690:3960 -e SVN_REPONAME=repos -d docker.io/elleflorio/svn-server
进入SVN容器内部
docker exec -it svn_server bash
进入容器后,可以参照物理机安装SVN的方式配置SVN仓库。
物理机安装Jenkins
注意:安装Jenkins之前需要安装JDK和Maven,我这里同样将Jenkins安装在Master节点(binghe101服务器)。
1.启用Jenkins库
运行以下命令以下载repo文件并导入GPG密钥:
wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
2.安装Jenkins
执行如下命令安装Jenkis。
yum install jenkins
接下来,修改Jenkins默认端口,如下所示。
vim /etc/sysconfig/jenkins
修改后的两项配置如下所示。
JENKINS_JAVA_CMD="/usr/local/jdk1.8.0_212/bin/java"
JENKINS_PORT="18080"
此时,已经将Jenkins的端口由8080修改为18080
3.启动Jenkins
在命令行输入如下命令启动Jenkins。
systemctl start jenkins
配置Jenkins开机自启动。
systemctl enable jenkins
查看Jenkins的运行状态。
[root@test10 ~]# systemctl status jenkins
● jenkins.service - LSB: Jenkins Automation Server
Loaded: loaded (/etc/rc.d/init.d/jenkins; generated)
Active: active (running) since Tue 2020-05-12 04:33:40 EDT; 28s ago
Docs: man:systemd-sysv-generator(8)
Tasks: 71 (limit: 26213)
Memory: 550.8M
说明,Jenkins启动成功。
配置Jenkins运行环境
1.登录Jenkins
首次安装后,需要配置Jenkins的运行环境。首先,在浏览器地址栏访问链接http://192.168.0.10:18080,打开Jenkins界面。
根据提示使用如下命令到服务器上找密码值,如下所示。
[root@binghe101 ~]# cat /var/lib/jenkins/secrets/initialAdminPassword
71af861c2ab948a1b6efc9f7dde90776
将密码71af861c2ab948a1b6efc9f7dde90776复制到文本框,点击继续。会跳转到自定义Jenkins页面,如下所示。
这里,可以直接选择“安装推荐的插件”。之后会跳转到一个安装插件的页面,如下所示。
此步骤可能有下载失败的情况,可直接忽略。
2.安装插件
需要安装的插件
- Kubernetes Cli Plugin:该插件可直接在Jenkins中使用kubernetes命令行进行操作。
- Kubernetes plugin: 使用kubernetes则需要安装该插件
- Kubernetes Continuous Deploy Plugin:kubernetes部署插件,可根据需要使用
还有更多的插件可供选择,可点击 系统管理->管理插件进行管理和添加,安装相应的Docker插件、SSH插件、Maven插件。其他的插件可以根据需要进行安装。如下图所示。
3.配置Jenkins
(1)配置JDK和Maven
在Global Tool Configuration中配置JDK和Maven,如下所示,打开Global Tool Configuration界面。
接下来就开始配置JDK和Maven了。
由于我在服务器上将Maven安装在/usr/local/maven-3.6.3目录下,所以,需要在“Maven 配置”中进行配置,如下图所示。
接下来,配置JDK,如下所示。
注意:不要勾选“Install automatically”
接下来,配置Maven,如下所示。
注意:不要勾选“Install automatically”
(2)配置SSH
进入Jenkins的Configure System界面配置SSH,如下所示。
找到 SSH remote hosts 进行配置。
配置完成后,点击Check connection按钮,会显示 Successfull connection。如下所示。
至此,Jenkins的基本配置就完成了。
Jenkins发布Docker项目到K8s集群
1.调整SpringBoot项目的配置
实现,SpringBoot项目中启动类所在的模块的pom.xml需要引入打包成Docker镜像的配置,如下所示。
<properties>
<docker.repostory>192.168.0.10:1180</docker.repostory>
<docker.registry.name>test</docker.registry.name>
<docker.image.tag>1.0.0</docker.image.tag>
<docker.maven.plugin.version>1.4.10</docker.maven.plugin.version>
</properties>
<build>
<finalName>test-starter</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<!-- docker的maven插件,官网:https://github.com/spotify/docker‐maven‐plugin -->
<!-- Dockerfile maven plugin -->
<plugin>
<groupId>com.spotify</groupId>
<artifactId>dockerfile-maven-plugin</artifactId>
<version>${docker.maven.plugin.version}</version>
<executions>
<execution>
<id>default</id>
<goals>
<!--如果package时不想用docker打包,就注释掉这个goal-->
<goal>build</goal>
<goal>push</goal>
</goals>
</execution>
</executions>
<configuration>
<contextDirectory>${project.basedir}</contextDirectory>
<!-- harbor 仓库用户名及密码-->
<useMavenSettingsForAuth>useMavenSettingsForAuth>true</useMavenSettingsForAuth>
<repository>${docker.repostory}/${docker.registry.name}/${project.artifactId}</repository>
<tag>${docker.image.tag}</tag>
<buildArgs>
<JAR_FILE>target/${project.build.finalName}.jar</JAR_FILE>
</buildArgs>
</configuration>
</plugin>
</plugins>
<resources>
<!-- 指定 src/main/resources下所有文件及文件夹为资源文件 -->
<resource>
<directory>src/main/resources</directory>
<targetPath>${project.build.directory}/classes</targetPath>
<includes>
<include>**/*</include>
</includes>
<filtering>true</filtering>
</resource>
</resources>
</build>
接下来,在SpringBoot启动类所在模块的根目录创建Dockerfile,内容示例如下所示。
#添加依赖环境,前提是将Java8的Docker镜像从官方镜像仓库pull下来,然后上传到自己的Harbor私有仓库中
FROM 192.168.0.10:1180/library/java:8
#指定镜像制作作者
MAINTAINER binghe
#运行目录
VOLUME /tmp
#将本地的文件拷贝到容器
ADD target/*jar app.jar
#启动容器后自动执行的命令
ENTRYPOINT [ "java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/app.jar" ]
根据实际情况,自行修改。
注意:FROM 192.168.0.10:1180/library/java:8的前提是执行如下命令。
docker pull java:8
docker tag java:8 192.168.0.10:1180/library/java:8
docker login 192.168.0.10:1180
docker push 192.168.0.10:1180/library/java:8
在SpringBoot启动类所在模块的根目录创建yaml文件,录入叫做test.yaml文件,内容如下所示。
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-starter
labels:
app: test-starter
spec:
replicas: 1
selector:
matchLabels:
app: test-starter
template:
metadata:
labels:
app: test-starter
spec:
containers:
- name: test-starter
image: 192.168.0.10:1180/test/test-starter:1.0.0
ports:
- containerPort: 8088
nodeSelector:
clustertype: node12
---
apiVersion: v1
kind: Service
metadata:
name: test-starter
labels:
app: test-starter
spec:
ports:
- name: http
port: 8088
nodePort: 30001
type: NodePort
selector:
app: test-starter
2.Jenkins配置发布项目
将项目上传到SVN代码库,例如地址为svn://192.168.0.10/test
接下来,在Jenkins中配置自动发布。步骤如下所示。
点击新建Item。
在描述文本框中输入描述信息,如下所示。
接下来,配置SVN信息。
注意:配置GitLab的步骤与SVN相同,不再赘述。
定位到Jenkins的“构建模块”,使用Execute Shell来构建发布项目到K8S集群。
执行的命令依次如下所示。
#删除本地原有的镜像,不会影响Harbor仓库中的镜像
docker rmi 192.168.0.10:1180/test/test-starter:1.0.0
#使用Maven编译、构建Docker镜像,执行完成后本地Docker容器中会重新构建镜像文件
/usr/local/maven-3.6.3/bin/mvn -f ./pom.xml clean install -Dmaven.test.skip=true
#登录 Harbor仓库
docker login 192.168.0.10:1180 -u binghe -p Binghe123
#上传镜像到Harbor仓库
docker push 192.168.0.10:1180/test/test-starter:1.0.0
#停止并删除K8S集群中运行的
/usr/bin/kubectl delete -f test.yaml
#将Docker镜像重新发布到K8S集群
/usr/bin/kubectl apply -f test.yaml
Fantastic read! I was especially impressed by the depth provided on the topic, offering a perspective I hadn’t considered. Your insight adds significant value to the conversation. For future articles, it would be fascinating to explore more to dive deeper into this subject. Could you also clarify more about the topic? It caught my interest, and I’d love to understand more about it. Keep up the excellent work!
Fantastic read! I was especially impressed by the depth provided on the topic, offering a perspective I hadn’t considered. Your insight adds significant value to the conversation. For future articles, it would be fascinating to explore more to dive deeper into this subject. Could you also clarify more about the topic? It caught my interest, and I’d love to understand more about it. Keep up the excellent work!
Fantastic read! I was especially impressed by the depth provided on the topic, offering a perspective I hadn’t considered. Your insight adds significant value to the conversation. For future articles, it would be fascinating to explore more to dive deeper into this subject. Could you also clarify more about the topic? It caught my interest, and I’d love to understand more about it. Keep up the excellent work!
Fantastic read! I was especially impressed by the depth provided on the topic, offering a perspective I hadn’t considered. Your insight adds significant value to the conversation. For future articles, it would be fascinating to explore more to dive deeper into this subject. Could you also clarify more about the topic? It caught my interest, and I’d love to understand more about it. Keep up the excellent work!
Positive feedback reminiscent of an engaging social media campaign – building a supportive community.
Dedication making a significant impact, much like a successful influencer marketing campaign.
Hi Neat post Theres an issue together with your web site in internet explorer may test this IE still is the marketplace chief and a good component of people will pass over your fantastic writing due to this problem
Dedication to improvement, much like an A/B test – refining and optimizing for success.
Attention to detail mirrors the best data-driven marketing approach out there.
You’ve really helped me understand the nuances of this topic. Thanks for your clear and detailed approach!
This is some of the best analysis I’ve seen on the topic—well done!
I’ve been following your blog for quite some time now, and I’m continually impressed by the quality of your content. Your ability to blend information with entertainment is truly commendable.
I do not even know how I ended up here but I thought this post was great I dont know who you are but definitely youre going to a famous blogger if you arent already Cheers
HmXeCMlrRFLGPp
Your writing is like a breath of fresh air in the often stale world of online content. Your unique perspective and engaging style set you apart from the crowd. Thank you for sharing your talents with us.
Wonderful beat I wish to apprentice while you amend your web site how could i subscribe for a blog web site The account aided me a acceptable deal I had been a little bit acquainted of this your broadcast provided bright clear idea
Your ability to distill complex concepts into digestible nuggets of wisdom is truly remarkable. I always come away from your blog feeling enlightened and inspired. Keep up the phenomenal work!
Fantastic beat I would like to apprentice while you amend your web site how could i subscribe for a blog site The account helped me a acceptable deal I had been a little bit acquainted of this your broadcast offered bright clear concept
Every time I visit your website, I’m greeted with thought-provoking content and impeccable writing. You truly have a gift for articulating complex ideas in a clear and engaging manner.
Acaptivatingexplorationof[topic].
Hey people!!!!!
Good mood and good luck to everyone!!!!!
Your blog is a constant source of inspiration for me. Your passion for your subject matter shines through in every post, and it’s clear that you genuinely care about making a positive impact on your readers.
At Flash Upload, we’ve designed a user-friendly interface that lets you easily upload your files, set up monetization in just a few clicks, and share your unique creations with the world. Each upload is an opportunity for you to earn money while sharing your talent and expertise.
GhDVcFmY
Your blog is a treasure trove of valuable insights and thought-provoking commentary. Your dedication to your craft is evident in every word you write. Keep up the fantastic work!
Your writing is like a breath of fresh air in the often stale world of online content. Your unique perspective and engaging style set you apart from the crowd. Thank you for sharing your talents with us.
Every time I visit your website, I’m greeted with thought-provoking content and impeccable writing. You truly have a gift for articulating complex ideas in a clear and engaging manner.
Your blog is like a beacon of light in the vast expanse of the internet. Your thoughtful analysis and insightful commentary never fail to leave a lasting impression. Thank you for all that you do.
Dogs Cane Corso in Moscow
cane-corso.moscow
Здравствуйте! Меня зовут Шестакова Татьяна Викторовна, я дерматолог с многолетним опытом работы в области эстетической медицины. Сегодня я отвечу на ваши запросы и поделюсь полезной информацией о уходе за кожей. Моя цель — помочь вам понять, как правильно ухаживать за кожей и какие процедуры могут помочь вам достичь желаемого результата.
Как долго длится процедура ботокса для губ?
Процедура ботокса для губ обычно занимает от 10 до 20 минут. Время проведения может варьироваться в зависимости от количества инъекций и сложности случая.
How long does the Botox procedure for lips take?
The Botox procedure for lips usually takes 10 to 20 minutes. The duration may vary depending on the number of injections and the complexity of the case.
Какой период восстановления после удаления папиллом лазером?
Период восстановления после удаления папиллом лазером обычно занимает от нескольких дней до недели. В это время рекомендуется избегать физических нагрузок, загара и использования косметических средств на обработанной области, а также следовать всем рекомендациям врача.
What is the recovery period after laser removal of papillomas?
The recovery period after laser removal of papillomas usually takes from a few days to a week. During this time, it is recommended to avoid physical exertion, tanning, and using cosmetic products on the treated area, as well as follow all the doctor’s recommendations.
Какие противопоказания существуют для лазерного удаления папиллом?
Противопоказания для лазерного удаления папиллом включают острые воспалительные процессы, инфекции кожи в зоне обработки, онкологические заболевания, тяжелые нарушения свертываемости крови и беременность. Врач обязательно проведет осмотр и, при необходимости, дополнительные исследования перед процедурой.
What are the contraindications for laser removal of papillomas?
Contraindications for laser removal of papillomas include acute inflammatory processes, skin infections in the treatment area, cancer, severe blood clotting disorders, and pregnancy. The doctor will conduct an examination and, if necessary, additional tests before the procedure.
Добрый день! Меня зовут Шестаков Юрий Иванович, я врач-косметолог с многолетним опытом работы в области эстетической медицины. Сейчас я отвечу на ваши запросы и расскажу полезной информацией о удалении папиллом с помощью лазера. Моя цель — помочь вам понять, как безопасно и эффективно избавиться от папиллом и какие преимущества имеет лазерное удаление.
Существуют ли побочные эффекты после лазерного удаления папиллом?
Побочные эффекты после лазерного удаления папиллом обычно минимальны и могут включать покраснение, отек, легкое кровотечение и образование корочки на месте удаления. В редких случаях могут возникнуть инфекционные осложнения или изменения пигментации кожи. Все побочные эффекты обычно проходят самостоятельно в течение нескольких дней.
Are there any side effects after laser removal of papillomas?
Side effects after laser removal of papillomas are usually minimal and may include redness, swelling, slight bleeding, and crust formation at the removal site. In rare cases, infectious complications or skin pigmentation changes may occur. All side effects typically resolve on their own within a few days.
Hello! I’m from Russia, i’m never kissed russian girl, for 32 years.
If you want stop war in Ukraine, block Russia users on all foreign adult web-sites. And stop host Russia adult projects.
All “adult” in Russia, it’s foreign 18+ sites and video.
Be careful, Russia girl it’s scam.
Что делать, если после удаления папилломы появилась корочка?
Появление корочки после лазерного удаления папилломы является нормальным этапом заживления. Важно не срывать корочку и избегать ее повреждения. Корочка обычно отпадает самостоятельно через несколько дней, оставляя здоровую кожу. Если появляются признаки инфекции или сильное покраснение, необходимо обратиться к врачу.
What to do if a scab appears after papilloma removal?
The appearance of a scab after laser removal of a papilloma is a normal part of the healing process. It is important not to pick at the scab and avoid damaging it. The scab usually falls off on its own within a few days.
Могут ли папилломы появиться снова после удаления лазером?
Хотя лазерное удаление эффективно устраняет существующие папилломы, вирус, вызывающий их, может оставаться в организме. Это означает, что существует вероятность повторного появления папиллом, особенно если не соблюдать профилактические меры, такие как укрепление иммунной системы и поддержание гигиены.
Can papillomas reappear after laser removal?
While laser removal effectively eliminates existing papillomas, the virus causing them can remain in the body. This means there is a possibility of papillomas reappearing, especially if preventive measures such as strengthening the immune system and maintaining hygiene are not followed.
When I encounter problems with a SoundCloud to OGG converter, I troubleshoot by clearing my
browser cache or trying a different tool. To improve your download speed, you should
close unnecessary applications and tabs while downloading songs from SoundCloud.
FtKQEeoDm
Какие меры предосторожности необходимо соблюдать после удаления папиллом лазером?
После удаления папиллом лазером рекомендуется избегать прямого солнечного света на обработанную область, не использовать косметические средства, содержащие спирт, и не трогать обработанную область руками. Следует также избегать посещения саун и бассейнов в течение нескольких дней после процедуры. Важно следовать всем рекомендациям врача для предотвращения осложнений и ускорения процесса заживления.
What precautions should be taken after laser removal of papillomas?
After laser removal of papillomas, it is recommended to avoid direct sunlight on the treated area, not to use cosmetic products containing alcohol, and not to touch the treated area with your hands. It is also advised to avoid visiting saunas and swimming pools for a few days after the procedure. Following all doctor’s recommendations is important to prevent complications and speed up the healing process.
Modern life and our eating practices can be demanding to our systems, and if we don’t act, these
stressors can take a toll on our bodies and minds. So whether it’s an overload of chocolate from the
Easter binge, your office chair makes your back beg for mercy, or another long day
with the one in charge makes your head throb, standard Thai massage might be simply what you need.
The advantages of Thai massage might be really 21st century however
they have their roots in ancient Eastern practices developed over
thousands of years. It’s all a long way from the experience you may have had
of requirement, Westernised massage. Performed on a special Thai mat in loose-fitting clothes, a therapist extends your body through a series of techniques comparable to yoga applies and postures pressure to particular points along the energy lines of your body.
Thai massage is excellent both physically and psychologically.
It can increase flexibility, enhance muscle tone, increase movement, release muscle stress, correct poor posture and fight stress.
How? Well, conventional specialists think that by
assisting energy to stream easily around the body once again they’re
able to ease the symptoms of persistent discomfort and improve psychological wellbeing.
A 2015 review of scientific trials found that
every released study reports benefits in post-treatment pain reduction. Some research study concludes that standard Thai massage results in a reduction in discomfort of as much as 80% with the benefits kept
for over 3 months at a time, consisting of enhancements in muscle stress, versatility and even anxiety.
Findings published in the International Journal of Neuroscience recommend that massage therapy has favorable impacts on the hormonal agents and chemicals that regulate state of minds by decreasing cortisol and increasing both serotonin and dopamine.
This effective triple action can help in reducing depression and promote
happiness. Obviously, beyond the dry science, Thai yoga massage also simply feels actually, really nice.
Guided by an experienced therapist, purposeful extending combined with well-established pressure methods helps soothes
the muscles and enhance blood flow, leaving you relaxed and calm, ready to handle anything.
Welcome to pain and stress management, Thai-style.
We welcome you to come in and experience the magic of a Thai yoga Massage
on your own. To reserve your Thai yoga massage at Castle Thai Spa,
click here.
Body-to-body massage, also known as B2B massage, is a sensual and intimate kind of massage that includes the use of
not simply hands, however the whole body to offer relaxation and enjoyment.
In this post, we will explore the different elements of body-to-body massage, including its origins, methods, benefits, and
security precautions. Whether you are a specialist or somebody thinking
about experiencing this unique kind of massage, this post will offer you with valuable insights and details.
Let’s begin by acquiring a much deeper understanding of body-to-body massage.
This ancient practice has its roots in numerous cultures
throughout history. Body-to-body massage, also referred to as B2B massage, is a kind of massage
that goes beyond the standard techniques. It is an intimate and sensual experience
that involves the close connection in between the provider and
receiver. Unlike other kinds of massage, B2B massage includes using numerous body parts, such as
the arms, chest, and legs, to provide a more immersive and satisfying experience.
Body-to-body massage has been practiced for centuries
in various parts of the world. Its origins can be traced back to ancient Asian cultures, such as India and China, where it was believed to enhance spiritual and physical wellness.
In these ancient customs, body-to-body massage was considered a holistic approach to healing
and relaxation. In India, body-to-body massage was called “abhyanga,” which suggests
“caring hands.” It was a spiritual practice that was believed to balance the body’s energy and
promote total wellness. In China, body-to-body massage became part of the Taoist custom
and was utilized to promote the circulation of vital energy,
known as “qi,” throughout the body. Throughout history, body-to-body massage has actually evolved and adapted to various cultures and societies.
In some cultures, it was seen as a kind of art and charm, while in others, it
was considered a therapeutic practice. Today, body-to-body massage is extensively recognized for its
capability to supply both psychological and physical benefits.
At its core, body-to-body massage includes the
intimate connection in between the provider and receiver, developing a sense of trust and
intimacy.
The provider utilizes their body to apply pressure and perform numerous massage techniques on the receiver’s
body. During a body-to-body massage, the provider may
use their arms to glide efficiently over the receiver’s back, offering a peaceful and calming experience.
The chest might be utilized to use gentle pressure to particular locations, launching stress
and promoting deep relaxation. The legs might be utilized to carry out extending motions, enhancing flexibility and increasing blood circulation. One
of the crucial concepts of body-to-body massage is the focus on today minute.
Both the giver and receiver are motivated to be fully present and conscious
of their feelings and bodies. This mindfulness permits a much
deeper connection and enhances the overall experience.
Another important concept of body-to-body massage is interaction. The giver and receiver should honestly communicate their
choices, boundaries, and any pain they might experience during the session. This guarantees that both parties are comfy and can fully take pleasure in the experience.
Overall, body-to-body massage is an intimate and distinct practice that uses a variety of physical and psychological advantages.
The advantages of Thai massage may be really 21st century however they have their
roots in ancient Eastern practices established over thousands of
years. We invite you to come in and experience the magic of
a Thai yoga Massage for yourself. Body-to-body massage,
also known as B2B massage, is a sensuous and intimate type of massage that includes the use
of not just hands, however the whole body to supply relaxation and enjoyment.
Body-to-body massage, also known as B2B massage, is
a kind of massage that goes beyond the conventional techniques.
The provider utilizes their body to use pressure and perform numerous massage
strategies on the receiver’s body.
Your exemplary work demonstrates remarkable potential. With strategic enhancements, this foundation can be transformed into a pioneering contribution that advances the field and solidifies your position as an industry authority.
zRgmsn newsWu jHyzp baMZL tYCzz
RVCPgipcKQWSOsL
cdpPRYDQoCvxWuH
Hey people!!!!!
Good mood and good luck to everyone!!!!!
Thank you for sharing this insightful post! Your writing is clear, informative, and engaging. I appreciate how you’ve broken down complex concepts into easily digestible parts. It’s evident that you have a deep understanding of the topic, and your tips are practical and actionable. I particularly liked the way you addressed [specific point from the article], as it resonated with my own experiences. This kind of content is invaluable for readers looking to expand their knowledge and apply new strategies effectively. Looking forward to reading more from you. Keep up the excellent work!
Thank you for sharing this insightful post! Your writing is clear, informative, and engaging. I appreciate how you’ve broken down complex concepts into easily digestible parts. It’s evident that you have a deep understanding of the topic, and your tips are practical and actionable. I particularly liked the way you addressed [specific point from the article], as it resonated with my own experiences. This kind of content is invaluable for readers looking to expand their knowledge and apply new strategies effectively. Looking forward to reading more from you. Keep up the excellent work!
Thank you for the auspicious writeup It in fact was a amusement account it Look advanced to more added agreeable from you By the way how could we communicate
Hi my family member I want to say that this post is awesome nice written and come with approximately all significant infos I would like to peer extra posts like this
lCVrNbonYf
Hey people!!!!!
Good mood and good luck to everyone!!!!!
You’re fantastic
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Thank you for sharing this insightful post! Your writing is clear, informative, and engaging. I appreciate how you’ve broken down complex concepts into easily digestible parts. It’s evident that you have a deep understanding of the topic, and your tips are practical and actionable. I particularly liked the way you addressed [specific point from the article], as it resonated with my own experiences. This kind of content is invaluable for readers looking to expand their knowledge and apply new strategies effectively. Looking forward to reading more from you. Keep up the excellent work!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Great post! I found your insights really enlightening. I’m curious, what inspired you to start writing about this subject? Also, do you have any recommendations for further reading or resources on this topic? Looking forward to your response!
Thank you for sharing this insightful post! Your writing is clear, informative, and engaging. I appreciate how you’ve broken down complex concepts into easily digestible parts. It’s evident that you have a deep understanding of the topic, and your tips are practical and actionable. I particularly liked the way you addressed [specific point from the article], as it resonated with my own experiences. This kind of content is invaluable for readers looking to expand their knowledge and apply new strategies effectively. Looking forward to reading more from you. Keep up the excellent work!
I loved as much as you will receive carried out right here The sketch is attractive your authored material stylish nonetheless you command get got an impatience over that you wish be delivering the following unwell unquestionably come more formerly again since exactly the same nearly a lot often inside case you shield this hike
I was recommended this website by my cousin I am not sure whether this post is written by him as nobody else know such detailed about my trouble You are amazing Thanks
Hey people!!!!!
Good mood and good luck to everyone!!!!!
Hi there, I just finished reading your post, and I have to say it’s really insightful! You’ve covered some key points that many people often overlook. I especially appreciated the way you explainedthis article. It’s something I can definitely relate to and I think it will benefit a lot of readers. I also wanted to add that I’ve been exploring a similar topic over at my website, where I discuss [mention something relevant to the post but related to your content]. It’s interesting to see how our ideas align in some areas, and I’d love to hear your thoughts on it! Thanks again for sharing such valuable information. Keep up the great work!
Hi there, I just finished reading your post, and I have to say it’s really insightful! You’ve covered some key points that many people often overlook. I especially appreciated the way you explainedthis article. It’s something I can definitely relate to and I think it will benefit a lot of readers. I also wanted to add that I’ve been exploring a similar topic over at my website, where I discuss [mention something relevant to the post but related to your content]. It’s interesting to see how our ideas align in some areas, and I’d love to hear your thoughts on it! Thanks again for sharing such valuable information. Keep up the great work!
9jne4g
Your blog has quickly become one of my favorites. Your writing is both insightful and thought-provoking, and I always come away from your posts feeling inspired. Keep up the phenomenal work!
What i do not understood is in truth how you are not actually a lot more smartlyliked than you may be now You are very intelligent You realize therefore significantly in the case of this topic produced me individually imagine it from numerous numerous angles Its like men and women dont seem to be fascinated until it is one thing to do with Woman gaga Your own stuffs nice All the time care for it up
Your blog is a breath of fresh air in the crowded online space. I appreciate the unique perspective you bring to every topic you cover. Keep up the fantastic work!
I’m impressed by the clarity and detail in your writing—well done!
Your perspective is truly unique!
Hello!
Good cheer to all on this beautiful day!!!!!
Good luck 🙂
Hi Neat post Theres an issue together with your web site in internet explorer may test this IE still is the marketplace chief and a good component of people will pass over your fantastic writing due to this problem
Your blog is a constant source of inspiration for me. Your passion for your subject matter is palpable, and it’s clear that you pour your heart and soul into every post. Keep up the incredible work!
Magnificent beat I would like to apprentice while you amend your site how can i subscribe for a blog web site The account helped me a acceptable deal I had been a little bit acquainted of this your broadcast offered bright clear idea
you are in reality a good webmaster The website loading velocity is amazing It sort of feels that youre doing any distinctive trick Also The contents are masterwork you have done a fantastic job in this topic