kubernetes Jenkins gitlab搭建CI/CD环境 (二)

docker, 云计算, 虚拟化
5月 24, 2018
0 评论

接前一篇文章 kubernetes Jenkins gitlab搭建CI/CD环境–(一)，本文介绍在Kubernetes上安装Jenkins。

Jenkins的安装有多种，可以在独立的服务器安装，结合K8S的话可以使用helm，参考：https://github.com/kubernetes/charts/tree/master/stable/jenkins
chart中使用的Jenkins基础镜像 jenkins/jenkins:lts，也可以通过Dockerfile自己定制。
下面是我基于jenkins/jenkins:lts定制的Dockerfile，增加了docker，docker-compose，kubectl和maven，然后使用yaml文件手动部署的。
Jenkins Master的 Dockerfile 如下：

FROM jenkins/jenkins:lts
MAINTAINER Fisher.yu <yu2hei@gmail.com>

EXPOSE 8080 50000
ENV DOCKER_VERSION=17.04.0-ce DOCKER_COMPOSE_VERSION=1.21.2 KUBECTL_VERSION=v1.10.1

# Use Root to setup kubectl, docker-ce, docker-compose
USER root
WORKDIR /usr/local/bin

############
# change debian source to 163
############

RUN sed -i 's/deb.debian.org/mirrors.163.com/g' /etc/apt/sources.list
RUN sed -i 's/security.debian.org/mirrors.163.com\/debian-security/g' /etc/apt/sources.list

############
# Update packages
############

RUN apt-get update && apt-get install -y --no-install-recommends \
    make \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

#######
# docker-ce
#######

#RUN curl -fsSLO https://get.docker.com/builds/Linux/x86_64/docker-${DOCKER_VERSION}.tgz \
RUN curl -fsSLO http://sudops.com/docker-${DOCKER_VERSION}.tgz \
		&& tar --strip-components=1 -xvzf docker-${DOCKER_VERSION}.tgz -C /usr/local/bin \
		&& chmod -R +x /usr/local/bin/docker


#######
# docker-compose
#######

#RUN curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose \
RUN curl -L http://sudops.com/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose \
    && chmod +x /usr/local/bin/docker-compose


#######
# kubectl
#######

RUN curl -L  http://sudops.com/kubectl-${KUBECTL_VERSION} -o /usr/local/bin/kubectl \
    && chmod +x /usr/local/bin/kubectl

#######
# Maven
#######

# Preparation

ENV MAVEN_VERSION 3.5.3
ENV MAVEN_HOME /etc/maven-${MAVEN_VERSION}

# Installation

RUN cd /tmp
RUN wget http://archive.apache.org/dist/maven/maven-3/$MAVEN_VERSION/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz
RUN mkdir maven-${MAVEN_VERSION}
RUN tar -zxvf apache-maven-${MAVEN_VERSION}-bin.tar.gz --directory maven-${MAVEN_VERSION} --strip-components=1
RUN mv maven-${MAVEN_VERSION} ${MAVEN_HOME}
ENV PATH ${PATH}:${MAVEN_HOME}/bin

# Cleanup

RUN rm apache-maven-${MAVEN_VERSION}-bin.tar.gz
RUN unset MAVEN_VERSION

#######
# Back to Jenkins home
#######

USER jenkins
WORKDIR $JENKINS_HOME

#build docker 镜像

docker build -t repo.ky.in/webcola/jenkins-docker-kubectl:v0.0.1 --no-cache .

#将build好的docker镜像push到私有docker-harbor中

docker push repo.ky.in/webcola/jenkins-docker-kubectl:v0.0.1

创建Jenkins部署文件
*** 本文中kubernetes使用namespace均为devns
创建jenkins PersistentVolumeClaim yaml文件

# cat jenkins-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: jenkins-pvc
 namespace: devns
spec:
 accessModes:
    - ReadWriteOnce
 resources:
   requests:
     storage: 60Gi
 storageClassName: kyglustersc
** 说明，这里指定了之前创建的storageclass： kyglustersc

创建jenkins deployment文件

# cat jenkins-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kyjenkins
  namespace: devns
  labels:
    app: kyjenkins
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: kyjenkins
        tier: kyjenkins
    spec:
      containers:
      - image: repo.ky.in/webcola/jenkins-docker-kubectl:v0.0.1
        name: kyjenkins
        securityContext:
          privileged: true
        ports:
        - containerPort: 8080
          name: kyjenkins
        - containerPort: 50000
          name: agent
          protocol: TCP
        volumeMounts:
        - name: docker
          mountPath: /var/run/docker.sock
        - name: jenkins-persistent-storage
          mountPath: /var/jenkins_home
        - name: kube-config
          mountPath: /root/.kube/config
      volumes:
      - name: docker
        hostPath:
          path: /var/run/docker.sock
      - name: jenkins-persistent-storage
        persistentVolumeClaim:
          claimName: jenkins-pvc
      - name: kube-config
        hostPath:
          path: /root/.kube/config

创建jenkins-service

# cat jenkins-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: kyjenkins
  namespace: devns
  labels:
    app: kyjenkins
spec:
  ports:
    - port: 8080
      targetPort: 8080
      name: kyjenkins
    - port: 50000
      targetPort: 50000
      name: agent
  selector:
    app: kyjenkins
    tier: kyjenkins

然后执行

kubectl create -f .

待jenkisn pods running后，describe查看pods情况：

Events:
  Type     Reason                 Age              From                   Message
  ----     ------                 ----             ----                   -------
  Warning  FailedScheduling       2d (x5 over 2d)  default-scheduler      pod has unbound PersistentVolumeClaims (repeated 9 times)
  Normal   Scheduled              2d               default-scheduler      Successfully assigned kyjenkins-8685458884-j8qjg to node_178
  Normal   SuccessfulMountVolume  2d               kubelet, node_178  MountVolume.SetUp succeeded for volume "docker"
  Normal   SuccessfulMountVolume  2d               kubelet, node_178  MountVolume.SetUp succeeded for volume "kube-config"
  Normal   SuccessfulMountVolume  2d               kubelet, node_178  MountVolume.SetUp succeeded for volume "default-token-fxgsp"
  Normal   SuccessfulMountVolume  2d               kubelet, node_178  MountVolume.SetUp succeeded for volume "pvc-d7ad4f36-5d73-11e8-bcc6-5254006a334a"
  Normal   Pulled                 2d               kubelet, node_178  Container image "repo.ky.in/webcola/jenkins-docker-kubectl:v0.0.1" already present on machine
  Normal   Created                2d               kubelet, node_178  Created container
  Normal   Started                2d               kubelet, node_178  Started container

Jenkins Master运行没问题，然后配置Ingress，并reload配置。

  - host: jenkins.kydev.in
    http:
      paths:
      - backend:
          serviceName: kyjenkins
          servicePort: 8080

访问 http://jenkins.kydev.in

解锁密钥可以在kubernetes上执行 kubectl logs kyjenkins-8685458884-j8qjg -f 查看，也可以exec 进入到pods内部查看文件。
可以安装下社区推荐的插件：

Jenkins一些常用的插件，可以根据自己的实际情况安装：

locale
php-jenkins-plugins
checkstyle
cloverphp
dry
htmlpublisher
jdepend
plot
pmd
violations
xunit
php
git
phing
build-pipeline-plugin
bouncycastle API		
PHP Built-in Web Server
ElasticBox Jenkins Kubernetes CI/CD
Javadoc
Maven Integration
OWASP Markup Formatter
Static Analysis Utilities
DRY
GitLab Logo
Kubernetes :: Pipeline :: Arquillian Steps
Gitlab Merge Request Builder
Kubernetes Cli
Kubernetes :: Pipeline :: DevOps Steps
Kubernetes :: Pipeline :: Kubernetes Steps
Kubernetes :: Pipeline :: Aggregator
Windows Slaves
Matrix Authorization Strategy
Phing
Plot
Gitlab Authentication
Violation Comments to GitLab
Token Macro
Ant
PAM Authentication
LDAP
External Monitor Job Type
Run Condition
Conditional BuildStep
Parameterized Trigger
jQuery
Build Pipeline
ruby-runtime
Gitlab Hook
xUnit
Pipeline Aggregator
GitHub API
GitHub
GitHub Branch Source
Pipeline: GitHub
Clover PHP
JDepend
Dashboard View
Delivery Pipeline
HTML Publisher
PMD
Checkstyle
Violations
php
Checkstyle
Azure Commons
Kubernetes Continuous Deploy
Kubernetes Credentials Provider
GitLab
Kubernetes :: Pipeline :: Kubernetes Steps
Kubernetes :: Pipeline :: DevOps Steps

如果使用默认源比较慢，可以多试试其他几个源：

https://updates.jenkins.io/update-center.json (默认：)
http://updates.jenkins-ci.org/update-center.json
https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json
http://mirror.xmission.com/jenkins/updates/current/update-center.json

系统管理–云–增加Kubernets连接配置

接下来可以验证Master的工作情况：
创建一个freestyle的job kube-test。

可以看到任务执行成功，kubectl get pods -n devns 可以打印出结果。
接下来配置Jenkins slave
同样Jenkins slave也是采用自定义Dockerfile的方式

# cat Dockerfile
FROM jenkins/jnlp-slave

MAINTAINER Fisher.yu <yu2hei@gmail.com>

ENV KUBECTL_VERSION=v1.10.1 DOCKER_VERSION=17.04.0-ce

USER root

##########
# Maven-3.5.3
##########

COPY maven /usr/share/maven/
RUN chmod +x /usr/share/maven/bin/mvn && ln -s /usr/share/maven/bin/mvn /usr/local/bin/mvn

##########
# kubectl-1.10.1
##########

COPY kubectl /usr/local/bin/
RUN chmod +x /usr/local/bin/kubectl

##########
# 预配置 kubectl
##########

##########
# 在运行时由 ConfigMap 挂载Volume
##########

ENV CERT_DIR /etc/kubernetes/conf
ARG DOCKER_SOCK_DIR=/var/run/docker.sock
RUN mkdir -p ${CERT_DIR} \
mkidr -p /root/.kube \
touch ${CERT_DIR}/k8s-root-ca.pem \
touch ${CERT_DIR}/admin.pem \
touch ${CERT_DIR}/admin-key.pem
COPY config /usr/local/bin/
RUN export KUBECONFIG=/usr/local/bin/config && kubectl config view

##########
# docker-ce
##########
#------------------------------------------------#
copy docker /usr/bin/docker
RUN apt-get -y update && apt-get install -y apt-utils iptables libdevmapper1.02.1 libltdl7 libseccomp2 && apt-get -y autoremove && chmod +x /usr/bin/docker

##########
# 挂载volume
##########

VOLUME ${CERT_DIR}
VOLUME ${DOCKER_SOCK_DIR}

## 当前目录结果
# tree .
.
├── config
├── docker
├── docker-compose
├── Dockerfile
├── kubectl
└── maven
    ├── bin
    ├── conf
    ├── lib
    └── README.txt

# cat config
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /etc/kubernetes/conf/k8s-root-ca.pem
    server: http://k8sapi.kydev.in
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: kube-system
    user: admin
  name: kubernetes
current-context: kubernetes
kind: Config
preferences: {}
users:
- name: admin
  user:
    client-certificate: /etc/kubernetes/conf/admin.pem
    client-key: /etc/kubernetes/conf/admin-key.pem

将build的镜像push到私有repo中。

docker build -t repo.ky.in/webcola/jnlp-slave-docker-kubectl-ky:v0.0.1 --no-cache .
docker push repo.ky.in/webcola/jnlp-slave-docker-kubectl-ky:v0.0.1

# (1). 创建 kubectl-cert-cm 资源对象
       kubectl create configmap kubectl-cert-cm --from-file=/etc/kubernetes/ssl -n=devns
            
# (2). 查看所创建的 kubectl-cert-cm 资源对象
       kubectl describe configmap kubectl-cert-cm -n=devns

下面继续配置Jenkins的Slave节点。
Kubernetes Pod Template 中配置，详见截图

好了，现在Jenkins的jnlp-slave已经配置完毕
下面开始测试：
创建pipeline任务

pipeline script

def label = "mypod-${UUID.randomUUID().toString()}"
podTemplate(label: label, containers: [
    containerTemplate(name: 'maven', image: 'repo.ky.in/webcola/jdk-maven-ant:v1.0.0', ttyEnabled: true, command: 'cat')
  ]) {

    node(label) {
        stage('Get a Maven project') {
            container('maven') {
                stage('wait for exec check'){
                    sh 'sleep 10'
                }

                stage('get maven env') {
                    sh 'cat /etc/resolv.conf'
                    sh 'cat /etc/issue'
                    sh 'uname -a'
                    sh 'env'
                    sh 'echo "$(sed \'s/options ndots:5/#options ndots:5/g\' /etc/resolv.conf)" > /etc/resolv.conf'
                }
                
                stage('code checkout') {
                    git url: 'http://gitlab.ky.in/devops/tomcatwartest.git', credentialsId: '29efa3cc-4fe7-42e2-95a4-a03c18d56603', branch: 'master'
                    sh 'mvn clean package'
                    sh 'sleep 300'
                }
            }
        }
    }
}

任务运行成功。

在配置Jenkins-slave是有几个地方需要注意：
（1）Jenkins slave使用自定义镜像调用kubectl时freestyle的任务可以在构建环境中指定『setup kubernetes CLI』，但是在pipeline中就需要在镜像里面做文章了，可以参考jenkins-slave的Dockerfile，需要将三个证书文件和config文件传入到镜像中，创建kubectl-cert-cm的configmap。
（2）pipeline中的podTemplate指定了自定义的image，但是实际启动中还是会默认再启动一个jenkins/jnlp-slave:alpine的镜像。然而这个alpine本身就有很多”坑”，比如dns解析的问题，在slave的镜像里面git代码会报Can’t resolve git server host。解决方式就是注释掉/etc/resolv.conf里面的”options ndots:5″ 这行，具体见我之前的文章：kubernetes 使用基于 alpine 镜像无法正常解析外网DNS

好了，接下来的文章会介绍kubernetes上安装gitlab，配置钩子自动出发流程，以及本文涉及的一个简单的tomcat-maven测试环境。