为了账号安全,请及时绑定邮箱和手机立即绑定

kubernetes1.13.1集群结合ceph rbd部署最新版本jenkins

标签:
SQL Server

参考文档

https://blog.csdn.net/aixiaoyang168/article/details/79767649https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/kuberneteshttps://www.cnblogs.com/cocowool/p/kubernetes_statefulset.htmlhttps://www.cnblogs.com/cocowool/p/kubernetes_storage.html

文档目录


简介

jenkins-kubernetes-plugin
Jenkins plugin to run dynamic agents in a Kubernetes cluster.
Based on the Scaling Docker with Kubernetes article, automates the scaling of Jenkins agents running in Kubernetes.
The plugin creates a Kubernetes Pod for each agent started, defined by the Docker image to run, and stops it after each build.
Agents are launched using JNLP, so it is expected that the image connects automatically to the Jenkins master. For that some environment variables are automatically injected:

  • JENKINS_URL: Jenkins web interface url

  • JENKINS_SECRET: the secret key for authentication

  • JENKINS_AGENT_NAME: the name of the Jenkins agent

  • JENKINS_NAME: the name of the Jenkins agent (Deprecated. Only here for backwards compatibility)

基本环境

k81集群1.13.1版本

[root@elasticsearch01 ~]# kubectl get nodesNAME        STATUS   ROLES    AGE   VERSION10.2.8.34   Ready    <none>   25d   v1.13.110.2.8.65   Ready    <none>   25d   v1.13.1

ceph集群 luminous版本

[root@ceph01 ~]# ceph -s
  services:
    mon: 3 daemons, quorum ceph01,ceph02,ceph03    mgr: ceph03(active), standbys: ceph02, ceph01    osd: 24 osds: 24 up, 24 in
    rgw: 3 daemons active

操作步骤

一、使用ceph rbd创建pv、pvc

官网使用的是自带创建pv与pvc这里使用的是手动创建
1、创建pv

[root@elasticsearch01 jenkins]# cat jenkins-pv.yaml apiVersion: v1kind: PersistentVolumemetadata:
  name: jenkins-home-pvspec:
  capacity:
    storage: 40Gi  accessModes:
    - ReadWriteOnce  rbd:
    monitors:
      - '10.0.4.10:6789'
      - '10.0.4.13:6789'
      - '10.0.4.15:6789'
    pool: rbd-k8s    image: cephimage3    user: admin    secretRef:
      name: ceph-secret    fsType: ext4    readOnly: false
  persistentVolumeReclaimPolicy: Recycle

[root@elasticsearch01 jenkins]# kubectl create -f jenkins-pv.yaml persistentvolume/jenkins-home-pv created
[root@elasticsearch01 jenkins]# kubectl get pvNAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                       STORAGECLASS   REASON   AGE
ceph-rbd-pv       20Gi       RWO            Recycle          Bound       default/ceph-rbd-pv-claim                           22h
jenkins-home-pv   40Gi       RWO            Recycle          Available                                                       4s

2、创建pvc

[root@elasticsearch01 jenkins]# cat jenkins-pvc.yaml apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-home-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

[root@elasticsearch01 jenkins]# kubectl create -f jenkins-pvc.yaml persistentvolumeclaim/jenkins-home-pvc created

[root@elasticsearch01 jenkins]# kubectl get pvcNAME                STATUS   VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ceph-rbd-pv-claim   Bound    ceph-rbd-pv       20Gi       RWO                           22h
jenkins-home-pvc    Bound    jenkins-home-pv   40Gi       RWO                           3s
[root@elasticsearch01 jenkins]# kubectl get pvNAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS   REASON   AGE
ceph-rbd-pv       20Gi       RWO            Recycle          Bound    default/ceph-rbd-pv-claim                           22h
jenkins-home-pv   40Gi       RWO            Recycle          Bound    default/jenkins-home-pvc                            77s

二、跟进实际情况修改jenkins.yml文件

主要修改的配置从上到下分别是:
1、拉取镜像策略

          imagePullPolicy: IfNotPresent

2、自动存储storage class改成voulumes的pvc方式实现

      volumes:
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: jenkins-home-pvc

3、ingress的host改成实际的

    host: jenkins.minminmsn.com

4、ingres的tls证书改成实际的

  tls:  - hosts:    - jenkins.minminmsn.com
    secretName: ingress-secret

5、具体如下

[root@elasticsearch01 jenkins]# cat jenkins.yml ---apiVersion: apps/v1beta1kind: StatefulSetmetadata:
  name: jenkins  labels:
    name: jenkinsspec:
  serviceName: jenkins  replicas: 1
  updateStrategy:
    type: RollingUpdate  template:
    metadata:
      name: jenkins      labels:
        name: jenkins    spec:
      terminationGracePeriodSeconds: 10
      serviceAccountName: jenkins      containers:
        - name: jenkins          image: jenkins/jenkins:lts-alpine          imagePullPolicy: IfNotPresent          ports:
            - containerPort: 8080
            - containerPort: 50000
          resources:
            limits:
              cpu: 1
              memory: 1Gi            requests:
              cpu: 0.5
              memory: 500Mi          env:
            - name: LIMITS_MEMORY              valueFrom:
                resourceFieldRef:
                  resource: limits.memory                  divisor: 1Mi
            - name: JAVA_OPTS              # value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
              value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
          volumeMounts:
            - name: jenkins-home              mountPath: /var/jenkins_home              readOnly: false
          livenessProbe:
            httpGet:
              path: /login              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12 # ~2 minutes
          readinessProbe:
            httpGet:
              path: /login              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12 # ~2 minutes
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkins-home        persistentVolumeClaim:
          claimName: jenkins-home-pvc

---apiVersion: v1kind: Servicemetadata:
  name: jenkinsspec:
  # type: LoadBalancer
  selector:
    name: jenkins  # ensure the client ip is propagated to avoid the invalid crumb issue when using LoadBalancer (k8s >=1.7)
  #externalTrafficPolicy: Local
  ports:
    -      name: http      port: 80
      targetPort: 8080
      protocol: TCP
    -      name: agent      port: 50000
      protocol: TCP

---apiVersion: extensions/v1beta1kind: Ingressmetadata:
  name: jenkins  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/tls-acme: "true"
    # "413 Request Entity Too Large" uploading plugins, increase client_max_body_size
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
    # For nginx-ingress controller < 0.9.0.beta-18
    ingress.kubernetes.io/ssl-redirect: "true"
    # "413 Request Entity Too Large" uploading plugins, increase client_max_body_size
    ingress.kubernetes.io/proxy-body-size: 50m
    ingress.kubernetes.io/proxy-request-buffering: "off"spec:
  rules:
  - http:
      paths:
      - path: /        backend:
          serviceName: jenkins          servicePort: 80
    host: jenkins.minminmsn.com  tls:
  - hosts:
    - jenkins.minminmsn.com    secretName: ingress-secret

三、创建状态集、svc、pod、ingress

1、创建rbac认证角色

[root@elasticsearch01 jenkins]# kubectl create -f service-account.yml serviceaccount/jenkins created
role.rbac.authorization.k8s.io/jenkins created
rolebinding.rbac.authorization.k8s.io/jenkins created

2、创建jenkins服务等

[root@elasticsearch01 jenkins]# kubectl create -f jenkins.yml statefulset.apps/jenkins created
service/jenkins created
ingress.extensions/jenkins created        4s
[root@elasticsearch01 jenkins]# kubectl get podsNAME               READY   STATUS              RESTARTS   AGE
busybox            1/1     Running             454        18d
ceph-rbd-pv-pod1   1/1     Running             1          21h
jenkins-0          0/1     ContainerCreating   0          7s
[root@elasticsearch01 jenkins]# kubectl get podsNAME               READY   STATUS    RESTARTS   AGE
busybox            1/1     Running   454        18d
ceph-rbd-pv-pod1   1/1     Running   1          21h
jenkins-0          1/1     Running   0          4m52s

四、通过ingress访问

获取ingress-nginx对外端口,https://jenkins.minminmsn.com:47215/访问即可,需要配置dns解析到pod所在node的ip

[root@elasticsearch01 jenkins]# kubectl get svc -n ingress-nginx|grep ingress-nginxingress-nginx       LoadBalancer   10.254.125.151   <pending>     80:33003/TCP,443:47215/TCP   14d

五、初始化jenkins

1、查找密码

[root@elasticsearch02 ~]# df -h|grep rbd/dev/rbd0                  493G  163G  306G  35% /data
/dev/rbd1                   20G   45M   20G   1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/rbd-k8s-image-cephimage2
/dev/rbd2                   40G  138M   40G   1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/rbd-k8s-image-cephimage3
[root@elasticsearch02 ~]# cd //var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/rbd-k8s-image-cephimage3[root@elasticsearch02 rbd-k8s-image-cephimage3]# lsconfig.xml                     init.groovy.d                        jobs              nodes                     secrets      war
copy_reference_file.log        jenkins.CLI.xml                      logs              plugins                   updates
hudson.model.UpdateCenter.xml  jenkins.install.UpgradeWizard.state  lost+found        secret.key                userContent
identity.key.enc               jenkins.telemetry.Correlator.xml     nodeMonitors.xml  secret.key.not-so-secret  users
[root@elasticsearch02 rbd-k8s-image-cephimage3]# cat secrets/initialAdminPassword 92c145b796cc48b0af8b5ef0f7afce28

webp

2、选择安装插件

webp


webp

3、创建初始管理账号

webp

4、设置jenkins url默认https://jenkins.minminmsn.com:47215/

webp

5、开始使用jenkins

webp

6、jenkins控制台界面,主要配置都在系统管理中

[图片上传失败...(image-ce2d85-1548296164745)]

总结

使用ceph rbd 这种只能读写一次的设备不能用在线上,线上应该使用分布式存储例如nfs,cephfs,glusterfs等,这里只是测试jenkins结合ceph,pv,pvc完成有状态pod的测试



作者:三杯水Plus
链接:https://www.jianshu.com/p/e8207c9c999b


点击查看更多内容
TA 点赞

若觉得本文不错,就分享一下吧!

评论

作者其他优质文章

正在加载中
  • 推荐
  • 评论
  • 收藏
  • 共同学习,写下你的评论
感谢您的支持,我会继续努力的~
扫码打赏,你说多少就多少
赞赏金额会直接到老师账户
支付方式
打开微信扫一扫,即可进行扫码打赏哦
今天注册有机会得

100积分直接送

付费专栏免费学

大额优惠券免费领

立即参与 放弃机会
意见反馈 帮助中心 APP下载
官方微信

举报

0/150
提交
取消