{"msg":"操作成功","code":200,"data":{"createBy":"admin","createTime":"2021-11-28 17:55:40","updateBy":"admin","updateTime":"2021-11-28 17:55:40","remark":null,"id":74,"articleTitle":"Kubernetes（十二）搭建Redis-Cluster集群","articleUrl":"k8s_redis_cluster","articleThumbnail":"https://www.asumimoe.com/imgfiles/20220906/f93daad129a04b8db74eed70cd45263b.png","articleFlag":"0","draftStatus":"1","reprintStatement":"1","articleSummary":"在k8s集群中搭建三主三从cluster集群","articleContent":"## 集群准备\n\n在Kubernetes上创建redis三主三从集群需要明确以下需求：\n\n- 将配置文件以configMap的方式挂载到Pod中方便修改。\n- redis开启数据持久化，为此需要挂载数据卷满足数据可用性。注意nfs目录的权限\n- 无状态应用须使用Statefulset控制保证Pod迁移后集群和数据的可用性。\n- 创建Headless Service暴露集群入口。\n\n## 编写yaml文件\n\n```yaml\n# 创建一个独立的namespace，所有redis集群相关资源均绑定在此namespace中\napiVersion: v1\nkind: Namespace\nmetadata:\n  name: redis-cluster\n  labels:\n    app: redis-cluster\n\n---\n# 创建PV及PVC供redis节点保存数据使用\napiVersion: v1\nkind: PersistentVolume\nmetadata:\n  name: nfs-pv1\n  namespace: redis-cluster\nspec:\n  capacity:\n    storage: 100Mi\n  accessModes:\n    - ReadWriteMany\n  persistentVolumeReclaimPolicy: Recycle\n  storageClassName: redis\n  nfs:\n    server: 192.168.52.211\n    path: \"/nfs/redis/pv1\"\n---\napiVersion: v1\nkind: PersistentVolume\nmetadata:\n  name: nfs-pv2\n  namespace: redis-cluster\nspec:\n  capacity:\n    storage: 100Mi\n  accessModes:\n    - ReadWriteMany\n  persistentVolumeReclaimPolicy: Recycle\n  storageClassName: redis\n  nfs:\n    server: 192.168.52.211\n    path: \"/nfs/redis/pv2\"\n---\napiVersion: v1\nkind: PersistentVolume\nmetadata:\n  name: nfs-pv3\n  namespace: redis-cluster\nspec:\n  capacity:\n    storage: 100Mi\n  accessModes:\n    - ReadWriteMany\n  persistentVolumeReclaimPolicy: Recycle\n  storageClassName: redis\n  nfs:\n    server: 192.168.52.211\n    path: \"/nfs/redis/pv3\"\n---\napiVersion: v1\nkind: PersistentVolume\nmetadata:\n  name: nfs-pv4\n  namespace: redis-cluster\nspec:\n  capacity:\n    storage: 100Mi\n  accessModes:\n    - ReadWriteMany\n  persistentVolumeReclaimPolicy: Recycle\n  storageClassName: redis\n  nfs:\n    server: 192.168.52.211\n    path: \"/nfs/redis/pv4\"\n---\napiVersion: v1\nkind: PersistentVolume\nmetadata:\n  name: nfs-pv5\n  namespace: redis-cluster\nspec:\n  capacity:\n    storage: 100Mi\n  accessModes:\n    - ReadWriteMany\n  persistentVolumeReclaimPolicy: Recycle\n  storageClassName: redis\n  nfs:\n    server: 192.168.52.211\n    path: \"/nfs/redis/pv5\"\n---\napiVersion: v1\nkind: PersistentVolume\nmetadata:\n  name: nfs-pv6\n  namespace: redis-cluster\nspec:\n  capacity:\n    storage: 100Mi\n  accessModes:\n    - ReadWriteMany\n  persistentVolumeReclaimPolicy: Recycle\n  storageClassName: redis\n  nfs:\n    server: 192.168.52.211\n    path: \"/nfs/redis/pv6\"\n---\n# 创建configMap\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: redis-cm\n  namespace: redis-cluster\ndata:\n  fix-ip.sh: |\n    #!/bin/sh\n    CLUSTER_CONFIG=\"/data/cluster-node.conf\"\n    if [ -f ${CLUSTER_CONFIG} ]; then\n      if [ -z \"${POD_IP}\" ]; then\n        echo \"Unable to determine Pod IP address!\"\n        exit 1\n      fi\n      echo \"Updating my IP to ${POD_IP} in ${CLUSTER_CONFIG}\"\n      sed -i.bak -e '/myself/ s/[0-9]\\{1,3\\}\\.[0-9]\\{1,3\\}\\.[0-9]\\{1,3\\}\\.[0-9]\\{1,3\\}/'${POD_IP}'/' ${CLUSTER_CONFIG}\n    fi\n    exec \"$@\"\n  redis.conf: |\n    port 6379\n    daemonize no\n    databases 16\n    save 900 1\n    dir /data\n    dbfilename \"dump.rdb\"\n    appendonly yes\n    cluster-enabled yes\n    cluster-config-file \"/data/cluster-node.conf\"\n---\n# 创建Headless Service提供集群访问入口\napiVersion: v1\nkind: Service\nmetadata:\n  name: redis-cluster-svc\n  namespace: redis-cluster\n  labels:\n    app: redis\nspec:\n  selector:\n    app: redis\n    cluster: redis-cluster\n  clusterIP: None\n  ports:\n  - port: 6379\n    targetPort: 6379\n---\n# 创建Statefulset控制Redis集群\napiVersion: apps/v1\nkind: StatefulSet\nmetadata:\n  name: cluster-node\n  namespace: redis-cluster\n  labels:\n    app: redis\n    cluster: redis-cluster\nspec:\n  selector:\n    matchLabels:\n      app: redis\n      cluster: redis-cluster\n  serviceName: redis-cluster-svc\n  replicas: 6\n  template:\n    metadata:\n      labels:\n        app: redis\n        cluster: redis-cluster  \n    spec:\n      containers:\n      - name: cluster-node\n        image: redis:6.2.6\n        env:\n        - name: TZ\n          value: \"Asia/Shanghai\"\n        - name: POD_IP\n          valueFrom:\n            fieldRef:\n              fieldPath: status.podIP\n        ports:\n        - containerPort: 6379\n          protocol: TCP\n          name: redis\n        - containerPort: 16379\n          protocol: TCP\n          name: cluster\n        volumeMounts:\n        - name: redis-pvc\n          mountPath: \"/data\"\n        - name: redis-conf\n          mountPath: \"/etc/redis\"\n        command: [\"/etc/redis/fix-ip.sh\",\"redis-server\",\"/etc/redis/redis.conf\"]\n      volumes:\n      - name: redis-conf\n        configMap:\n          name: redis-cm\n          defaultMode: 0755\n  volumeClaimTemplates:\n  - metadata:\n      name: redis-pvc\n      namespace: redis-cluster\n    spec:\n      accessModes: [\"ReadWriteMany\"]\n      storageClassName: redis\n      resources:\n        requests:\n          storage: 100Mi\n```\n\n## 创建集群\n\n### 创建所需资源\n\n```shell\n[root@master ~]# kubectl apply -f redis-cluster.yml \nnamespace/redis-cluster created\npersistentvolume/nfs-pv1 created\npersistentvolume/nfs-pv2 created\npersistentvolume/nfs-pv3 created\npersistentvolume/nfs-pv4 created\npersistentvolume/nfs-pv5 created\npersistentvolume/nfs-pv6 created\nconfigmap/redis-cm created\nservice/redis-cluster-svc created\nstatefulset.apps/cluster-node created\n[root@master ~]# kubectl get po -n redis-cluster\nNAME             READY   STATUS    RESTARTS   AGE\ncluster-node-0   1/1     Running   0          52s\ncluster-node-1   1/1     Running   0          48s\ncluster-node-2   1/1     Running   0          26s\ncluster-node-3   1/1     Running   0          21s\ncluster-node-4   1/1     Running   0          16s\ncluster-node-5   1/1     Running   0          12s\n[root@master ~]# kubectl get pvc -n redis-cluster\nNAMESPACE       NAME                       STATUS   VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE\nredis-cluster   redis-pvc-cluster-node-0   Bound    nfs-pv6   100Mi      RWX            redis          59s\nredis-cluster   redis-pvc-cluster-node-1   Bound    nfs-pv2   100Mi      RWX            redis          55s\nredis-cluster   redis-pvc-cluster-node-2   Bound    nfs-pv3   100Mi      RWX            redis          33s\nredis-cluster   redis-pvc-cluster-node-3   Bound    nfs-pv5   100Mi      RWX            redis          28s\nredis-cluster   redis-pvc-cluster-node-4   Bound    nfs-pv1   100Mi      RWX            redis          23s\nredis-cluster   redis-pvc-cluster-node-5   Bound    nfs-pv4   100Mi      RWX            redis          19s\n```\n\n### 为集群设置三主三从\n\nredis节点创建完成后，每个redis会获得一个唯一的、不变的域名。Pod失效后，IP可能会变，但是域名不会变。格式为`$(podname).$(service name).$(namespace).svc.cluster.local`即\n\n```shell\ncluster-node-0.redis-cluster-svc.redis-cluster.svc.cluster.local\ncluster-node-1.redis-cluster-svc.redis-cluster.svc.cluster.local\n......\n```\n\nPod节点均启动后我们开始初始化集群\n\n```shell\n# 获取PodIP\nkubectl get pods -n redis-cluster -owide\nNAME             READY   STATUS    RESTARTS   AGE     IP            NODE    NOMINATED NODE   READINESS GATES\ncluster-node-0   1/1     Running   0          2m45s   10.244.1.23   node1   <none>           <none>\ncluster-node-1   1/1     Running   0          2m40s   10.244.2.26   node2   <none>           <none>\ncluster-node-2   1/1     Running   0          2m36s   10.244.1.24   node1   <none>           <none>\ncluster-node-3   1/1     Running   0          2m33s   10.244.2.27   node2   <none>           <none>\ncluster-node-4   1/1     Running   0          2m32s   10.244.1.25   node1   <none>           <none>\ncluster-node-5   1/1     Running   0          2m28s   10.244.2.28   node2   <none>           <none>\n# 获取redis各Pod的IP\nkubectl get pods -l app=redis -n redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 '\n10.244.1.23:6379 10.244.2.26:6379 10.244.1.24:6379 10.244.2.27:6379 10.244.1.25:6379 10.244.2.28:6379\n\n这里特别注意一下：\n上面命令最后一个单引号前面一定要有空格！！\n因为接下来进行Redis Cluster集群初始化的时候，集群节点间的ip+port之间要通过空格隔开。\n\n# 初始化集群\nkubectl exec -it -n redis-cluster cluster-node-0 -- redis-cli --cluster create 10.244.1.23:6379 10.244.2.26:6379 10.244.1.24:6379 10.244.2.27:6379 10.244.1.25:6379 10.244.2.28:6379 --cluster-re\nplicas 1>>> Performing hash slots allocation on 6 nodes...\nMaster[0] -> Slots 0 - 5460\nMaster[1] -> Slots 5461 - 10922\nMaster[2] -> Slots 10923 - 16383\nAdding replica 10.244.1.25:6379 to 10.244.1.23:6379\nAdding replica 10.244.2.28:6379 to 10.244.2.26:6379\nAdding replica 10.244.2.27:6379 to 10.244.1.24:6379\nM: 07cd98b778e813a3d200bf26eb2cee1e77e7dea6 10.244.1.23:6379\n   slots:[0-5460] (5461 slots) master\nM: 7c2c32886acf881f92cb6a30f108edf97af06fb2 10.244.2.26:6379\n   slots:[5461-10922] (5462 slots) master\nM: 0b45a9b75b2452e68e1ae6a327a247f614ee848e 10.244.1.24:6379\n   slots:[10923-16383] (5461 slots) master\nS: 703ca3255705b6227e992f6b8eec5b9302baf558 10.244.2.27:6379\n   replicates 0b45a9b75b2452e68e1ae6a327a247f614ee848e\nS: e56032cb1414ca6a3dd6a58f5dd4e73bd26ab195 10.244.1.25:6379\n   replicates 07cd98b778e813a3d200bf26eb2cee1e77e7dea6\nS: 1ee5a4193548b084b38eb65476911f946ea65f70 10.244.2.28:6379\n   replicates 7c2c32886acf881f92cb6a30f108edf97af06fb2\nCan I set the above configuration? (type 'yes' to accept): yes\n```\n\n### 验证集群\n\n```shell\nkubectl exec -it -n redis-cluster cluster-node-0 -- redis-cli cluster nodes\n1ee5a4193548b084b38eb65476911f946ea65f70 10.244.2.28:6379@16379 slave 7c2c32886acf881f92cb6a30f108edf97af06fb2 0 1649054816013 2 connected\n07cd98b778e813a3d200bf26eb2cee1e77e7dea6 10.244.1.23:6379@16379 myself,master - 0 1649054808000 1 connected 0-5460\n0b45a9b75b2452e68e1ae6a327a247f614ee848e 10.244.1.24:6379@16379 master - 0 1649054815000 3 connected 10923-16383\n7c2c32886acf881f92cb6a30f108edf97af06fb2 10.244.2.26:6379@16379 master - 0 1649054811000 2 connected 5461-10922\n703ca3255705b6227e992f6b8eec5b9302baf558 10.244.2.27:6379@16379 slave 0b45a9b75b2452e68e1ae6a327a247f614ee848e 0 1649054813000 3 connected\ne56032cb1414ca6a3dd6a58f5dd4e73bd26ab195 10.244.1.25:6379@16379 slave 07cd98b778e813a3d200bf26eb2cee1e77e7dea6 0 1649054815004 1 connected\n```\n\n删除Pod查看是否能自动切换主从\n\n```shell\n[root@master ~]# kubectl delete po cluster-node-4 -n redis-cluster\npod \"cluster-node-4\" deleted\n[root@master ~]# kubectl get po -n redis-cluster\nNAME             READY   STATUS              RESTARTS   AGE\ncluster-node-0   1/1     Running             0          9m1s\ncluster-node-1   1/1     Running             0          8m56s\ncluster-node-2   1/1     Running             0          8m52s\ncluster-node-3   1/1     Running             0          8m49s\ncluster-node-4   0/1     ContainerCreating   0          1s\ncluster-node-5   1/1     Running             0          8m44s\n[root@master ~]# kubectl get po -n redis-cluster\nNAME             READY   STATUS    RESTARTS   AGE\ncluster-node-0   1/1     Running   0          9m4s\ncluster-node-1   1/1     Running   0          8m59s\ncluster-node-2   1/1     Running   0          8m55s\ncluster-node-3   1/1     Running   0          8m52s\ncluster-node-4   1/1     Running   0          4s\ncluster-node-5   1/1     Running   0          8m47s\n[root@master ~]# kubectl exec -it -n redis-cluster cluster-node-0 -- redis-cli cluster nodes\n1ee5a4193548b084b38eb65476911f946ea65f70 10.244.2.28:6379@16379 slave 7c2c32886acf881f92cb6a30f108edf97af06fb2 0 1649054884765 2 connected\n07cd98b778e813a3d200bf26eb2cee1e77e7dea6 10.244.1.23:6379@16379 myself,master - 0 1649054883000 1 connected 0-5460\n0b45a9b75b2452e68e1ae6a327a247f614ee848e 10.244.1.24:6379@16379 master - 0 1649054883742 3 connected 10923-16383\n7c2c32886acf881f92cb6a30f108edf97af06fb2 10.244.2.26:6379@16379 master - 0 1649054883000 2 connected 5461-10922\n703ca3255705b6227e992f6b8eec5b9302baf558 10.244.2.27:6379@16379 slave 0b45a9b75b2452e68e1ae6a327a247f614ee848e 0 1649054882000 3 connected\ne56032cb1414ca6a3dd6a58f5dd4e73bd26ab195 10.244.1.26:6379@16379 slave 07cd98b778e813a3d200bf26eb2cee1e77e7dea6 0 1649054880703 1 connected\n[root@master ~]# kubectl get po -n redis-cluster -owide\nNAME             READY   STATUS    RESTARTS   AGE     IP            NODE    NOMINATED NODE   READINESS GATES\ncluster-node-0   1/1     Running   0          9m23s   10.244.1.23   node1   <none>           <none>\ncluster-node-1   1/1     Running   0          9m18s   10.244.2.26   node2   <none>           <none>\ncluster-node-2   1/1     Running   0          9m14s   10.244.1.24   node1   <none>           <none>\ncluster-node-3   1/1     Running   0          9m11s   10.244.2.27   node2   <none>           <none>\ncluster-node-4   1/1     Running   0          23s     10.244.1.26   node1   <none>           <none>\ncluster-node-5   1/1     Running   0          9m6s    10.244.2.28   node2   <none>           <none>\n```\n\n可以看到`cluster-node-4`被删除后，该节点会被自动重建，重建后IP发生了变化。但由于我们设置了自动更新IP的脚本。所以它可以重新加入集群。","categoryId":10,"viewCount":1466,"categoryName":"Kubernetes","author":"球接子","authorAvatar":null,"tagIds":[16],"tagNames":["Kubernetes"]}}