k8s对接云实现自动扩容pod

发表于 | 分类于 | 浏览 次

介绍

在之前的文章我介绍了下 Custom Metric 怎么实现自动扩容的。k8s基于自定义指标实现自动扩容

实际上Kubernetes定义了三种不同的监控数据接口，分别是Resource Metric，Custom Metric以及External Metric。

一般来说Resource Metric是通过metrics-server采集；

Custom Metric是通过prometheus来实现自定义扩容。

External Metric就是针对云场景的了，比方说通过获取slb最大连接数来实现自动扩容。

下面我来说下具体怎么实现的。

部署

安装alibaba-cloud-metrics-adapter，以下是yaml文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: alibaba-cloud-metrics-adapter
  name: alibaba-cloud-metrics-adapter
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alibaba-cloud-metrics-adapter
  template:
    metadata:
      labels:
        app: alibaba-cloud-metrics-adapter
      name: alibaba-cloud-metrics-adapter
    spec:
      serviceAccountName: admin
      containers:
      - name: alibaba-cloud-metrics-adapter
        image: registry.cn-beijing.aliyuncs.com/acs/alibaba-cloud-metrics-adapter-amd64:v0.2.0-alpha-e8f8c17f
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 443
          name: https
        - containerPort: 8080
          name: http
        volumeMounts:
        - mountPath: /tmp
          name: temp-vol
        - name: tz-config
          mountPath: /etc/localtime
          readOnly: true
      volumes:
      - name: temp-vol
        emptyDir: {}
      - name: tz-config
        hostPath:
          path: /etc/localtime
---
apiVersion: v1
kind: Service
metadata:
  name: alibaba-cloud-metrics-adapter
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    targetPort: 443
  - name: http
    port: 80
    targetPort: 8080
  selector:
    app: alibaba-cloud-metrics-adapter
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.external.metrics.k8s.io
spec:
  service:
    name: alibaba-cloud-metrics-adapter
    namespace: kube-system
  group: external.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-metrics-server-resources
rules:
- apiGroups:
  - external.metrics.k8s.io
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-metrics-resource-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-metrics-server-resources
subjects:
- kind: ServiceAccount
  name: horizontal-pod-autoscaler
  namespace: kube-system

可以使用下面的命令来检测是否生效了： 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
# kubectl get --raw="/apis/external.metrics.k8s.io/v1beta1" | jq
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "sls_ingress_qps",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "slb_l4_connection_utilization",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "slb_l7_qps",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "ahas_sentinel_total_qps",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "ahas_sentinel_avg_rt",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "k8s_workload_cpu_util",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "k8s_workload_memory_request",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "k8s_workload_memory_cache",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    。。。。。。。。。

简单说下各个指标的含义，方便之后去选择哪个指标去自动扩容。

slb_l4_traffic_rx 每秒流入

slb_l4_packet_tx 每秒流入的数据包数

slb_l4_active_connection 活动连接

slb_l4_max_connection 最大连接数

slb_l7_qps QPS

slb_l7_status_2xx 2xx个请求（每秒）

slb_l7_upstream_4xx 上游服务4xx请求（每秒）

sls_ingress_qps 特定入口路由的QPS

sls_ingress_inflow 入口流入带宽

k8s_workload_memory_usage 内存使用情况

k8s_workload_memory_rss rss

扩容例子

根据slb_l4_active_connection这个指标，实现自动扩容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
apiVersion: apps/v1beta2 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment-basic
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9 # replace it with your exactly <image_name:tags>
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: LoadBalancer
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: slb-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta2
    kind: Deployment
    name: nginx-deployment-basic
  minReplicas: 5
  maxReplicas: 10
  metrics:
    - type: External
      external:
        metric:
          name: slb_l4_active_connection
          selector:
            matchLabels:
              # slb.instance.id: "lb-2ze2locy5fk8at1cfx47y"
              slb.instance.id: ""
              # slb.instance.port: "80"
              slb.instance.port: ""
        target:
          type: Value
          value: 100

这样就实现了通过External Metric自动扩容。

参考链接

1
https://github.com/AliyunContainerService/alibaba-cloud-metrics-adapter
Donate