當前位置：首頁 > 编程资源 > 编程问答 >内容正文

编程问答

如何自定义hpa metric server

發布時間：2024/3/12 编程问答 27 豆豆

生活随笔收集整理的這篇文章主要介紹了如何自定义hpa metric server 小編覺得挺不錯的,現在分享給大家,幫大家做個參考.

1. custom-metrics-apiserver簡介
2. 定制自己的metric server
- 2.1 代碼部署和編譯
- 2.2 創建 Sv and APIService
- 2.3 system:anonymous授權
3. 創建hpa驗證是否成功
4. 追蹤整個過程
5. 總結

本章重點： 如何基于 custom-metrics-apiserver 項目，打造自己的 metric server

1. custom-metrics-apiserver簡介

項目地址： https://github.com/kubernetes-sigs/custom-metrics-apiserver/tree/master

自定義metric server，具體來說需要做以下幾個事情：

（1）實現 custom-metrics-apiserver 的三個接口，如下：

type CustomMetricsProvider interface {// 定義metric。例如 pod_cpu_used_1mListAllMetrics() []CustomMetricInfo// 如何根據 metric的信息，得到具體的值GetMetricByName(name types.NamespacedName, info CustomMetricInfo) (*custom_metrics.MetricValue, error)// 如何根據 metric selector的信息，得到具體的值GetMetricBySelector(namespace string, selector labels.Selector, info CustomMetricInfo) (*custom_metrics.MetricValueList, error) }

GetMetricBySelectorm, GetMetricByName 在reststorage.go被使用。

https://github.com/kubernetes-sigs/custom-metrics-apiserver/blob/master/pkg/registry/custom_metrics/reststorage.go

restful接口在installer.go中被定義。

https://github.com/kubernetes-sigs/custom-metrics-apiserver/blob/master/pkg/apiserver/installer/installer.go

總的來說，可以認為

（1）基于custom-metrics-apiserver這個項目，你只要實現上述三個接口就行。其他的事情這個包在你new provider的時候都自動實現了。

（2）ListAllMetrics 注冊了所有的Metric，讓api-server 知道有哪些自定義metric。

（3）GetMetricByName， GetMetricBySelector 都是返回具體的Metric數據。

（4）一般api server都是調用GetMetricBySelector，因為hpa的對象基本都是deploy, GetMetricBySelector會循環調用GetMetricByName取得deploy所有pod的metric信息。

2. 定制自己的metric server

2.1 代碼部署和編譯

這里我做了如下的修改。對于metric server而言，無論訪問什么metric，都返回10。

func (p *monitorProvider) GetMetricByName(name types.NamespacedName,info provider.CustomMetricInfo,metricSelector labels.Selector, ) (*custom_metrics.MetricValue, error) {ref, err := helpers.ReferenceFor(p.mapper, name, info)if err != nil {return nil, err}return &custom_metrics.MetricValue{DescribedObject: ref,// MetricName: ? ? info.Metric,Metric: custom_metrics.MetricIdentifier{Name: info.Metric,},Timestamp: metav1.Time{time.Unix(int64(10), 0)},Value: ? ? *resource.NewMilliQuantity(int64(10*1000.0), resource.DecimalSI),}, nil }

更詳細的可以參考: https://github.com/kubernetes-sigs/custom-metrics-apiserver/tree/master/test-adapter

編譯生成自己的鏡像：zoux/hpa:v1。然后生成一下的deployment。

apiVersion: apps/v1 kind: Deployment metadata:labels:app: kube-hpaname: kube-hpanamespace: kube-system spec:replicas: 1selector:matchLabels:app: kube-hpatemplate:metadata:labels:app: kube-hpaname: kube-hpaspec:hostNetwork: truecontainers:- name: kube-hpaimage: zoux/hpa:v1imagePullPolicy: IfNotPresentcommand:- /metric-serverargs:- --master-url=XXX- --kube-config=/pkc/config- --tls-private-key-file=/pkc/server-key.pem- --secure-port=9997- --v=10ports:- containerPort: 9997resources:limits:cpu: 2memory: 2048Mirequests:cpu: 0.5memory: 500MivolumeMounts:- name: pkcmountPath: /pkcreadOnly: truevolumes:- name: pkchostPath:path: /opt/kubernetes/ssl

驗證部署成功

root@k8s-master:~/testyaml/hpa# kubectl get pod -n kube-system -o wide NAME ? ? ? ? ? ? ? ? ? ? ? READY ? STATUS ? RESTARTS ? AGE ? ? IP ? ? ? ? ? NODE ? ? ? ? NOMINATED NODE ? READINESS GATES kube-hpa-84c884f994-gd5fl ? 1/1 ? ? Running ? 0 ? ? ? ? 3d13h ? 192.168.0.5 ? 192.168.0.5 ? <none> ? ? ? ? ? <none>

2.2 創建 Sv and APIService

上面雖然部署成功了，但是apiserver還是訪問不到。

k8s-master:~/testyaml/hpa# kubectl get --raw "/apis/custom.metrics.k8s.io/v1 Error from server (NotFound): the server could not find the requested resource

原因在于，apiserver不知道如何找到kube-hpa-84c884f994-gd5fl這個pod進行訪問。所以需要創建下面的svc和apiserver。

root@k8s-master:~/testyaml/hpa# cat tls.yaml apiVersion: v1 kind: Service metadata:name: kube-hpanamespace: kube-system spec:clusterIP: Noneports:- name: https-hpa-dont-edit-itport: 9997targetPort: 9997selector:app: kube-hpa --- apiVersion: apiregistration.k8s.io/v1beta1 kind: APIService metadata:name: v1beta1.custom.metrics.k8s.io spec:service:name: kube-hpanamespace: kube-systemport: 9997group: custom.metrics.k8s.ioversion: v1beta1insecureSkipTLSVerify: truegroupPriorityMinimum: 100versionPriority: 100

創建完成，驗證是否成功：

root@k8s-master:~/testyaml/hpa# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" {"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[{"name":"pods/pod_cpu_used_1m","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/pod_cpu_used_5m","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/container_cpu_used_1m","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":...}

如果報錯。查看該apiserver哪里報錯了

root@k8s-master:~/testyaml/hpa# kubectl get APIService v1beta1.custom.metrics.k8s.io -oyaml apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata:creationTimestamp: "2021-06-13T13:22:01Z"name: v1beta1.custom.metrics.k8s.ioresourceVersion: "1590641"selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.custom.metrics.k8s.iouid: d488d6a8-7e79-4311-a1e9-0b12e4591375 spec:group: custom.metrics.k8s.iogroupPriorityMinimum: 100insecureSkipTLSVerify: trueservice:name: kube-hpanamespace: kube-systemport: 9997version: v1beta1versionPriority: 100 status:conditions:- lastTransitionTime: "2021-06-13T13:42:17Z"message: all checks passedreason: Passedstatus: "True"type: Available

或者直接curl訪問：

curl -k https://nodeip:9997/apis/custom.metrics.k8s.io/v1beta1

2.3 system:anonymous授權

如果沒有出現類似問題，這一步直接跳過

又是會出現如下的錯誤或者上述的APIService沒有運行成功。都是因為system:anonymous權限不夠

annotations:autoscaling.alpha.kubernetes.io/conditions: '[{"type":"AbleToScale","status":"True","lastTransitionTime":"2021-06-13T13:33:12Z","reason":"SucceededGetScale","message":"theHPA controller was able to get the target''s current scale"},{"type":"ScalingActive","status":"False","lastTransitionTime":"2021-06-13T13:33:12Z","reason":"FailedGetPodsMetric","message":"theHPA was unable to compute the replica count: unable to get metric pod_cpu_usage_for_limit_1m:unable to fetch metrics from custom metrics API: pods.custom.metrics.k8s.io\"*\" is forbidden: User \"system:anonymous\" cannot get resource \"pods/pod_cpu_usage_for_limit_1m\"in API group \"custom.metrics.k8s.io\" in the namespace \"default\""}]'autoscaling.alpha.kubernetes.io/metrics: '[{"type":"Pods","pods":{"metricName":"pod_cpu_usage_for_limit_1m","targetAverageValue":"60"}}]'metric-containerName: zx-hpacreationTimestamp: "2021-06-13T13:32:56Z"name: nginx-hpa-zx-1namespace: defaultresourceVersion: "1589301"selfLink: /apis/autoscaling/v1/namespaces/default/horizontalpodautoscalers/

這是可以直接綁定clusterrole https://github.com/kubernetes-sigs/metrics-server/issues/81

我這里是直接給了 cluster-admin 權限，實際情況可以按照需求賦權。

kubectl create clusterrolebinding anonymous-role-binding --clusterrole=cluster-admin --user=system:anonymous

3. 創建hpa驗證是否成功

可以看出來都是10

root@k8s-master:~/testyaml/hpa# kubectl get hpa NAME ? ? ? ? ? ? REFERENCE ? ? ? ? ? TARGETS ? MINPODS ? MAXPODS ? REPLICAS ? AGE nginx-hpa-zx-1 ? Deployment/zx-hpa ? 10/60 ? ? 1 ? ? ? ? 3 ? ? ? ? 3 ? ? ? ? 9m55s root@k8s-master:~/testyaml/hpa# kubectl get hpa NAME ? ? ? ? ? ? REFERENCE ? ? ? ? ? TARGETS ? MINPODS ? MAXPODS ? REPLICAS ? AGE nginx-hpa-zx-1 ? Deployment/zx-hpa ? 10/60 ? ? 1 ? ? ? ? 3 ? ? ? ? 3 ? ? ? ? 9m57s

4. 追蹤整個過程

第一步 Kcm（hpa controller）發送的請求。

I0613 23:12:36.498740 ? 9879 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/pod_aa_100m?labelSelector=app%3Dzx-hpa-test: (35.302304ms) 200 [kube-controller-manager/v1.17.4 (linux/amd64) kubernetes/8d8aa39/horizontal-pod-autoscaler 192.168.0.4:42750]

要在url里使用不安全字符，就需要使用轉義。

%2A = *

%3D = =（等號）

第二步 apiserver進行了 url轉換。

kcm訪問的是： /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/pod_aa_100m?labelSelector=app%3Dzx-hpa-test

但是由于第二步創建了 Sv and APIService。所以訪問這個url會被轉換為：

https://192.168.0.5:9997/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/pod_aa_100m?labelSelector=app%3Dzx-hpa-test

192.168.0.5是pod kube-hpa-84c884f994-gd5fl 所在的節點ip也是podia(hostNetwork模式)。 9997是定義的端口。

第三步： 訪問 https://192.168.0.5:9997/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/pod_aa_100m?labelSelector=app%3Dzx-hpa-test

直接在master節點上（masterip=192.168.0.4）通過curl模擬

root@k8s-master:~/testyaml/hpa# curl -k https://192.168.0.5:9997/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/pod_aa_100m?labelSelector=app%3Dzx-hpa-test {"kind": "MetricValueList","apiVersion": "custom.metrics.k8s.io/v1beta1","metadata": {"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/pod_aa_100m"},"items": [{"describedObject": {"kind": "Pod","namespace": "default","name": "zx-hpa-7b56cddd95-5j6r4|","apiVersion": "/v1"},"metricName": "pod_aa_100m","timestamp": "1970-01-01T00:00:10Z","value": "10","selector": null},{"describedObject": {"kind": "Pod","namespace": "default","name": "zx-hpa-7b56cddd95-lthbz|","apiVersion": "/v1"},"metricName": "pod_aa_100m","timestamp": "1970-01-01T00:00:10Z","value": "10","selector": null},{"describedObject": {"kind": "Pod","namespace": "default","name": "zx-hpa-7b56cddd95-n9ft9|","apiVersion": "/v1"},"metricName": "pod_aa_100m","timestamp": "1970-01-01T00:00:10Z","value": "10","selector": null}] }

5. 總結

（1）如何定制自己的metric-server，包括代碼編寫和環境搭建

（2）Kubernetes 里的 Custom Metrics 機制，也是借助 Aggregator APIServer 擴展機制來實現的。這里的具體原理是，當你把 Custom Metrics APIServer 啟動之后，Kubernetes 里就會出現一個叫作custom.metrics.k8s.io的 API。而當你訪問這個 URL 時，Aggregator 就會把你的請求轉發給 Custom Metrics APIServer 。

這里一定要注意： kube-apiserver啟動參數一定要包含： -enable-swagger-ui=true

（3）ListAllMetrics()并沒有將metric注冊到apiserver。 apiserver并沒有對metric進行驗證。上文中，我metric server的ListAllMetrics()并沒有注冊 pod_aa_100m這個metric，但是可以正常使用。

原因：apiserver并沒有進行驗證，apiserver只進行url轉發，如果有返回數據，apiserver就認為這個metric是正確的。所以這一點可以用來自定義metric。

總結

以上是生活随笔為你收集整理的如何自定义hpa metric server的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。