k8s springboot 生产实践(高可用部署、基于qps动态扩缩容、prometheus监控)

k8s springboot 生产实践(高可用部署、基于qps动态扩缩容、prometheus监控)

以下均为亲自实践并应用 概要 1、k8s sprintboot eureka高可用配置、部署、优化 2、k8s sprintboot 微服务高可用配置、部署、优化 3、springboot安装prometheus依赖并获取metric 4、helm安装prometheus operator、prometheus adapter(custom metric) 5、配置promet...

以下均为亲自实践并应用

概要

1、k8s sprintboot eureka高可用配置、部署、优化
2、k8s sprintboot 微服务高可用配置、部署、优化
3、springboot安装prometheus依赖并获取metric
4、helm安装prometheus operator、prometheus adapter(custom metric)
5、配置prometheus及adapter获取应用qps
6、配置k8s hpa(HorizontalPodAutoscaler)
7、配置grafana展示监控数据

效果展示

1、k8s sprintboot eureka高可用配置、部署、优化

简介:eureka做集群,需要eureka互相注册,为方便注册,使用statefulSet方式部署
eureka server配置:

eureka:
  server:
    enable-self-preservation: false
    eviction-interval-timer-in-ms: 5000
  client:
    fetch-registry: true  
    register-with-eureka: true #互相注册
    serviceUrl:
      defaultZone: ${EUREKA_URL}
  instance:
    prefer-ip-address: true
    lease-renewal-interval-in-seconds: 3
    lease-expiration-duration-in-seconds: 10

eureka_url:将多实例地址均填写,方便配置

EUREKA_URL="http://eureka-0.eureka.default.svc.cluster.local:8761/eureka/,http://eureka-1.eureka.default.svc.cluster.local:8761/eureka/"

eureka statefulSet配置:

apiVersion:  apps/v1
kind: StatefulSet
metadata:
  labels:
    app: eureka
  name: eureka
  namespace: default
spec:
  replicas: 2
  serviceName: eureka
  podManagementPolicy: "Parallel"
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: eureka
  template:
    metadata:
      labels:
        app: eureka
      annotations:   #用于prometheus收集数据,需micrometer-registry-prometheus插件
        prometheus.io/path: "/actuator/prometheus"
        prometheus.io/port: "8761"
        prometheus.io/scrape: "true"
    spec:
      containers:
      - image: eureka:latest              #镜像名
        imagePullPolicy: IfNotPresent
        name: eureka
        envFrom:
        - configMapRef:
            name: app-configmap           #环境变量
        env:
        - name: MEMORY
          value: "800"
        resources:
          limits:
            cpu: "0"
            memory: 1000Mi
          requests:
            cpu: "0"
            memory: 1000Mi
        readinessProbe:
          tcpSocket:
            port: 8761
          failureThreshold: 1
          initialDelaySeconds: 20
        livenessProbe:
          tcpSocket:
            port: 8761
          failureThreshold: 1
          initialDelaySeconds: 60
        volumeMounts:
        - mountPath: /logs
          name: log-vol
      volumes:
      - name: log-vol
        nfs:
          path: /logs/eureka         #多实例日志可统一在此次查看
          server: 127.0.0.1           #nfs地址
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: registry-harbor
      restartPolicy: Always

2、k8s sprintboot 微服务高可用配置、部署、优化

这边有思考点,client以什么地址注册eureka?
a、以deployment直接部署,默认情况会将容器名称注册到eureka,k8s不识别deployment方式容器名称域名,导致无法访问
b、使用statefulSet部署,通过pod名称注册,并且可以识别域名,以pod名称注册并使用,感觉完美,然而需要弹性扩缩容时候,pod缩容,注册信息不会立马从eureka server去除,有延迟,会导致请求到这个销毁的pod上,error!(如果不考虑hpa这方式ok)
c、使用service注册,通过endpoint分发请求,pod销毁后pod ip会里面从endpoint里去除,不会出现缩容时候请求到销毁pod上,扩容时候,服务没完全起来请求转发到此pod上会出现问题,不过可以通过健康检查解决
最终选择c方案:
client配置如下:
HOSTNAME即k8s service名称,注册时候会将service注册到eureka

#服务注册相关配置
eureka:
  client:
    serviceUrl:
      defaultZone: ${EUREKA_URL}
  instance:
    prefer-ip-address: true
    instance-id: ${HOSTNAME}:${server.port}
    ip-address: ${HOSTNAME}

deployment和service配置

apiVersion:  apps/v1
kind: Deployment
metadata:
  labels:
    app: client 
  name: client 
  namespace: default
spec:
  replicas: 2
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: client 
  template:
    metadata:
      labels:
        app: client 
      annotations:
        prometheus.io/path: "/actuator/prometheus"
        prometheus.io/port: "8024"
        prometheus.io/scrape: "true"
    spec:
      containers:
      - image: client:latest
        imagePullPolicy: IfNotPresent
        name: client 
        envFrom:
        - configMapRef:
            name: client-configmap
        env:
        - name: MEMORY
          value: "1300"
        - name: HOSTNAME
          value: client 
        resources:
          limits:
            cpu: "0"
            memory: 1500Mi
          requests:
            cpu: "0"
            memory: 1500Mi
        readinessProbe:
          tcpSocket:
            port: 8024
          failureThreshold: 1
          initialDelaySeconds: 60
        livenessProbe:
          tcpSocket:
            port: 8024
          failureThreshold: 1
          initialDelaySeconds: 90
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: registry-harbor
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: client 
  namespace: default
  labels:
    app: client  
spec:
  ports:
  - name: http-8024
    port: 8024
    protocol: TCP
    targetPort: 8024
  selector:
    app: client  

3、springboot安装prometheus依赖并获取metric

高可用部署完成后就需要弹性扩缩容, 需要获取qps数据,这里用的是prometheus,直接贴配置

增加pom依赖:

        io.micrometermicrometer-registry-prometheus

增加配置文件:

management:
  endpoints:
    web:
      exposure:
        include: '*'
  metrics:
    tags:
      application: ${spring.application.name}

主函数增加bean

import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryCustomizer;
import org.springframework.context.annotation.Bean;

public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
    @Bean
    MeterRegistryCustomizer configurer(
            @Value("${spring.application.name}") String applicationName) {
        return (registry) -> registry.config().commonTags("application", applicationName);
    }
}

配置完成后访问:http:localhost:8024/actuator/prometheus,返回数据如下(重点关注下http_server_requests_seconds_count,后面会用到)

# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
# TYPE jvm_memory_max_bytes gauge
jvm_memory_max_bytes{application="portal-eureka",area="heap",id="Par Survivor Space",} 2.7918336E7
jvm_memory_max_bytes{application="portal-eureka",area="nonheap",id="Compressed Class Space",} 1.073741824E9
jvm_memory_max_bytes{application="portal-eureka",area="heap",id="Tenured Gen",} 5.59284224E8
jvm_memory_max_bytes{application="portal-eureka",area="nonheap",id="Metaspace",} -1.0
jvm_memory_max_bytes{application="portal-eureka",area="heap",id="Par Eden Space",} 2.23739904E8
jvm_memory_max_bytes{application="portal-eureka",area="nonheap",id="Code Cache",} 2.5165824E8
# HELP logback_events_total Number of error level events that made it to the logs
# TYPE logback_events_total counter
logback_events_total{application="portal-eureka",level="info",} 804.0
http_server_requests_seconds_count{application="portal-eureka",exception="None",method="POST",outcome="SUCCESS",status="204",uri="root",} 8.0
http_server_requests_seconds_sum{application="portal-eureka",exception="None",method="POST",outcome="SUCCESS",status="204",uri="root",} 0.29997471
http_server_requests_seconds_count{application="portal-eureka",exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 114.0
...

4、helm安装prometheus operator、prometheus adapter(custom metric)

a、下载prometheus-operator

helm fetch stable/prometheus-operator --version 8.13.0

b、解压

tar -zxvf prometheus-operator-8.13.3.tgz

c、修改values.yaml文件,增加采集pod job(operator默认没有搜集kubernetes-pod的配置)

vim prometheus-operator/values.yaml

增加

    additionalScrapeConfigs: 
      - job_name: 'kubernetes-pods'
        honor_labels: false
        kubernetes_sd_configs:
        - role: pod
        tls_config:
          insecure_skip_verify: true
        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: pod
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::d+)?;(d+)
          replacement: $1:$2
          target_label: __address__
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (.+)

d、安装

helm install -n prometheus-operator prometheus-operator --namespace monitor

e、暴露端口

kubectl edit svc prometheus-operator-prometheus -nmonitor #增加externalIPs或者nodeport

f、安装stable/prometheus-adapter

helm fetch stable/prometheus-adapter --version 2.3.1

解压

tar -zxvf prometheus-adapter-2.3.1.tgz

修改prometheus server配置

vim prometheus-adapter/values.yaml #将prometheus.url替换为http://prometheus-operator-prometheus.monitor.svc

安装

helm install -n prometheus-adapter prometheus-adapter   --namespace monitor

查看api

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

5、配置prometheus及adapter获取应用qps

a、增加adapter规则

kubectl edit cm -nmonitor prometheus-adapter

增加规则,将http_server_requests_seconds_count以每5分钟为时间段计算为qps,metric名称为http_server_requests_seconds

    - seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
      seriesFilters:
      - isNot: .*_seconds_total
      resources:
        template: >
      name:
        matches: ^(.*)_count$
        as: ""
      metricsQuery: sum(rate(>{>}[5m])) by (>)

b、重启adapter

kubectl -nmonitor delete pod prometheus-adapter-...

c、查看自定义metric qps

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_server_requests_seconds"

6、配置k8s hpa(HorizontalPodAutoscaler)

直接贴配置:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: client
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: client
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_server_requests_seconds
      target:
        type: AverageValue
        averageValue: 50
kubectl get hpa #查看hpa是否获取到metric

7、配置grafana展示监控数据

展示应用qps sql(不包含/actuator/prometheus地址)

sum(rate(http_server_requests_seconds_count{job="kubernetes-pods",uri!="/actuator/prometheus"  }[5m])) by (application)
image.png

dashboard模板地址:https://grafana.com/grafana/dashboards?dataSource=prometheus&direction=asc&orderBy=name&search=spring

推荐Spring Boot 2.1 Statistics模板,导入方法如下图,复制id或者json即可

image.png
0

评论0

鱼翔浅底,鹰击长空,驼走大漠
没有账号? 注册  忘记密码?