本文已参与好文召集令活动，点击查看：后端、大前端双赛道投稿，2万元奖池等你挑战

1 VPA 简介

VPA 全称 Vertical Pod Autoscaler，即垂直 Pod 自动扩缩容，它根据容器资源使用率自动设置 CPU 和内存的requests，从而允许在节点上进行适当的调度，以便为每个 Pod 提供适当的资源。
它既可以缩小过度请求资源的容器，也可以根据其使用情况随时提升资源不足的容量。
PS: VPA不会改变Pod的资源limits值。

废话不多说，直接上图，看VPA工作流程

2 部署metrics-server

2.1 下载部署清单文件

1	bash复制代码[root@VM-10-48-centos ~]# wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml

2.2 修改components.yaml文件

修改了镜像地址为：scofield/metrics-server:v0.3.7
修改了metrics-server启动参数args

yaml复制代码- name: metrics-server
  image: scofield/metrics-server:v0.3.7
  imagePullPolicy: IfNotPresent
  args:
    - --cert-dir=/tmp
    - --secure-port=4443
    - /metrics-server
    - --kubelet-insecure-tls
    - --kubelet-preferred-address-types=InternalIP

2.3 执行部署

1	bash复制代码[root@VM-10-48-centos ~]# kubectl apply -f components.yaml

2.4 验证

bash复制代码[root@VM-10-48-centos ~]# kubectl get po -n kube-system | grep metrics-server
metrics-server-5b58f4df77-f7nks                          1/1     Running   0          35d

# 能获取要top信息视为成功
[root@VM-10-48-centos ~]# kubectl top nodes
NAME        CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
10.1.2.15   138m         3%     4207Mi          29%       
10.1.2.16   159m         4%     3138Mi          45%       
10.1.2.17   147m         3%     4118Mi          59%       
10.1.50.2   82m          4%     1839Mi          55%

3 部署vertical-pod-autoscaler

3.1 克隆autoscaler项目

1	bash复制代码[root@VM-10-48-centos ~]# git clone https://github.com/kubernetes/autoscaler.git

3.2 修改部署文件

bash复制代码[root@VM-10-48-centos ~]# cd autoscaler/vertical-pod-autoscaler/deploy
admission-controller-deployment.yaml
us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-admission-controller:0.8.0
改为
scofield/vpa-admission-controller:0.8.0

recommender-deployment.yaml
us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-recommender:0.8.0
改为
image: scofield/vpa-recommender:0.8.0

updater-deployment.yaml
us.gcr.io/k8s-artifacts-prod/autoscaling/vpa-updater:0.8.0
改为
scofield/vpa-updater:0.8.0

3.3 部署

bash复制代码[root@VM-10-48-centos ~]# cd autoscaler/vertical-pod-autoscaler
[root@VM-10-48-centos ~]# ./hack/vpa-up.sh
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io created
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io created
clusterrole.rbac.authorization.k8s.io/system:metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:vpa-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:evictioner created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictionter-binding created
serviceaccount/vpa-admission-controller created
clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding created
serviceaccount/vpa-updater created
deployment.apps/vpa-updater created
serviceaccount/vpa-recommender created
deployment.apps/vpa-recommender created
Generating certs for the VPA Admission Controller in /tmp/vpa-certs.
Generating RSA private key, 2048 bit long modulus (2 primes)
............................................................................+++++
.+++++
e is 65537 (0x010001)
Generating RSA private key, 2048 bit long modulus (2 primes)
............+++++
...........................................................................+++++
e is 65537 (0x010001)
Signature ok
subject=CN = vpa-webhook.kube-system.svc
Getting CA Private Key
Uploading certs to the cluster.
secret/vpa-tls-certs created
Deleting /tmp/vpa-certs.
deployment.apps/vpa-admission-controller created
service/vpa-webhook created

这里如果出现错误：ERROR: Failed to create CA certificate for self-signing. If the error is “unknown option -addext”, update your openssl version or deploy VPA from the vpa-release-0.8 branch

需要升级openssl的版本解决

bash复制代码[root@VM-10-48-centos ~]# yum install gcc gcc-c++ -y
[root@VM-10-48-centos ~]# openssl version -a
[root@VM-10-48-centos ~]# wget https://www.openssl.org/source/openssl-1.1.1k.tar.gz && tar zxf openssl-1.1.1k.tar.gz && cd openssl-1.1.1k
[root@VM-10-48-centos ~]# ./config
[root@VM-10-48-centos ~]# make && make install
[root@VM-10-48-centos ~]# mv /usr/local/bin/openssl /usr/local/bin/openssl.bak
[root@VM-10-48-centos ~]# mv apps/openssl /usr/local/bin
[root@VM-10-48-centos ~]# openssl version -a
OpenSSL 1.1.1k  25 Mar 2021 (Library: OpenSSL 1.1.1g FIPS  21 Apr 2020)
built on: Mon Mar 29 23:48:12 2021 UTC
platform: linux-x86_64
options:  bn(64,64) rc4(16x,int) des(int) idea(int) blowfish(ptr) 
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wa,--noexecstack -Wa,--generate-missing-build-notes=yes -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DZLIB -DNDEBUG -DPURIFY -DDEVRANDOM="\"/dev/urandom\""
OPENSSLDIR: "/etc/pki/tls"
ENGINESDIR: "/usr/lib64/engines-1.1"
Seeding source: os-specific

再次执行vertical-pod-autoscaler/pkg/admission-controller/gencerts.sh

3.4 查看结果

可以看到metrics-server和vpa都已经正常运行了

bash复制代码[root@VM-10-48-centos ~]# kubectl get po -n kube-system | grep -E "metrics-server|vpa"
metrics-server-5b58f4df77-f7nks                          1/1     Running   0          35d
vpa-admission-controller-7ff888c959-tvtmk                1/1     Running   0          104m
vpa-recommender-74f69c56cb-zmzwg                         1/1     Running   0          104m
vpa-updater-79b88f9c55-m4xx5                             1/1     Running   0          103m

4 示例

4.1 updateMode: Off

1 首先我们部署一个nginx服务,部署到namespace: vpa中

yaml复制代码apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: vpa
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        resources:
          requests:
            cpu: 100m
            memory: 250Mi

看下结果，正常运行了2个pod

bash复制代码[root@VM-10-48-centos ~]# kubectl get po -n vpa
NAME                     READY   STATUS    RESTARTS   AGE
nginx-59fdffd754-cb5dn   1/1     Running   0          8s
nginx-59fdffd754-cw8d7   1/1     Running   0          9s

2 创建一个NodePort类型的service

bash复制代码[root@VM-10-48-centos ~]# cat svc.yaml 
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: vpa
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx

[root@VM-10-48-centos ~]# kubectl get svc -n vpa | grep nginx
nginx   NodePort   10.255.253.166   <none>        80:30895/TCP   54s

[root@VM-2-16-centos ~]# curl -I 10.1.2.16:30895
HTTP/1.1 200 OK
Server: nginx/1.21.1
Date: Fri, 09 Jul 2021 09:54:58 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 06 Jul 2021 14:59:17 GMT
Connection: keep-alive
ETag: "60e46fc5-264"
Accept-Ranges: bytes

3 创建VPA
这里先使用updateMode: "Off"模式，这种模式仅获取资源推荐，但不更新Pod

bash复制代码[root@VM-10-48-centos ~]# cat nginx-vpa-demo.yaml
apiVersion: autoscaling.k8s.io/v1beta2
kind: VerticalPodAutoscaler
metadata:
  name: nginx-vpa
  namespace: vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: nginx
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
    - containerName: "nginx"
      minAllowed:
        cpu: "250m"
        memory: "100Mi"
      maxAllowed:
        cpu: "2000m"
        memory: "2048Mi"

4 查看部署结果

1
2
3

bash复制代码[root@VM-10-48-centos ~]# kubectl get vpa -n vpa
NAME        MODE   CPU   MEM   PROVIDED   AGE
nginx-vpa   Off                           7s

5 使用describe查看vpa详情，主要关注Container Recommendations

bash复制代码[root@VM-10-48-centos ~]# kubectl describe vpa nginx-vpa -n vpa
Name:         nginx-vpa
Namespace:    vpa
Spec:
  Resource Policy:
    Container Policies:
      Container Name:  nginx
      Max Allowed:
        Cpu:     2000m
        Memory:  2048Mi
      Min Allowed:
        Cpu:     250m
        Memory:  100Mi
  Target Ref:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         nginx
  Update Policy:
    Update Mode:  Off
Status:
  Conditions:
    Last Transition Time:  2021-07-09T09:59:50Z
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  nginx
      Lower Bound:
        Cpu:     250m
        Memory:  262144k
      Target:
        Cpu:     250m
        Memory:  262144k
      Uncapped Target:
        Cpu:     25m
        Memory:  262144k
      Upper Bound:
        Cpu:     670m
        Memory:  700542995

其中

json复制代码Lower Bound:                 下限值
Target:                      推荐值
Upper Bound:                 上限值
Uncapped Target:             如果没有为VPA提供最小或最大边界，则表示目标利用率
上述结果表明，推荐的 Pod 的 CPU 请求为 25m，推荐的内存请求为 262144k 字节。

6 现在对nginx进行压测
执行压测命令

text复制代码[root@VM-10-48-centos ~]# ab -c 100 -n 10000000 http://10.1.2.16:30895/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.1.2.16 (be patient)

Completed 1000000 requests
Completed 2000000 requests
Completed 3000000 requests

7 几分钟后再观察VPA Recommendation变化

bash复制代码[root@VM-10-48-centos ~]# kubectl describe vpa -n vpa nginx-vpa | tail -n 20
  Conditions:
    Last Transition Time:  2021-07-09T09:59:50Z
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  nginx
      Lower Bound:
        Cpu:     250m
        Memory:  262144k
      Target:
        Cpu:     1643m
        Memory:  262144k
      Uncapped Target:
        Cpu:     1643m
        Memory:  262144k
      Upper Bound:
        Cpu:     2
        Memory:  562581530
Events:          <none>

从输出信息可以看出，VPA对Pod给出了推荐值：Cpu: 1643m，因为我们这里设置了updateMode: "Off"，所以不会更新Pod

4.2 updateMode: Auto

1 把updateMode: “Auto”，看看VPA会有什么动作
这里把resources改为：memory: 50Mi，cpu: 100m

bash复制代码[root@VM-10-48-centos ~]# kubectl get po -n vpa
NAME                     READY   STATUS    RESTARTS   AGE
nginx-5594c66dc6-lzs67   1/1     Running   0          26s
nginx-5594c66dc6-zk6h9   1/1     Running   0          21s

2 再次部署vpa，这里VPA部署文件nginx-vpa-demo.yaml只改了updateMode: "Auto"

text复制代码[root@k8s-node001 examples]# cat  nginx-vpa-demo.yaml
apiVersion: autoscaling.k8s.io/v1beta2
kind: VerticalPodAutoscaler
metadata:
  name: nginx-vpa-2
  namespace: vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: nginx
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: "nginx"
      minAllowed:
        cpu: "250m"
        memory: "100Mi"
      maxAllowed:
        cpu: "2000m"
        memory: "2048Mi"

3 再次压测

1	bash复制代码[root@VM-10-48-centos ~]# ab -c 100 -n 10000000 http://10.1.2.16:30895/

4 几分钟后，使用describe查看vpa详情，同样只关注Container Recommendations

bash复制代码[root@VM-10-48-centos ~]# kubectl describe vpa nginx-vpa  -n vpa | tail -n 20
  Conditions:
    Last Transition Time:  2021-07-09T09:59:50Z
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  nginx
      Lower Bound:
        Cpu:     250m
        Memory:  262144k
      Target:
        Cpu:     1643m
        Memory:  262144k
      Uncapped Target:
        Cpu:     1643m
        Memory:  262144k
      Upper Bound:
        Cpu:     2
        Memory:  511550327
Events:          <none>

Target变成了Cpu:1643m ，Memory:262144k

5、来看下event事件

bash复制代码[root@VM-10-48-centos ~]# kubectl get event -n vpa
LAST SEEN   TYPE     REASON                 OBJECT                        MESSAGE
38s         Normal   Scheduled              pod/nginx-5594c66dc6-d8d6h    Successfully assigned vpa/nginx-5594c66dc6-d8d6h to 10.1.2.16
38s         Normal   Pulling                pod/nginx-5594c66dc6-d8d6h    Pulling image "nginx"
37s         Normal   Pulled                 pod/nginx-5594c66dc6-d8d6h    Successfully pulled image "nginx"
37s         Normal   Created                pod/nginx-5594c66dc6-d8d6h    Created container nginx
37s         Normal   Started                pod/nginx-5594c66dc6-d8d6h    Started container nginx
3m10s       Normal   Scheduled              pod/nginx-5594c66dc6-lzs67    Successfully assigned vpa/nginx-5594c66dc6-lzs67 to 10.1.2.15
3m9s        Normal   Pulling                pod/nginx-5594c66dc6-lzs67    Pulling image "nginx"
3m5s        Normal   Pulled                 pod/nginx-5594c66dc6-lzs67    Successfully pulled image "nginx"
3m5s        Normal   Created                pod/nginx-5594c66dc6-lzs67    Created container nginx
3m5s        Normal   Started                pod/nginx-5594c66dc6-lzs67    Started container nginx
99s         Normal   EvictedByVPA           pod/nginx-5594c66dc6-lzs67    Pod was evicted by VPA Updater to apply resource recommendation.
99s         Normal   Killing                pod/nginx-5594c66dc6-lzs67    Stopping container nginx
98s         Normal   Scheduled              pod/nginx-5594c66dc6-tdmnh    Successfully assigned vpa/nginx-5594c66dc6-tdmnh to 10.1.2.15
98s         Normal   Pulling                pod/nginx-5594c66dc6-tdmnh    Pulling image "nginx"
97s         Normal   Pulled                 pod/nginx-5594c66dc6-tdmnh    Successfully pulled image "nginx"
97s         Normal   Created                pod/nginx-5594c66dc6-tdmnh    Created container nginx
97s         Normal   Started                pod/nginx-5594c66dc6-tdmnh    Started container nginx
3m5s        Normal   Scheduled              pod/nginx-5594c66dc6-zk6h9    Successfully assigned vpa/nginx-5594c66dc6-zk6h9 to 10.1.2.17
3m4s        Normal   Pulling                pod/nginx-5594c66dc6-zk6h9    Pulling image "nginx"
3m          Normal   Pulled                 pod/nginx-5594c66dc6-zk6h9    Successfully pulled image "nginx"
2m59s       Normal   Created                pod/nginx-5594c66dc6-zk6h9    Created container nginx
2m59s       Normal   Started                pod/nginx-5594c66dc6-zk6h9    Started container nginx
39s         Normal   EvictedByVPA           pod/nginx-5594c66dc6-zk6h9    Pod was evicted by VPA Updater to apply resource recommendation.
39s         Normal   Killing                pod/nginx-5594c66dc6-zk6h9    Stopping container nginx
3m10s       Normal   SuccessfulCreate       replicaset/nginx-5594c66dc6   Created pod: nginx-5594c66dc6-lzs67
3m5s        Normal   SuccessfulCreate       replicaset/nginx-5594c66dc6   Created pod: nginx-5594c66dc6-zk6h9
99s         Normal   SuccessfulCreate       replicaset/nginx-5594c66dc6   Created pod: nginx-5594c66dc6-tdmnh
38s         Normal   SuccessfulCreate       replicaset/nginx-5594c66dc6   Created pod: nginx-5594c66dc6-d8d6h
35m         Normal   Scheduled              pod/nginx-59fdffd754-cb5dn    Successfully assigned vpa/nginx-59fdffd754-cb5dn to 10.1.2.16
35m         Normal   Pulling                pod/nginx-59fdffd754-cb5dn    Pulling image "nginx"
35m         Normal   Pulled                 pod/nginx-59fdffd754-cb5dn    Successfully pulled image "nginx"
35m         Normal   Created                pod/nginx-59fdffd754-cb5dn    Created container nginx
35m         Normal   Started                pod/nginx-59fdffd754-cb5dn    Started container nginx
3m5s        Normal   Killing                pod/nginx-59fdffd754-cb5dn    Stopping container nginx
35m         Normal   Scheduled              pod/nginx-59fdffd754-cw8d7    Successfully assigned vpa/nginx-59fdffd754-cw8d7 to 10.1.2.16
35m         Normal   Pulling                pod/nginx-59fdffd754-cw8d7    Pulling image "nginx"
35m         Normal   Pulled                 pod/nginx-59fdffd754-cw8d7    Successfully pulled image "nginx"
35m         Normal   Created                pod/nginx-59fdffd754-cw8d7    Created container nginx
35m         Normal   Started                pod/nginx-59fdffd754-cw8d7    Started container nginx
2m58s       Normal   Killing                pod/nginx-59fdffd754-cw8d7    Stopping container nginx
35m         Normal   SuccessfulCreate       replicaset/nginx-59fdffd754   Created pod: nginx-59fdffd754-cw8d7
35m         Normal   SuccessfulCreate       replicaset/nginx-59fdffd754   Created pod: nginx-59fdffd754-cb5dn
3m5s        Normal   SuccessfulDelete       replicaset/nginx-59fdffd754   Deleted pod: nginx-59fdffd754-cb5dn
2m58s       Normal   SuccessfulDelete       replicaset/nginx-59fdffd754   Deleted pod: nginx-59fdffd754-cw8d7
35m         Normal   ScalingReplicaSet      deployment/nginx              Scaled up replica set nginx-59fdffd754 to 2
34m         Normal   EnsuringService        service/nginx                 Deleted Loadbalancer
34m         Normal   EnsureServiceSuccess   service/nginx                 Service Sync Success. RetrunCode: S2000
3m10s       Normal   ScalingReplicaSet      deployment/nginx              Scaled up replica set nginx-5594c66dc6 to 1
3m5s        Normal   ScalingReplicaSet      deployment/nginx              Scaled down replica set nginx-59fdffd754 to 1
3m5s        Normal   ScalingReplicaSet      deployment/nginx              Scaled up replica set nginx-5594c66dc6 to 2
2m58s       Normal   ScalingReplicaSet      deployment/nginx              Scaled down replica set nginx-59fdffd754 to 0

从输出信息可以了解到，vpa执行了EvictedByVPA，自动停掉了nginx，然后使用 VPA推荐的资源启动了新的nginx
，我们查看下nginx的pod可以得到确认

bash复制代码[root@VM-10-48-centos ~]# kubectl describe po -n vpa nginx-5594c66dc6-d8d6h
Name:         nginx-5594c66dc6-d8d6h
Namespace:    vpa
Priority:     0
Node:         10.1.2.16/10.1.2.16
Start Time:   Fri, 09 Jul 2021 18:09:26 +0800
Labels:       app=nginx
              pod-template-hash=5594c66dc6
Annotations:  tke.cloud.tencent.com/networks-status:
                [{
                    "name": "tke-bridge",
                    "interface": "eth0",
                    "ips": [
                        "10.252.1.50"
                    ],
                    "mac": "e6:38:26:0b:c5:97",
                    "default": true,
                    "dns": {}
                }]
              vpaObservedContainers: nginx
              vpaUpdates: Pod resources updated by nginx-vpa: container 0: cpu request, memory request
Status:       Running
IP:           10.252.1.50
IPs:
  IP:           10.252.1.50
Controlled By:  ReplicaSet/nginx-5594c66dc6
Containers:
  nginx:
    Container ID:   docker://42e45f5f122ba658e293395d78a073cfe51534c773f9419a179830fd6d1698ea
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:8df46d7414eda82c2a8c9c50926545293811ae59f977825845dda7d558b4125b
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 09 Jul 2021 18:09:27 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        1643m
      memory:     262144k
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-m2j2z (ro)

看重点Requests：cpu: 1643m，memory: 262144k
再回头看看部署文件

1
2
3

yaml复制代码requests:
  cpu: 100m
  memory: 50Mi

现在可以知道VPA做了哪些事了吧。当然，随着服务的负载的变化，VPA的推荐之也会不断变化。当目前运行的pod的资源达不到VPA的推荐值，就会执行pod驱逐，重新部署新的足够资源的服务。

4.3 VPA使用限制

不能与HPA（Horizontal Pod Autoscaler ）一起使用
Pod比如使用副本控制器，例如属于Deployment或者StatefulSet

4.4 VPA有啥好处

Pod 资源用其所需，所以集群节点使用效率高。
Pod 会被安排到具有适当可用资源的节点上。
不必运行基准测试任务来确定 CPU 和内存请求的合适值。
VPA 可以随时调整 CPU 和内存请求，无需人为操作，因此可以减少维护时间。

本文转载自: 掘金

开发者博客 – 和开发相关的这里全都有