Config management using Kustomize

发表于 更新于
20210525113423

WHY

我们通过一系列的yaml来Config application deployed on Kubernetes.

我们的每个application往往需要一组K8s resource, 即一组yaml files.

问题, 对于Kubernetes:

  • 我们如何高效管理不同application的manifest: “分治法”, 不同application不同的folder, 之下是其group of manifest.
  • 不同环境的manifest, 大体相似，细微config差异，如何reuse manifest, 但区别不同的环境?

在Kustomize之前，已经有很多tool来管理不同环境的manifest, 可以看出对其的需求之强, 参见Config management design doc

在Kustomize之前，我们组用ktmpl来apply manifest到不同环境; 机制是 Parameterization Templates: 所有的环境共享一致的template, template含有不同的parameters, 不同环境定制这些parameters.

Kustomize是Kunernetes官方的config management tool, 解决第二个问题. 机制是对将可复用的manifest定义在base文件中，不同环境对其做patch: 即base file + patch(in different env)组成最终apply的manifest.

Config组织结构:

  • base存放common manifest
  • overlays存放不同环境的patches
1
2
3
4
5
6
7
8
9
10
11
12
├── base
│   ├── configMap.yaml
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   └── service.yaml
└── overlays
    ├── production
    │   ├── deployment.yaml
    │   └── kustomization.yaml
    └── staging
        ├── kustomization.yaml
        └── map.yaml

可以看出:

  • overlays下每个folder代表一个环境: staging, prod
  • 每个folder下都有kustomization.yaml: 为Kustomize的配置文件.

Kustomize feature

通过filekustomization.yaml来配置:

  • Name prefix: 不同overlays给k8s resource以不同的name prefix
  • Common label, annotation
  • ConfigMap, secret generator: 生成ConfigMap, Secret, with hash suffix; 并自动reference.
  • Diff of overlays: 以diff输出的形式，查看overlays对base的patch.
  • Image tag: 配置image’s Tag
  • Namespace: 配置resource的namespace

Hello world

代码见这里, 示例来自Demo: hello world with variants

helloword folder存放demo app的manifest, tree base

1
2
3
4
5
base
├── configMap.yaml
├── deployment.yaml
├── kustomization.yaml
└── service.yaml

kustomize build base会将base下所有manifest混合在一起输出, 一般这么用: kustomize build base | k apply -f -

加入staging和prod的patches, in overlays folder, 变成: 

1
2
3
4
5
6
7
8
9
10
11
12
├── base
│   ├── configMap.yaml
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   └── service.yaml
└── overlays
    ├── production
    │   ├── deployment.yaml
    │   └── kustomization.yaml
    └── staging
        ├── kustomization.yaml
        └── map.yaml

Base’s Kustomization

base/kustomization.yamlconfig common labels for all resources: 

1
2
commonLabels:
  app: hello

我们修改为app: my-hello, 验证是否所有base下resource都发生了改变: 

1
2
sed -i.bak 's/app: hello/app: my-hello/' \
    base/kustomization.yaml

通过kustomize build base可以看到效果:

20210525104303

resources列出basefolder包含的resource.

Overlays intro

以staging的kustomization.yaml为例: 

1
2
3
4
5
6
7
8
9
10
namePrefix: staging-
commonLabels:
  variant: staging
  org: acmeCorporation
commonAnnotations:
  note: Hello, I am staging!
bases:
- ../../base
patchesStrategicMerge:
- map.yaml
  • namePrefix表明以staging-prefix各resource name
  • commonLabels和commonAnnotations类似上述base的case
  • bases: 指明base manifest
  • patchesStrategicMerge: 以merge的形式update相应的resource: 
1
2
3
4
5
6
7
apiVersion: v1
kind: ConfigMap
metadata:
  name: the-map # (1)
data:
  altGreeting: "Have a pineapple!"
  enableRisky: "true"
  • (1)指明了update target, name必须match.

Diff

直接看下staging和production的不同(同理base与具体环境): 

1
2
3
4
diff \
  <(kustomize build overlays/staging) \
  <(kustomize build overlays/production) |\
  more
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
3,4c3,4
<   altGreeting: Have a pineapple!
<   enableRisky: "true"
---
>   altGreeting: Good Morning!
>   enableRisky: "false"
8c8
<     note: Hello, I am staging!
---
>     note: Hello, I am production!
12,13c12,13
<     variant: staging
<   name: staging-the-map
---
>     variant: production
>   name: production-the-map
...(truncate)

Patches总结

Kustomize支持两种patch:

  • patchesStrategicMerge: 给出partial object, 对原object做Upsert, design doc: Strategic Merge Patch
  • JSON patch: 根据JSON patch 标准, 对JSON object施加标准operations, 达到update的效果. 比merge更为imperative, operation有:
    • add
    • remove
    • replace
    • move
    • copy

JSON patch

示例见这里

json_patch为demo folder, 下面kustomization.yaml: 

1
2
3
4
5
6
7
8
9
resources:
- ingress.yaml
patches:
- path: ingress_patch.json
  target:
    group: networking.k8s.io
    version: v1beta1
    kind: Ingress
    name: my-ingress
  • resources: 待patch的manifest
  • patches: JSON patch文件，格式也为JSON

Verify patch diff: 

1
2
3
4
diff \                     
  <(cat json_patch/ingress.yaml) \
  <(kustomize build json_patch) |\       
  more
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
7c7
<   - host: foo.bar.com
---
>   - host: foo.bar.io
10,11c10
<       - path: /
<         backend:
---
>       - backend:
13,15c12,17
<           servicePort: 8888
<       - path: /api
<         backend:
---
>           servicePort: 80
>         path: /
>       - backend:
>           servicePort: 7700
>         path: /healthz
>       - backend:
18,19c20,21
<       - path: /test
<         backend:
---
>         path: /api
>       - backend:
21a24
>         path: /test

Note:

  • JSON patch并不要求object为JSON, 也可为yaml, 见 ingress_patch.yaml为example.
  • patches之前版本也写为patchesJson6902

相关Kustomize文档:

Reference

简洁的Intro slide
JSON Patching
Patching multiple resources at once