VPC 主要用于有多租户网络强隔离的场景，部分 Kubernetes 网络功能在多租户网络下存在冲突。 例如节点和 Pod 互访，NodePort 功能，基于网络访问的健康检查和 DNS 能力在多租户网络场景暂不支持。 为了方便常见 Kubernetes 的使用场景，Kube-OVN 默认 VPC 做了特殊设计，该 VPC 下的 Subnet 可以满足 Kubernetes 规范。用户自定义 VPC 支持本文档介绍的静态路由，EIP 和 NAT 网关等功能。 常见隔离需求可通过默认 VPC 下的网络策略和子网 ACL 实现，在使用自定义 VPC 前请明确是否需要 VPC 级别的隔离，并了解自定义 VPC 下的限制。 在 Underlay 网络下，物理交换机负责数据面转发，VPC 无法对 Underlay 子网进行隔离。

vpc 参数介绍

vpc 参数用法介绍

1
2
3
4
5
6
7
8
9

apiVersion: ovn.yusur.tech/v1
kind: Vpc
metadata:
  name: vpc1
spec:
  cidrBlock: 10.1.0.0/16/24 #必选 规定vpc 大网段，支持多个段写法，16表示大网段，24表示subnet的子网掩码， 格式必须是x.x.x.x/x/x
  enableExternal: true #可选 流量是否能出vpc，可结合bgp，宣告vpc内网段到tor，支持以vpc 内的ip地址出k8s集群,vpc 启用 enableExternal 会自动创建 lrp 关联到上述指定的公网
  enableK8sNode: true #可选 vpc流量是否能相互访问k8s node节点
  externalGatewaySwitch: join # 可选, vpc 出外网链接那个sw，默认链接join，可配置右边网关的sw 名称如：external3

创建自定义 VPC

创建完全隔离vpc

1
2
3
4
5
6

apiVersion: ovn.yusur.tech/v1
kind: Vpc
metadata:
  name: vpc1
spec:
  cidrBlock: 10.1.0.0/16/24 #必选 规定vpc 大网段，支持多个段写法，16表示大网段，24表示subnet的子网掩码

创建vpc 使用默认网关(ovn-k8s的GR_xx网关)

1
2
3
4
5
6
7

apiVersion: ovn.yusur.tech/v1
kind: Vpc
metadata:
  name: vpc1
spec:
  cidrBlock: 10.1.0.0/16/24 #必选 规定vpc 大网段，支持多个段写法，16表示大网段，24表示subnet的子网掩码
  enableExternal: true #可选 流量是否能出vpc，可结合bgp，宣告vpc内网段到tor，支持以vpc 内的ip地址出k8s集群,vpc 启用 enableExternal 会自动创建 lrp 关联到上述指定的公网

或者

1
2
3
4
5
6
7
8

apiVersion: ovn.yusur.tech/v1
kind: Vpc
metadata:
  name: vpc1
spec:
  cidrBlock: 10.1.0.0/16/24 #必选 规定vpc 大网段，支持多个段写法，16表示大网段，24表示subnet的子网掩码
  enableExternal: true #可选 流量是否能出vpc，可结合bgp，宣告vpc内网段到tor，支持以vpc 内的ip地址出k8s集群,vpc 启用 enableExternal 会自动创建 lrp 关联到上述指定的公网
  externalGatewaySwitch: join # 可选, vpc 出外网链接那个sw，默认链接join。

创建vpc 使用自定义网关

前提：external3 网关sw 必须提前创建完成，参考创建自定义网关

1
2
3
4
5
6
7
8

apiVersion: ovn.yusur.tech/v1
kind: Vpc
metadata:
  name: vpc1
spec:
  cidrBlock: 10.1.0.0/16/24 #必选 规定vpc 大网段，支持多个段写法，16表示大网段，24表示subnet的子网掩码
  enableExternal: true #可选 流量是否能出vpc，可结合bgp，宣告vpc内网段到tor，支持以vpc 内的ip地址出k8s集群,vpc 启用 enableExternal 会自动创建 lrp 关联到上述指定的公网
  externalGatewaySwitch: external3 # 可选, vpc 出外网链接那个sw，默认链接join。

创建vpc与k8s node节点互通

1
2
3
4
5
6
7

apiVersion: ovn.yusur.tech/v1
kind: Vpc
metadata:
  name: vpc1
spec:
  cidrBlock: 10.1.0.0/16/24 #必选 规定vpc 大网段，支持多个段写法，16表示大网段，24表示subnet的子网掩码
  enableK8sNode: true #可选 流量是否能相互访问k8s node节点

VPC 内容器访问外部网络需要通过 VPC 网关，VPC 网关可以打通物理网络和租户网络，并提供 浮动 IP，SNAT 和 DNAT 功能。

准备 underlay 公网网络

1
2
3
4
5
6
7

apiVersion: ovn.yusur.tech/v1
kind: ProviderNetwork
metadata:
  name: external3
spec:
  defaultBridge: brenp0s10 # 复用非dpu节点brenp0s10，用于非dpu节点
  defaultBridgeDpu: brp0 # 复用dpu节点brp0，用于dpu节点

1
2
3
4
5
6
7

apiVersion: ovn.yusur.tech/v1
kind: Vlan
metadata:
  name: vlan0
spec:
  id: 0
  provider: external3

1
2
3
4
5
6
7
8
9
10
11

apiVersion: ovn.yusur.tech/v1
kind: Subnet
metadata:
  name: external3
spec:
  cidrBlock: 172.16.10.0/24
  excludeIps:
  - 172.16.10.1..172.16.10.206
  gateway: 172.16.10.1
  vpc: default # 必须写default
  vlan: vlan0 # 必须设置vlan

注意：同一 VPC 下的不同子网不能包含同样的 IP，以及基于 VPC peering 或者 VPN 互相打通的不同子网不能包含同样的 IP。

subnet 参数用法介绍

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

apiVersion: ovn.yusur.tech/v1
kind: Subnet
metadata:
  name: subnet1
spec:
  vpc: vpc1 #必选 设置属于那个vpc
  cidrBlock: 10.1.1.0/24 #必须 设置subnet的cidr，必须在vpc指定的大段内
  gateway: 10.1.1.1 #必选 设置subnet的网关ip地址
  excludeIps: #可选 设置排除ip地址，不会分配给pod 使用
  - 10.1.1.1
  default: false #可选 是否作为vpc 默认的subnet，创建pod 不指定subnet则是 defualt 为true的subnet
  dhcpV4Options: "" #可选 自定义dhcp_options的ipv4 uuid
  dhcpV6Options: "" #可选 自定义dhcp_options的ipv6 uuid
  enableDHCP: false #可选 是否在该网段设置dhcp 功能
  ipv6RAConfigs: "" #可选 配置ipv6参数
  enableIPv6RA: false #可选 开启ipv6 参数
  enableLb: #可选，是否在该switch 上开启lb功能，用于实现k8s svc功能
  mtu: 0 #可选 设置dhcp服务mtu参数，可被dhcp client 使用
  natOutgoing: false #可选，设置整个subnet网段snat 规则，用于出vpc访问
  vlan: "" #可选，设置vlan，则管理网段为Underlay 网段
  vips: #可选，设置该网段，保留vip 的ip地址信息
status: #subnet 资源创建过程中，状态展示
  conditions: # 不包含type：Error的时候，表示subnet 成功创建
  - lastTransitionTime: "2024-03-25T08:01:17Z"
    lastUpdateTime: "2024-03-25T08:01:17Z"
    reason: ResetVpcLogicalSwitchSuccess
    status: "True"
    type: Ready
  dhcpV4OptionsUUID: "" # 设置ipv4 dhcp的uuid
  dhcpV6OptionsUUID: "" # 设置ipv6 dhcp的uuid
  standby: true # subnet 是否可用，false 表示subnet不可用或者创建失败
  v4availableIPs: 253 # ipv4 可用ip数量
  v4usingIPs: 0 # ipv4 已使用ip数量
  v6availableIPs: 0 # ipv6 可用ip数量
  v6usingIPs: 0 # ipv4 已使用ip数量

在Kubernetes集群网络中有一些最基本的核心网络需求诸如：

• Pod上的每一个网络接口都必须有自己唯一的IP；
• POD应具备在没有NAT的情况下与Kubernetes集群中的任何其他POD通信的能力；
• 节点上的Agents(例如：系统守护进程，kubelet)可以与该节点上的所有POD通信。

然而kubernetes本身并没有内置工具或组件解决上述核心网络需求，而是通过遵循Container Network Interface(CNI)规范的overlay网络插件来提供上述核心网络能力的。ovn-kubernetes和诸如Flannel,Calico等知名CNI一样，为kubernetes集群网提供核心网络所需能力。

OVN-Kubernetes使用OVN（Open Virtual Network）作为kubernetes网络的控制器来抽象并管理工作节点上的overlay网络流量，支持使用Vxlan/Geneve协议在各节点间创建underlay网络，并通过一个运行在DPU SOC上的OVS将转发面下沉到DPU上。开启DPU hardware offload特性后，可将kubernetes网络流量完全卸载到DPU上以实现高吞吐和极低时延的极致性能提升。

软件版本

• Kubernetes >= 1.24。
• Docker >= 1.12.6, Containerd >= 1.3.4。
• 操作系统: CentOS 7/8, Ubuntu 16.04/18.04/20.04。

一键安装

修改配置文件 设置 –vpc-enable为true

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

dpu_mode_enable: true
dpu_host_pf0: "enp0s10"
dpu_host_pf1: "enp0s11"
dpu_host_vf_num: "2-7"
dpu_host_vf0: "enp0s10v0"
gateway_mode: "shared"
tunnel_type: "vxlan"
pod_cidr: "10.244.0.0/16"
configureable_parameters:
  - --gateway-nexthop=172.16.10.1
  - --multicast-enabled=true
  - --mtu=1400
  - --disable-pkt-mtu-check=true
  - --routable_mtu=1400
  - --encap_csum=false
  - --vpc-enable=true

host_registry: "harbor.yusur.tech/yusur_ovn/ovn-kubernetes:test_push_0319"
dpu_registry: "harbor.yusur.tech/yusur_ovn/ovn-kubernetes:test_push_0319"

enable_calico: false
calico_ippool: "10.144.0.0/16"
calico_network_dev: "enp0s10"

roles:
  - name: master
    label: "k8s.ovn.org/role: \"master\""
  - name: worker
    label: "k8s.ovn.org/role: \"worker\""
  - name: dpu-host
    label: "k8s.ovn.org/dpu-host:"
  - name: dpu
    label: "k8s.ovn.org/dpu:"

nodes:
  - hostname: "master"
    role: "master"
    k8s_ip: "172.16.10.100"
    user: "root"
    passwd: "root"
    dpu_mode: false

  - hostname: "worker1"
    role: "worker"
    k8s_ip: "172.16.10.101"
    user: "root"
    passwd: "root"
    dpu_mode: false

  - hostname: "host1"
    role: "worker"
    k8s_ip: "172.16.10.200"
    user: "root"
    passwd: "root"
    dpu_mode: true
    dpu_ip: "172.16.10.201"
    dpu_user: "root"
    dpu_passwd: "root"
    dpu_network_name: "brp0"
    dpu_pci0: "00:0a.0"
    dpu_pci1: "00:0b.0"
    vdpa_mode: false
    ready_delete: true

  - hostname: "host2"
    role: "worker"
    k8s_ip: "172.16.10.202"
    user: "root"
    passwd: "root"
    dpu_mode: true
    dpu_ip: "172.16.10.203"
    dpu_user: "root"
    dpu_passwd: "root"
    dpu_network_name: "brp0"
    dpu_pci0: "00:0a.0"
    dpu_pci1: "00:0b.0"
    vdpa_mode: true

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

cat /opt/yusur/hados/scripts/ovn-kubernetes/hados_ovn-kubernetes-deploy/scripts/install.sh
 
img=harbor.yusur.tech/hados/hados_ovn-kubernetes:1.0.24.2  
kubeconfig=/etc/kubernetes/admin.conf
nerdctl run -d --name install --net host --entrypoint python3 \
-w /root/autodeploy/ \
-v $PWD/autodeploy:/root/autodeploy/ \
-v $kubeconfig:/root/.kube/config \
-v $PWD/:/var/log/install/ $img install.py
 
# 运行日志查看
nerdctl logs -f install
  
# 删除部署容器
nerdctl rm install
  
# 查看详细日志
tail -fn200 install.log

详细参考

一键卸载

1
2
3
4
5
6
7
8
9
10

img=harbor.yusur.tech/hados/hados_ovn-kubernetes:1.0.24.2  
kubeconfig=/etc/kubernetes/admin.conf
nerdctl run -it --rm --net host --entrypoint python3 \
-w /root/autodeploy/ \
-v $PWD/autodeploy:/root/autodeploy/ \
-v $kubeconfig:/root/.kube/config \
-v $PWD/:/var/log/install/ $img clear.py
 
# 查看详细日志
tail -fn200 install.log

详细参考

作为运维工程师，你需要掌握一系列技能和知识，涵盖了多个领域，从操作系统管理到云计算、自动化和网络等。以下是一个运维工程师学习知识树的一般框架，你可以根据自己的实际情况和兴趣深入研究每个领域：

操作系统管理：

Linux 和 Windows 操作系统基础知识
用户管理、权限管理和文件系统管理
进程管理和系统监控
网络配置和故障排除

书籍：
《鸟哥的 Linux 私房菜》（第五版）
《Linux 基础教程》（第四版）
《UNIX 环境高级编程》
《Windows PowerShell 脚本管理教程》

云计算平台：

AWS、Azure、Google Cloud 等主流云服务提供商的基本概念和服务
虚拟机、存储、网络和身份认证的管理
云上应用的部署和扩展

网络基础：

TCP/IP 协议栈和网络协议
子网划分和路由
防火墙和安全性基础

自动化和脚本编程：

Shell 脚本（Bash）
Python 或其他脚本语言
自动化配置管理工具（如Ansible、Puppet、Chef）

监控和日志管理：

监控系统的设置和使用
使用日志分析工具（如ELK Stack：Elasticsearch、Logstash、Kibana）
告警设置和故障排除

容器化和编排：

Docker 容器基础知识
Kubernetes 或其他容器编排工具

数据库管理：

基本数据库知识（MySQL、PostgreSQL、MongoDB等）
数据库备份、恢复和性能优化

安全和身份认证：

安全最佳实践
SSL/TLS 和加密基础
身份认证和访问控制

项目管理和团队协作：

版本控制系统（如Git）
敏捷方法论和项目管理工具（如Jira）

持续集成和持续交付：

CI/CD 流程
Jenkins 或其他持续集成工具

学习资源和社区参与：

参与技术社区（如Stack Overflow、GitHub）
专业技术博客和在线教程
相关认证课程和培训

学习链接

linux

Linux 视频

LAMP兄弟连-易第优教育PHPPHP培训北京PHP培训PHP教程PHP学习Java培训LAMP培训pythonpython培训就业大学生就业PHP视频Linux视频

Unix技术网 = 全球最大的Linux-Unix应用与开发者社区 = IT人的网上家园

韩顺平 linux视频教程–华为网盘|资源共享-文件备份-免费网络硬盘

韩顺平 linux视频教程–华为网盘 资源共享-文件备份-免费网络硬盘

Linux 系统官网

Index of /centos/6.4/isos/x86_64/

BackTrack Linux - Penetration Testing Distribution

The NetBSD Project

The FreeBSD Project

The Linux Kernel

Index of /centos/7.0.1406

Linux From Scratch

Kube-proxy 是 Kubernetes 集群中的一个关键组件，负责实现 Kubernetes 服务发现和负载均衡。它运行在每个节点上，以确保网络流量正确路由到集群中的服务。
以下是 Kube-proxy 的一些主要功能和特点：

服务代理：Kube-proxy会监视 Kubernetes API Server 中的 Service 和 Endpoints 对象，以了解集群中的服务和其后端 Pod。当有新服务创建或更新时，Kube-proxy会自动更新代理规则，以确保服务可用性。

IP 负载均衡：Kube-proxy通过维护 IP 负载均衡规则，将请求分发到服务的后端 Pod。这有助于实现服务的负载均衡，确保请求能够均匀地分发到多个 Pod 上。

面向用户的服务发现：Kube-proxy允许用户通过 Kubernetes 的服务抽象来访问应用程序，而不需要了解后端 Pod 的详细信息。用户只需使用服务名称和端口号，而不必担心具体的 Pod IP 地址。

支持多种代理模式：Kube-proxy支持多种代理模式，包括iptables、IPVS、和 Windows 等。这些模式允许根据集群的具体需求选择不同的代理方式。

节点故障处理：Kube-proxy会监视节点上的 Pod 和服务，并在节点故障时自动处理重新路由流量，以确保服务的高可用性。

安全策略支持：Kube-proxy也可以与网络策略 (Network Policies) 集成，允许管理员定义网络访问控制规则，以保护集群中的服务。

总之，Kube-proxy是 Kubernetes 集群中的一个关键组件，它简化了服务发现和负载均衡的管理，使用户能够轻松地访问和管理其应用程序，同时确保高可用性和网络安全。不过需要注意的是，Kubernetes 1.20版本及以后，Kube-proxy引入了一个新的模式，即”Kube-proxy mode”，用于简化其设计和维护。因此，Kube-proxy的具体实现细节可能会随着Kubernetes版本的演进而变化。

图解

kube-proxy是工作在每个节点上面的组件，它会去watch apiserver，watch apiserver之后kube-proxy，假设kube-proxy工作在iptables模式下面，它就会去调用iptables的命令，去生成这样一些规则，无非就是说你的目标IP是某个clusterIP，那我就应该去写一些dnat的跳转规则，让后端数据包的目标地址变为后端pod的IP地址，这是kube-proxy希望去实现的一个目标。
经过IP tables之后，你访问cluster ip，你要将数据包转到后端的pod里面去，这是我们要达到的一个目的。

iptables 模式

这个chain和原生的linux kernel里面内置的hook点是怎么产生绑定关系的呢？可以看到将kube-service这个chain加入到pretouting和output这两个hook点里面，

也就意味着，所有进入到Linux内核的数据包，以及linux系统里面的进程，向外发起请求的这些数据包全部都需要走kube-servcie，那么就意味着任何请求都会经过这些规则去处理。

假设本地有一个pod，要去访问nginx-basic这个服务，这个数据包意味是从本地的pod发出的，所以它先要走本地的kernel的output表，output就会去执行kube-service，走kube-service就会去看如果你的clusterip是符合我刚才service的clusterip的话，它就会经由展示的那些规则一个一个去命中。命中的话就将clusterip换成了目标pod的IP。，这样就完成了一个负载均衡的目标，这一切是在k8s节点的主机网络空间里面去做的。

ipvs是netfilter另外一种工作模式，另外一种plugin，ipvs本身是lvs的一部分，或者说二者是可以等价的，lvs更多的是为负载均衡服务，所以对nat网络包的处理，有更好的支持。

ipvs和iptables不一样，它所支持的hook点不一样，它没有prerouting的支持，它只有local_in的支持，有local_out的支持，只有local int out可以让我们去添加一些规则。

因为iptables模式下面，当你去访问clusterip，在做路由判决之前就将目标地址换掉了，进来的数据包是通过prerouting去做的dnat，这个时候路由判决还没有做，所以那个clusterip可以不绑在任何的路由设备上面，在路由判决的时候那个IP已经变成了pod的真实IP。

针对ipvs来说，从外面发进来的一个包，由Linux kernel来处理，如果访问的是clusterIP，那么在prerouting这里是没有ipvs的hook点的，也就是路由判决的时候，这个ip还是你service对应的clusterip，也就说在路由判决的时候，linux kernel会去判断，你所访问的目标地址，是不是在我本机一个有效地址，如果是无效就将这个包丢掉，针对ipvs模式，有一个额外的操作，要将所有的service cluster ip绑在当前节点的dump设备上面，这样的话才能判断我这个本机上面是有这个IP的，它是一个有效IP，又因为它是本机的，它会走到local_in这个hook点，在这里就可以插入一些ipvs规则的，出去的包也一样在local_out去做。
这是本质的区别，针对ipvs这种模式，你的clusterip是要绑在本机的某个interface上面的。

参数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-proxy
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeadm:node-proxier
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-proxier
subjects:
- kind: ServiceAccount
  name: kube-proxy
  namespace: kube-system

---
apiVersion: v1
data:
  config.conf: |-
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    bindAddress: 0.0.0.0
    bindAddressHardFail: false
    clientConnection:
      acceptContentTypes: ""
      burst: 0
      contentType: ""
      kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
      qps: 0
    clusterCIDR: 10.144.0.0/16
    configSyncPeriod: 0s
    conntrack:
      maxPerCore: null
      min: null
      tcpCloseWaitTimeout: null
      tcpEstablishedTimeout: null
    detectLocal:
      bridgeInterface: ""
      interfaceNamePrefix: ""
    detectLocalMode: ""
    enableProfiling: false
    healthzBindAddress: ""
    hostnameOverride: ""
    iptables:
      masqueradeAll: false
      masqueradeBit: null
      minSyncPeriod: 0s
      syncPeriod: 0s
    ipvs:
      excludeCIDRs: null
      minSyncPeriod: 0s
      scheduler: ""
      trictARP: false
      syncPeriod: 0s
      tcpFinTimeout: 0s
      tcpTimeout: 0s
      udpTimeout: 0s
    kind: KubeProxyConfiguration
    metricsBindAddress: ""
    mode: "iptables"
    nodePortAddresses: null
    oomScoreAdj: null
    portRange: ""
    showHiddenMetricsForVersion: ""
    udpIdleTimeout: 0s
    winkernel:
      enableDSR: false
      forwardHealthCheckVip: false
      networkName: ""
      rootHnsEndpointName: ""
      sourceVip: ""
  kubeconfig.conf: |-
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://192.168.201.10:6443
      name: default
    contexts:
    - context:
        cluster: default
        namespace: default
        user: default
      name: default
    current-context: default
    users:
    - name: default
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
kind: ConfigMap
metadata:
  labels:
    app: kube-proxy
  name: kube-proxy
  namespace: kube-system

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: kube-proxy
  name: kube-proxy
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: kube-proxy
  template:
    metadata:
      labels:
        k8s-app: kube-proxy
    spec:
      containers:
      - command:
        - /usr/local/bin/kube-proxy
        - --config=/var/lib/kube-proxy/config.conf
        - --hostname-override=$(NODE_NAME)
        - --exclude-cluster-cidr=10.244.0.0/16
        - --v=5
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        image: flftuu/kube-proxy-amd64:v1.24.10-dirty
        imagePullPolicy: IfNotPresent
        name: kube-proxy
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/kube-proxy
          name: kube-proxy
        - mountPath: /run/xtables.lock
          name: xtables-lock
        - mountPath: /lib/modules
          name: lib-modules
          readOnly: true
      dnsPolicy: ClusterFirst
      hostNetwork: true
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: kube-proxy
      serviceAccountName: kube-proxy
      terminationGracePeriodSeconds: 30
      tolerations:
      - operator: Exists
      volumes:
      - configMap:
          defaultMode: 420
          name: kube-proxy
        name: kube-proxy
      - hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
        name: xtables-lock
      - hostPath:
          path: /lib/modules
          type: ""
        name: lib-modules
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate

编译

1
2
3
4
5

git clone https://github.com/kubernetes/kubernetes.git
cd kubernetes
export KUBE_GIT_VERSION=v1.24.10
make clean
make quick-release-images

镜像

1

docker pull flftuu/kube-proxy-amd64:v1.24.10-dirty

效果

iptables 模式

1
2
3
4
5
6
7
8
9
10
11
12
13

# 查看iptables nat 表的规则

# 存在calico ip：10.96.67.75 信息
[root@fanyi-worker1 ~]# iptables -L -vn -t nat | grep 10.96.67.75 -A 2
    0     0 KUBE-SVC-RH2G6EKXTJDCSNM7  tcp  --  *      *       0.0.0.0/0            10.96.67.75          /* default/nginx-calico cluster IP */ tcp dpt:80
   12   720 KUBE-NODEPORTS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

--
    0     0 KUBE-MARK-MASQ  tcp  --  *      *      !10.144.0.0/16        10.96.67.75          /* default/nginx-calico cluster IP */ tcp dpt:80
    0     0 KUBE-SEP-ZMODNQBWAQHJ2KQZ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/nginx-calico -> 10.144.154.71:80 */

# 不存在ovn ip：10.96.95.98  信息
[root@fanyi-worker1 ~]# iptables -L -vn -t nat | grep 10.96.95.98 -A 2

ipvs 模式

1
2
3
4
5
6
7
8
9
10
11
12

# 查看ipvs 规则和 kube-ipvs0网卡上的ip信息

# 存在calico ip：10.96.67.75 信息
[root@fanyi-worker1 ~]# ip add | grep  10.96.67.75
    inet 10.96.67.75/32 scope global kube-ipvs0
[root@fanyi-worker1 ~]# ipvsadm -l -n | grep  10.96.67.75 -A 1
TCP  10.96.67.75:80 rr
  -> 10.144.154.71:80             Masq    1      0          0  

# 不存在ovn ip：10.96.95.98  信息
[root@fanyi-worker1 ~]# ip add | grep  10.96.95.98 
[root@fanyi-worker1 ~]# ipvsadm -l -n | grep  10.96.95.98 

multi-homing功能简介

具有多个网络接口的 K8s Pod被称为 multi-homed。  Network Plumbing Working Group 提出了一个标准，描述如何指定附加网络接口的配置。

有几个插件（Multus、Genie）实现了此标准。

在特定 Kubernetes 节点上调度 pod 后，kubelet 将调用委托插件来准备 pod 的联网。然后，此元插件将调用负责设置 pod 的默认集群网络的 CNI，然后迭代 pod 上的其他附件列表，调用相应的委托 CNI 实现将 pod 附加到该特定网络的逻辑。

环境搭建

ovn-k8s 开启multi-homing

 OVN_IMAGE=harbor.yusur.tech/yusur_ovn/ovn-daemonset-f:second-cni-latest ./daemonset.sh \--image=$OVN_IMAGE \--net-cidr=10.124.0.0/16 \--svc-cidr=10.86.0.0/16 \--gateway-mode="local" \--gateway-options="--gateway-interface=ens3f2 --gateway-nexthop=192.168.200.1" \--k8s-apiserver=https://192.168.122.10:6443 \--multicast-enabled=false \--disable-snat-multiple-gws=true \--disable-pkt-mtu-check=true \--multi-network-enable=true \ # 开启multi-homing 功能--master-loglevel=5 --node-loglevel=5 #设置日志级别5# 安装yamlkubectl apply -f ../yaml/ovn-setup.yamlkubectl apply -f ../yaml/k8s.ovn.org_adminpolicybasedexternalroutes.yamlkubectl apply -f ../yaml/ovnkube-db.yamlkubectl apply -f ../yaml/ovnkube-master.yamlkubectl apply -f ../yaml/ovnkube-node.yaml

安装multus-cni 组件

# Note:#   This deployment file is designed for 'quickstart' of multus, easy installation to test it,#   hence this deployment yaml does not care about following things intentionally.#     - various configuration options#     - minor deployment scenario#     - upgrade/update/uninstall scenario#   Multus team understand users deployment scenarios are diverse, hence we do not cover#   comprehensive deployment scenario. We expect that it is covered by each platform deployment.---apiVersion: apiextensions.k8s.io/v1kind: CustomResourceDefinitionmetadata:  name: network-attachment-definitions.k8s.cni.cncf.iospec:  group: k8s.cni.cncf.io  scope: Namespaced  names:    plural: network-attachment-definitions    singular: network-attachment-definition    kind: NetworkAttachmentDefinition    shortNames:    - net-attach-def  versions:    - name: v1      served: true      storage: true      schema:        openAPIV3Schema:          description: 'NetworkAttachmentDefinition is a CRD schema specified by the Network Plumbing            Working Group to express the intent for attaching pods to one or more logical or physical            networks. More information available at: https://github.com/k8snetworkplumbingwg/multi-net-spec'          type: object          properties:            apiVersion:              description: 'APIVersion defines the versioned schema of this represen                tation of an object. Servers should convert recognized schemas to the                latest internal value, and may reject unrecognized values. More info:                https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'              type: string            kind:              description: 'Kind is a string value representing the REST resource this                object represents. Servers may infer this from the endpoint the client                submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'              type: string            metadata:              type: object            spec:              description: 'NetworkAttachmentDefinition spec defines the desired state of a network attachment'              type: object              properties:                config:                  description: 'NetworkAttachmentDefinition config is a JSON-formatted CNI configuration'                  type: string---kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata:  name: multusrules:  - apiGroups: ["k8s.cni.cncf.io"]    resources:      - '*'    verbs:      - '*'  - apiGroups:      - ""    resources:      - pods      - pods/status    verbs:      - get      - update  - apiGroups:      - ""      - events.k8s.io    resources:      - events    verbs:      - create      - patch      - update---kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata:  name: multusroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: multussubjects:- kind: ServiceAccount  name: multus  namespace: kube-system---apiVersion: v1kind: ServiceAccountmetadata:  name: multus  namespace: kube-system---kind: ConfigMapapiVersion: v1metadata:  name: multus-cni-config  namespace: kube-system  labels:    tier: node    app: multusdata:  # NOTE: If you'd prefer to manually apply a configuration file, you may create one here.  # In the case you'd like to customize the Multus installation, you should change the arguments to the Multus pod  # change the "args" line below from  # - "--multus-conf-file=auto"  # to:  # "--multus-conf-file=/tmp/multus-conf/70-multus.conf"  # Additionally -- you should ensure that the name "70-multus.conf" is the alphabetically first name in the  # /etc/cni/net.d/ directory on each node, otherwise, it will not be used by the Kubelet.  cni-conf.json: |    {      "name": "multus-cni-network",      "type": "multus",      "capabilities": {        "portMappings": true      },      "delegates": [        {          "cniVersion": "0.3.1",          "name": "default-cni-network",          "plugins": [            {              "type": "flannel",              "name": "flannel.1",                "delegate": {                  "isDefaultGateway": true,                  "hairpinMode": true                }              },              {                "type": "portmap",                "capabilities": {                  "portMappings": true                }              }          ]        }      ],      "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"    }---apiVersion: apps/v1kind: DaemonSetmetadata:  name: kube-multus-ds  namespace: kube-system  labels:    tier: node    app: multus    name: multusspec:  selector:    matchLabels:      name: multus  updateStrategy:    type: RollingUpdate  template:    metadata:      labels:        tier: node        app: multus        name: multus    spec:      hostNetwork: true      tolerations:      - operator: Exists        effect: NoSchedule      - operator: Exists        effect: NoExecute      serviceAccountName: multus      containers:      - name: kube-multus        image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot        command: ["/thin_entrypoint"]        args:        - "--multus-conf-file=auto"        - "--multus-autoconfig-dir=/host/etc/cni/net.d"        - "--cni-conf-dir=/host/etc/cni/net.d"        resources:          requests:            cpu: "100m"            memory: "50Mi"          limits:            cpu: "100m"            memory: "50Mi"        securityContext:          privileged: true        volumeMounts:        - name: cni          mountPath: /host/etc/cni/net.d        - name: cnibin          mountPath: /host/opt/cni/bin        - name: multus-cfg          mountPath: /tmp/multus-conf      initContainers:        - name: install-multus-binary          image: ghcr.io/k8snetworkplumbingwg/multus-cni:snapshot          command: ["/install_multus"]          args:            - "--type"            - "thin"          resources:            requests:              cpu: "10m"              memory: "15Mi"          securityContext:            privileged: true          volumeMounts:            - name: cnibin              mountPath: /host/opt/cni/bin              mountPropagation: Bidirectional      terminationGracePeriodSeconds: 10      volumes:        - name: cni          hostPath:            path: /etc/cni/net.d        - name: cnibin          hostPath:            path: /opt/cni/bin        - name: multus-cfg          configMap:            name: multus-cni-config            items:            - key: cni-conf.json              path: 70-multus.conf

安装multi-networkpolicy 组件

简介：multi-network policy通过iptables 或者 tc 实现了对net-attach-def的网络策略功能，并为net-attach-def网络提供网络安全保护。

参考：https://github.com/k8snetworkplumbingwg/multi-networkpolicy

---apiVersion: apiextensions.k8s.io/v1kind: CustomResourceDefinitionmetadata:  name: multi-networkpolicies.k8s.cni.cncf.iospec:  group: k8s.cni.cncf.io  scope: Namespaced  names:    plural: multi-networkpolicies    singular: multi-networkpolicy    kind: MultiNetworkPolicy    shortNames:    - multi-policy  versions:    - name: v1beta1      served: true      storage: true      schema:        openAPIV3Schema:          description: "MultiNetworkPolicy is a CRD schema to provide NetworkPolicy            mechanism for net-attach-def which is specified by the Network Plumbing            Working Group. MultiNetworkPolicy is identical to Kubernetes NetworkPolicy,            See: https://kubernetes.io/docs/concepts/services-networking/network-policies/ ."          properties:            spec:              description: 'Specification of the desired behavior for this MultiNetworkPolicy.'              properties:                egress:                  description: "List of egress rules to be applied to the selected pods.                    Outgoing traffic is allowed if there are no NetworkPolicies selecting                    the pod (and cluster policy otherwise allows the traffic), OR if the                    traffic matches at least one egress rule across all of the NetworkPolicy                    objects whose podSelector matches the pod. If this field is empty                    then this NetworkPolicy limits all outgoing traffic (and serves solely                    to ensure that the pods it selects are isolated by default). This                    field is beta-level in 1.8"                  items:                    description: "NetworkPolicyEgressRule describes a particular set of                      traffic that is allowed out of pods matched by a NetworkPolicySpec's                      podSelector. The traffic must match both ports and to. This type                      is beta-level in 1.8"                    properties:                      ports:                        description: "List of destination ports for outgoing traffic. Each                          item in this list is combined using a logical OR. If this field                          is empty or missing, this rule matches all ports (traffic not                          restricted by port). If this field is present and contains at                          least one item, then this rule allows traffic only if the traffic                          matches at least one port in the list."                        items:                          description: "NetworkPolicyPort describes a port to allow traffic on"                          properties:                            port:                              anyOf:                                - type: integer                                - type: string                              description: "The port on the given protocol. This can either                                be a numerical or named port on a pod. If this field is                                not provided, this matches all port names and numbers."                              x-kubernetes-int-or-string: true                            protocol:                              description: "The protocol (TCP, UDP, or SCTP) which traffic                                must match. If not specified, this field defaults to TCP."                              type: string                          type: object                        type: array                      to:                        description: "List of destinations for outgoing traffic of pods                          selected for this rule. Items in this list are combined using                          a logical OR operation. If this field is empty or missing, this                          rule matches all destinations (traffic not restricted by destination).                          If this field is present and contains at least one item, this                          rule allows traffic only if the traffic matches at least one                          item in the to list."                        items:                          description: "NetworkPolicyPeer describes a peer to allow traffic                        from. Only certain combinations of fields are allowed"                          properties:                            ipBlock:                              description: "IPBlock defines policy on a particular IPBlock.                                If this field is set then neither of the other fields                                can be."                              properties:                                cidr:                                  description: "CIDR is a string representing the IP Block                                    Valid examples are '192.168.1.1/24'"                                  type: string                                except:                                  description: "Except is a slice of CIDRs that should                                    not be included within an IP Block Valid examples                                    are '192.168.1.1/24' Except values will be rejected                                    if they are outside the CIDR range"                                  items:                                    type: string                                  type: array                              required:                              - cidr                              type: object                            namespaceSelector:                              description: "Selects Namespaces using cluster-scoped labels.                                This field follows standard label selector semantics;                                if present but empty, it selects all namespaces. \n If                                PodSelector is also set, then the NetworkPolicyPeer as                                a whole selects the Pods matching PodSelector in the Namespaces                                selected by NamespaceSelector. Otherwise it selects all                                Pods in the Namespaces selected by NamespaceSelector."                              properties:                                matchExpressions:                                  description: "matchExpressions is a list of label selector                                    requirements. The requirements are ANDed."                                  items:                                    description: "A label selector requirement is a selector                                      that contains values, a key, and an operator that                                      relates the key and values."                                    properties:                                      key:                                        description: "key is the label key that the selector                                          applies to."                                        type: string                                      operator:                                        description: "operator represents a key's relationship                                          to a set of values. Valid operators are In,                                          NotIn, Exists and DoesNotExist."                                        type: string                                      values:                                        description: "values is an array of string values.                                          If the operator is In or NotIn, the values array                                          must be non-empty. If the operator is Exists                                          or DoesNotExist, the values array must be empty.                                          This array is replaced during a strategic merge                                          patch."                                        items:                                          type: string                                        type: array                                    required:                                    - key                                    - operator                                    type: object                                  type: array                                matchLabels:                                  additionalProperties:                                    type: string                                  description: "matchLabels is a map of {key,value} pairs.                                    A single {key,value} in the matchLabels map is equivalent                                    to an element of matchExpressions, whose key field                                    is 'key', the operator is 'In', and the values array                                    contains only 'value'. The requirements are ANDed."                                  type: object                              type: object                            podSelector:                              description: "This is a label selector which selects Pods.                                This field follows standard label selector semantics;                                if present but empty, it selects all pods. \n If NamespaceSelector                                is also set, then the NetworkPolicyPeer as a whole selects                                the Pods matching PodSelector in the Namespaces selected                                by NamespaceSelector. Otherwise it selects the Pods matching                                PodSelector in the policy's own Namespace."                              properties:                                matchExpressions:                                  description: "matchExpressions is a list of label selector                                    requirements. The requirements are ANDed."                                  items:                                    description: "A label selector requirement is a selector                                      that contains values, a key, and an operator that                                      relates the key and values."                                    properties:                                      key:                                        description: key is the label key that the selector                                          applies to.                                        type: string                                      operator:                                        description: "operator represents a key's relationship                                          to a set of values. Valid operators are In,                                          NotIn, Exists and DoesNotExist."                                        type: string                                      values:                                        description: "values is an array of string values.                                          If the operator is In or NotIn, the values array                                          must be non-empty. If the operator is Exists                                          or DoesNotExist, the values array must be empty.                                          This array is replaced during a strategic merge                                          patch."                                        items:                                          type: string                                        type: array                                    required:                                    - key                                    - operator                                    type: object                                  type: array                                matchLabels:                                  additionalProperties:                                    type: string                                  description: "matchLabels is a map of {key,value} pairs.                                    A single {key,value} in the matchLabels map is equivalent                                    to an element of matchExpressions, whose key field                                    is 'key', the operator is 'In', and the values array                                    contains only 'value'. The requirements are ANDed."                                  type: object                              type: object                          type: object                        type: array                    type: object                  type: array                ingress:                  description: "List of ingress rules to be applied to the selected pods.                    Traffic is allowed to a pod if there are no NetworkPolicies selecting                    the pod (and cluster policy otherwise allows the traffic), OR if the                    traffic source is the pod's local node, OR if the traffic matches                    at least one ingress rule across all of the NetworkPolicy objects                    whose podSelector matches the pod. If this field is empty then this                    NetworkPolicy does not allow any traffic (and serves solely to ensure                    that the pods it selects are isolated by default)"                  items:                    description: "NetworkPolicyIngressRule describes a particular set of                      traffic that is allowed to the pods matched by a NetworkPolicySpec's                      podSelector. The traffic must match both ports and from."                    properties:                      from:                        description: "List of sources which should be able to access the                          pods selected for this rule. Items in this list are combined                          using a logical OR operation. If this field is empty or missing,                          this rule matches all sources (traffic not restricted by source).                          If this field is present and contains at least one item, this                          rule allows traffic only if the traffic matches at least one                          item in the from list."                        items:                          description: NetworkPolicyPeer describes a peer to allow traffic                            from. Only certain combinations of fields are allowed                          properties:                            ipBlock:                              description: "IPBlock defines policy on a particular IPBlock.                                If this field is set then neither of the other fields                                can be."                              properties:                                cidr:                                  description: "CIDR is a string representing the IP Block                                    Valid examples are '192.168.1.1/24'"                                  type: string                                except:                                  description: "Except is a slice of CIDRs that should                                    not be included within an IP Block Valid examples                                    are '192.168.1.1/24' Except values will be rejected                                    if they are outside the CIDR range"                                  items:                                    type: string                                  type: array                              required:                              - cidr                              type: object                            namespaceSelector:                              description: "Selects Namespaces using cluster-scoped labels.                                This field follows standard label selector semantics;                                if present but empty, it selects all namespaces. \n If                                PodSelector is also set, then the NetworkPolicyPeer as                                a whole selects the Pods matching PodSelector in the Namespaces                                selected by NamespaceSelector. Otherwise it selects all                                Pods in the Namespaces selected by NamespaceSelector."                              properties:                                matchExpressions:                                  description: "matchExpressions is a list of label selector                                    requirements. The requirements are ANDed."                                  items:                                    description: "A label selector requirement is a selector                                      that contains values, a key, and an operator that                                      relates the key and values."                                    properties:                                      key:                                        description: "key is the label key that the selector                                          applies to."                                        type: string                                      operator:                                        description: "operator represents a key's relationship                                          to a set of values. Valid operators are In,                                          NotIn, Exists and DoesNotExist."                                        type: string                                      values:                                        description: "values is an array of string values.                                          If the operator is In or NotIn, the values array                                          must be non-empty. If the operator is Exists                                          or DoesNotExist, the values array must be empty.                                          This array is replaced during a strategic merge                                          patch."                                        items:                                          type: string                                        type: array                                    required:                                    - key                                    - operator                                    type: object                                  type: array                                matchLabels:                                  additionalProperties:                                    type: string                                  description: "matchLabels is a map of {key,value} pairs.                                    A single {key,value} in the matchLabels map is equivalent                                    to an element of matchExpressions, whose key field                                    is 'key', the operator is 'In', and the values array                                    contains only 'value'. The requirements are ANDed."                                  type: object                              type: object                            podSelector:                              description: "This is a label selector which selects Pods.                                This field follows standard label selector semantics;                                if present but empty, it selects all pods. \n If NamespaceSelector                                is also set, then the NetworkPolicyPeer as a whole selects                                the Pods matching PodSelector in the Namespaces selected                                by NamespaceSelector. Otherwise it selects the Pods matching                                PodSelector in the policy's own Namespace."                              properties:                                matchExpressions:                                  description: "matchExpressions is a list of label selector                                    requirements. The requirements are ANDed."                                  items:                                    description: "A label selector requirement is a selector                                      that contains values, a key, and an operator that                                      relates the key and values."                                    properties:                                      key:                                        description: "key is the label key that the selector                                          applies to."                                        type: string                                      operator:                                        description: "operator represents a key's relationship                                          to a set of values. Valid operators are In,                                          NotIn, Exists and DoesNotExist."                                        type: string                                      values:                                        description: "values is an array of string values.                                          If the operator is In or NotIn, the values array                                          must be non-empty. If the operator is Exists                                          or DoesNotExist, the values array must be empty.                                          This array is replaced during a strategic merge                                          patch."                                        items:                                          type: string                                        type: array                                    required:                                    - key                                    - operator                                    type: object                                  type: array                                matchLabels:                                  additionalProperties:                                    type: string                                  description: "matchLabels is a map of {key,value} pairs.                                    A single {key,value} in the matchLabels map is equivalent                                    to an element of matchExpressions, whose key field                                    is 'key', the operator is 'In', and the values array                                    contains only 'value'. The requirements are ANDed."                                  type: object                              type: object                          type: object                        type: array                      ports:                        description: "List of ports which should be made accessible on                          the pods selected for this rule. Each item in this list is combined                          using a logical OR. If this field is empty or missing, this                          rule matches all ports (traffic not restricted by port). If                          this field is present and contains at least one item, then this                          rule allows traffic only if the traffic matches at least one                          port in the list."                        items:                          description: NetworkPolicyPort describes a port to allow traffic                            on                          properties:                            port:                              anyOf:                                - type: integer                                - type: string                              description: "The port on the given protocol. This can either                                be a numerical or named port on a pod. If this field is                                not provided, this matches all port names and numbers."                              x-kubernetes-int-or-string: true                            protocol:                              description: "The protocol (TCP, UDP, or SCTP) which traffic                                must match. If not specified, this field defaults to TCP."                              type: string                          type: object                        type: array                    type: object                  type: array                podSelector:                  description: "This is a label selector which selects Pods.                    This field follows standard label selector semantics;                    if present but empty, it selects all pods. \n If NamespaceSelector                    is also set, then the NetworkPolicyPeer as a whole selects                    the Pods matching PodSelector in the Namespaces selected                    by NamespaceSelector. Otherwise it selects the Pods matching                    PodSelector in the policy's own Namespace."                  properties:                    matchExpressions:                      description: "matchExpressions is a list of label selector                        requirements. The requirements are ANDed."                      items:                        description: "A label selector requirement is a selector                          that contains values, a key, and an operator that                          relates the key and values."                        properties:                          key:                            description: "key is the label key that the selector applies to."                            type: string                          operator:                            description: "operator represents a key's relationship                              to a set of values. Valid operators are In,                              NotIn, Exists and DoesNotExist."                            type: string                          values:                            description: "values is an array of string values.                              If the operator is In or NotIn, the values array                              must be non-empty. If the operator is Exists                              or DoesNotExist, the values array must be empty.                              This array is replaced during a strategic merge                              patch."                            items:                              type: string                            type: array                        required:                        - key                        - operator                        type: object                      type: array                    matchLabels:                      additionalProperties:                        type: string                        description: "matchLabels is a map of {key,value} pairs.                          A single {key,value} in the matchLabels map is equivalent                          to an element of matchExpressions, whose key field                          is 'key', the operator is 'In', and the values array                          contains only 'value'. The requirements are ANDed."                      type: object                  type: object                policyTypes:                  description: "List of rule types that the NetworkPolicy relates to. Valid                    options are 'Ingress', 'Egress', or 'Ingress,Egress'. If this field                    is not specified, it will default based on the existence of Ingress                    or Egress rules; policies that contain an Egress section are assumed                    to affect Egress, and all policies (whether or not they contain an                    Ingress section) are assumed to affect Ingress. If you want to write                    an egress-only policy, you must explicitly specify policyTypes [ 'Egress'                    ]. Likewise, if you want to write a policy that specifies that no                    egress is allowed, you must specify a policyTypes value that include                    'Egress' (since such a policy would not include an Egress section                    and would otherwise default to just [ 'Ingress' ]). This field is                    beta-level in 1.8"                  items:                    description: "Policy Type string describes the NetworkPolicy type This                      type is beta-level in 1.8"                    type: string                  type: array              required:              - podSelector              type: object          type: object    - name: v1beta2      served: false      storage: false      schema:        openAPIV3Schema:          description: "MultiNetworkPolicy is a CRD schema to provide NetworkPolicy            mechanism for net-attach-def which is specified by the Network Plumbing            Working Group. MultiNetworkPolicy is identical to Kubernetes NetworkPolicy,            See: https://kubernetes.io/docs/concepts/services-networking/network-policies/ ."          properties:            spec:              description: 'Specification of the desired behavior for this MultiNetworkPolicy.'              properties:                egress:                  description: "List of egress rules to be applied to the selected pods.                    Outgoing traffic is allowed if there are no NetworkPolicies selecting                    the pod (and cluster policy otherwise allows the traffic), OR if the                    traffic matches at least one egress rule across all of the NetworkPolicy                    objects whose podSelector matches the pod. If this field is empty                    then this NetworkPolicy limits all outgoing traffic (and serves solely                    to ensure that the pods it selects are isolated by default). This                    field is beta-level in 1.8"                  items:                    description: "NetworkPolicyEgressRule describes a particular set of                      traffic that is allowed out of pods matched by a NetworkPolicySpec's                      podSelector. The traffic must match both ports and to. This type                      is beta-level in 1.8"                    properties:                      ports:                        description: "List of destination ports for outgoing traffic. Each                          item in this list is combined using a logical OR. If this field                          is empty or missing, this rule matches all ports (traffic not                          restricted by port). If this field is present and contains at                          least one item, then this rule allows traffic only if the traffic                          matches at least one port in the list."                        items:                          description: "NetworkPolicyPort describes a port to allow traffic on"                          properties:                            port:                              anyOf:                                - type: integer                                - type: string                              description: "The port on the given protocol. This can either                                be a numerical or named port on a pod. If this field is                                not provided, this matches all port names and numbers."                              x-kubernetes-int-or-string: true                            endPort:                              type: integer                              description: "If set, indicates that the range of ports from                                port to endPort, inclusive, should be allowed by the policy.                                This field cannot be defined if the port field is not                                defined or if the port field is defined as a named (string)                                port. The endPort must be equal or greater than port."                            protocol:                              description: "The protocol (TCP, UDP, or SCTP) which traffic                                must match. If not specified, this field defaults to TCP."                              type: string                          type: object                        type: array                      to:                        description: "List of destinations for outgoing traffic of pods                          selected for this rule. Items in this list are combined using                          a logical OR operation. If this field is empty or missing, this                          rule matches all destinations (traffic not restricted by destination).                          If this field is present and contains at least one item, this                          rule allows traffic only if the traffic matches at least one                          item in the to list."                        items:                          description: "NetworkPolicyPeer describes a peer to allow traffic                        from. Only certain combinations of fields are allowed"                          properties:                            ipBlock:                              description: "IPBlock defines policy on a particular IPBlock.                                If this field is set then neither of the other fields                                can be."                              properties:                                cidr:                                  description: "CIDR is a string representing the IP Block                                    Valid examples are '192.168.1.1/24'"                                  type: string                                except:                                  description: "Except is a slice of CIDRs that should                                    not be included within an IP Block Valid examples                                    are '192.168.1.1/24' Except values will be rejected                                    if they are outside the CIDR range"                                  items:                                    type: string                                  type: array                              required:                              - cidr                              type: object                            namespaceSelector:                              description: "Selects Namespaces using cluster-scoped labels.                                This field follows standard label selector semantics;                                if present but empty, it selects all namespaces. \n If                                PodSelector is also set, then the NetworkPolicyPeer as                                a whole selects the Pods matching PodSelector in the Namespaces                                selected by NamespaceSelector. Otherwise it selects all                                Pods in the Namespaces selected by NamespaceSelector."                              properties:                                matchExpressions:                                  description: "matchExpressions is a list of label selector                                    requirements. The requirements are ANDed."                                  items:                                    description: "A label selector requirement is a selector                                      that contains values, a key, and an operator that                                      relates the key and values."                                    properties:                                      key:                                        description: "key is the label key that the selector                                          applies to."                                        type: string                                      operator:                                        description: "operator represents a key's relationship                                          to a set of values. Valid operators are In,                                          NotIn, Exists and DoesNotExist."                                        type: string                                      values:                                        description: "values is an array of string values.                                          If the operator is In or NotIn, the values array                                          must be non-empty. If the operator is Exists                                          or DoesNotExist, the values array must be empty.                                          This array is replaced during a strategic merge                                          patch."                                        items:                                          type: string                                        type: array                                    required:                                    - key                                    - operator                                    type: object                                  type: array                                matchLabels:                                  additionalProperties:                                    type: string                                  description: "matchLabels is a map of {key,value} pairs.                                    A single {key,value} in the matchLabels map is equivalent                                    to an element of matchExpressions, whose key field                                    is 'key', the operator is 'In', and the values array                                    contains only 'value'. The requirements are ANDed."                                  type: object                              type: object                            podSelector:                              description: "This is a label selector which selects Pods.                                This field follows standard label selector semantics;                                if present but empty, it selects all pods. \n If NamespaceSelector                                is also set, then the NetworkPolicyPeer as a whole selects                                the Pods matching PodSelector in the Namespaces selected                                by NamespaceSelector. Otherwise it selects the Pods matching                                PodSelector in the policy's own Namespace."                              properties:                                matchExpressions:                                  description: "matchExpressions is a list of label selector                                    requirements. The requirements are ANDed."                                  items:                                    description: "A label selector requirement is a selector                                      that contains values, a key, and an operator that                                      relates the key and values."                                    properties:                                      key:                                        description: key is the label key that the selector                                          applies to.                                        type: string                                      operator:                                        description: "operator represents a key's relationship                                          to a set of values. Valid operators are In,                                          NotIn, Exists and DoesNotExist."                                        type: string                                      values:                                        description: "values is an array of string values.                                          If the operator is In or NotIn, the values array                                          must be non-empty. If the operator is Exists                                          or DoesNotExist, the values array must be empty.                                          This array is replaced during a strategic merge                                          patch."                                        items:                                          type: string                                        type: array                                    required:                                    - key                                    - operator                                    type: object                                  type: array                                matchLabels:                                  additionalProperties:                                    type: string                                  description: "matchLabels is a map of {key,value} pairs.                                    A single {key,value} in the matchLabels map is equivalent                                    to an element of matchExpressions, whose key field                                    is 'key', the operator is 'In', and the values array                                    contains only 'value'. The requirements are ANDed."                                  type: object                              type: object                          type: object                        type: array                    type: object                  type: array                ingress:                  description: "List of ingress rules to be applied to the selected pods.                    Traffic is allowed to a pod if there are no NetworkPolicies selecting                    the pod (and cluster policy otherwise allows the traffic), OR if the                    traffic source is the pod's local node, OR if the traffic matches                    at least one ingress rule across all of the NetworkPolicy objects                    whose podSelector matches the pod. If this field is empty then this                    NetworkPolicy does not allow any traffic (and serves solely to ensure                    that the pods it selects are isolated by default)"                  items:                    description: "NetworkPolicyIngressRule describes a particular set of                      traffic that is allowed to the pods matched by a NetworkPolicySpec's                      podSelector. The traffic must match both ports and from."                    properties:                      from:                        description: "List of sources which should be able to access the                          pods selected for this rule. Items in this list are combined                          using a logical OR operation. If this field is empty or missing,                          this rule matches all sources (traffic not restricted by source).                          If this field is present and contains at least one item, this                          rule allows traffic only if the traffic matches at least one                          item in the from list."                        items:                          description: NetworkPolicyPeer describes a peer to allow traffic                            from. Only certain combinations of fields are allowed                          properties:                            ipBlock:                              description: "IPBlock defines policy on a particular IPBlock.                                If this field is set then neither of the other fields                                can be."                              properties:                                cidr:                                  description: "CIDR is a string representing the IP Block                                    Valid examples are '192.168.1.1/24'"                                  type: string                                except:                                  description: "Except is a slice of CIDRs that should                                    not be included within an IP Block Valid examples                                    are '192.168.1.1/24' Except values will be rejected                                    if they are outside the CIDR range"                                  items:                                    type: string                                  type: array                              required:                              - cidr                              type: object                            namespaceSelector:                              description: "Selects Namespaces using cluster-scoped labels.                                This field follows standard label selector semantics;                                if present but empty, it selects all namespaces. \n If                                PodSelector is also set, then the NetworkPolicyPeer as                                a whole selects the Pods matching PodSelector in the Namespaces                                selected by NamespaceSelector. Otherwise it selects all                                Pods in the Namespaces selected by NamespaceSelector."                              properties:                                matchExpressions:                                  description: "matchExpressions is a list of label selector                                    requirements. The requirements are ANDed."                                  items:                                    description: "A label selector requirement is a selector                                      that contains values, a key, and an operator that                                      relates the key and values."                                    properties:                                      key:                                        description: "key is the label key that the selector                                          applies to."                                        type: string                                      operator:                                        description: "operator represents a key's relationship                                          to a set of values. Valid operators are In,                                          NotIn, Exists and DoesNotExist."                                        type: string                                      values:                                        description: "values is an array of string values.                                          If the operator is In or NotIn, the values array                                          must be non-empty. If the operator is Exists                                          or DoesNotExist, the values array must be empty.                                          This array is replaced during a strategic merge                                          patch."                                        items:                                          type: string                                        type: array                                    required:                                    - key                                    - operator                                    type: object                                  type: array                                matchLabels:                                  additionalProperties:                                    type: string                                  description: "matchLabels is a map of {key,value} pairs.                                    A single {key,value} in the matchLabels map is equivalent                                    to an element of matchExpressions, whose key field                                    is 'key', the operator is 'In', and the values array                                    contains only 'value'. The requirements are ANDed."                                  type: object                              type: object                            podSelector:                              description: "This is a label selector which selects Pods.                                This field follows standard label selector semantics;                                if present but empty, it selects all pods. \n If NamespaceSelector                                is also set, then the NetworkPolicyPeer as a whole selects                                the Pods matching PodSelector in the Namespaces selected                                by NamespaceSelector. Otherwise it selects the Pods matching                                PodSelector in the policy's own Namespace."                              properties:                                matchExpressions:                                  description: "matchExpressions is a list of label selector                                    requirements. The requirements are ANDed."                                  items:                                    description: "A label selector requirement is a selector                                      that contains values, a key, and an operator that                                      relates the key and values."                                    properties:                                      key:                                        description: "key is the label key that the selector                                          applies to."                                        type: string                                      operator:                                        description: "operator represents a key's relationship                                          to a set of values. Valid operators are In,                                          NotIn, Exists and DoesNotExist."                                        type: string                                      values:                                        description: "values is an array of string values.                                          If the operator is In or NotIn, the values array                                          must be non-empty. If the operator is Exists                                          or DoesNotExist, the values array must be empty.                                          This array is replaced during a strategic merge                                          patch."                                        items:                                          type: string                                        type: array                                    required:                                    - key                                    - operator                                    type: object                                  type: array                                matchLabels:                                  additionalProperties:                                    type: string                                  description: "matchLabels is a map of {key,value} pairs.                                    A single {key,value} in the matchLabels map is equivalent                                    to an element of matchExpressions, whose key field                                    is 'key', the operator is 'In', and the values array                                    contains only 'value'. The requirements are ANDed."                                  type: object                              type: object                          type: object                        type: array                      ports:                        description: "List of ports which should be made accessible on                          the pods selected for this rule. Each item in this list is combined                          using a logical OR. If this field is empty or missing, this                          rule matches all ports (traffic not restricted by port). If                          this field is present and contains at least one item, then this                          rule allows traffic only if the traffic matches at least one                          port in the list."                        items:                          description: NetworkPolicyPort describes a port to allow traffic                            on                          properties:                            port:                              anyOf:                                - type: integer                                - type: string                              description: "The port on the given protocol. This can either                                be a numerical or named port on a pod. If this field is                                not provided, this matches all port names and numbers."                              x-kubernetes-int-or-string: true                            endPort:                              type: integer                              description: "If set, indicates that the range of ports from                                port to endPort, inclusive, should be allowed by the policy.                                This field cannot be defined if the port field is not                                defined or if the port field is defined as a named (string)                                port. The endPort must be equal or greater than port."                            protocol:                              description: "The protocol (TCP, UDP, or SCTP) which traffic                                must match. If not specified, this field defaults to TCP."                              type: string                          type: object                        type: array                    type: object                  type: array                podSelector:                  description: "This is a label selector which selects Pods.                    This field follows standard label selector semantics;                    if present but empty, it selects all pods. \n If NamespaceSelector                    is also set, then the NetworkPolicyPeer as a whole selects                    the Pods matching PodSelector in the Namespaces selected                    by NamespaceSelector. Otherwise it selects the Pods matching                    PodSelector in the policy's own Namespace."                  properties:                    matchExpressions:                      description: "matchExpressions is a list of label selector                        requirements. The requirements are ANDed."                      items:                        description: "A label selector requirement is a selector                          that contains values, a key, and an operator that                          relates the key and values."                        properties:                          key:                            description: "key is the label key that the selector applies to."                            type: string                          operator:                            description: "operator represents a key's relationship                              to a set of values. Valid operators are In,                              NotIn, Exists and DoesNotExist."                            type: string                          values:                            description: "values is an array of string values.                              If the operator is In or NotIn, the values array                              must be non-empty. If the operator is Exists                              or DoesNotExist, the values array must be empty.                              This array is replaced during a strategic merge                              patch."                            items:                              type: string                            type: array                        required:                        - key                        - operator                        type: object                      type: array                    matchLabels:                      additionalProperties:                        type: string                        description: "matchLabels is a map of {key,value} pairs.                          A single {key,value} in the matchLabels map is equivalent                          to an element of matchExpressions, whose key field                          is 'key', the operator is 'In', and the values array                          contains only 'value'. The requirements are ANDed."                      type: object                  type: object                policyTypes:                  description: "List of rule types that the NetworkPolicy relates to. Valid                    options are 'Ingress', 'Egress', or 'Ingress,Egress'. If this field                    is not specified, it will default based on the existence of Ingress                    or Egress rules; policies that contain an Egress section are assumed                    to affect Egress, and all policies (whether or not they contain an                    Ingress section) are assumed to affect Ingress. If you want to write                    an egress-only policy, you must explicitly specify policyTypes [ 'Egress'                    ]. Likewise, if you want to write a policy that specifies that no                    egress is allowed, you must specify a policyTypes value that include                    'Egress' (since such a policy would not include an Egress section                    and would otherwise default to just [ 'Ingress' ]). This field is                    beta-level in 1.8"                  items:                    description: "Policy Type string describes the NetworkPolicy type This                      type is beta-level in 1.8"                    type: string                  type: array              required:              - podSelector              type: object          type: object---kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata:  name: multi-networkpolicyrules:  - apiGroups: ["k8s.cni.cncf.io"]    resources:      - '*'    verbs:      - '*'  - apiGroups:      - ""    resources:      - pods      - namespaces    verbs:      - list      - watch      - get  # Watch for changes to Kubernetes NetworkPolicies.  - apiGroups: ["networking.k8s.io"]    resources:      - networkpolicies    verbs:      - watch      - list  - apiGroups:      - ""      - events.k8s.io    resources:      - events    verbs:      - create      - patch      - update---kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata:  name: multi-networkpolicyroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: multi-networkpolicysubjects:- kind: ServiceAccount  name: multi-networkpolicy  namespace: kube-system---apiVersion: v1kind: ServiceAccountmetadata:  name: multi-networkpolicy  namespace: kube-system---kind: ConfigMapapiVersion: v1metadata:  name: multi-networkpolicy-custom-v4-rules  namespace: kube-system  labels:    tier: node    app: multi-networkpolicydata:  custom-v4-rules.txt: |    # accept redirect    -p icmp --icmp-type redirect -j ACCEPT    # accept fragmentation-needed (for MTU discovery)    -p icmp --icmp-type fragmentation-needed -j ACCEPT---kind: ConfigMapapiVersion: v1metadata:  name: multi-networkpolicy-custom-v6-rules  namespace: kube-system  labels:    tier: node    app: multi-networkpolicydata:  custom-v6-rules.txt: |    # accept NDP    -p icmpv6 --icmpv6-type neighbor-solicitation -j ACCEPT    -p icmpv6 --icmpv6-type neighbor-advertisement -j ACCEPT    # accept RA/RS    -p icmpv6 --icmpv6-type router-solicitation -j ACCEPT    -p icmpv6 --icmpv6-type router-advertisement -j ACCEPT    # accept redirect    -p icmpv6 --icmpv6-type redirect -j ACCEPT    # accept packet-too-big (for MTU discovery)    -p icmpv6 --icmpv6-type packet-too-big -j ACCEPT---apiVersion: apps/v1kind: DaemonSetmetadata:  name: multi-networkpolicy-ds-amd64  namespace: kube-system  labels:    tier: node    app: multi-networkpolicy    name: multi-networkpolicyspec:  selector:    matchLabels:      name: multi-networkpolicy  updateStrategy:    type: RollingUpdate  template:    metadata:      labels:        tier: node        app: multi-networkpolicy        name: multi-networkpolicy    spec:      hostNetwork: true      nodeSelector:        kubernetes.io/arch: amd64      tolerations:      - operator: Exists        effect: NoSchedule      serviceAccountName: multi-networkpolicy      containers:      - name: multi-networkpolicy        image: harbor.yusur.tech/yusur_ovn/multi-networkpolicy-iptables:latest        imagePullPolicy: Always        command: ["/usr/bin/multi-networkpolicy-iptables"]        args:        - "--host-prefix=/host"        # change this if runtime is different that crio default        - "--container-runtime-endpoint=/run/containerd/containerd.sock"        # uncomment this if you want to store iptables rules        - "--pod-iptables=/var/lib/multi-networkpolicy/iptables"        # uncomment this if you need to accept link-local address traffic        #- "--allow-ipv6-src-prefix=fe80::/10"        #- "--allow-ipv6-dst-prefix=fe80::/10"        # uncomment this if you need to add custom iptables rules defined above configmap        #- "--custom-v4-ingress-rule-file=/etc/multi-networkpolicy/rules/custom-v4-rules.txt"        #- "--custom-v4-egress-rule-file=/etc/multi-networkpolicy/rules/custom-v4-rules.txt"        #- "--custom-v6-ingress-rule-file=/etc/multi-networkpolicy/rules/custom-v6-rules.txt"        #- "--custom-v6-egress-rule-file=/etc/multi-networkpolicy/rules/custom-v6-rules.txt"        # uncomment if you want to accept ICMP/ICMPv6 traffic        #- "--accept-icmp"        #- "--accept-icmpv6"        resources:          requests:            cpu: "100m"            memory: "80Mi"          limits:            cpu: "100m"            memory: "150Mi"        securityContext:          privileged: true          capabilities:            add: ["SYS_ADMIN", "NET_ADMIN"]        volumeMounts:        - name: host          mountPath: /host        - name: var-lib-multinetworkpolicy          mountPath: /var/lib/multi-networkpolicy        - name: multi-networkpolicy-custom-rules          mountPath: /etc/multi-networkpolicy/rules          readOnly: true      volumes:        - name: host          hostPath:            path: /        - name: var-lib-multinetworkpolicy          hostPath:            path: /var/lib/multi-networkpolicy        - name: multi-networkpolicy-custom-rules          projected:            sources:              - configMap:                  name: multi-networkpolicy-custom-v4-rules              - configMap:                  name: multi-networkpolicy-custom-v6-rules

配置辅助网络

ovn-k8s 默认网络架构图

Routed - layer 3 - topology

逻辑网络拓扑

++注意：默认网络与第二网络并不相通++

• 此拓扑是集群默认网络拓扑的简化 - 但没有出口。

• 每个节点都有一个逻辑交换机（每个节点都有不同的子网）以及一个互连所有逻辑交换机的路由器。

• 网络仅允许东/西流量。

创建网络

apiVersion: k8s.cni.cncf.io/v1kind: NetworkAttachmentDefinitionmetadata:  name: l3-network  namespace: kube-systemspec:  config: |2    {            "cniVersion": "0.3.1",            "name": "l3-network",            "type": "ovn-k8s-cni-overlay",            "topology": "layer3",            "subnets": "10.33.0.0/16/24", #subnets 属性指示跨集群和每个节点的子网。上面的示例意味着您的网络有一个 /16 子网，但每个节点都有一个 /24 子网            "mtu": 1400,            "netAttachDefName": "kube-system/l3-network"    }

网络配置参考:

name (string, required): 网络的名称。

type (string, required): “ovn-k8s-cni-overlay”。 必须是这个值

topology (string, required): “layer3”。

subnets (string, optional):  以逗号分隔的子网列表。当提供多个子网时，用户将从每个子网获得一个IP。

mtu (integer, optional): 显式将 MTU 设置为指定值。默认为内核选择的值。

netAttachDefName (string, required): 必须与对象的 / 匹配。

OVN 信息

# ovn中会创建一个逻辑路由器和多个逻辑交换机# 查看逻辑路由器[root@master001 ~]# ovn-nbctl lr-list caed4518-4b52-40d8-b79f-2a39628856cd (GR_master001)ed509ce6-e784-4082-b6e9-355ffb7d7f76 (GR_work001)cac0d9f6-7e5a-4521-a2a4-b4aac6e87067 (GR_work002)601cfe5f-975c-43c8-9257-7235125ca147 (l3.network_ovn_cluster_router) # 第二网络创建逻辑路由器f624c445-8b54-4722-91af-ae4008bf66ec (ovn_cluster_router)[root@master001 ~]# ovn-nbctl find Logical_Router name=l3.network_ovn_cluster_router_uuid               : 601cfe5f-975c-43c8-9257-7235125ca147copp                : 7b79199c-5e38-442d-b552-1865844cf460enabled             : []external_ids        : {k8s-cluster-router=yes, k8s-ovn-topo-version="5", "k8s.ovn.org/network"=l3-network, "k8s.ovn.org/topology"=layer3}load_balancer       : []load_balancer_group : []name                : l3.network_ovn_cluster_routernat                 : []options             : {always_learn_from_arp_request="false"}policies            : []ports               : [4391bd42-60f4-41a6-93b0-8cfa0b3fe964, f235ac3d-b99f-4cee-9854-758c8b51ac5f, f956c1d6-1f09-498a-a5eb-cb5a4115e905]static_routes       : []# 查看逻辑交换机，每个节点对应一个[root@master001 ~]# ovn-nbctl ls-list81b94dd9-a395-474c-9c0b-150e95bdbe38 (ext_master001)aa556a3e-1aa3-4aff-b242-b48c58b229b2 (ext_work001)4d54339b-c5ff-4479-a4f0-8e12646d9fa8 (ext_work002)d8ee50bc-ee80-4576-ac2e-79a065f800bd (join)e447923e-6d33-4acd-950f-17b57aa3d687 (l3.network_master001) #第二网络创建逻辑交换机0a8b2d9b-431b-4625-b715-044ab3bab41d (l3.network_work001)#第二网络创建逻辑交换机47270897-5c07-4115-bca9-90cb61e57d43 (l3.network_work002)#第二网络创建逻辑交换机be036c87-2c44-4168-9dea-da52eeb8b695 (master001)b8ec646d-d9c7-4e5a-bdbe-ba0815eb6dfd (work001)ebec33e6-4cd8-41fb-8e27-73cb6dd0e337 (work002)[root@master001 ~]# ovn-nbctl find Logical_Switch  name=l3.network_master001_uuid               : e447923e-6d33-4acd-950f-17b57aa3d687acls                : []copp                : []dns_records         : []external_ids        : {"k8s.ovn.org/network"=l3-network, "k8s.ovn.org/topology"=layer3}forwarding_groups   : []load_balancer       : []load_balancer_group : []name                : l3.network_master001other_config        : {exclude_ips="10.33.0.2", subnet="10.33.0.0/24"}ports               : [30574080-f288-4488-a35b-488258fbd1eb]qos_rules           : []

使用网络

apiVersion: v1kind: Servicemetadata:  name: nginx1-l3  namespace: defaultspec:  selector:    app: nginx1-l3  ports:  - protocol: TCP    port: 80    targetPort: server---apiVersion: v1kind: Podmetadata:  labels:    app: nginx1-l3  annotations:    k8s.v1.cni.cncf.io/networks: kube-system/l3-network@eth1 #标识使用多网络  name: nginx1-l3  namespace: defaultspec:  containers:  - image: harbor.yusur.tech/leid/ubuntu:22.04    imagePullPolicy: IfNotPresent    name: nginx1-l3    ports:    - containerPort: 80      name: server    securityContext:      runAsUser: 0      privileged: true  restartPolicy: Always  nodeSelector:    kubernetes.io/hostname: work001# 查看ovn 信息# 查看work001 节点对应交换机信息[root@master001 ~]# ovn-nbctl find Logical_Switch  name=l3.network_work001_uuid               : 0a8b2d9b-431b-4625-b715-044ab3bab41dacls                : []copp                : []dns_records         : []external_ids        : {"k8s.ovn.org/network"=l3-network, "k8s.ovn.org/topology"=layer3}forwarding_groups   : []load_balancer       : []load_balancer_group : []name                : l3.network_work001other_config        : {exclude_ips="10.33.2.2", subnet="10.33.2.0/24"}ports               : [ab1ecf25-8a92-47e0-b913-c79e33fd5e23, b675abc1-be1a-4248-9f73-b8dff7454957]qos_rules           : []# 查看pod 对应的 port 信息[root@master001 ~]# ovn-nbctl find Logical_Switch_Port |grep -A 16 b675abc1-be1a-4248-9f73-b8dff7454957_uuid               : b675abc1-be1a-4248-9f73-b8dff7454957addresses           : ["0a:58:0a:21:02:03 10.33.2.3"]dhcpv4_options      : []dhcpv6_options      : []dynamic_addresses   : []enabled             : []external_ids        : {"k8s.ovn.org/nad"="kube-system/l3-network", "k8s.ovn.org/network"=l3-network, "k8s.ovn.org/topology"=layer3, namespace=default, pod="true"}ha_chassis_group    : []mirror_rules        : []name                : kube.system.l3.network_default_nginx1-l3options             : {iface-id-ver="a7fad855-9a37-4918-8aaa-b10627cdfe91", requested-chassis=work001}parent_name         : []port_security       : ["0a:58:0a:21:02:03 10.33.2.3"]tag                 : []tag_request         : []type                : ""up                  : true

测试网络

#查看pod1 ip信息，默认网络ip：10.124.1.7，第二网络ip：10.33.2.3root@work001:~# kubectl  get pod nginx1-l3 -o=jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'[{    "name": "ovn-kubernetes",    "interface": "eth0",    "ips": [        "10.124.1.7"    ],    "mac": "0a:58:0a:7c:01:07",    "default": true,    "dns": {}},{    "name": "kube-system/l3-network",    "interface": "eth1",    "ips": [        "10.33.2.3"    ],    "mac": "0a:58:0a:21:02:03",    "dns": {}}]#查看pod2 ip信息，默认网络ip：10.124.2.6，第二网络ip：10.33.1.3root@work001:~# kubectl  get pod nginx2-l3 -o=jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'[{    "name": "ovn-kubernetes",    "interface": "eth0",    "ips": [        "10.124.2.6"    ],    "mac": "0a:58:0a:7c:02:06",    "default": true,    "dns": {}},{    "name": "kube-system/l3-network",    "interface": "eth1",    "ips": [        "10.33.1.3"    ],    "mac": "0a:58:0a:21:01:03",    "dns": {}}]

pod 与 pod

# pod1 ping pod2，pod2 tcpdump 抓包# 查看pod1 的网络信息root@work001:~# kubectl  exec -it  nginx1-l3      -- route -n Kernel IP routing tableDestination     Gateway         Genmask         Flags Metric Ref    Use Iface0.0.0.0         10.124.1.1      0.0.0.0         UG    0      0        0 eth010.33.0.0       10.33.2.1       255.255.0.0     UG    0      0        0 eth110.33.2.0       0.0.0.0         255.255.255.0   U     0      0        0 eth110.86.0.0       10.124.1.1      255.255.0.0     UG    0      0        0 eth010.124.0.0      10.124.1.1      255.255.0.0     UG    0      0        0 eth010.124.1.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0100.64.0.0      10.124.1.1      255.255.0.0     UG    0      0        0 eth0# pod1 ping pod2:10.33.1.3root@work001:~# kubectl  exec -it nginx1-l3 -- ping -c 1 10.33.1.3PING 10.33.1.3 (10.33.1.3) 56(84) bytes of data.64 bytes from 10.33.1.3: icmp_seq=1 ttl=63 time=1.41 ms--- 10.33.1.3 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 1.405/1.405/1.405/0.000 ms# pod2 抓包root@work002:~# kubectl  exec -it nginx2-l3 -- tcpdump -enn -i eth1 icmptcpdump: verbose output suppressed, use -v[v]... for full protocol decodelistening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes09:03:49.189769 0a:58:0a:21:01:01 > 0a:58:0a:21:01:03, ethertype IPv4 (0x0800), length 98: 10.33.2.3 > 10.33.1.3: ICMP echo request, id 29942, seq 1, length 6409:03:49.189800 0a:58:0a:21:01:03 > 0a:58:0a:21:01:01, ethertype IPv4 (0x0800), length 98: 10.33.1.3 > 10.33.2.3: ICMP echo reply, id 29942, seq 1, length 64

物理网络拓扑

Switched - layer 2 - topology

逻辑网络拓扑

++注意：默认网络与第二网络并不相通++

创建网络

apiVersion: k8s.cni.cncf.io/v1kind: NetworkAttachmentDefinitionmetadata:  name: l2-network  namespace: kube-systemspec:  config: |2    {            "cniVersion": "0.3.1",            "name": "l2-network",            "type": "ovn-k8s-cni-overlay",            "topology":"layer2",            "subnets": "10.22.0.0/16",            "mtu": 1400,            "netAttachDefName": "kube-system/l2-network",            "excludeSubnets": "10.22.200.0/24"    }

网络配置参考:

name (string, required): 网络的名称。

type (string, required): “ovn-k8s-cni-overlay”。 必须是这个值

topology (string, required): “layer2”。

subnets (string, optional):  以逗号分隔的子网列表。当提供多个子网时，用户将从每个子网获得一个IP。

mtu (integer, optional): 显式将 MTU 设置为指定值。默认为内核选择的值。

netAttachDefName (string, required): 必须与对象的 / 匹配。

excludeSubnets (string, optional): 以逗号分隔的 CIDR/IP 列表。这些 IP 将从可分配的 IP 池中删除，并且永远不会移交给 Pod。

注意：

• 当省略subnets属性时，实现网络的逻辑交换机将仅提供第2层通信，用户必须为Pod配置IP。端口安全只能防止 MAC 欺骗。

• switched - layer2 - 辅助网络仅允许东/西流量。

• 当对多个区域启用互连功能时，不支持此拓扑。

OVN 信息

# 新建一个逻辑交换机[root@master001 ~]# ovn-nbctl  ls-list 81b94dd9-a395-474c-9c0b-150e95bdbe38 (ext_master001)aa556a3e-1aa3-4aff-b242-b48c58b229b2 (ext_work001)4d54339b-c5ff-4479-a4f0-8e12646d9fa8 (ext_work002)d8ee50bc-ee80-4576-ac2e-79a065f800bd (join)43f53a4a-2952-4ad1-bd84-ece4ef70dc38 (l2.network_ovn_layer2_switch) # 第二网络新建逻辑交换机be036c87-2c44-4168-9dea-da52eeb8b695 (master001)b8ec646d-d9c7-4e5a-bdbe-ba0815eb6dfd (work001)ebec33e6-4cd8-41fb-8e27-73cb6dd0e337 (work002)# 查看交换机信息[root@master001 ~]# ovn-nbctl  find Logical_Switch name=l2.network_ovn_layer2_switch_uuid               : 43f53a4a-2952-4ad1-bd84-ece4ef70dc38acls                : []copp                : []dns_records         : []external_ids        : {k8s-ovn-topo-version="5", "k8s.ovn.org/network"=l2-network, "k8s.ovn.org/topology"=layer2}forwarding_groups   : []load_balancer       : []load_balancer_group : []name                : l2.network_ovn_layer2_switchother_config        : {subnet="10.22.0.0/16"}ports               : [43484d88-1b3e-4634-8cf6-b267d58642f0, d288ffdc-bc30-485b-a8fd-bb062d5e76f3]qos_rules           : []

使用网络

apiVersion: v1kind: Servicemetadata:  name: nginx1-l2  namespace: defaultspec:  selector:    app: nginx1-l2  ports:  - protocol: TCP    port: 80    targetPort: server---apiVersion: v1kind: Podmetadata:  labels:    app: nginx1-l2  annotations:    k8s.v1.cni.cncf.io/networks: kube-system/l2-network@eth1 #指定第二网络  name: nginx1-l2  namespace: defaultspec:  containers:  - image: harbor.yusur.tech/leid/ubuntu:22.04    imagePullPolicy: IfNotPresent    name: nginx1-l2    ports:    - containerPort: 80      name: server    securityContext:      runAsUser: 0      privileged: true  restartPolicy: Always  nodeSelector:    kubernetes.io/hostname: work001# 查看逻辑交换机 port信息# pod1 的port 信息[root@master001 ~]# ovn-nbctl find Logical_Switch_Port | grep -A 16 43484d88-1b3e-4634-8cf6-b267d58642f0 # pod1 port信息_uuid               : 43484d88-1b3e-4634-8cf6-b267d58642f0addresses           : ["0a:58:0a:16:00:01 10.22.0.1"]dhcpv4_options      : []dhcpv6_options      : []dynamic_addresses   : []enabled             : []external_ids        : {"k8s.ovn.org/nad"="kube-system/l2-network", "k8s.ovn.org/network"=l2-network, "k8s.ovn.org/topology"=layer2, namespace=default, pod="true"}ha_chassis_group    : []mirror_rules        : []name                : kube.system.l2.network_default_nginx1-l2options             : {iface-id-ver="f4bb6a91-019b-4612-b6aa-88fc663dd515", requested-chassis=work001}parent_name         : []port_security       : ["0a:58:0a:16:00:01 10.22.0.1"]tag                 : []tag_request         : []type                : ""up                  : true# pod2 的port 信息[root@master001 ~]# ovn-nbctl find Logical_Switch_Port | grep -A 16 d288ffdc-bc30-485b-a8fd-bb062d5e76f3_uuid               : d288ffdc-bc30-485b-a8fd-bb062d5e76f3addresses           : ["0a:58:0a:16:00:02 10.22.0.2"]dhcpv4_options      : []dhcpv6_options      : []dynamic_addresses   : []enabled             : []external_ids        : {"k8s.ovn.org/nad"="kube-system/l2-network", "k8s.ovn.org/network"=l2-network, "k8s.ovn.org/topology"=layer2, namespace=default, pod="true"}ha_chassis_group    : []mirror_rules        : []name                : kube.system.l2.network_default_nginx2-l2options             : {iface-id-ver="56ffc5ab-b5eb-456e-b5cc-b77252c29ae3", requested-chassis=work002}parent_name         : []port_security       : ["0a:58:0a:16:00:02 10.22.0.2"]tag                 : []tag_request         : []type                : ""up                  : true

测试网络

#查看pod1 ip信息，默认网络ip：10.124.1.8，第二网络ip：10.22.0.1root@work001:~# kubectl  get pod nginx1-l2 -o=jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'[{    "name": "ovn-kubernetes",    "interface": "eth0",    "ips": [        "10.124.1.8"    ],    "mac": "0a:58:0a:7c:01:08",    "default": true,    "dns": {}},{    "name": "kube-system/l2-network",    "interface": "eth1",    "ips": [        "10.22.0.1"    ],    "mac": "0a:58:0a:16:00:01",    "dns": {}}]#查看pod2 ip信息，默认网络ip：10.124.2.7，第二网络ip：10.22.0.2root@work001:~# kubectl  get pod nginx2-l2 -o=jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'[{    "name": "ovn-kubernetes",    "interface": "eth0",    "ips": [        "10.124.2.7"    ],    "mac": "0a:58:0a:7c:02:07",    "default": true,    "dns": {}},{    "name": "kube-system/l2-network",    "interface": "eth1",    "ips": [        "10.22.0.2"    ],    "mac": "0a:58:0a:16:00:02",    "dns": {}}]

pod 与 pod

# pod1 ping pod2，pod2 tcpdump 抓包# 查看pod1 的网络信息root@work001:~# kubectl  exec -it  nginx1-l2 -- route -n Kernel IP routing tableDestination     Gateway         Genmask         Flags Metric Ref    Use Iface0.0.0.0         10.124.1.1      0.0.0.0         UG    0      0        0 eth010.22.0.0       0.0.0.0         255.255.0.0     U     0      0        0 eth110.86.0.0       10.124.1.1      255.255.0.0     UG    0      0        0 eth010.124.0.0      10.124.1.1      255.255.0.0     UG    0      0        0 eth010.124.1.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0100.64.0.0      10.124.1.1      255.255.0.0     UG    0      0        0 eth0# pod1 ping pod2:10.22.0.2root@work001:~# kubectl  exec -it  nginx1-l2 -- ping -c1 10.22.0.2PING 10.22.0.2 (10.22.0.2) 56(84) bytes of data.64 bytes from 10.22.0.2: icmp_seq=1 ttl=64 time=1.02 ms--- 10.22.0.2 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 1.024/1.024/1.024/0.000 ms# pod2 抓包root@work002:~# kubectl  exec -it nginx2-l3 -- tcpdump -enn -i eth1 icmptcpdump: verbose output suppressed, use -v[v]... for full protocol decodelistening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes10:02:05.653790 0a:58:0a:16:00:01 > 0a:58:0a:16:00:02, ethertype IPv4 (0x0800), length 98: 10.22.0.1 > 10.22.0.2: ICMP echo request, id 7475, seq 1, length 6410:02:05.653967 0a:58:0a:16:00:02 > 0a:58:0a:16:00:01, ethertype IPv4 (0x0800), length 98: 10.22.0.2 > 10.22.0.1: ICMP echo reply, id 7475, seq 1, length 64

物理网络拓扑

Switched - localnet - topology

逻辑网络拓扑

注意：默认网络与第二网络并不相通

创建网络

apiVersion: k8s.cni.cncf.io/v1kind: NetworkAttachmentDefinitionmetadata:  name: localnet-network  namespace: kube-systemspec:  config: |2    {            "cniVersion": "0.3.1",            "name": "localnet-network",            "type": "ovn-k8s-cni-overlay",            "topology":"localnet",            "subnets": "192.168.100.0/24", #设置与物理网络同网段            "mtu": 1500,            "netAttachDefName": "kube-system/localnet-network",            "excludeSubnets": "192.168.100.1/32,192.168.100.10/32,192.168.100.101/32,192.168.100.102/32" #排除物理网段上已经使用的ip    }

注意：为了连接到物理网络，需要在 node 针对该本地网络正确配置 ovn-bridge-mappings。

网络配置参考:

name (string, required): 网络的名称。

type (string, required): “ovn-k8s-cni-overlay”。 必须是这个值

topology (string, required): “localnet”。

subnets (string, optional):  以逗号分隔的子网列表。当提供多个子网时，用户将从每个子网获得一个IP。

mtu (integer, optional): 显式将 MTU 设置为指定值。默认为内核选择的值。

netAttachDefName (string, required): 必须与对象的 / 匹配。

excludeSubnets (string, optional): 以逗号分隔的 CIDR/IP 列表。这些 IP 将从可分配的 IP 池中删除，并且永远不会移交给 Pod。

vlanID (integer, optional):分配 VLAN 标记。默认为无。

注意：

• 当省略subnets属性时，实现网络的逻辑交换机将仅提供第2层通信，用户必须为Pod配置IP。端口安全只能防止 MAC 欺骗。

• 当对多个区域启用互连功能时，不支持此拓扑。

OVN 信息

逻辑交换机localnet.network_ovn_localnet_switch

[root@master001 ~]# ovn-nbctl ls-list 81b94dd9-a395-474c-9c0b-150e95bdbe38 (ext_master001)aa556a3e-1aa3-4aff-b242-b48c58b229b2 (ext_work001)4d54339b-c5ff-4479-a4f0-8e12646d9fa8 (ext_work002)d8ee50bc-ee80-4576-ac2e-79a065f800bd (join)54c4887d-8c53-480f-a2bd-917357211726 (localnet.network_ovn_localnet_switch) #新增逻辑交换机be036c87-2c44-4168-9dea-da52eeb8b695 (master001)b8ec646d-d9c7-4e5a-bdbe-ba0815eb6dfd (work001)ebec33e6-4cd8-41fb-8e27-73cb6dd0e337 (work002)# 查看ls信息[root@master001 ~]# ovn-nbctl find Logical_Switch name=localnet.network_ovn_localnet_switch_uuid               : 54c4887d-8c53-480f-a2bd-917357211726acls                : []copp                : []dns_records         : []external_ids        : {k8s-ovn-topo-version="5", "k8s.ovn.org/network"=localnet-network, "k8s.ovn.org/topology"=localnet}forwarding_groups   : []load_balancer       : []load_balancer_group : []name                : localnet.network_ovn_localnet_switchother_config        : {subnet="192.168.100.0/24"}ports               : [c1b80942-bae4-433d-a228-7b0638fe6150]qos_rules           : []#查看lsp信息[root@master001 ~]# ovn-nbctl find Logical_Switch_Port | grep -A 16 c1b80942-bae4-433d-a228-7b0638fe6150_uuid               : c1b80942-bae4-433d-a228-7b0638fe6150addresses           : [unknown]dhcpv4_options      : []dhcpv6_options      : []dynamic_addresses   : []enabled             : []external_ids        : {}ha_chassis_group    : []mirror_rules        : []name                : localnet.network_ovn_localnet_portoptions             : {network_name=localnet.network_br-localnet}parent_name         : []port_security       : []tag                 : []tag_request         : []type                : localnetup                  : false

配置外网

设置与物理网络连接需要在每个 node 节点上设置ovn-bridge-mappings 逻辑和物理交换机映射关系

介绍：ovn-bridge-mappings参数是Open Virtual Network（OVN）中的一个配置选项，它用于定义逻辑交换机与物理交换机之间的映射关系。

在OVN中，逻辑交换机（Logical Switch）是虚拟网络中的一个组件，它负责虚拟机之间的通信。而物理交换机（Physical Switch）是实际的网络设备，负责物理网络的连接。

ovn-bridge-mappings参数的作用是将逻辑交换机与物理交换机之间的端口进行映射。它定义了逻辑交换机和物理交换机之间的对应关系，使得虚拟机可以通过逻辑交换机与物理交换机进行通信。

在每个节点上执行如下操作：

ovs-vsctl set open_vswitch .  external_ids:ovn-bridge-mappings="physnet:brens3f2,localnet.network_br-localnet:br-out"# 添加brovs-vsctl add-br br-out # 添加port，选用ens3f1物理网络通信ovs-vsctl add-port br-out ens3f1# 配置br-out ip信息ifconfig ens3f1 0.0.0.0ifconfig br-out 192.168.100.101/24 up# 查看配置结果root@work001:~# ovs-vsctl  show 6345984f-07df-4b81-924c-7c4d55e1f5ef    Bridge brens3f2 #默认网络的配置        fail_mode: standalone        Port brens3f2            Interface brens3f2                type: internal        Port patch-brens3f2_work001-to-br-int            Interface patch-brens3f2_work001-to-br-int                type: patch                options: {peer=patch-br-int-to-brens3f2_work001}        Port ens3f2            Interface ens3f2    Bridge br-out #第二网络配置        Port ens3f1            Interface ens3f1        Port br-out            Interface br-out                type: internal        Port patch-localnet.network_ovn_localnet_port-to-br-int            Interface patch-localnet.network_ovn_localnet_port-to-br-int                type: patch                options: {peer=patch-br-int-to-localnet.network_ovn_localnet_port}    Bridge br-int        fail_mode: secure        datapath_type: system        Port br-int            Interface br-int                type: internal        Port ovn-5bd00c-0            Interface ovn-5bd00c-0                type: geneve                options: {csum="true", key=flow, remote_ip="192.168.122.102"}        Port patch-br-int-to-brens3f2_work001            Interface patch-br-int-to-brens3f2_work001                type: patch                options: {peer=patch-brens3f2_work001-to-br-int}        Port ovn-k8s-mp0            Interface ovn-k8s-mp0                type: internal        Port patch-br-int-to-localnet.network_ovn_localnet_port            Interface patch-br-int-to-localnet.network_ovn_localnet_port                type: patch                options: {peer=patch-localnet.network_ovn_localnet_port-to-br-int}        Port ovn-6bad89-0            Interface ovn-6bad89-0                type: geneve                options: {csum="true", key=flow, remote_ip="192.168.122.10"}    ovs_version: "2.17.5"

使用网络

apiVersion: v1kind: Servicemetadata:  name: nginx1-lo  namespace: defaultspec:  selector:    app: nginx1-lo  ports:  - protocol: TCP    port: 80    targetPort: server---apiVersion: v1kind: Podmetadata:  labels:    app: nginx1-lo  annotations:    k8s.v1.cni.cncf.io/networks: kube-system/localnet-network@eth1 #标注使用localnet网络  name: nginx1-lo  namespace: defaultspec:  containers:  - image: harbor.yusur.tech/leid/ubuntu:22.04    imagePullPolicy: IfNotPresent    name: nginx1-lo    ports:    - containerPort: 80      name: server    securityContext:      runAsUser: 0      privileged: true  restartPolicy: Always  nodeSelector:    kubernetes.io/hostname: work001# 查看ovn 逻辑交换机端口lsp信息[root@master001 ~]# ovn-nbctl find Logical_Switch_Port | grep -A 16 3dacea72-9ad8-4c5a-8912-ffe30d369f43_uuid               : 3dacea72-9ad8-4c5a-8912-ffe30d369f43addresses           : ["0a:58:c0:a8:64:04 192.168.100.4"]dhcpv4_options      : []dhcpv6_options      : []dynamic_addresses   : []enabled             : []external_ids        : {"k8s.ovn.org/nad"="kube-system/localnet-network", "k8s.ovn.org/network"=localnet-network, "k8s.ovn.org/topology"=localnet, namespace=default, pod="true"}ha_chassis_group    : []mirror_rules        : []name                : kube.system.localnet.network_default_nginx1-looptions             : {iface-id-ver="1c22a591-3c56-4663-b9b2-983effab96b1", requested-chassis=work001}parent_name         : []port_security       : ["0a:58:c0:a8:64:04 192.168.100.4"]tag                 : []tag_request         : []type                : ""up                  : true[root@master001 ~]# ovn-nbctl find Logical_Switch_Port | grep -A 16 f47cf104-a178-4a2c-a6af-a8d2845ef336_uuid               : f47cf104-a178-4a2c-a6af-a8d2845ef336addresses           : ["0a:58:c0:a8:64:05 192.168.100.5"]dhcpv4_options      : []dhcpv6_options      : []dynamic_addresses   : []enabled             : []external_ids        : {"k8s.ovn.org/nad"="kube-system/localnet-network", "k8s.ovn.org/network"=localnet-network, "k8s.ovn.org/topology"=localnet, namespace=default, pod="true"}ha_chassis_group    : []mirror_rules        : []name                : kube.system.localnet.network_default_nginx2-looptions             : {iface-id-ver="daecc092-6d3e-4b22-a5aa-59eb6b430029", requested-chassis=work002}parent_name         : []port_security       : ["0a:58:c0:a8:64:05 192.168.100.5"]tag                 : []tag_request         : []type                : ""up                  : true

测试网络

#查看pod1 ip信息，默认网络ip：10.124.1.6，第二网络ip：192.168.100.4root@master001:~/multi-homing# kubectl  get pod nginx1-lo -o=jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'[{    "name": "ovn-kubernetes",    "interface": "eth0",    "ips": [        "10.124.1.6"    ],    "mac": "0a:58:0a:7c:01:06",    "default": true,    "dns": {}},{    "name": "kube-system/localnet-network",    "interface": "eth1",    "ips": [        "192.168.100.4"    ],    "mac": "0a:58:c0:a8:64:04",    "dns": {}}]#查看pod2 ip信息，默认网络ip：10.124.2.5，第二网络ip：192.168.100.5root@master001:~/multi-homing# kubectl  get pod nginx2-lo -o=jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}'[{    "name": "ovn-kubernetes",    "interface": "eth0",    "ips": [        "10.124.2.5"    ],    "mac": "0a:58:0a:7c:02:05",    "default": true,    "dns": {}},{    "name": "kube-system/localnet-network",    "interface": "eth1",    "ips": [        "192.168.100.5"    ],    "mac": "0a:58:c0:a8:64:05",    "dns": {}}]

pod 与 pod

# pod1 ping pod2，pod2 tcpdump 抓包# pod1 网络信息root@work001:~# kubectl  exec -it nginx1-lo -- route -n Kernel IP routing tableDestination     Gateway         Genmask         Flags Metric Ref    Use Iface0.0.0.0         10.124.1.1      0.0.0.0         UG    0      0        0 eth010.86.0.0       10.124.1.1      255.255.0.0     UG    0      0        0 eth010.124.0.0      10.124.1.1      255.255.0.0     UG    0      0        0 eth010.124.1.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0100.64.0.0      10.124.1.1      255.255.0.0     UG    0      0        0 eth0192.168.100.0   0.0.0.0         255.255.255.0   U     0      0        0 eth1root@work001:~# kubectl  exec -it nginx1-lo -- ifconfigeth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1400        inet 10.124.1.6  netmask 255.255.255.0  broadcast 10.124.1.255        inet6 fe80::858:aff:fe7c:106  prefixlen 64  scopeid 0x20<link>        ether 0a:58:0a:7c:01:06  txqueuelen 0  (Ethernet)        RX packets 6  bytes 516 (516.0 B)        RX errors 0  dropped 0  overruns 0  frame 0        TX packets 14  bytes 1076 (1.0 KB)        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500        inet 192.168.100.4  netmask 255.255.255.0  broadcast 192.168.100.255        inet6 fe80::858:c0ff:fea8:6404  prefixlen 64  scopeid 0x20<link>        ether 0a:58:c0:a8:64:04  txqueuelen 0  (Ethernet)        RX packets 12  bytes 796 (796.0 B)        RX errors 0  dropped 0  overruns 0  frame 0        TX packets 14  bytes 1076 (1.0 KB)        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536        inet 127.0.0.1  netmask 255.0.0.0        inet6 ::1  prefixlen 128  scopeid 0x10<host>        loop  txqueuelen 1000  (Local Loopback)        RX packets 0  bytes 0 (0.0 B)        RX errors 0  dropped 0  overruns 0  frame 0        TX packets 0  bytes 0 (0.0 B)        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0# pod1 ping pod2:192.168.100.5root@work001:~# kubectl  exec -it nginx1-lo -- ping -c1 192.168.100.5PING 192.168.100.5 (192.168.100.5) 56(84) bytes of data.64 bytes from 192.168.100.5: icmp_seq=1 ttl=64 time=0.883 ms--- 192.168.100.5 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 0.883/0.883/0.883/0.000 ms# pod2 抓包root@work002:~# kubectl  exec -it nginx2-lo -- tcpdump -enn -i eth1 icmp02:29:36.582503 0a:58:c0:a8:64:04 > 0a:58:c0:a8:64:05, ethertype IPv4 (0x0800), length 98: 192.168.100.4 > 192.168.100.5: ICMP echo request, id 25198, seq 1, length 6402:29:36.582541 0a:58:c0:a8:64:05 > 0a:58:c0:a8:64:04, ethertype IPv4 (0x0800), length 98: 192.168.100.5 > 192.168.100.4: ICMP echo reply, id 25198, seq 1, length 64

pod 与 node

# ping node2： work002 节点# 查看work002 节点ip信息root@work002:~# ip addr  | grep 'ens3\|br-out'2: ens3f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000    inet 192.168.122.102/24 brd 192.168.122.255 scope global ens3f03: ens3f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 10004: ens3f2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master ovs-system state UP group default qlen 1000    inet 192.168.200.102/24 brd 192.168.200.255 scope global ens3f29: brens3f2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000    inet 192.168.200.102/24 brd 192.168.200.255 scope global brens3f2    inet 169.254.169.2/29 brd 169.254.169.7 scope global brens3f237: br-out: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000    inet 192.168.100.102/24 brd 192.168.100.255 scope global br-out# ping 192.168.100.102root@work001:~# kubectl  exec -it nginx1-lo -- ping -c1 192.168.100.102PING 192.168.100.102 (192.168.100.102) 56(84) bytes of data.64 bytes from 192.168.100.102: icmp_seq=1 ttl=64 time=1.11 ms--- 192.168.100.102 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 1.106/1.106/1.106/0.000 ms# ping 192.168.200.102root@work001:~# kubectl  exec -it nginx1-lo -- ping -c1 192.168.200.102PING 192.168.200.102 (192.168.200.102) 56(84) bytes of data.64 bytes from 192.168.200.102: icmp_seq=1 ttl=63 time=1.66 ms--- 192.168.200.102 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 1.660/1.660/1.660/0.000 ms# ping 192.168.122.102root@work001:~# kubectl  exec -it nginx1-lo -- ping -c1 192.168.122.102PING 192.168.122.102 (192.168.122.102) 56(84) bytes of data.64 bytes from 192.168.122.102: icmp_seq=1 ttl=63 time=0.601 ms--- 192.168.122.102 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 0.601/0.601/0.601/0.000 ms

pod 与 web

# pod1 添加路由信息 114.114.114.114root@work001:~# kubectl  exec -it nginx1-lo -- route add -host 114.114.114.114/32 gw 192.168.100.1 dev eth1 root@work001:~# kubectl  exec -it nginx1-lo -- route -n Kernel IP routing tableDestination     Gateway         Genmask         Flags Metric Ref    Use Iface0.0.0.0         10.124.1.1      0.0.0.0         UG    0      0        0 eth010.86.0.0       10.124.1.1      255.255.0.0     UG    0      0        0 eth010.124.0.0      10.124.1.1      255.255.0.0     UG    0      0        0 eth010.124.1.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0100.64.0.0      10.124.1.1      255.255.0.0     UG    0      0        0 eth0114.114.114.114 192.168.100.1   255.255.255.255 UGH   0      0        0 eth1192.168.100.0   0.0.0.0         255.255.255.0   U     0      0        0 eth1# ping 114.114.114.114 , 第一次ping，存在丢包现象。以后正常root@work001:~# kubectl  exec -it nginx1-lo -- ping -c1 114.114.114.114 PING 114.114.114.114 (114.114.114.114) 56(84) bytes of data.64 bytes from 114.114.114.114: icmp_seq=1 ttl=94 time=15.1 ms--- 114.114.114.114 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 15.081/15.081/15.081/0.000 ms

物理网络拓扑

参考文献

ovn localnet 介绍：https://l8liliang.github.io/2021/06/11/ovn-ls.html#bridged-logical-switch-vs-overlay-logical-switch

ovn localnet 配置：https://www.jianshu.com/p/204db500cdaa

安装 k8s lb 控制器

《k8s lb服务metallb(bgp模式)介绍》

配置 Linux 软路由器

网络架构图：

配置路由 bgp 功能

BGP 是一种外部网关协议 （EGP），用于在不同自治系统 （AS） 中的路由器之间交换路由信息。BGP 路由信息包括到每个目标的完整路由。BGP 使用路由信息来维护网络可访问性信息数据库，该数据库与其他 BGP 系统交换。BGP 使用网络可访问性信息来构建 AS 连接图，使 BGP 能够在 AS 级别移除路由环路并实施策略决策。

bgp as 网络架构图 例子：

方法 1：gobgpd

利用 gobgpd 搭建，GoBGP 是一种开源边界网关协议 (BGP) 实现，从头开始为现代环境设计，并以现代编程语言 Go 编程语言实现。

一些开源软件工具包，例如 frr/zerba 也能实现，简单介绍下另一个开源路由软件：gobgp，基于go开发的一款工具，性能自然是杠杠的了。此外，该工具仅支持BGP，并不像zerba那样，支持ospf，rip，bgp等其他路由协议。

参考文档：https://github.com/osrg/gobgp/blob/master/docs/sources/zebra.md

1， 配置 bgp server

mkdir gobgpcat <<EOF >gobgp/gobgp.conf[global.config]  as = 65001  port = 179  router-id = "192.168.122.1"[[peer-groups]]  [peer-groups.config]    peer-group-name = "example"    peer-as = 64512  [[peer-groups.afi-safis]]    [peer-groups.afi-safis.config]      afi-safi-name = "ipv4-unicast"  [[peer-groups.afi-safis]]    [peer-groups.afi-safis.config]      afi-safi-name = "ipv4-flowspec"  [peer-groups.transport]    [peer-groups.transport.config]      remote-port = 7946[[dynamic-neighbors]]  [dynamic-neighbors.config]    prefix = "192.168.122.0/24"    peer-group = "example"[zebra.config]  enabled = true  url = "unix:/var/run/frr/zserv.api"  redistribute-route-type-list = ["connect"]  version = 6EOF

2，启动 bgp server

docker run --privileged -it -d --network=host --rm --name gobgp -v $PWD/gobgp:/etc/gobgp:rw flftuu/frr:v8.1.0# 启动gobgpd 服务docker exec -it gobgp gobgpd -f /etc/gobgp/gobgp.conf  -l debug --sdnotify

3，查看 bgp 信息

# 查看配置信息[root@yusur-55 ~]# docker exec -it gobgp gobgp  globalAS:        65001Router-ID: 192.168.122.1Listening Port: 179, Addresses: 0.0.0.0, ::# 查看所有neighbor信息[root@yusur-55 ~]# docker exec -it gobgp gobgp  neighbor Peer               AS  Up/Down State       |#Received  Accepted192.168.122.102 64512 00:01:47 Establ      |        2         2# 查看neighbor 是否宣告了路由信息[root@yusur-55 ~]# docker exec -it gobgp gobgp  neighbor  192.168.122.102 adj-in   ID  Network              Next Hop             AS_PATH              Age        Attrs   0   172.19.0.0/32        192.168.122.102      64512                00:02:45   [{Origin: i}]   0   172.19.0.100/32      192.168.122.102      64512                00:02:45   [{Origin: i}]# 查看是否有bgp路由信息[root@yusur-55 ~]# docker exec -it gobgp gobgp global rib   Network              Next Hop             AS_PATH              Age        Attrs*> 10.2.20.0/23         0.0.0.0                                   00:04:37   [{Origin: i} {Med: 0}]*> 10.244.0.0/24        0.0.0.0                                   00:04:37   [{Origin: i} {Med: 0}]*> 10.244.0.0/32        0.0.0.0                                   00:04:37   [{Origin: i} {Med: 0}]*> 172.19.0.0/32        192.168.122.102      64512                00:04:30   [{Origin: i} {Med: 0}]*> 172.19.0.100/32      192.168.122.102      64512                00:04:30   [{Origin: i} {Med: 0}]*> 192.168.100.0/24     0.0.0.0                                   00:04:37   [{Origin: i} {Med: 0}]*> 192.168.122.0/24     0.0.0.0                                   00:04:37   [{Origin: i} {Med: 0}]*> 192.168.200.0/24     0.0.0.0                                   00:04:37   [{Origin: i} {Med: 0}]# 查看bgp 路由是否已经同步到linux 内核路由中[root@yusur-55 ~]# ip route list proto bgp172.19.0.0 via 192.168.122.102 dev virbr0 metric 20 172.19.0.100 via 192.168.122.102 dev virbr0 metric 20 

3，附件

详细开发文档

《gobgp 本地调试介绍》

方法 2：frr

使用 frr 工具包搭建，FRRouting (FRR) 是适用于 Linux 和 Unix 平台的免费开源 Internet 路由协议套件。它实现了 BGP、OSPF、RIP、IS-IS、PIM、LDP、BFD、Babel、PBR、OpenFabric 和 VRRP，并支持 EIGRP 和 NHRP。

FRR 与本地 Linux/Unix IP 网络堆栈的无缝集成使其成为适用于各种用例的通用路由堆栈，包括将主机/VM/容器连接到网络、广告网络服务、LAN 交换和路由、Internet 访问路由器、和互联网对等。

参考文档：https://frrouting.org/

系统架构:

传统的路由软件是作为一个进程程序，提供所有的路由协议功能。FRR采取了不同的方法。FRR是一组守护程序，它们共同构建路由表。每个主要协议都在自己的守护进程中实现，这些守护进程与中间守护进程（zebra）通信，后者负责协调路由决策并与数据平面通信。体系结构图：

+----+  +----+  +-----+  +----+  +----+  +----+  +-----+|bgpd|  |ripd|  |ospfd|  |ldpd|  |pbrd|  |pimd|  |.....|+----+  +----+  +-----+  +----+  +----+  +----+  +-----+     |       |        |       |       |       |        |+----v-------v--------v-------v-------v-------v--------v|                                                      ||                         Zebra                        ||                                                      |+------------------------------------------------------+       |                    |                   |       |                    |                   |+------v------+   +---------v--------+   +------v------+|             |   |                  |   |             || *NIX Kernel |   | Remote dataplane |   | ........... ||             |   |                  |   |             |+-------------+   +------------------+   +-------------+

1，配置 frr 服务

mkdir frrtouch frr/vtysh.confcat <<EOF >frr/daemonsbgpd=yesEOF

2，启动 frr 服务

docker run  --privileged  -it -d --network=host --rm --name frr8  -v  $PWD/frr:/etc/frr:rw flftuu/frr:v8.1.0

3，vtysh 方式配置 bgp

# 进入vtysh 中vtysh# 查看当前配置show running-config# 进行配置模式config t# bgp配置router bgp 65001bgp router-id 192.168.122.1neighbor 192.168.122.101 remote-as 64512neighbor 192.168.122.102 remote-as 64512no bgp ebgp-requires-policyaddress-family ipv4 unicastexit-address-familydebug bgp keepalivesdebug bgp neighbor-eventsdebug bgp nhtdebug bgp updates indebug bgp updates outdebug bgp zebradebug zebra nhtdebug zebra rib# 退出config-router 模式exit# 退出config模式exit

或者写如配置文件中，重启frr8 容器

cat <<EOF >frr/bgpd.confrouter bgp 65001bgp router-id 192.168.122.1neighbor 192.168.122.101 remote-as 64512neighbor 192.168.122.102 remote-as 64512no bgp ebgp-requires-policyaddress-family ipv4 unicastexit-address-familydebug bgp keepalivesdebug bgp neighbor-eventsdebug bgp nhtdebug bgp updates indebug bgp updates outdebug bgp zebradebug zebra nhtdebug zebra ribEOF# 重启frr8容器docker rm -f frr8docker run  --privileged  -it -d --network=host --rm --name frr8  -v  $PWD/frr:/etc/frr:rw flftuu/frr:v8.1.0

4，查看 bgp 信息

# 进入vtyshdocker exec -it frr8 vtysh yusur-55# show ip bgpBGP table version is 2, local router ID is 192.168.122.1, vrf id 0Default local pref 100, local AS 65001Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,               i internal, r RIB-failure, S Stale, R RemovedNexthop codes: @NNN nexthops vrf id, < announce-nh-selfOrigin codes:  i - IGP, e - EGP, ? - incompleteRPKI validation codes: V valid, I invalid, N Not found   Network          Next Hop            Metric LocPrf Weight Path*> 172.19.0.0/32    192.168.122.102                        0 64512 i*> 172.19.0.100/32  192.168.122.102                        0 64512 i# 查看 summary 信息yusur-55# show ip bgp summaryIPv4 Unicast Summary (VRF default):BGP router identifier 192.168.122.1, local AS number 65001 vrf-id 0BGP table version 2RIB entries 3, using 552 bytes of memoryPeers 1, using 716 KiB of memoryNeighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc192.168.122.102 4      64512         9         8        0    0    0 00:02:42            2        2 N/ATotal number of neighbors 1# 查看bgp route 信息yusur-55# show ip routeCodes: K - kernel route, C - connected, S - static, R - RIP,       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,       f - OpenFabric,       > - selected route, * - FIB route, q - queued, r - rejected, b - backup       t - trapped, o - offload failureK>* 0.0.0.0/0 [0/100] via 10.2.20.1, enp3s0, 00:05:23C>* 10.2.20.0/23 is directly connected, enp3s0, 00:05:23C>* 10.244.0.0/24 is directly connected, cni0, 00:05:23C>* 10.244.0.0/32 is directly connected, flannel.1, 00:05:23B>* 172.19.0.0/32 [20/0] via 192.168.122.102, virbr0, weight 1, 00:03:49B>* 172.19.0.100/32 [20/0] via 192.168.122.102, virbr0, weight 1, 00:03:49C>* 192.168.100.0/24 is directly connected, virbr1, 00:05:23C>* 192.168.122.0/24 is directly connected, virbr0, 00:05:23C>* 192.168.200.0/24 is directly connected, virbr2, 00:05:23# 查看bgp 路由是否已经同步到linux 内核路由中[root@yusur-55 ~]# ip route list proto bgp172.19.0.0 via 192.168.122.102 dev virbr0 metric 20 172.19.0.100 via 192.168.122.102 dev virbr0 metric 20 

5，附件

show ip bgp [x.x.x.x](longer)show ip bgp summaryshow ip route [x.x.x.x](longer)show ip bgp neighbor [x.x.x.x]show ip bgp neighbor 192.168.122.1 advertised-routesshow ip bgp neighbor x.x.x.x received-routes debug bgp alldebug bgp flowspecclear bgp * (soft in/soft out)

配置路由转发功能

要将CentOS配置为路由器，您需要执行以下步骤：

配置网络接口：

首先，确保您有至少两个网络接口，一个用于连接到互联网（外部接口），另一个用于连接到本地网络（内部接口）。

您可以使用`ifconfig`或`ip addr`命令来查看可用的网络接口。假设您的外部接口是enp3s0，内部接口是virbr0。

2. 启用IP转发

打开`/etc/sysctl.conf`文件并查找以下行：

#net.ipv4.ip_forward=1#将其取消注释，或者如果没有该行，则添加以下行：net.ipv4.ip_forward=1#保存文件并运行以下命令使更改生效：sysctl -p

3. 配置网络地址转换（NAT）

使用以下命令配置iptables规则，以实现网络地址转换（NAT）：

# 前提清空现有iptables 规则iptables -t nat -I POSTROUTING 1 -o enp3s0 -j MASQUERADEiptables -I FORWARD 1 -i enp3s0 -o virbr0 -m state --state RELATED,ESTABLISHED -j ACCEPTiptables -I FORWARD 1 -i virbr0 -o enp3s0 -j ACCEPT

这些规则将允许转发从内部接口（virbr0）到外部接口（enp3s0）的流量，并自动处理源地址转换。

4. 查看路由规则

# 该规则由 frr 利用bgp协议 自动学习k8s中配置svc 的lb ip地址#route add -host 172.19.0.100/32 gw 192.168.122.101# 查看bgp路由是否写入linux 内核路由中[root@yusur-55 ~]# ip route list proto bgp172.19.0.0 via 192.168.122.102 dev virbr0 metric 20 172.19.0.100 via 192.168.122.102 dev virbr0 metric 20

流量测试

手动 配置 gw 为10.2.20.4

# 添加172.19.0.0/24 的默认网关地址(为centos的enp3s0地址)route add -net 172.19.0.0/24 gw 10.2.20.4# 或者在本机开启frr bgp 功能从10.2.20.4 路由器学习bgp路由

在客户端访问 curl 172.19.0.100 地址

[root@yusur-25 ~]# curl 172.19.0.100<!DOCTYPE html><html><head><title>Welcome to nginx!</title><style>    body {        width: 35em;        margin: 0 auto;        font-family: Tahoma, Verdana, Arial, sans-serif;    }</style></head><body><h1>Welcome to nginx!</h1><p>If you see this page, the nginx web server is successfully installed andworking. Further configuration is required.</p><p>For online documentation and support please refer to<a href="http://nginx.org/">nginx.org</a>.<br/>Commercial support is available at<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p></body></html>[root@yusur-25 ~]# curl 172.19.0.0<!DOCTYPE html><html><head><title>Welcome to nginx!</title><style>    body {        width: 35em;        margin: 0 auto;        font-family: Tahoma, Verdana, Arial, sans-serif;    }</style></head><body><h1>Welcome to nginx!</h1><p>If you see this page, the nginx web server is successfully installed andworking. Further configuration is required.</p><p>For online documentation and support please refer to<a href="http://nginx.org/">nginx.org</a>.<br/>Commercial support is available at<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p></body></html>

查看 curl 转发流程

iptables dnat 信息

root@work002:~# iptables -vn -t nat -L  OVN-KUBE-EXTERNALIPChain OVN-KUBE-EXTERNALIP (2 references) pkts bytes target     prot opt in     out     source               destination             8   480 DNAT       tcp  --  *      *       0.0.0.0/0            172.19.0.0           tcp dpt:80 to:10.86.210.196:80# pkts 包的数量发生变化

本机路由匹配

root@work002:~# ip route  | grep 10.86.0.0/1610.86.0.0/16 via 169.254.169.4 dev brens3f0 mtu 1400 # 根据dst 转发到 ovs网桥

ovs 转发流表

root@work002:~# ovs-appctl dpctl/dump-flows -m | grep 10.86.210.196ufid:0bd14815-d0f3-44b4-8c31-7624e3d19ff1, recirc_id(0x13d),dp_hash(0/0),skb_priority(0/0),in_port(brens3f0),skb_mark(0/0),ct_state(0x21/0x25),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=00:00:00:00:00:00/00:00:00:00:00:00,dst=00:00:00:00:00:00/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=10.86.210.196,proto=6,tos=0/0,ttl=0/0,frag=no),tcp(src=0/0,dst=80),tcp_flags(0/0), packets:0, bytes:0, used:never, dp:ovs, actions:hash(l4(0)),recirc(0x13e)ufid:f745e2c4-b92f-4072-ad11-980114e3b9b3, recirc_id(0x13c),dp_hash(0/0),skb_priority(0/0),in_port(brens3f0),skb_mark(0/0),ct_state(0x21/0x23),ct_zone(0/0),ct_mark(0/0x2),ct_label(0/0),eth(src=52:54:00:e0:72:df,dst=0a:58:a9:fe:a9:04),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=10.86.210.196,proto=0/0,tos=0/0,ttl=62,frag=no), packets:0, bytes:0, used:never, dp:ovs, actions:set(eth(dst=52:54:00:e0:72:df)),ct(zone=8,nat),recirc(0x13d)ufid:aefc60dd-20ac-4f29-94b7-548a5ad8ffb9, recirc_id(0x13c),dp_hash(0/0),skb_priority(0/0),in_port(brens3f0),skb_mark(0/0),ct_state(0x22/0x23),ct_zone(0/0),ct_mark(0/0x2),ct_label(0/0),eth(src=52:54:00:e0:72:df,dst=0a:58:a9:fe:a9:04),eth_type(0x0800),ipv4(src=128.0.0.0/192.0.0.0,dst=10.86.210.196,proto=0/0,tos=0/0,ttl=62,frag=no), packets:3, bytes:198, used:2.332s, flags:F., dp:ovs, actions:set(eth(dst=52:54:00:e0:72:df)),ct(zone=8,nat),recirc(0x13d)

pod 里抓包

# pod里面抓包root@work002:~# kubectl exec -it nginx1-ovn -- tcpdump port 80 -nnvvtcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes09:20:21.653960 IP (tos 0x0, ttl 60, id 22325, offset 0, flags [DF], proto TCP (6), length 60)    100.64.0.4.60608 > 10.124.0.3.80: Flags [S], cksum 0x90f3 (correct), seq 2453724959, win 29200, options [mss 1460,sackOK,TS val 292120159 ecr 0,nop,wscale 7], length 0

MetalLB是一个为基础 Kubernetes集群提供负载均衡实现的工具，使用标准路由协议。

Kubernetes在基础集群中不提供网络负载均衡器（类型为LoadBalancer的服务）的实现。Kubernetes提供的网络负载平衡器实现都是调用各种IaaS平台（如GCP、AWS、Azure等）的接口代码。如果您没有运行在受支持的IaaS平台上（如GCP、AWS、Azure等），则创建时LoadBalancers将无限期处于“挂起”状态。

在基础集群中，操作员只有两个接口来将用户流量引入他们的集群，“NodePort”和“externalIPs”服务。

这两个选项在生产使用中都有显著的缺点，这使得基础集群成为 Kubernetes 生态系统中的二等公民。

MetalLB旨在通过提供与标准网络设备集成的网络负载均衡器实现来解决这种不平衡，以便基础群集上的external services尽可能“正常工作”。

前置要求

• 需要运行Kubernetes 1.13.0或更高版本的Kubernetes集群，该集群尚未具备网络负载平衡功能。

• 可以与MetalLB共存的集群网络（如 calico）。

• 一些IPv4地址供MetalLB分配。

• 当使用BGP操作模式时，您将需要一个或多个能够使用BGP协议进行通信的路由器。

• 当使用L2操作模式时，节点之间必须允许端口7946上（TCP和UDP，其他端口可配置）的流量通过，这是memberlist所需的。

安装 metallb 服务

如果您正在使用IPVS模式下的kube-proxy，则自Kubernetes v1.14.2起，您必须启用严格的ARP模式。

请注意，如果您使用kube-router作为服务代理，则不需要此操作，因为它默认启用了 strict ARP。

您可以通过编辑当前集群中的kube-proxy配置来实现此操作：

kubectl edit configmap -n kube-system kube-proxy# setapiVersion: kubeproxy.config.k8s.io/v1alpha1kind: KubeProxyConfigurationmode: "ipvs"ipvs:  strictARP: true# see what changes would be made, returns nonzero returncode if differentkubectl get configmap kube-proxy -n kube-system -o yaml | \sed -e "s/strictARP: false/strictARP: true/" | \kubectl diff -f - -n kube-system# actually apply the changes, returns nonzero returncode on errors onlykubectl get configmap kube-proxy -n kube-system -o yaml | \sed -e "s/strictARP: false/strictARP: true/" | \kubectl apply -f - -n kube-system# delete pods kubectl  delete pods -l k8s-app=kube-proxy -n kube-system

创建 metallb

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/main/config/manifests/metallb-native.yaml# 查看对应的metalb  controller控制器pod 是否启动root@master001:~/lb# kubectl  -n metallb-system  get pods -l app=metallbNAME                                                   READY   STATUS    RESTARTS   AGEcontroller-848965b68-8slv7                             1/1     Running   0          8m8sspeaker-5hx2t                                          1/1     Running   0          8m8sspeaker-8vbmh                                          1/1     Running   0          8m8sspeaker-r46dt                                          1/1     Running   0          8m8s# pod 介绍这将在 metallb-system 命名空间下部署 MetalLB。清单中包含以下组件：- metallb-system/controller 部署。它是处理 IP 地址分配的集群控制器。- metallb-system/speaker 守护进程集。它是使用您选择的协议来使服务可达的组件。- controller和 speaker 的服务帐户，以及组件需要运行所需的 RBAC 权限。

创建 IP pool 池

IPAddressPool 是定义要分配给负载均衡器服务的IP地址

使用所需的IP创建IPAddressPool。将autoAssign设置为false是有意义的，这样它就不会被其他服务错误地占用 - 我们的服务将显式请求该池。

cat <<EOF >ip.yamlapiVersion: metallb.io/v1beta1kind: IPAddressPoolmetadata:  name: example-pool  namespace: metallb-systemspec:  addresses:  - 172.19.0.0/24  autoAssign: falseEOF

IPAddressPools可以同时存在多个实例，地址可以通过CIDR，范围定义，并且可以分配IPv4和IPv6地址。

创建 bgp adv

cat <<EOF >adv.yamlapiVersion: metallb.io/v1beta1kind: BGPAdvertisementmetadata:  name: example-bgp-adv  namespace: metallb-systemspec:  ipAddressPools:  - example-pool    #  nodeSelectors:    #  - matchLabels:    #      egress-service.k8s.ovn.org/some-namespace-example-service: ""EOF

创建 bgp peer

cat <<EOF > peer.yamlapiVersion: metallb.io/v1beta2kind: BGPPeermetadata:  name: example  namespace: metallb-systemspec:  myASN: 64512  peerASN: 65001  peerAddress: 192.168.122.1  peerPort: 179EOF

配置路由器 bgp 功能

《LB服务和BGP路由器测试》

创建 LB SVC

cat <<EOF > svc.yamlapiVersion: v1kind: Servicemetadata:  name: example-service  annotations:    # 指定lb的地址池    metallb.universe.tf/address-pool: example-poolspec:  selector:    app: nginx1-ovn  ports:    - name: http      protocol: TCP      port: 80      targetPort: 80  type: LoadBalancerEOFkubectl apply -f svc.yaml

查看 LB SVC

kubectl  describe svc example-serviceName:                     example-serviceNamespace:                defaultLabels:                   <none>Annotations:              metallb.universe.tf/address-pool: example-pool                          metallb.universe.tf/ip-allocated-from-pool: example-poolSelector:                 app=nginx1-ovnType:                     LoadBalancerIP Family Policy:         SingleStackIP Families:              IPv4IP:                       10.86.73.55IPs:                      10.86.73.55LoadBalancer Ingress:     172.19.0.100 # 分配的lb地址Port:                     http  80/TCPTargetPort:               80/TCPNodePort:                 http  32541/TCPEndpoints:                10.124.0.3:80Session Affinity:         NoneExternal Traffic Policy:  Cluster

高级用法

参考：https://metallb.universe.tf/configuration/_advanced_bgp_configuration/

本文主要提供本地开发调试 BGP 程序的一个技巧，同时学习 BGP 协议，以及 go 包 gobgp https://github.com/osrg/gobgp

学习 BGP 协议必要性：K8s 容器网络目前我们使用 Cilium/Calico，都是使用 BGP 来宣告路由，使得 pod/service cidr 在 K8s 之外可达。所以，想要了解容器网络，有必要了解 BGP。

BGP 基本概念

BGP <https://datatracker.ietf.org/doc/html/rfc4271 是一个建立在 TCP 上的边界网关协议，可以用来动态学习路由。>

在 K8s Cilium/Calico 网络插件生态里，主要使用 BGP 协议宣告 pod/service 路由给交换机，使得 pod/service ip 可以在 K8s 之外访问。

比如，下图是把目的地址为 service ip 的路由宣告给交换机，可以见青云的 Porter 项目 https://github.com/kubesphere/openelb/blob/master/doc/bgp-mode.md

该项目比较简单，通过 k8s 这一层获取 service ip，和定制选择几个 Node IP 作为下一跳Next Hop，构造成一个路由，宣告给交换机。当然，也可以直接把 service cidr 宣告给交换机而不是一个个 service ip:

porter 项目和 metallbhttps://github.com/metallb/metallb项目基本一致，主要区别在porter 使用强大的 gobgp 库来实现 bgphttps://github.com/osrg/gobgp，且通过 K8s CRD 来配置，而 metallb 自己实现的一个简陋版本的 BGP 实现。两者都自己实现了 IPAM。对比可以参考：https://github.com/kubesphere/openelb/blob/master/doc/zh/compared\_with\_metallb.md，

感觉porter 更友好点，不过做的事情几乎一样。

pod cidr 也是通过 BGP 协议宣告给交换机，目前我们 K8s 集群的 Cilium/Calico 使用 bird 程序(C语言写的) 来宣告每一个 pod cidr subnet 给交换机，本机 Node IP 作为下一跳。比如 pod cidr 为 10.20.30.40/19，集群有两个node 100.100.100.100、200.200.200.200 ，并且切分成两个 pod cidr

subnet10.20.30.40/20、10.20.30.41/20，则两个 node 上的 bird 分别会把类似如下路由宣告给交换机：

10.20.30.40/20 via 100.100.100.10010.20.30.41/20 via 200.200.200.200

流量到了交换机那一侧，如果 pod ip 在10.20.30.40/20 子网内，则下一跳去100.100.100.100 node；如果 pod ip 在10.20.30.41/20 子网内，则下一跳去200.200.200.200 node。

调试 BGP 程序

需要本地开发一个 BGP 程序，比如类似 porter/metallb 那样的项目，但是没有真实的交换机/路由器，那就不太方便本地开发调试了，可以使用 gobgp <https://github.com/osrg/gobgp包来在本地运行作为 route server(类似交换机那一侧)。而且，建议使用强大的gobgp BGP 包，go 语言写的。>

或者也可以使用 bird，C 语言写的。不过程序需要集成 gobgp 包，也建议 gobgp。

先安装下 gobgpd 和 gobgp 两个 bin 文件，gobgp 是客户端：

wget -c https://github.com/osrg/gobgp/releases/download/v3.15.0/gobgp_3.15.0_linux_amd64.tar.gztar -xf -C /usr/local/bin gobgp_3.15.0_linux_amd64.tar.gz

然后本地运行 route-server，可以参考<https://github.com/osrg/gobgp/blob/master/docs/sources/route-server.md，配置文档 route-server-conf>.

conf 如下：

[global.config] as = 64512 router-id = "1.1.1.1" port = 1790 local-address-list = ["127.0.0.1"][[neighbors]] [neighbors.config] neighbor-address = "127.0.0.1" peer-as = 65001 [neighbors.transport.config] remote-port = 1791 passive-mode = true [neighbors.route-server.config] route-server-client = true

本地运行服务

gobgpd -f ./route-server-conf.conf -l debug --api-hosts ":50052" --pprof-disable

这时，以上 route server 就相当于交换机那一侧，等待 node 这一侧宣告路由过去。然后就是编写一个 bgp 程序，并使用 gobgp 包。一个简单的代码如

下所示，当然在可以加上k8s那一侧控制平面获取 pod/service cidr 等逻辑，类似 porter/metallb 项目。

测试代码里，本地这一侧会宣告一个路由: 目标地址 10.20.30.0/24 下一跳 30.30.30.30

package bgpimport ( "context" "github.com/golang/protobuf/ptypes" api "github.com/osrg/gobgp/api" gobgp "github.com/osrg/gobgp/pkg/server" "google.golang.org/protobuf/types/known/anypb" "testing")// gobgpd -f ./route-server-conf.conf -l debug --api-hosts ":50052" --pprof-disable// gobgpd -f ./route-client-conf.conf -l debug --api-hosts ":50053" --pprof-disable// nodegobgp -p 50053 -d global rib add -a ipv4 100.0.0.0/24 nexthop 20.20.20.20// gobgp -p 50052 -d neighbor 127.0.0.1 adj-in// gobgp -p 50052 -d global rib add -a ipv4 200.0.0.0/24 nexthop 20.20.20.20// nodegobgp -p 50053 -d neighbor 127.0.0.1 adj-infunc TestRouteServer(test *testing.T) { log.SetLevel(log.DebugLevel) ch := make(chan struct{}) // bgp1 s := gobgp.NewBgpServer(gobgp.GrpcListenAddress(":50053")) go s.Serve() _ = s.StartBgp(context.Background(), &api.StartBgpRequest{  Global: &api.Global{   As:         65001,     // AS Number, NetOPS API AS Number   RouterId:   "2.2.2.2", // IP   ListenPort: 1791,  }, }) defer s.StopBgp(context.Background(), &api.StopBgpRequest{}) // route-server() bgp-server p1 := &api.Peer{  Conf: &api.PeerConf{   NeighborAddress: "127.0.0.1",   PeerAs:          64512,  },  Transport: &api.Transport{   RemotePort: 1790,  }, } _ = s.AddPeer(context.Background(), &api.AddPeerRequest{Peer: p1}) nlri, _ := ptypes.MarshalAny(&api.IPAddressPrefix{  Prefix:    "10.20.30.0",  PrefixLen: 24, }) a1, _ := ptypes.MarshalAny(&api.OriginAttribute{  Origin: 0, }) a2, _ := ptypes.MarshalAny(&api.NextHopAttribute{  NextHop: "30.30.30.30", }) attrs := []*anypb.Any{a1, a2} s.AddPath(context.TODO(), &api.AddPathRequest{  Path: &api.Path{   Family: &api.Family{Afi: api.Family_AFI_IP, Safi: api.Family_SAFI_UNICAST},   Nlri:   nlri,   Pattrs: attrs,  }, }) <-ch}

k8s 技能树

master01: 172.16.10.11

master01: 172.16.10.12

master01: 172.16.10.13

外部LB：172.16.10.62

创建三个vm虚拟机

根据现有网络环境，vm 必须包好两张网卡，eth0(使用nat 网络模式，用于访问公网) eth1(使用桥接模式，和宿主机处于同一网络平面上，用户访问dpu节点)

创建网络

创建桥接模式网络

# 创建网桥brctl addbr br0# 加入宿主机网卡到网桥brctl addif br0 enp1s0

创建nat 模式网络

# 使用virsh 创建网络cat <EOF >default.xml<network connections='3'>  <name>default</name>  <uuid>414f2c31-93ba-46a3-9588-84a3b36bc409</uuid>  <forward mode='nat'>    <nat>      <port start='1024' end='65535'/>    </nat>  </forward>  <bridge name='virbr0' stp='on' delay='0'/>  <mac address='52:54:00:6e:b8:48'/>  <ip address='192.168.122.1' netmask='255.255.255.0'>    <dhcp>      <range start='192.168.122.2' end='192.168.122.254'/>    </dhcp>  </ip></network>EOFvirsh net-define default.xmlvirsh net-start default# 查看网桥信息brctl show virbr0

使用网络

vm xml信息

  # 添加nat模式网络，用于外网访问   <interface type='network'>      <mac address='fa:92:01:33:e4:fd'/>      <source network='default'/>      <model type='virtio'/>      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>    </interface>    # 添加桥接模式网络，用于k8s管理网段，访问dpu节点    <interface type='bridge'>      <mac address='fa:92:01:33:e4:fc'/>      <source bridge='br0'/>      <model type='virtio'/>      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>    </interface>

安装containerd

卸载旧版本

sudo apt-get remove docker docker-engine docker.io docker-ce docker-ce-cli containerd runc kubelet kubectl kubeadm  cri-tools sudo rm -rf /var/lib/dockersudo rm -rf /var/lib/containerdsudo rm -fr /etc/cni/net.dsudo rm -fr /opt/cni/bin

准备包环境，更新apt，允许使用https

sudo apt-get updatesudo apt-get install \    ca-certificates \    curl \    gnupg \    lsb-release

添加docker官方GPG key

sudo mkdir -p /etc/apt/keyringscurl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

设置软件仓库源

echo \  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null# 添加公钥sudo gpg --keyserver keyserver.ubuntu.com --recv B53DC80D13EDEF05sudo gpg --export --armor B53DC80D13EDEF05 | sudo apt-key add -

安装containerd

# 安装containerdsudo apt-get updatesudo apt-get install -y containerd.io# 如果是安装docker则执行：#sudo apt-get install docker-ce docker-ce-cli containerd.io# 查看运行状态systemctl enable containerdsystemctl status containerd

配置containerd

在 Linux 上，containerd 的默认 CRI 套接字是 /run/containerd/containerd.sock

生成默认配置

containerd config default > /etc/containerd/config.toml

修改CgroupDriver为systemdk8s官方推荐使用systemd类型的CgroupDriver

# 修改默认配置 /etc/containerd/config.tomlsandbox_image = "registry.aliyuncs.com/google_containers/pause:3.7" #修改pause地址  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]    SystemdCgroup = true #修改

重启containerd

systemctl enable containerdsystemctl restart containerd

安装外部LB服务

cat >EOF >nginx.confuser nginx;worker_processes auto;error_log /var/log/nginx/error.log;pid /run/nginx.pid;include /usr/share/nginx/modules/*.conf;events {    worker_connections 1024;}# 四层负载均衡，为两台Master apiserver组件提供负载均衡stream {    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';    access_log  /var/log/nginx/k8s-access.log  main;    upstream k8s-apiserver {       server 172.16.10.11:6443 max_fails=1 fail_timeout=10s;   # master01 APISERVER IP:PORT       server 172.16.10.12:6443 max_fails=1 fail_timeout=10s;   # master02 APISERVER IP:PORT       server 172.16.10.13:6443 max_fails=1 fail_timeout=10s;   # master03 APISERVER IP:PORT    }        server {       listen 16443;       proxy_pass k8s-apiserver;    }}http {}EOF# 启动lb服务docker run -d --name k8s-apiserver-lb -p 16443:16443 -v $PWD/nginx.conf:/etc/nginx/nginx.conf nginx:latest

安装kubernetes

所有节点执行

cat >EOF >/etc/apt/sources.list.d/kubernetes.list #deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial maindeb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial mainEOF# 添加公钥sudo gpg --keyserver keyserver.ubuntu.com --recv B53DC80D13EDEF05sudo gpg --export --armor B53DC80D13EDEF05 | sudo apt-key add -# 安装k8ssudo apt-get updatesudo apt-get install -y kubelet=1.24.10-00 kubeadm=1.24.10-00 kubectl=1.24.10-00 --allow-downgrades# 锁定版本不升级sudo apt-mark hold kubelet kubeadm kubectl

所有节点执行

# 设置参数cat <<EOF > /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1net.ipv4.ip_forward = 1vm.swappiness = 0EOF# 加载模块cat << EOF > /etc/modules-load.d/containerd.confoverlaybr_netfilterEOFmodprobe overlaymodprobe br_netfiltersysctl -p  /etc/sysctl.d/k8s.conf# 在所有节点(含Master)都安装ovssudo apt-get install openvswitch-switch

master01 节点安装

sudo kubeadm init --control-plane-endpoint 172.16.10.62:16443  --upload-certs --kubernetes-version=v1.24.10 --apiserver-advertise-address=172.16.10.11 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.123.0.0/16 --service-cidr=10.86.0.0/16# 输出如下：Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:  mkdir -p $HOME/.kube  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config  sudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run:  export KUBECONFIG=/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster.Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:  https://kubernetes.io/docs/concepts/cluster-administration/addons/You can now join any number of the control-plane node running the following command on each as root:  kubeadm join 172.16.10.62:16443 --token m7iv0r.uob1re8vw5xtht52 \ --discovery-token-ca-cert-hash sha256:07cfc6d8d1092f8ac9baf71bbe9e606d61138dad19a1988947baa14280aad8aa \ --control-plane --certificate-key 98adc3aac8c6caf80c2cd8da6e516bf43be98c2cf8acdff1a02892c8be88fc74Please note that the certificate-key gives access to cluster sensitive data, keep it secret!As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.Then you can join any number of worker nodes by running the following on each as root:kubeadm join 172.16.10.62:16443 --token m7iv0r.uob1re8vw5xtht52 \ --discovery-token-ca-cert-hash sha256:07cfc6d8d1092f8ac9baf71bbe9e606d61138dad19a1988947baa14280aad8aa 

master02 节点安装

添加–apiserver-advertise-address=172.16.10.12 参数

kubeadm join 172.16.10.62:16443 --apiserver-advertise-address=172.16.10.12 --token m7iv0r.uob1re8vw5xtht52 \ --discovery-token-ca-cert-hash sha256:07cfc6d8d1092f8ac9baf71bbe9e606d61138dad19a1988947baa14280aad8aa \ --control-plane --certificate-key 98adc3aac8c6caf80c2cd8da6e516bf43be98c2cf8acdff1a02892c8be88fc74

master03 节点安装

添加–apiserver-advertise-address=172.16.10.13 参数

kubeadm join 172.16.10.62:16443 --apiserver-advertise-address=172.16.10.13 --token m7iv0r.uob1re8vw5xtht52 \ --discovery-token-ca-cert-hash sha256:07cfc6d8d1092f8ac9baf71bbe9e606d61138dad19a1988947baa14280aad8aa \ --control-plane --certificate-key 98adc3aac8c6caf80c2cd8da6e516bf43be98c2cf8acdff1a02892c8be88fc74

work 节点安装

修改node ip参数：–node-ip=172.16.10.205

# 指定node ip地址cat >EOF >/etc/default/kubelet KUBELET_EXTRA_ARGS="--node-ip=172.16.10.205"EOFkubeadm join 172.16.10.62:16443 --token m7iv0r.uob1re8vw5xtht52 \ --discovery-token-ca-cert-hash sha256:07cfc6d8d1092f8ac9baf71bbe9e606d61138dad19a1988947baa14280aad8aa

环境为双网卡环境，默认部署使用单一网卡的ip 证书信任， 另外一网卡ip没做证书信任，现在需要重新生成证书来添加ip信任，让apiserver 支持多ip地址访问

备份 kubernetes 目录

cp -r /etc/kubernetes{,-bak}

查看证书内的 ip

for i in $(find /etc/kubernetes/pki -type f -name "*.crt");do echo ${i} && openssl x509 -in ${i} -text | grep 'DNS:';done

可以看到，只有 apiserver 和 etcd 的证书里面是包含了 ip 的

root@yusur-62:~# for i in $(find /etc/kubernetes/pki -type f -name "*.crt");do echo ${i} && openssl x509 -in ${i} -text | grep 'DNS:';done/etc/kubernetes/pki/etcd/server.crt                DNS:localhost, DNS:yusur-62, IP Address:192.168.28.62, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1/etc/kubernetes/pki/etcd/peer.crt                DNS:localhost, DNS:yusur-62, IP Address:192.168.28.62, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1/etc/kubernetes/pki/etcd/healthcheck-client.crt/etc/kubernetes/pki/etcd/ca.crt                DNS:etcd-ca/etc/kubernetes/pki/apiserver.crt                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:yusur-62, IP Address:10.86.0.1, IP Address:192.168.28.62/etc/kubernetes/pki/front-proxy-client.crt/etc/kubernetes/pki/apiserver-kubelet-client.crt/etc/kubernetes/pki/front-proxy-ca.crt                DNS:front-proxy-ca/etc/kubernetes/pki/apiserver-etcd-client.crt/etc/kubernetes/pki/ca.crt                DNS:kubernetes

生成集群配置

查看集群kubeadm 配置信息

# 查看集群kubeadm 配置信息kubectl -n kube-system get cm kubeadm-config -o jsonpath={.data.ClusterConfiguration} | tee /root/kubeadm.yamlapiServer:  extraArgs:    authorization-mode: Node,RBAC  timeoutForControlPlane: 4m0sapiVersion: kubeadm.k8s.io/v1beta3certificatesDir: /etc/kubernetes/pkiclusterName: kubernetescontrollerManager: {}dns: {}etcd:  local:    dataDir: /var/lib/etcdimageRepository: registry.aliyuncs.com/google_containerskind: ClusterConfigurationkubernetesVersion: v1.24.10networking:  dnsDomain: cluster.local  podSubnet: 10.123.0.0/16  serviceSubnet: 10.86.0.0/16

增加新ip

# 编辑配资文件vim /root/kubeadm.yamlapiServer:  # 增加下面的配置，添加master节点多个网卡ip地址  certSANs:  - 192.168.28.62  - 172.16.10.62  # 增加上面的配置  extraArgs:    authorization-mode: Node,RBAC  timeoutForControlPlane: 4m0sapiVersion: kubeadm.k8s.io/v1beta3certificatesDir: /etc/kubernetes/pkiclusterName: kubernetescontrollerManager: {}dns: {}etcd:  local:    dataDir: /var/lib/etcd    # 增加下面的配置，添加master节点多个网卡ip地址    serverCertSANs:    - 192.168.28.62    - 172.16.10.62    # 增加上面的配置imageRepository: registry.aliyuncs.com/google_containerskind: ClusterConfigurationkubernetesVersion: v1.24.10networking:  dnsDomain: cluster.local  podSubnet: 10.123.0.0/16  serviceSubnet: 10.86.0.0/16

删除原有的证书

需要保留 ca ，sa，front-proxy 这三个证书

rm -rf /etc/kubernetes/pki/{apiserver*,front-proxy-client*}rm -rf /etc/kubernetes/pki/etcd/{healthcheck*,peer*,server*}

重新生成证书

更新证书

kubeadm init phase certs all --config /root/kubeadm.yaml

再次查看证书内的 ip

for i in $(find /etc/kubernetes/pki -type f -name "*.crt");do echo ${i} && openssl x509 -in ${i} -text | grep 'DNS:';done

这里可以得到验证，不会覆盖之前证书内已经有的 ip，会将新的 ip 追加到后面

/etc/kubernetes/pki/etcd/server.crt                DNS:localhost, DNS:yusur-62, IP Address:192.168.28.62, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1, IP Address:172.16.10.62/etc/kubernetes/pki/etcd/peer.crt                DNS:localhost, DNS:yusur-62, IP Address:192.168.28.62, IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1/etc/kubernetes/pki/etcd/healthcheck-client.crt/etc/kubernetes/pki/etcd/ca.crt                DNS:etcd-ca/etc/kubernetes/pki/apiserver.crt                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:yusur-62, IP Address:10.86.0.1, IP Address:192.168.28.62, IP Address:172.16.10.62/etc/kubernetes/pki/front-proxy-client.crt/etc/kubernetes/pki/apiserver-kubelet-client.crt/etc/kubernetes/pki/front-proxy-ca.crt                DNS:front-proxy-ca/etc/kubernetes/pki/apiserver-etcd-client.crt/etc/kubernetes/pki/ca.crt                DNS:kubernetes

将配置更新到CM中

这样，以后有升级，或者增加其他 ip 时，也会将配置的 CertSANs 的 ip 保留下来，方便以后删减

kubeadm init phase upload-config kubeadm --config /root/kubeadm.yaml

使用脚本安装

Helm现在有个安装脚本可以自动拉取最新的Helm版本并在 本地安装。

您可以获取这个脚本并在本地执行。它良好的文档会让您在执行之前知道脚本都做了什么。

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3chmod 700 get_helm.sh./get_helm.sh

如果想直接执行安装，运行curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash。

通过包管理器安装

Helm社区成员贡献了针对Apt的一个 helm包通常是最新的。

curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/nullsudo apt-get install apt-transport-https --yesecho "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.listsudo apt-get updatesudo apt-get install helm

参考文档：https://helm.sh/zh/docs/

helm操作命令

添加公共仓库

# 也可以换成微软的源，速度快，内容和官方同步的 helm repo add stable http://mirror.azure.cn/kubernetes/charts              helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

查看仓库

helm repo lsNAME      URL                                                   my-repo   https://charts.bitnami.com/bitnami                    yusur-repohttp://10.2.20.7:8081                                 harbor    https://harbor.yusur.tech/chartrepo/leid              stable    http://mirror.azure.cn/kubernetes/charts              aliyun    https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts

helm安装插件

安装push插件

helm plugin install https://github.com/chartmuseum/helm-pushDownloading and installing helm-push v0.10.3 ...https://github.com/chartmuseum/helm-push/releases/download/v0.10.3/helm-push_0.10.3_darwin_arm64.tar.gzInstalled plugin: cm-push

制作helm包

创建chart包

# 创建新chart包helm create mychartCreating mychart

搭建私有helm仓库

Chartmuseum 除了给我们提供一个类似于web服务器的功能之外，还提供了其他有用的功能，便于日常我们私有仓库的管理。

• 根据chart文件自动生成index.yaml(无须使用helm repo index手动生成)

• helm push的插件，可以在helm命令之上实现将chart文件推送到chartmuseum上

• 相应的tls配置，Basic认证，JWT认证（Bearer token认证）

• 提供了Restful的api（可以使用curl命令操作）和可以使用的cli命令行工具

• 提供了各种后端存储的支持（Amazon s3, Google Cloud Storage, 阿里、百度、腾讯，开源对象存储等）

• 提供了Prometheus的集成，对外提供自己的监控信息。

• 没有用户的概念，但是基于目录实现了一定程度上的多租户的需求。

Chartmuseum 搭建

直接使用最简单的 docker run 方式，使用local 本地存储方式，通过 -v 映射到宿主机 /opt/charts

更多支持安装方式见官网

mkdir /opt/chartsdocker run -d \  --name chartmuseum \  -u 0:0 \  -p 8081:8080 \  -e DEBUG=1 \  -e STORAGE=local \  -e STORAGE_LOCAL_ROOTDIR=/charts \  -v /opt/charts:/charts \  chartmuseum/chartmuseum:latest# 使用 curl 测试下接口，没有报错就行，当前仓库内容还是空的# curl localhost:8081/api/charts{}

添加私有仓库Chartmuseum

helm repo add yusur-repo http://10.2.20.7:8081

cm-push chats包到Chartmuseum

# 添加cm-push 插件helm plugin install https://github.com/chartmuseum/helm-push# 使用cm-push上传chat 包helm cm-push mariadb-11.5.7.tgz yusur-repo# 使用cm-push上传chat 目录helm cm-push ./mychart yusur-repo

复用harbor仓库

harbor自带helm私有仓库的功能，不需要再部署一个helm私有仓库，在这里给大家介绍一下helm如何上传chart包到harbor

cm-push chats包到harbor

添加私有仓库harbor

# 添加repohelm repo add harbor https://harbor.yusur.tech/chartrepo/library --username xxx --password xxx# 使用cm-push 命令 上传chat包helm cm-push mariadb-11.5.7.tgz harbor# 上传cm-push chat目录helm cm-push ./mychart harbor

push chats包到harbor

使用oci协议

# 登录到注册中心helm registry login -u leid harbor.yusur.tech -p xxxxxx# 使用push 命令上传chat包helm push mychart-0.1.0.tgz oci://harbor.yusur.tech/leid# 使用pull 下载包helm pull oci://harbor.yusur.tech/leid/mychart --version 0.1.0# 查看chat 信息helm show all oci://harbor.yusur.tech/leid/mychart --version 0.1.0

构建同一个镜像支持amd和arm平台

方法一

前提条件

# 设置docker 参数export DOCKER_CLI_EXPERIMENTAL=enabled

登入amd机器上编译和制作镜像

OVN_IMAGE=harbor.yusur.tech/yusur_ovn/ovn-daemonset-f:second-cni-amd64# build 镜像docker build --build-arg BUILDPLATFORM=linux/amd64 --build-arg TARGETPLATFORM=linux/amd64 -t $OVN_IMAGE -f Dockerfile.fedora .# push 镜像docker push $OVN_IMAGE

登入arm机器上编译和制作镜像

OVN_IMAGE=harbor.yusur.tech/yusur_ovn/ovn-daemonset-f:second-cni-aarch64# build 镜像docker build --build-arg BUILDPLATFORM=linux/aarch64 --build-arg TARGETPLATFORM=linux/aarch64 -t $OVN_IMAGE -f Dockerfile.fedora .# push 镜像docker push $OVN_IMAGE

创建镜像manifest

# 创建 manifest 列表OVN_IMAGE=harbor.yusur.tech/yusur_ovn/ovn-daemonset-f:second-cniOVN_IMAGE_AMD64=harbor.yusur.tech/yusur_ovn/ovn-daemonset-f:second-cni-amd64OVN_IMAGE_ARM64=harbor.yusur.tech/yusur_ovn/ovn-daemonset-f:second-cni-aarch64# 当要修改一个 manifest 列表时，可以加入 -a 或 --amend 参数docker manifest create $OVN_IMAGE $OVN_IMAGE_AMD64 $OVN_IMAGE_ARM64# 推送 manifest 列表docker manifest push $OVN_IMAGE

设置 manifest 列表

# 设置 manifest 列表OVN_IMAGE=harbor.yusur.tech/yusur_ovn/ovn-daemonset-f:second-cniOVN_IMAGE_AMD64=harbor.yusur.tech/yusur_ovn/ovn-daemonset-f:second-cni-amd64OVN_IMAGE_ARM64=harbor.yusur.tech/yusur_ovn/ovn-daemonset-f:second-cni-aarch64docker manifest annotate $OVN_IMAGE $OVN_IMAGE_AMD64 --os linux --arch x86_64docker manifest annotate $OVN_IMAGE $OVN_IMAGE_ARM64 --os linux --arch arm64 --variant v8

查看 manifest 列表

# 查看 manifest 列表OVN_IMAGE=harbor.yusur.tech/yusur_ovn/ovn-daemonset-f:second-cnidocker manifest inspect $OVN_IMAGE# 输出如下信息{   "schemaVersion": 2,   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",   "manifests": [      {         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",         "size": 4902,         "digest": "sha256:8ed550698e5dabaee8dbafaf9a4c41f7173fe2aaf17437cd7fc4a8b163983e31",         "platform": {            "architecture": "arm64",            "os": "linux"         }      },      {         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",         "size": 4899,         "digest": "sha256:4c59e7cb42177d9a7aadc476dbf5bf88448bd3097ad763f8ff70753934ec3b16",         "platform": {            "architecture": "amd64",            "os": "linux"         }      }   ]}

方法二

使用amd架构的docker和arm架构的docker 两种docker 整合成一个docker编译平台

docker远程

编辑 /usr/lib/systemd/system/docker.service文件

在如下位置添加 -H tcp://0.0.0.0:2375

重启docker

systemctl daemon-reloadsystemctl restart docker

测试连接

curl http://localhost:2375/version

使用 docker buildx 构建跨平台 容器镜像。

默认的 docker build 命令无法完成跨平台构建任务，我们需要为 docker 命令行安装 buildx 插件扩展其功能。buildx 能够使用由 Moby BuildKit 提供的构建镜像额外特性，它能够创建多个 builder 实例，在多个节点并行地执行构建任务，以及跨平台构建。

buildx启用

macOS 或 Windows 系统的 Docker Desktop，以及 Linux 发行版通过 deb 或者 rpm 包所安装的 docker 内置了 buildx，不需要另行安装。

如果 docker 没有 buildx 命令，可以下载二进制包进行安装：

1. 首先从 Docker buildx 项目的 release 页面找到适合自己平台的二进制文件。

2. 下载二进制文件到本地并重命名为 docker-buildx，移动到 docker 的插件目录 ~/.docker/cli-plugins 或者 /usr/libexec/docker/cli-plugins 目录。

3. 向二进制文件授予可执行权限。

如果本地的 docker 版本高于 19.03，可以通过以下命令直接在本地构建并安装，这种方式更为方便：

DOCKER_BUILDKIT=1 docker build --platform=local -o . "https://github.com/docker/buildx.git"mkdir -p ~/.docker/cli-pluginsmv buildx ~/.docker/cli-plugins/docker-buildx

使用 buildx 进行构建的方法如下：

# docker 20.X 以下版本需要开启环境变量export DOCKER_CLI_EXPERIMENTAL=enabled# 查看docker 是否支持buildxdocker info | grep buildx  buildx: Docker Buildx (Docker Inc., v0.10.4)# buildx 进行构建的方法cdocker buildx build .

buildx 和 docker build 命令的使用体验基本一致，还支持 build 常用的选项如 -t、-f等。

buildx实例

docker buildx 通过 builder 实例对象来管理构建配置和节点，命令行将构建任务发送至 builder 实例，再由 builder 指派给符合条件的节点执行。我们可以基于同一个 docker 服务程序创建多个 builder 实例，提供给不同的项目使用以隔离各个项目的配置，也可以为一组远程 docker 节点创建一个 builder 实例组成构建阵列，并在不同阵列之间快速切换。

使用 docker buildx create 命令可以创建 builder 实例，这将以当前使用的 docker 服务为节点创建一个新的 builder 实例。要使用一个远程节点，可以在创建示例时通过 DOCKER_HOST 环境变量指定远程端口或提前切换到远程节点的 docker context。

下面首先以当前节点创建一个新的 builder 实例，并通过命令行选项指定实例名称、驱动以及当前节点的目标平台：

# docker 20.X 以下版本需要开启环境变量export DOCKER_CLI_EXPERIMENTAL=enabled# 创建新builder 实例 multi-builderdocker buildx create --driver docker-container --platform linux/amd64 --name multi-buildermulti-builder# 添加远程arm平台的buidlerexport DOCKER_HOST=tcp://192.168.100.2:2375 #(前提：远程docker开启远程访问)docker buildx create --name multi-builder --append --node remote-builder#启动新buidler实例 multi-builder[root@yusur-53 test]#docker buildx inspect --bootstrap multi-builderName:          multi-builderDriver:        docker-containerLast Activity: 2023-04-14 09:59:15 +0000 UTCNodes:Name:      multi-builder0Endpoint:  unix:///var/run/docker.sockStatus:    runningBuildkit:  v0.9.3Platforms: linux/amd64, linux/386Name:      remote-builderEndpoint:  tcp://192.168.100.2:2375Status:    runningBuildkit:  v0.11.5Platforms: linux/arm64, linux/arm/v7, linux/arm/v6# 查看现有builder所有实例[root@yusur-53 test]#docker buildx lsNAME/NODE        DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMSmulti-builder *  docker-container                               multi-builder0 unix:///var/run/docker.sock running v0.9.3   linux/amd64, linux/386  remote-builder tcp://192.168.100.2:2375    running v0.11.5  linux/arm64, linux/arm/v7, linux/arm/v6default          docker                                         default        default                     running 20.10.12 linux/arm64, linux/arm/v7, linux/arm/v6

启用新builder 实例

# 启用后才能使用 multi-builder 多架构编译镜像docker buildx use multi-builder

buildx构建

构建单个arm镜像

# 构建arm的镜像到本地OVN_IMAGE=harbor.yusur.tech/yusur_ovn/ovn-daemonset-f:second-cni-aarch64docker buildx build --build-arg BUILDPLATFORM=linux/aarch64 --build-arg TARGETPLATFORM=linux/aarch64  --platform=linux/arm64 -t $OVN_IMAGE -o type=docker -f Dockerfile.fedora .

构建单个amd镜像

# 构建amd的镜像到本地OVN_IMAGE=harbor.yusur.tech/yusur_ovn/ovn-daemonset-f:second-cni-amd64docker buildx build --build-arg BUILDPLATFORM=linux/amd64 --build-arg TARGETPLATFORM=linux/amd64  --platform=linux/amd64 -t $OVN_IMAGE -o type=docker -f Dockerfile.fedora .

构建多种arm, amd镜像

#编译完成后就直接push到hubOVN_IMAGE=harbor.yusur.tech/yusur_ovn/ovn-daemonset-f:second-cnidocker buildx build --push --platform linux/arm64,linux/amd64 -t $OVN_IMAGE -f Dockerfile.fedora .

go代码例子

cat <<EOF > demo.gopackage mainimport (    "fmt"    "log"    "net/http")func listenServer() {    http.Handle("/", http.HandlerFunc(lServe))    http.ListenAndServe(":8080", nil)}func lServe(w http.ResponseWriter, r *http.Request) {    log.Println("response success")    fmt.Fprintln(w, "hello world")}func main() {    listenServer()}EOF# 编写Dockerfilecat <<EOF > Dockerfile# Build the manager binaryFROM golang:1.19 as builderMAINTAINER leidWORKDIR /workspaceADD . ./ENV GO111MODULE=onENV GOPROXY="https://goproxy.io"RUN go build -o demo demo.goFROM alpine:latestWORKDIR /COPY --from=builder /workspace/demo .EXPOSE 8080ENTRYPOINT ["/demo"]EOF

编译该代码

# 编译镜像docker buildx build --push --platform linux/arm64,linux/amd64 -t harbor.yusur.tech/leid/demo:latest .# 查看镜像digestsdocker buildx imagetools inspect harbor.yusur.tech/leid/demo:latestName:      docker.io/flftuu/demo:latestMediaType: application/vnd.docker.distribution.manifest.list.v2+jsonDigest:    sha256:d2bfa8696da0f3e8e4bfe8d4f1b23c1e10dbb7e1dbea7dcfa59a851ae7a8fb8c           Manifests:   Name:      docker.io/flftuu/demo:latest@sha256:58c51a62aa94b9d138d39fdb79759d41a7b04f8c4d8c4901ebbf7ed16363e6a2  MediaType: application/vnd.docker.distribution.manifest.v2+json  Platform:  linux/arm64               Name:      docker.io/flftuu/demo:latest@sha256:00ec9b7455915a9b70c9ffa316326147c1a2225b5bd014367a4cdf97984f0916  MediaType: application/vnd.docker.distribution.manifest.v2+json  Platform:  linux/amd64

方法三

binfmt_misc启用

如果你使用的是 Docker 桌面版（MacOS 和 Windows），默认已经启用了 binfmt_misc，可以跳过这一步。

如果你使用的是 Linux，需要手动启用 binfmt_misc。大多数 Linux 发行版都很容易启用，不过还有一个更容易的办法，直接运行一个特权容器，容器里面写好了设置脚本：

$ docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64

验证是 binfmt_misc 否开启：

[root@centos8~]$ ls -al /proc/sys/fs/binfmt_misc/  total 0drwxr-xr-x 2 root root 0 Jun 16 18:46 .dr-xr-xr-x 1 root root 0 Jun  5 21:42 ..-rw-r--r-- 1 root root 0 Jun 16 19:22 qemu-aarch64-rw-r--r-- 1 root root 0 Jun 16 19:22 qemu-arm-rw-r--r-- 1 root root 0 Jun 16 19:22 qemu-ppc64le-rw-r--r-- 1 root root 0 Jun 16 19:22 qemu-riscv64-rw-r--r-- 1 root root 0 Jun 16 19:22 qemu-s390x--w------- 1 root root 0 Jun 16 18:46 register-rw-r--r-- 1 root root 0 Jun 16 18:46 status

验证是否启用了相应的处理器：

[root@centos8 ~]$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64  enabledinterpreter /usr/bin/qemu-aarch64flags: OCFoffset 0magic 7f454c460201010000000000000000000200b7mask ffffffffffffff00fffffffffffffffffeffff

注意

请将 Linux 内核版本升级到 4.x 以上，特别是 CentOS7 用户。

centos7 用户可以用以下命令开启

$ docker run --rm --privileged multiarch/qemu-user-static:register --reset$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64enabledinterpreter /usr/bin/qemu-aarch64-staticflags: offset 0magic 7f454c460201010000000000000000000200b700mask ffffffffffffff00fffffffffffffffffeffffff$ docker create -it --name dummy multiarch/qemu-user-static:x86_64-aarch64 bash$ docker container ls -aCONTAINER ID        IMAGE                                       COMMAND             CREATED             STATUS              PORTS               NAMES6ab622a76dfa        multiarch/qemu-user-static:x86_64-aarch64   "bash"              3 minutes ago       Created                                 dummy$ docker cp dummy:/usr/bin/qemu-aarch64-static qemu-aarch64-static$ ls qemu-aarch64-staticqemu-aarch64-static*$ docker rm -f dummy$ docker container ls -aCONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES$ docker run --rm -t -v $(pwd)/qemu-aarch64-static:/usr/bin/qemu-aarch64-static arm64v8/ubuntu uname -maarch64docker run --rm --privileged multiarch/qemu-user-static:register --reset

切换多平台构建器

由于 Docker 默认的 builder 实例不支持同时指定多个 --platform，我们必须首先创建一个新的 builder 实例手动替换

docker buildx create --use --name multi-builder

启动构建器：

root@master01:/tmp# docker buildx inspect multi-builder --bootstrap[+] Building 2.7s (1/1) FINISHED                                                                                                                                       => [internal] booting buildkit                                                                                                                                  2.7s => => pulling image moby/buildkit:buildx-stable-1                                                                                                               2.3s => => creating container buildx_buildkit_multi-builder0                                                                                                         0.4sName:          multi-builderDriver:        docker-containerLast Activity: 2023-04-17 03:08:42 +0000 UTCNodes:Name:      multi-builder0Endpoint:  unix:///var/run/docker.sockStatus:    runningBuildkit:  v0.11.5Platforms: linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

查看当前使用的构建器及构建器支持的 CPU 架构，可以看到支持很多 CPU 架构：

root@master01:/tmp# docker buildx lsNAME/NODE        DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMSmulti-builder *  docker-container                               multi-builder0 unix:///var/run/docker.sock running v0.11.5  linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6default          docker                                         default        default                     running 23.0.3   linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/arm/v7, linux/arm/v6

构建多平台镜像

新建 Dockerfile 文件,将该应用容器化

FROM --platform=$TARGETPLATFORM alpineRUN uname -a > /os.txtCMD cat /os.txt

现在就可以使用 buildx 构建一个支持 arm、arm64 和 amd64 多架构的 Docker 镜像了，同时将其推送到 Docker Hub：

--push 参数表示将构建好的镜像推送到 Docker 仓库。

提前使用docker login 命令登录认证 Docker Hub

docker buildx build --platform linux/arm,linux/arm64,linux/amd64 -t flftuu/demo . --push# 查看镜像信息docker buildx imagetools inspect flftuu/demo

背后的原理也很简单，之前已经提到过了，buildx 会通过

QEMU 和 binfmt_misc 分别为 3 个不同的 CPU 架构（arm，arm64 和 amd64）构建 3 个不同的镜像。构建完成后，就会创建一个 manifest list，其中包含了指向这 3 个镜像的指针。

如果想将构建好的镜像保存在本地，可以将

type 指定为 docker，但必须分别为不同的 CPU 架构构建不同的镜像，不能合并成一个镜像，即：

docker buildx build -t hello-arch:arm --platform=linux/arm -o type=docker .docker buildx build -t hello-arch:arm64 --platform=linux/arm64 -o type=docker .docker buildx build -t hello-arch:amd64 --platform=linux/amd64 -o type=docker .

测试多平台镜像

由于之前已经启用了 binfmt_misc，现在我们就可以运行任何 CPU 架构的 Docker 镜像了，因此可以在本地系统上测试之前生成的 3 个镜像是否有问题。

# arm$ docker run -it --rm hello-arch:armLinux buildkitsandbox 4.18.0-80.el8.x86_64 #1 SMP Tue Jun 4 09:19:46 UTC 2019 armv7l Linux# arm64$ docker run -it --rm hello-arch:arm64Linux buildkitsandbox 4.18.0-80.el8.x86_64 #1 SMP Tue Jun 4 09:19:46 UTC 2019 aarch64 Linux# amd64$ docker run -it --rm hello-arch:amd64 Linux buildkitsandbox 4.18.0-80.el8.x86_64 #1 SMP Tue Jun 4 09:19:46 UTC 2019 x86_64 Linux

第一种方法通过k8s标准资源networkpolicy实现ACL（推荐使用）

什么是网络策略？

首先NetworkPolicy是k8s的一种resource，可以通过以下三种维度对整个namespace或单一POD进行隔离：pod（允许某些pod访问）、namespace（允许某些namespace访问）、ip段（CIDR，但是pod所运行node节点始终可以访问该pod）当定义基于pod和namespace的NetworkPolicy时，需要根据标签来选择对应的pod或者namespace。

另外当需要使用NetworkPolicy资源时，k8s集群采用的网络插件必须支持，比如ovn-kubernetes等等

隔离和非隔离pod

默认情况所有的pod都是非隔离的，当pod被NetworkPolicy选择后，那么pod就是一个隔离的pod，出栈和入栈都要遵循NetworkPolicy的策略，并且如果有多个NetworkPolicy时，他们是叠加的而不是冲突的。

限制一个pod的NetworkPolicy需要从两个方向去考虑：ingress和egress。ingress表示入方向，egress表示出方向。

基于namespace的网络策略

以下yaml文件代表在app下创建一个NetworkPolicy资源，该策略作用在所有被打了app=nginx-test标签的pod，策略内容为只接受来自namespace为app的访问。

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: ns-network-policy

  namespace: app

spec:　

  #定义策略的类型：值是一个列表，支持Ingress和Egress两种，默认Egress不做任何限制

  policyTypes: [“Ingress”]

  #表示该策略作用的pod，以标签匹配，当该值是一个{}空字典时，代表所有POD

  podSelector:

    matchLabels:

      app: nginx-test

  #以下表示ingress策略

  ingress:

    - from:

       - namespaceSelector:

          matchLabels:

            kubernetes.io/metadata.name: app

基于POD的网络策略

以下yaml文件策略表示：允许带有app=nginx-test2的所有pod访问app=nginx-test的pod

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: ns-network-policy1

  namespace: app

spec:

  policyTypes: [“Ingress”]

  podSelector: 

    matchLabels:

      app: nginx-test

  ingress:

    - from:

       - podSelector: 

          matchLabels:

            app: nginx-test2

基于IP段的网络策略

以下yaml文件表示策略为：允许10.244.0.0/16段但是不包括10.244.1.0/24来访问带有app=nginx-test的pod

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: ns-network-policy

  namespace: olda

spec:

  policyTypes: [“Ingress”]

  podSelector: 

    matchLabels:

      app: nginx-test

  ingress:

    - from:

       - ipBlock: 

          cidr: 10.244.0.0/16

          except:

          - 10.244.1.0/24

关于端口号的限制

单个端口号，只允许app名称空间下的pod连接80端口，具体策略如下

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: test-network-policy

  namespace: app

spec:

  policyTypes: [“Ingress”]

  podSelector: 

    matchLabels:

      app: test

  ingress:

    - from:

       - namespaceSelector: 

          matchLabels:

            kubernetes.io/metadata.name: app

      ports:

      - protocol: TCP

        port: 80

多个端口号，在网络策略里增加3306端口

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: test-network-policy

  namespace: app

spec:

  policyTypes: [“Ingress”]

  podSelector: 

    matchLabels:

      app: test

  ingress:

    - from:

       - namespaceSelector: 

          matchLabels:

            kubernetes.io/metadata.name: app

      ports:

      - protocol: TCP

        port: 80

      - protocol: TCP

        port: 3306

常用的网络策略具体yaml文件示例

拒绝所有入站流量，具体yaml文件示例：

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: default-deny-ingress

spec:

  podSelector: {}

  policyTypes:

  - Ingress

这确保即使没有被任何其他 NetworkPolicy 选择的 Pod 仍将被隔离以进行入口。 此策略不影响任何 Pod 的出口隔离。

允许所有入站流量，具体yaml文件示例：

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: allow-all-ingress

spec:

  podSelector: {}

  ingress:

  - {}

  policyTypes:

  - Ingress

有了这个策略，任何额外的策略都不会导致到这些 Pod 的任何入站连接被拒绝。 此策略对任何 Pod 的出口隔离没有影响。

拒绝所有出站流量，具体yaml文件示例：

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: default-deny-egress

spec:

  podSelector: {}

  policyTypes:

  - Egress

此策略可以确保即使没有被其他任何 NetworkPolicy 选择的 Pod 也不会被允许流出流量。 此策略不会更改任何 Pod 的入站流量隔离行为。

允许所有出战流量，具体yaml文件示例：

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: allow-all-egress

spec:

  podSelector: {}

  egress:

  - {}

  policyTypes:

  - Egress

拒绝所有入站和出站流量，具体yaml文件示例：

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: default-deny-all

spec:

  podSelector: {}

  policyTypes:

  - Ingress

  - Egress

限制特定的pod访问，具体yaml文件示例：

kind: NetworkPolicy

apiVersion: networking.k8s.io/v1

metadata:

  name: app-test

spec:

  podSelector:

    matchLabels:

      app: bookstore

      role: api

  ingress:

  - from:

      - podSelector:

          matchLabels:

            app: bookstore

允许所有的namespace访问pod，具体yaml文件示例：

kind: NetworkPolicy

apiVersion: networking.k8s.io/v1

metadata:

  namespace: default

  name: web-allow-all-namespaces

spec:

  podSelector:

    matchLabels:

      app: web

  ingress:

  - from:

    - namespaceSelector: {}

允许特定的namespace访问pod，具体yaml文件示例：

kind: NetworkPolicy

apiVersion: networking.k8s.io/v1

metadata:

  name: web-allow-prod

spec:

  podSelector:

    matchLabels:

      app: web

  ingress:

  - from:

    - namespaceSelector:

        matchLabels:

          purpose: production

根据namespace和pod相结合的策略允许pod访问，具体yaml文件示例：

kind: NetworkPolicy

apiVersion: networking.k8s.io/v1

metadata:

  name: web-allow-all-ns-monitoring

  namespace: default

spec:

  podSelector:

    matchLabels:

      app: web

  ingress:

    - from:

      - namespaceSelector:     # chooses all pods in namespaces labelled with team=operations

          matchLabels:

            team: operations  

        podSelector:           # chooses pods with type=monitoring

          matchLabels:

            type: monitoring

根据特定的pod和port结合策略允许访问pod，具体yaml文件示例：

kind: NetworkPolicy

apiVersion: networking.k8s.io/v1

metadata:

  name: api-allow-5000

spec:

  podSelector:

    matchLabels:

      app: apiserver

  ingress:

  - ports:

    - port: 5000

    from:

    - podSelector:

        matchLabels:

          role: monitoring

限制访问外部的流量，但可以访问DNS，具体yaml文件示例：

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

  name: foo-deny-egress

spec:

  podSelector:

    matchLabels:

      app: foo

  policyTypes:

  - Egress

  egress:

  # allow DNS resolution

  - to:

    - namespaceSelector:

        matchLabels:

          kubernetes.io/metadata.name: kube-system

      podSelector:

        matchLabels:

          k8s-app: kube-dns

    ports:

      - port: 53

        protocol: UDP

      - port: 53

        protocol: TCP

设置了网络策略，如何在ovn里查看是否生成相应的ACL

登录ovn-kubernetes的环境，使用如下命令登录如图的pod：

kubectl exec -it -n ovn-kubernetes ovnkube-master-788b596d78-l7gkt -c ovnkube-master – bash -il

登录之后，使用如下命令，查看ACL的数据：

Ovn-nbctl find acl

此时没有设置网络策略，就没有多余的ACL，设置一个网络策略之后就会发现多一些ACL的数据：

注意：如果针对同一个规则同时设置了deny和allow的ACL，allow的优先级会比deny优先级高，所以会允许allow。

Networkpolicy使用限制：

通过网络策略（至少目前还）无法完成的工作，到 Kubernetes 1.26 为止，NetworkPolicy API 还不支持以下功能， 不过你可能可以使用操作系统组件（如 SELinux、OpenVSwitch、IPTables 等等） 或者第七层技术（Ingress 控制器、服务网格实现）或准入控制器来实现一些替代方案。 如果你对 Kubernetes 中的网络安全性还不太了解，了解使用 NetworkPolicy API 还无法实现下面的用户场景是很值得的。

• 强制集群内部流量经过某公用网关（这种场景最好通过服务网格或其他代理来实现）；

• 与 TLS 相关的场景（考虑使用服务网格或者 Ingress 控制器）；

• 特定于节点的策略（你可以使用 CIDR 来表达这一需求不过你无法使用节点在 Kubernetes 中的其他标识信息来辩识目标节点）；

• 基于名字来选择服务（不过，你可以使用 标签 来选择目标 Pod 或名字空间，这也通常是一种可靠的替代方案）；

• 创建或管理由第三方来实际完成的“策略请求”；

• 实现适用于所有名字空间或 Pods 的默认策略（某些第三方 Kubernetes 发行版本或项目可以做到这点）；

• 高级的策略查询或者可达性相关工具；

• 生成网络安全事件日志的能力（例如，被阻塞或接收的连接请求）；

• 显式地拒绝策略的能力（目前，NetworkPolicy 的模型默认采用拒绝操作， 其唯一的能力是添加允许策略）；

• 禁止本地回路或指向宿主的网络流量（Pod 目前无法阻塞 localhost 访问， 它们也无法禁止来自所在节点的访问请求）。

参考资料

++https://kubernetes.io/docs/concepts/services-networking/network-policies/++

第二种方法通过ovn-kubernetes自定义的crd并根据资源来控制实现ACL（因限制太多不推荐使用）

找到生成的crd的yaml文件k8s.ovn.org_egressfirewalls.yaml

在部署ovn的文件夹ovn/ovn-kubernetes/dist/yaml里，有一个k8s.ovn.org_egressfirewalls.yaml文件，是根据命令生成yaml文件时生产的：

./daemonset.sh –image=yusur/ovn-daemonset-f:fullmode –net-cidr=10.123.0.0/16 –svc-cidr=10.86.0.0/16 –gateway-mode=”local” –k8s-apiserver=https://192.168.122.189:6443

创建EgressFirewall资源

找到上述的yaml文件k8s.ovn.org_egressfirewalls.yaml，使用命令：

kubectl apply -f k8s.ovn.org_egressfirewalls.yaml

修改ovnkube-master.yaml里面的参数使EgressFirewall资源生效

如果在已经部署的环境中，请修改部署ovn的文件夹ovn/ovn-kubernetes/dist/yaml里的ovnkube-master.yaml，修改里面的- name: OVN_EGRESSFIREWALL_ENABLE为true

需要重新按修改后的yaml文件重新拉起这个pod

如果是即将部署的环境，请使用如下参数添加即可：

./daemonset.sh   –egress-firewall-enable=true

设置EgressFirewall规则

EgressFirewall特性允许集群管理员限制项目中的pod可以访问的外部主机。 

EgressFirewall对象规则适用于所有共享的pod带有egressfirewall对象的名称空间。仅一个名称空间，支持有一个EgressFirewallObject。

如下示例：

kind: EgressFirewall

apiVersion: k8s.ovn.org/v1

metadata:

  name: default  #该name必须设置为default，否则会创建失败

  namespace: default

spec:

  egress:

  - type: Allow

    to:

      dnsName: www.openvswitch.org

  - type: Allow

    to:

      cidrSelector: 1.2.3.0/24

  - type: Allow

    to:

      cidrSelector: 4.5.6.0/24

    ports:

      - protocol: UDP

        port: 55

  - type: Deny

    to:

      cidrSelector: 0.0.0.0/0

这个例子允许连接到默认名称空间中的Pods，www.openvswitch.org转换为的主机，任何外部主机在1.2.3.0到1.2.3.255范围内的主机，并且允许添加只有端口上的UDP协议才会将流量设置为4.5.6.0到4.5.6.255第55号，并拒绝所有其他外部主机的流量。端口是可选的，允许用户指定特定的端口To和协议允许或拒绝流量。规则的优先级由其在出口中的位置决定数组中。较早的规则在较晚的规则之前被处理。在前面的例子，如果规则颠倒过来，所有流量都被拒绝，包括到1.2.3.0/24 CIDR块中的主机的任何流量。使用DNS特性假设已定位节点和主节点 在与添加到ovn的DNS条目类似的位置数据库是由主机生成的。 注意:在deny规则中使用DNS名称时，请谨慎使用。DNS拦截器将永远不会完美地工作，并可能允许访问一个被拒绝的主机，如果节点上的DNS解析与主节点上的DNS解析不同。

使用kubectl apply -f xxx.yaml文件之后，同样会在ovn-master的pod里面看见新增加的ACL

Multus CNI enables attaching multiple network interfaces to pods in Kubernetes.

以上是 Multus CNI 项目官方对其存在意义的精简描述，它的存在就是帮助 K8s 的 Pod（可简单理解为一组容器的集合，是 K8s 可管理的最小“容器”单位）建立多网络接口。

Multus CNI 本身不提供网络配置功能，它是通过用其他满足 CNI 规范的插件进行容器的网络配置。

创建CNI网络

calico网络的NetworkAttachmentDefinition

apiVersion: "k8s.cni.cncf.io/v1"kind: NetworkAttachmentDefinitionmetadata:  name: calico-net #网络名称  namespace: kube-systemspec:  config: '{  "name": "calico-net",  "cniVersion": "0.3.1",  "plugins": [    {      "type": "calico",      "log_level": "info",      "log_file_path": "/var/log/calico/cni/cni.log",      "datastore_type": "kubernetes",      "mtu": 0,      "ipam": {          "type": "calico-ipam"      },      "policy": {          "type": "k8s"      },      "kubernetes": {          "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"      }    },    {      "type": "portmap",      "snat": true,      "capabilities": {"portMappings": true}    },    {      "type": "bandwidth",      "capabilities": {"bandwidth": true}    }  ]}'

ovn网络的NetworkAttachmentDefinition

apiVersion: "k8s.cni.cncf.io/v1"kind: NetworkAttachmentDefinitionmetadata:  name: ovn-net #网络名称  namespace: kube-system  annotations:    k8s.v1.cni.cncf.io/resourceName: yusur.tech/sriov_dpuspec:  config: '{    "cniVersion": "0.4.0",    "name": "ovn-kubernetes",    "type": "ovn-k8s-cni-overlay",    "ipam": {},    "dns": {},    "logFile": "/var/log/ovn-kubernetes/ovn-k8s-cni-overlay.log",    "logLevel": "5",    "logfile-maxsize": 100,    "logfile-maxbackups": 5,    "logfile-maxage": 5    }'

介绍CNI网络

关键字段说明

:::
v1.multus-cni.io/default-network: calico-net 

#该字段信息表示pod使用calico 网络， calico-net 必须是上面创建的NetworkAttachmentDefinition 对象名称

k8s.v1.cni.cncf.io/networks: ‘[{

      “name”: “ovn-net”,

      “namespace”: “kube-system”

      “interface”: “eth1”,

      “default-route”: [“10.124.0.1”]

    }]‘ #该字段表示使用多个辅助cni，ovn-net 和kube-system 必须是上面创建NetworkAttachmentDefinition对象名称
:::

使用CNI网络

单CNI

Calico网络

apiVersion: v1kind: Servicemetadata:  name: nginx2-calico  namespace: defaultspec:  selector:    app: nginx2-calico  ports:  - protocol: TCP    port: 80    targetPort: server---apiVersion: v1kind: Podmetadata:  name: nginx2-calico  namespace: default  annotations:    #单独使用calico网络    v1.multus-cni.io/default-network: calico-net  labels:    app: nginx2-calicospec:  containers:  - image: ubuntu:22.04-yusur    command: ['/bin/sh', '-c', 'nginx -g "daemon off;"']    imagePullPolicy: IfNotPresent    name: nginx2-calico    securityContext:      runAsUser: 0      privileged: true    ports:    - containerPort: 80      name: server  restartPolicy: Always  nodeSelector:    kubernetes.io/hostname: host-205

OVN网络

apiVersion: v1kind: Servicemetadata:  name: nginx2-ovn  namespace: defaultspec:  selector:    app: nginx2-ovn  ports:  - protocol: TCP    port: 80    targetPort: server---apiVersion: v1kind: Podmetadata:  name: nginx2-ovn  namespace: default  annotations:    #单独使用ovn网络    v1.multus-cni.io/default-network: ovn-net  labels:    app: nginx2-ovnspec:  containers:  - image: ubuntu:22.04-yusur    command: ['/bin/sh', '-c', 'nginx -g "daemon off;"']    imagePullPolicy: IfNotPresent    name: nginx2-ovn    securityContext:      runAsUser: 0      privileged: true    ports:    - containerPort: 80      name: server    resources:      requests:        yusur.tech/sriov_dpu: '1'      limits:        yusur.tech/sriov_dpu: '1'  restartPolicy: Always  nodeSelector:    kubernetes.io/hostname: host-205    k8s.ovn.org/dpu-host: ""

双CNI

Calico为主，OVN为辅

默认路由为Calico

++暂时不支持，需研发（该期需求）++

默认路由为OVN

apiVersion: v1kind: Servicemetadata:  name: nginx2-calico-ovn  namespace: defaultspec:  selector:    app: nginx2-calico-ovn  ports:  - protocol: TCP    port: 80    targetPort: server---apiVersion: v1kind: Podmetadata:  name: nginx2-calico-ovn  namespace: default  annotations:    #使用calico网络为主，ovn为辅    v1.multus-cni.io/default-network: calico-net    k8s.v1.cni.cncf.io/networks: '[{      "name": "ovn-net",      "namespace": "kube-system",      "interface": "eth1",      "default-route": ["10.124.2.1"]    }]'  labels:    app: nginx2-calico-ovnspec:  containers:  - image: ubuntu:22.04-yusur    command: ['/bin/sh', '-c', 'nginx -g "daemon off;"']    imagePullPolicy: IfNotPresent    name: nginx2-calico-ovn    securityContext:      runAsUser: 0      privileged: true    ports:    - containerPort: 80      name: server    resources:      requests:        yusur.tech/sriov_dpu: '1'      limits:        yusur.tech/sriov_dpu: '1'  restartPolicy: Always  nodeSelector:    kubernetes.io/hostname: host-205    k8s.ovn.org/dpu-host: ""

OVN为主，Calico为辅

默认路由为Calico

apiVersion: v1kind: Servicemetadata:  name: nginx2-ovn-calico  namespace: defaultspec:  selector:    app: nginx2-ovn-calico  ports:  - protocol: TCP    port: 80    targetPort: server---apiVersion: v1kind: Podmetadata:  name: nginx2-ovn-calico  namespace: default  annotations:    #使用ovn网络为主，calico为辅    v1.multus-cni.io/default-network: ovn-net    k8s.v1.cni.cncf.io/networks: '[{      "name": "calico-net",      "namespace": "kube-system",      "interface": "eth1",      "default-route": ["169.254.1.1"]    }]'  labels:    app: nginx2-ovn-calicospec:  containers:  - image: ubuntu:22.04-yusur    command: ['/bin/sh', '-c', 'nginx -g "daemon off;"']    imagePullPolicy: IfNotPresent    name: nginx2-ovn-calico    securityContext:      runAsUser: 0      privileged: true    ports:    - containerPort: 80      name: server    resources:      requests:        yusur.tech/sriov_dpu: '1'      limits:        yusur.tech/sriov_dpu: '1'  restartPolicy: Always  nodeSelector:    kubernetes.io/hostname: host-205    k8s.ovn.org/dpu-host: ""

默认路由为OVN

++暂时不支持++

当容器使用ovn网络的时候，需要给容器里面注入ovn 分配对应的vf网卡的ip信息

注入方法

给每个容器挂在volume，注入vf信息固定写法

#yaml容器段，定义挂载目录地址    volumeMounts:    - name: vfus      mountPath: /opt/yusur_ovn#yaml pod段，定义volume类容volumes:    - name: vfus      downwardAPI:        items:          - path: "config.ini"            fieldRef:              fieldPath: metadata.annotations['k8s.ovn.org/vfus-networks']

Pods资源

yaml 案例:

apiVersion: v1kind: Podmetadata:  name: nginx1-ovn-pod  namespace: default  annotations:    v1.multus-cni.io/default-network: calico-net    k8s.v1.cni.cncf.io/networks: kube-system/ovn-net@eth1 #配置辅助cni为ovn  labels:    app: nginx1-ovn-podspec:  containers:  - image: ubuntu:22.04-yusur    command: ['/bin/sh', '-c', 'nginx -g "daemon off;"']    imagePullPolicy: IfNotPresent    name: nginx1-ovn-pod    securityContext:      runAsUser: 0      privileged: true    ports:    - containerPort: 80    volumeMounts:    - name: vfus      mountPath: /opt/yusur_ovn    resources:      requests:        yusur.tech/sriov_dpu: '1'      limits:        yusur.tech/sriov_dpu: '1'  restartPolicy: Always  volumes:    - name: vfus      downwardAPI:        items:          - path: "config.ini"            fieldRef:              fieldPath: metadata.annotations['k8s.ovn.org/vfus-networks']

查看结果：

kubectl exec -it nginx1-ovn-pod -- cat /opt/yusur_ovn/config.ini | jq{  "port0": {    "addr": "10.124.1.18",    "netmask": "255.255.255.0",    "broadcast": "10.124.1.255",    "gateway": "10.124.1.1"  }}

Deployments资源

yaml 案例:

apiVersion: apps/v1kind: Deploymentmetadata:  namespace: default  name: nginx1-ovn-deployspec:  replicas: 1  selector:    matchLabels:      app: nginx1-ovn-deploy  template:    metadata:      labels:        app: nginx1-ovn-deploy      annotations:        v1.multus-cni.io/default-network: calico-net         k8s.v1.cni.cncf.io/networks: kube-system/ovn-net@eth1 #配置辅助cni为ovn    spec:      nodeSelector:        k8s.ovn.org/dpu-host: ""      containers:      - name: nginx1-ovn-deploy        image: ubuntu:22.04-yusur        imagePullPolicy: IfNotPresent        resources:          requests:            yusur.tech/sriov_dpu: '1'          limits:            yusur.tech/sriov_dpu: '1'        ports:        - containerPort: 80        volumeMounts:        - name: vfus          mountPath: /opt/yusur_ovn      volumes:       - name: vfus         downwardAPI:           items:             - path: "config.ini"               fieldRef:                 fieldPath: metadata.annotations['k8s.ovn.org/vfus-networks']

查看结果：

kubectl exec -it nginx1-ovn-deploy-c474975db-l642g -- cat /opt/yusur_ovn/config.ini | jq{  "port0": {    "addr": "10.124.1.17",    "netmask": "255.255.255.0",    "broadcast": "10.124.1.255",    "gateway": "10.124.1.1"  }}

Daemonsets资源

yaml 案例:

apiVersion: apps/v1kind: DaemonSetmetadata:  name: nginx1-ovn-ds  labels:    app: nginx1-ovn-dsspec:  selector:    matchLabels:      octopusexport: OctopusExport  updateStrategy:    type: RollingUpdate  template:    metadata:      labels:        app: nginx1-ovn-ds        octopusexport: OctopusExport      annotations:        v1.multus-cni.io/default-network: calico-net        k8s.v1.cni.cncf.io/networks: kube-system/ovn-net@eth1    spec:      volumes:        - name: vfus          downwardAPI:            items:              - path: config.ini                fieldRef:                  fieldPath: 'metadata.annotations[''k8s.ovn.org/vfus-networks'']'      containers:        - name: nginx1-ovn-ds          image: ubuntu:22.04-yusur          imagePullPolicy: IfNotPresent          ports:            - name: server              containerPort: 80              protocol: TCP          volumeMounts:            - name: vfus              mountPath: /opt/yusur_ovn              subPath: ''          resources:            requests:              yusur.tech/sriov_dpu: '1'            limits:              yusur.tech/sriov_dpu: '1'          securityContext:            privileged: true            runAsNonRoot: false

查看结果：

kubectl exec -it nginx1-ovn-ds-mt747 -- cat /opt/yusur_ovn/config.ini | jq{  "port0": {    "addr": "10.124.1.21",    "netmask": "255.255.255.0",    "broadcast": "10.124.1.255",    "gateway": "10.124.1.1"  }}

Statefulsets资源

yaml 案例:

apiVersion: apps/v1kind: StatefulSetmetadata:  name: nginx1-ovn-sts  labels:    app: nginx1-ovn-stsspec:  selector:    matchLabels:      octopusexport: OctopusExport  replicas: 1  serviceName: nginx1-ovn-sts  updateStrategy:    type: RollingUpdate  template:    metadata:      labels:        app: nginx1-ovn-sts        octopusexport: OctopusExport      annotations:        v1.multus-cni.io/default-network: calico-net        k8s.v1.cni.cncf.io/networks: kube-system/ovn-net@eth1    spec:      volumes:        - name: vfus          downwardAPI:            items:              - path: config.ini                fieldRef:                  fieldPath: 'metadata.annotations[''k8s.ovn.org/vfus-networks'']'      containers:        - name: nginx1-ovn-sts          image: ubuntu:22.04-yusur          imagePullPolicy: IfNotPresent          ports:            - name: server              containerPort: 80              protocol: TCP          volumeMounts:            - name: vfus              mountPath: /opt/yusur_ovn          resources:            requests:              yusur.tech/sriov_dpu: '1'            limits:              yusur.tech/sriov_dpu: '1'          securityContext:            privileged: true            runAsNonRoot: false

查看结果：

kubectl exec -it nginx1-ovn-sts-0 -- cat /opt/yusur_ovn/config.ini | jq{  "port0": {    "addr": "10.124.1.22",    "netmask": "255.255.255.0",    "broadcast": "10.124.1.255",    "gateway": "10.124.1.1"  }}

多容器注入

每个container 都需挂载volume

yaml 案例:

apiVersion: v1kind: Podmetadata:  name: nginx1-ovn-pod2  namespace: default  annotations:    v1.multus-cni.io/default-network: calico-net    k8s.v1.cni.cncf.io/networks: kube-system/ovn-net@eth1 #配置辅助cni为ovn  labels:    app: nginx1-ovn-pod2spec:  containers:  - image: ubuntu:22.04-yusur    command: ['/bin/sh', '-c', 'nginx -g "daemon off;"']    imagePullPolicy: IfNotPresent    name: nginx1-ovn-pod2    securityContext:      runAsUser: 0      privileged: true    ports:    - containerPort: 80    volumeMounts:    - name: vfus      mountPath: /opt/yusur_ovn    resources:      requests:        yusur.tech/sriov_dpu: '1'      limits:        yusur.tech/sriov_dpu: '1'  - image: ubuntu:22.04-yusur    command: ['/bin/sh', '-c', 'sleep 355555']    imagePullPolicy: IfNotPresent    name: nginx1-ovn-pod2-2    securityContext:      runAsUser: 0      privileged: true    ports:    - containerPort: 81    volumeMounts:    - name: vfus      mountPath: /opt/yusur_ovn    resources:      requests:        yusur.tech/sriov_dpu: '1'      limits:        yusur.tech/sriov_dpu: '1'  restartPolicy: Always  volumes:    - name: vfus      downwardAPI:        items:          - path: "config.ini"            fieldRef:              fieldPath: metadata.annotations['k8s.ovn.org/vfus-networks']

查看结果：

root@yusur-62:/home/leid/cni/test/vfus# kubectl exec -it nginx1-ovn-pod2 -c nginx1-ovn-pod2 -- cat /opt/yusur_ovn/config.ini | jq {  "port0": {    "addr": "10.124.1.4",    "netmask": "255.255.255.0",    "broadcast": "10.124.1.255",    "gateway": "10.124.1.1"  }}root@yusur-62:/home/leid/cni/test/vfus# kubectl exec -it nginx1-ovn-pod2 -c nginx1-ovn-pod2-2 -- cat /opt/yusur_ovn/config.ini | jq {  "port0": {    "addr": "10.124.1.4",    "netmask": "255.255.255.0",    "broadcast": "10.124.1.255",    "gateway": "10.124.1.1"  }}

参考工具

yaml生成工具https://k8syaml.com/

背景目的

在流量分离双CNI的场景下，实现实时路由下发，控制SVC服务从指定CNI网络出去；

双CNI方案架构：

1. Calico作为默认网络，承载管理流量与非低延时业务流量；

2. OVN-Kubernetes作为第二网络，承载低延时业务流量；

最终路由

路由最终效果展示：eth0: 为calico 网络，eth1: 为OVN网络

Kernel IP routing tableDestination     Gateway         Genmask         Flags Metric Ref    Use Iface0.0.0.0         169.254.1.1     0.0.0.0         UG    0      0        0 eth010.86.69.17     0.0.0.0         255.255.255.255 UH    0      0        0 eth010.86.205.109   0.0.0.0         255.255.255.255 UH    0      0        0 eth110.124.0.0      10.124.2.1      255.255.0.0     UG    0      0        0 eth110.124.2.0      0.0.0.0         255.255.255.0   U     0      0        0 eth1169.254.1.1     0.0.0.0         255.255.255.255 UH    0      0        0 eth0172.16.10.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0

路由解释：

1. 默认路由0.0.0.0 走calico网络eth0

2. SVC网段10.86.0.0/16走calico 网络，不用配置，走默认路由即可实现

3. 单一SVC：10.86.69.17/32走calico网络eth0

4. 单一SVC：10.86.205.109/32走OVN网络eth1

5. OVN段10.124.0.0/16， 10.124.2.0/24走OVN网络eth1

6. 自定义网段172.16.10.0/24 走OVN网络eth1

7. 自定义网段192.168.2.0/24 走calico网络eth0

研发需求

路由配置需求：

• 默认路由为calico网络（ovn-k8s组件来实现）

• svc整段出calico网络（ovn-k8s组件来实现）

• 单一SVC可实时配置走不同网络calico or ovn

• 自定义网段可实时配置走不同网络calico or ovn

• 支持修改默认路由

配置文件需求：

• 支持json，yaml格式：eg: cr-config.yaml, cr-config.json

• 支持SVC名称写法：eg: {“svc”: “/“, “dev”: “网卡名”} ,   {“svc”: “default/nginx1”, “dev”: “eth0”} 

• 支持网段写法：eg: {“route”:”172.16.0.0/16 “, “dev”: “eth1”}

路由下发需求：

• k8s集群中所有pod都下发自定义路由（目前需求）

• pod中无指定网卡，则对应路由不下发（目前需求，无双cni的pod中会出现）

• 宿主网络pod 不下发路由（目前需求，hostnetwork 类型pod）

• 指定特定pod下发自定义路由

• 指定特定namespace中所有pod 下发自定义路由

• 指定deploy， ds，sts 对应的所有pod 下发自定义路由

• 排除特定pod下发自定义路由

• 排除特定namespace中所有pod 下发自定义路由

技术方案

开发工具：Kubebuilder@3.4.1，****code-generator@0.24.0

相关教程：

https://xieys.club/code-generator-crd/

https://jimmysong.io/kubernetes-handbook/develop/kubebuilder-example.html

使用K8S中的crd 实现 路由配置规则定义和管理

使用Kubebuilder 来构建crd资源的 API、Controller

api定义

group: k8s.ovn.org

kind: route

version:v1

---apiVersion: apiextensions.k8s.io/v1kind: CustomResourceDefinitionmetadata:  annotations:    controller-gen.kubebuilder.io/version: v0.8.0  creationTimestamp: null  name: routes.k8s.ovn.orgspec:  group: k8s.ovn.org  names:    kind: Route    listKind: RouteList    plural: routes    singular: route  scope: Cluster  versions:  - additionalPrinterColumns:    - jsonPath: .status.status      name: Route Status      type: string    name: v1    schema:      openAPIV3Schema:        description: Route is the Schema for the routes API        properties:          apiVersion:            description: 'APIVersion defines the versioned schema of this representation              of an object. Servers should convert recognized schemas to the latest              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'            type: string          kind:            description: 'Kind is a string value representing the REST resource this              object represents. Servers may infer this from the endpoint the client              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'            type: string          metadata:            type: object          spec:            description: RouteSpec defines the desired state of Route            properties:              feature_gate:                additionalProperties:                  type: string                type: object              route:                items:                  description: RouteRule defines custom route info                  properties:                    dev:                      description: 网卡名称                      type: string                    dst:                      description: 自定义网络段                      type: string                    via:                      description: 网关地址                      type: string                  required:                  - dev                  - dst                  type: object                type: array              svc:                items:                  description: SvcRule defines svc info                  properties:                    dev:                      type: string                    name:                      type: string                    namespace:                      type: string                    via:                      type: string                  required:                  - dev                  - name                  - namespace                  - via                  type: object                type: array            required:            - route            - svc            type: object          status:            description: RouteStatus defines the observed state of Route            properties:              status:                description: 'INSERT ADDITIONAL STATUS FIELD - define observed state                  of cluster Important: Run "make" to regenerate code after modifying                  this file'                type: string            type: object        type: object    served: true    storage: true    subresources:      status: {}status:  acceptedNames:    kind: ""    plural: ""  conditions: []  storedVersions: []

demo

apiVersion: k8s.ovn.org/v1kind: Routemetadata:  name: demospec:  route:    - dst: "172.16.10.0/24"      dev: "eth1"    - dst: "172.16.11.0/24"      dev: "eth0"  svc:    - namespace: "default"      name: "nginx"      dev: "eth1"    - namespace: "kube-system"      name: "nginx1"      dev: "eth0"    - namespace: "kube-system"      name: "nginx2"      dev: "eth0"

自定义字段信息：

/*Copyright 2023.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at    http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.*/package v1import (    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1")// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.// RouteSpec defines the desired state of Routetype RouteSpec struct {    // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster    // Important: Run "make" to regenerate code after modifying this file    Route []RouteRule `json:"route"`    Svc   []SvcRule   `json:"svc"`    // +optional    FeatureGate map[string]string `json:"feature_gate"` // 可拓展参数，保证crd的扩展性}// RouteRule defines custom route infotype RouteRule struct {    // 自定义网络段    Dst string `json:"dst"`    // 网卡名称    Dev string `json:"dev"`    // 网关地址    // +optional    Via string `json:"via"`}// SvcRule defines svc infotype SvcRule struct {    Namespace string `json:"namespace"`    Name      string `json:"name"`    Dev       string `json:"dev"`    // +optional    Via string `json:"via"`}// RouteStatus defines the observed state of Routetype RouteStatus struct {    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster    // Important: Run "make" to regenerate code after modifying this file    Status string `json:"status,omitempty"`}//+genclient//+k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object//+kubebuilder:object:root=true//+kubebuilder:resource:shortName=route//+kubebuilder:resource:scope=Cluster//+kubebuilder:subresource:status//+kubebuilder:printcolumn:name="Route Status",type=string,JSONPath=".status.status"// Route is the Schema for the routes APItype Route struct {    metav1.TypeMeta   `json:",inline"`    metav1.ObjectMeta `json:"metadata,omitempty"`    Spec   RouteSpec   `json:"spec,omitempty"`    Status RouteStatus `json:"status,omitempty"`}//+kubebuilder:object:root=true// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object// RouteList contains a list of Routetype RouteList struct {    metav1.TypeMeta `json:",inline"`    metav1.ListMeta `json:"metadata,omitempty"`    Items           []Route `json:"items"`}func init() {    SchemeBuilder.Register(&Route{}, &RouteList{})}

Controller 定义

/*Copyright 2023.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at    http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.*/package controllersimport (    "context"    "k8s.io/apimachinery/pkg/runtime"    ctrl "sigs.k8s.io/controller-runtime"    "sigs.k8s.io/controller-runtime/pkg/client"    "sigs.k8s.io/controller-runtime/pkg/log")// RouteReconciler reconciles a Route objecttype RouteReconciler struct {    client.Client    Scheme *runtime.Scheme}//+kubebuilder:rbac:groups=k8s.ovn.org,resources=routes,verbs=get;list;watch;create;update;patch;delete//+kubebuilder:rbac:groups=k8s.ovn.org,resources=routes/status,verbs=get;update;patch//+kubebuilder:rbac:groups=k8s.ovn.org,resources=routes/finalizers,verbs=update// Reconcile is part of the main kubernetes reconciliation loop which aims to// move the current state of the cluster closer to the desired state.// TODO(user): Modify the Reconcile function to compare the state specified by// the Route object against the actual cluster state, and then// perform operations to make the cluster state reflect the state specified by// the user.//// For more details, check Reconcile and its Result here:// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.11.2/pkg/reconcilefunc (r *RouteReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {    _ = log.FromContext(ctx)    // TODO(user): your logic here    return ctrl.Result{}, nil}// SetupWithManager sets up the controller with the Manager.func (r *RouteReconciler) SetupWithManager(mgr ctrl.Manager) error {    return ctrl.NewControllerManagedBy(mgr).        // Uncomment the following line adding a pointer to an instance of the controlled resource as an argument        // For().        Complete(r)}

技术问题

POD的网络NS怎么获取到？

方案1: 在pod启动时候cni创建网络时，把pod对应的ns 写入 pod的annotations eg: POD_NS: /var/run/netns/cni-219052a8-dc07-14ad-2249-d380a989e501

方案2: 读取/var/lib/cni/results 目录下的*-lo 结尾的文件信息, 读取pod uid 和 ns 地址

解释：-lo 结尾文件是kubelet 使用 loopback cni组件给pod 添加lo网卡(回环网卡)，每个非hostnetwork的pod都会添加的网卡

root@yusur-62:/var/lib/cni/results# cat /var/lib/cni/results/cni-loopback-501bfc5a143cef8f0ed14c68112bbb321eae430e2ff3ec40357e226608c1a5cd-lo|jq{  "kind": "cniCacheV1",  "containerId": "501bfc5a143cef8f0ed14c68112bbb321eae430e2ff3ec40357e226608c1a5cd",  "config": "ewoiY25pVmVyc2lvbiI6ICIwLjMuMSIsCiJuYW1lIjogImNuaS1sb29wYmFjayIsCiJwbHVnaW5zIjogW3sKICAidHlwZSI6ICJsb29wYmFjayIKfV0KfQ==",  "ifName": "lo",  "networkName": "cni-loopback",  "cniArgs": [    [      "K8S_POD_NAMESPACE",      "kube-system"    ],    [      "K8S_POD_NAME",      "calico-kube-controllers-7f7bf765bb-whl8g"    ],    [      "K8S_POD_INFRA_CONTAINER_ID",      "501bfc5a143cef8f0ed14c68112bbb321eae430e2ff3ec40357e226608c1a5cd"    ],    [      "K8S_POD_UID", #pod 的uid      "174eecc3-58fa-42d1-8501-7f9004a28b76"    ],    [      "IgnoreUnknown",      "1"    ]  ],  "capabilityArgs": {    "dns": {      "Servers": [        "10.86.0.10"      ],      "Searches": [        "kube-system.svc.cluster.local",        "svc.cluster.local",        "cluster.local"      ],      "Options": [        "ndots:5"      ]    },    "io.kubernetes.cri.pod-annotations": {      "k8s.ovn.org/pod-networks": "{\"default\":{\"ip_addresses\":[\"10.124.0.3/24\"],\"mac_address\":\"0a:58:0a:7c:00:03\",\"gateway_ips\":[\"10.124.0.1\"],\"ip_address\":\"10.124.0.3/24\",\"gateway_ip\":\"10.124.0.1\"}}",      "kubernetes.io/config.seen": "2023-03-02T16:41:18.088181536+08:00",      "kubernetes.io/config.source": "api"    }  },  "result": {    "cniVersion": "0.3.1",    "dns": {},    "interfaces": [      {        "mac": "00:00:00:00:00:00",        "name": "lo",        "sandbox": "/var/run/netns/cni-219052a8-dc07-14ad-2249-d380a989e501"         # pod的网络ns地址      }    ],    "ips": [      {        "address": "127.0.0.1/8",        "interface": 0,        "version": ""      },      {        "address": "::1/128",        "interface": 0,        "version": ""      }    ]  }}

有效字段：

[      "K8S_POD_UID", #pod 的uid      "174eecc3-58fa-42d1-8501-7f9004a28b76"    ],"interfaces": [      {        "mac": "00:00:00:00:00:00",        "name": "lo",        "sandbox": "/var/run/netns/cni-219052a8-dc07-14ad-2249-d380a989e501"         # pod的网络ns地址      }    ],

代码地址

http://192.168.2.114/PRD/DSPG/CAPT/ovnkube-route.git

服务部署

# 部署方法kubectl apply -f deploy/# 查看效果root@yusur-62:~# kubectl  get pod -n ovn-kubernetes | grep route ovnkube-route-77887c678b-9gs6f    1/1     Running   0             16h# 需要修改crd的manager.yaml 为ds类型

系统架构

整个OVNKUBE-ROUTE逻辑层分为两个层面，功能层分为五大模块。

用户层是用户可以根据自身业务需求自定义低延迟业务走DPU网络，非低延迟业务可以走普通网络，提供一种基于用户自身业务层的网络性能优化能力。

功能层api模块首先定义crd资源来实现 路由配置规则定义和管理，controller模块来监听处理k8s集群中crd定义的路由规则资源cr的状态变化如添加，删除等，netlink模块则同步cr资源中路由规则到pod中生成对应的路由规则，而filter模块则是来过滤用户指定k8s哪种资源对象的pod能同步对应路由规则，达到细致化处理不同业务pod的网络使用规划。

API模块：利用K8S的crd功能实现扩展定义k8s的资源类型。

Controller 模块：是监听crd定义资源cr对象的变化控制模块，模块包含如下几个功能。

Netlink 模块：根据cr对象中路由规则和svc规则的变更，对指定POD进行路由规则变更，支持修改默认路由，删除路由，添加路由，类似linux系统命令，ip route 命令操作路由规则。

删除路由类似命令，ip route del 172.16.10.0/24

添加路由类似命令，ip route add 172.16.20.0/24 via 172.16.20.1

Filter 模块：根据cr对象中的过滤规则，进行对指定对象包括pods，deployments，daemonsets，namespaces，jobs，cronjobs等资源进行筛选过滤。

拓展工具

进入pod对应的网络ns中

方法1， 利用ip命令

#!/usr/bin/env bash# 进入pod：a , namespace：default 的网络空间，ns a default  退出 exitmountEnter(){        POD=$1        NAMESPACE=$2        PODID=$(crictl pods --namespace $NAMESPACE --name $POD --no-trunc |awk '{print $1}' |grep -v POD)        CID=$(crictl ps --no-trunc| grep $PODID | awk 'NR == 1 {print $1}')        PID=$(crictl inspect $CID | grep pid |awk 'NR == 1 {print $2}'|sed 's/,//g')        NSMOUNT=$(cat /var/lib/cni/results/cni-loopback-$PODID-lo | jq |grep sandbox |awk '{print $2}'|sed 's/"//g')        NS=$(echo $NSMOUNT | awk -F/ '{print $5}')        ip netns exec $NS bash}POD=$1NAMESPACE=$2mountEnter $1 $2

方法2， 利用nsenter命令

#!/usr/bin/env bash# 进入pod：a , namespace：default 的网络空间，ns a default  退出 exitpidEnter() {        POD=$1        NAMESPACE=$2        PODID=$(crictl pods --namespace $NAMESPACE --name $POD --no-trunc |awk '{print $1}' |grep -v POD)        CID=$(crictl ps --no-trunc| grep $PODID | awk 'NR == 1 {print $1}')        PID=$(crictl inspect $CID | grep pid |awk 'NR == 1 {print $2}'|sed 's/,//g')        NSMOUNT=$(cat /var/lib/cni/results/cni-loopback-$PODID-lo | jq |grep sandbox |awk '{print $2}'|sed 's/"//g')        NS=$(echo $NSMOUNT | awk -F/ '{print $5}')        nsenter -n -t  $PID bash}POD=$1NAMESPACE=$2pidEnter $1 $2

进入ns查看效果

root@yusur-62:/tmp# ./execns.sh coredns-74586cf9b6-tdbmb kube-systemroot@yusur-62:/tmp# route -nKernel IP routing tableDestination     Gateway         Genmask         Flags Metric Ref    Use Iface0.0.0.0         10.124.1.1      0.0.0.0         UG    0      0        0 eth010.124.0.0      10.124.1.1      255.255.0.0     UG    0      0        0 eth010.124.1.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0100.64.0.0      10.124.1.1      255.255.0.0     UG    0      0        0 eth0

dl 即是 data link 的缩写。

dl_type=ethertype

匹配以太网协议类型以太类型，以10到65535之间的整数（包括0和65535）指定，以十进制或以0x前缀的十六进制数表示，示例如下。

• dl_type=0x0800 匹配IPv4数据包，等同于dl_type=ip 。

• dl_type=0x086dd 匹配IPv6数据包，等同于dl_type=ipv6 。

• dl_type=0x0806 匹配ARP数据包，等同于dl_type=arp 。

• dl_type=0x8035 匹配RARP数据包，等同于 dl_type=rarp。

dl_vlan=vlan

数据包的 VLAN Tag 值，范围是 0-4095，0xffff 代表不包含 VLAN Tag 的数据包

dl_vlan_pcp=priority

VLAN 优先级，取值区间为[0-7]。数字越大，表示优先级越高。

dl_src=xx:xx:xx:xx:xx:xx

dl_dst=xx:xx:xx:xx:xx:xx

源或目的的 MAC地址

• 地址01:00:00:00:00:00/01:00:00:00:00:00 代表广播

• 地址00:00:00:00:00:00/01:00:00:00:00:00 代表单播

• 地址fe:ff:ff:ff:ff:ff 匹配除多播位以外的所有位，基本上不会用到。

• 地址ff:ff:ff:ff:ff:ff 完全匹配（等同于省略子网掩码）。

• 地址00:00:00:00:00:00 匹配全部位（等同于 dl_dst=*）。

zone: 表示ct的区域

config.Default.ConntrackZone //64000HostMasqCTZone     = config.Default.ConntrackZone + 1 //64001 host区域OVNMasqCTZone      = HostMasqCTZone + 1               //64002 ovn区域HostNodePortCTZone = config.Default.ConntrackZone + 3 //64003 nodeport区域

ovn-k8s 的 brens3f2(brp0) 网桥中的流表分析

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341

#查看信息

root@work02:~# ovs-vsctl  show 

89531dd2-795e-4072-bd31-803e740041e7

    Bridge br-int

        fail\_mode: secure

        datapath\_type: system

        Port patch-br-int-to-brens3f2\_work02

            Interface patch-br-int-to-brens3f2\_work02

                type: patch

                options: {peer=patch-brens3f2\_work02-to-br-int}

        Port ovn-bfec55-0

            Interface ovn-bfec55-0

                type: vxlan

                options: {csum="true", key=flow, remote\_ip="192.168.122.10"}

        Port ovn-eca30b-0

            Interface ovn-eca30b-0

                type: vxlan

                options: {csum="true", key=flow, remote\_ip="192.168.122.101"}

        Port "7ac47439675847a"

            Interface "7ac47439675847a"

        Port br-int

            Interface br-int

                type: internal

        Port ovn-k8s-mp0

            Interface ovn-k8s-mp0

                type: internal

    Bridge brens3f2

        fail\_mode: standalone

        Port ens3f2

            Interface ens3f2

        Port patch-brens3f2\_work02-to-br-int

            Interface patch-brens3f2\_work02-to-br-int

                type: patch

                options: {peer=patch-br-int-to-brens3f2\_work02}

        Port brens3f2

            Interface brens3f2

                type: internal

    ovs\_version: "2.17.3"

#查看flows流表

root@work02:~# ovs-ofctl  dump-flows brens3f2

 cookie=0xdeff105, duration=447743.773s, table=0, n\_packets=0, n\_bytes=0, priority=500,ip,in\_port="patch-brens3f2\_",nw\_src=192.168.200.102,nw\_dst=169.254.169.2 actions=ct(commit,table=4,zone=64001,nat(dst=192.168.200.102))

#table=0 0表第一个表格

#priority 级别500，数字越大级别越大

#ip 匹配协议

#in\_port 流入口port patch-brens3f2\_

#nw\_src 匹配源IPv4 地址，可以使 IP 地址或者域名 192.168.200.102

#nw\_dst 匹配目标IPv4 地址，可以使 IP 地址或者域名 169.254.169.2

#ct开始commit跟踪一个链接，复制数据跳转到表4，设置64001区

#actions "ct"动作将数据报文发送给链接跟踪模块。

支持的参数如下：

commit: 请求链接跟踪模块开始跟踪一个链接。

table=number: 将数据报流水线复制为两份。原始的数据报作为未被跟踪的数据报在当前的动作流水线中继续处理。被复制的数据报被发送到链接跟踪模块，这些数据报会重新回到Openflow流水线中触发后续的处理，但是这些数据报中的ct\_state和其他的ct字段被置位。

zone=value OR zone=src\[start..end\]: 16位的上下文id，其被用于将链接隔离到不同的域，将叠加的网络映射到不同的区域中。zone的默认值为0。设置ct\_zone，在不同的域中隔离链接，从而容许IP重复

nat: 为被跟踪的链接指明地址和端口nat。(作dnat 169.254.169.2 -> 192.168.200.102）

 cookie=0xdeff105, duration=447743.773s, table=0, n\_packets=0, n\_bytes=0, priority=500,ip,in\_port="patch-brens3f2\_",nw\_src=192.168.200.102,nw\_dst=192.168.100.102 actions=ct(commit,table=4,zone=64001)

#ip 匹配协议

#in\_port 流入口port patch-brens3f2\_

#nw\_src 匹配源IPv4 地址，可以使 IP 地址或者域名 192.168.200.102

#nw\_dst 匹配目标IPv4 地址，可以使 IP 地址或者域名 192.168.100.102

#ct开始commit跟踪一个链接，复制数据跳转到表4，设置64001区

 cookie=0xdeff105, duration=447743.773s, table=0, n\_packets=0, n\_bytes=0, priority=500,ip,in\_port="patch-brens3f2\_",nw\_src=192.168.200.102,nw\_dst=192.168.122.102 actions=ct(commit,table=4,zone=64001)

#ip 匹配协议

#in\_port 流入口port patch-brens3f2\_

#nw\_src 匹配源IPv4 地址，可以使 IP 地址或者域名 192.168.200.102

#nw\_dst 匹配目标IPv4 地址，可以使 IP 地址或者域名 192.168.122.102

#ct开始commit跟踪一个链接，复制数据跳转到表4，设置64001区

 cookie=0xdeff105, duration=447743.773s, table=0, n\_packets=0, n\_bytes=0, priority=500,ip,in\_port=LOCAL,nw\_dst=169.254.169.1 actions=ct(table=5,zone=64002,nat)

#ip 匹配协议

#in\_port 流入口port 本地网桥接口brens3f2

#nw\_dst 匹配目标IPv4 地址，可以使 IP 地址或者域名 169.254.169.1 

#ct开始commit跟踪一个链接，复制数据跳转到表5，设置64002区，做nat

 cookie=0xdeff105, duration=447743.773s, table=0, n\_packets=5, n\_bytes=490, priority=500,ip,in\_port=LOCAL,nw\_dst=10.86.0.0/16 actions=ct(commit,table=2,zone=64001,nat(src=169.254.169.2))

#ip 匹配协议

#in\_port 流入口port 本地网桥接口brens3f2

#nw\_dst 匹配目标IPv4 地址，可以使 IP 地址或者域名 10.86.0.0/16  

#ct开始commit跟踪一个链接，复制数据跳转到表2，设置64001区，做nat设置snat为169.254.169.2

 cookie=0xdeff105, duration=447743.773s, table=0, n\_packets=0, n\_bytes=0, priority=500,ip,in\_port="patch-brens3f2\_",nw\_src=10.86.0.0/16,nw\_dst=169.254.169.2 actions=ct(table=3,zone=64001,nat)

#ip 匹配协议

#in\_port 流入口port patch-brens3f2\_

#nw\_src 匹配源IPv4 地址，可以使 IP 地址或者域名 10.86.0.0/16 

#nw\_dst 匹配目标IPv4 地址，可以使 IP 地址或者域名 169.254.169.2

#ct开始commit跟踪一个链接，复制数据跳转到表3，设置64001区，做nat

 cookie=0xdeff105, duration=447743.773s, table=0, n\_packets=0, n\_bytes=0, priority=205,udp,in\_port=ens3f2,dl\_dst=52:54:00:a3:9e:42,tp\_dst=6081 actions=LOCAL

#udp 匹配协议

#in\_port 流入口port ens3f2

#dl\_dst 匹配目标mac，52:54:00:a3:9e:42 （ens3f2，brens3f2 网卡mac地址）

#tp\_dst 匹配目标port，6081 （使用udp 封装数据，端口6081）

#actions 动作为local（一般是转发给本地网桥）

 cookie=0xdeff105, duration=447743.773s, table=0, n\_packets=0, n\_bytes=0, priority=200,udp,in\_port=ens3f2,tp\_dst=6081 actions=NORMAL

#udp 匹配协议

#in\_port 流入口port ens3f2

#tp\_dst 匹配目标port，6081 （使用udp 封装数据，端口6081）

#actions 动作为normal， 动作为normal（转为L2/L3处理流程）

 cookie=0xdeff105, duration=447743.773s, table=0, n\_packets=0, n\_bytes=0, priority=200,udp,in\_port=LOCAL,tp\_dst=6081 actions=output:ens3f2

#udp 匹配协议

#in\_port 流入口port 本地网桥接口brens3f2

#tp\_dst 匹配目标port，6081 （使用udp 封装数据，端口6081）

#actions 输出端口ens3f2， output:port: 输出数据包到指定的端口。port 是指端口的 OpenFlow 端口编号

 cookie=0xdeff105, duration=447743.773s, table=0, n\_packets=4, n\_bytes=392, priority=100,ip,in\_port="patch-brens3f2\_" actions=ct(commit,zone=64000,exec(load:0x1->NXM\_NX\_CT\_MARK\[\])),output:ens3f2

#ip 匹配协议

#in\_port 流入口port patch-brens3f2\_

#nw\_src 匹配源IPv4 地址，可以使 IP 地址或者域名 10.86.0.0/16 

#nw\_dst 匹配目标IPv4 地址，可以使 IP 地址或者域名 169.254.169.2

#ct开始commit跟踪一个链接，设置64000区，输出端口ens3f2网口

 cookie=0xdeff105, duration=447743.773s, table=0, n\_packets=0, n\_bytes=0, priority=100,ip,in\_port=LOCAL actions=ct(commit,zone=64000,exec(load:0x2->NXM\_NX\_CT\_MARK\[\])),output:ens3f2

#ip 匹配协议

#in\_port 流入口port 本地网桥接口brens3f2

#actions ct开始commit跟踪一个链接，设置64000区，输出端口ens3f2网口

#exec 设置ct执行参数load:0x2->NXM\_NX\_CT\_MARK\[\] 设置ct\_mark为0x2

 cookie=0xdeff105, duration=447743.773s, table=0, n\_packets=0, n\_bytes=0, priority=50,ip,in\_port=ens3f2 actions=ct(table=1,zone=64000)

#ip 匹配协议

#in\_port 流入口port ens3f2

#ct开始commit跟踪一个链接，复制数据跳转到表1，设置64000区

 cookie=0xdeff105, duration=447743.773s, table=0, n\_packets=3, n\_bytes=126, priority=10,in\_port=ens3f2,dl\_dst=52:54:00:a3:9e:42 actions=output:"patch-brens3f2\_",LOCAL

#in\_port 流入口port ens3f2

#dl\_dst 匹配目标mac，52:54:00:a3:9e:42 （ens3f2，brens3f2 网卡mac地址）

#ct开始commit跟踪一个链接，复制数据跳转到表1，设置64000区

#actions local 转发给本地网桥， 输出端口为 patch-brens3f2\_

 cookie=0x0, duration=447744.377s, table=0, n\_packets=224667, n\_bytes=11696694, priority=0 actions=NORMAL

#actions=NORMAL 使数据包经过设备的常规L2 / L3处理，正常处理

 cookie=0xdeff105, duration=447743.773s, table=1, n\_packets=0, n\_bytes=0, priority=100,ct\_state=+est+trk,ct\_mark=0x1,ip actions=output:"patch-brens3f2\_"

#table 表1 规则

#ip 匹配协议

#ct\_state 匹配链路状态 ct\_state=+est+trk 匹配已经建立连接,数据包过Conntrack

#ct\_mark 设置0x1 32位的元数据被提交给数据报所对应的链接。其值为ct动作exec的参数。

#actions 输出端口为 patch-brens3f2\_

 cookie=0xdeff105, duration=447743.773s, table=1, n\_packets=0, n\_bytes=0, priority=100,ct\_state=+est+trk,ct\_mark=0x2,ip actions=LOCAL

#table 表1 规则

#ip 匹配协议

#ct\_state 匹配链路状态 ct\_state=+est+trk 匹配已经建立连接,数据包过Conntrack

#ct\_mark 设置0x2 32位的元数据被提交给数据报所对应的链接。其值为ct动作exec的参数。

#actions local转发到本地网桥

 cookie=0xdeff105, duration=447743.773s, table=1, n\_packets=0, n\_bytes=0, priority=100,ct\_state=+rel+trk,ct\_mark=0x1,ip actions=output:"patch-brens3f2\_"

#table 表1 规则

#ip 匹配协议

#ct\_state 匹配链路状态 ct\_state=+rel+trk 匹配已经存在连接相关联,数据包过Conntrack

#ct\_mark 设置0x1 32位的元数据被提交给数据报所对应的链接。其值为ct动作exec的参数。

#actions 输出端口为 patch-brens3f2\_

 cookie=0xdeff105, duration=447743.773s, table=1, n\_packets=0, n\_bytes=0, priority=100,ct\_state=+rel+trk,ct\_mark=0x2,ip actions=LOCAL

#table 表1 规则

#ip 匹配协议

#ct\_state 匹配链路状态 ct\_state=+rel+trk 匹配已经存在连接相关联,数据包过Conntrack

#ct\_mark 设置0x2 32位的元数据被提交给数据报所对应的链接。其值为ct动作exec的参数。

#actions local转发到本地网桥

 cookie=0xdeff105, duration=447743.773s, table=1, n\_packets=0, n\_bytes=0, priority=13,udp,in\_port=ens3f2,tp\_dst=3784 actions=output:"patch-brens3f2\_",LOCAL

#table 表1 规则

#udp 匹配协议

#in\_port 流入口port 本地网桥接口ens3f2

#tp\_dst 匹配目标port，3784 （BFD，端口3784）

#actions 输出端口为 patch-brens3f2\_，local转发到本地网桥

 cookie=0xdeff105, duration=447743.773s, table=1, n\_packets=0, n\_bytes=0, priority=10,dl\_dst=52:54:00:a3:9e:42 actions=LOCAL

#table 表1 规则

#dl\_dst 匹配目标mac：52:54:00:a3:9e:42 （ens3f2，brens3f2 网卡mac地址）

#actions local转发到本地网桥

 cookie=0xdeff105, duration=447743.773s, table=1, n\_packets=0, n\_bytes=0, priority=0 actions=NORMAL

#table 表1 规则

#actions=NORMAL 使数据包经过设备的常规L2 / L3处理，正常处理

 cookie=0xdeff105, duration=11.200s, table=2, n\_packets=5, n\_bytes=490, actions=mod\_dl\_dst:52:54:00:a3:9e:42,output:"patch-brens3f2\_"

#table 表2 规则

#actions 输出端口为 patch-brens3f2\_，mod\_dl\_dst修改目标mac地址为52:54:00:a3:9e:42

 cookie=0xdeff105, duration=11.200s, table=3, n\_packets=0, n\_bytes=0, actions=move:NXM\_OF\_ETH\_DST\[\]->NXM\_OF\_ETH\_SRC\[\],mod\_dl\_dst:52:54:00:a3:9e:42,LOCAL

#table 表3 规则

#actions 将已命名的位从字段dst复制到字段src mod\_dl\_dst修改目标mac地址为52:54:00:a3:9e:42，local转发到本地网桥

 cookie=0xdeff105, duration=447743.772s, table=4, n\_packets=0, n\_bytes=0, ip actions=ct(commit,table=3,zone=64002,nat(src=169.254.169.1))

#table 表4规则

#ip 匹配协议

#actions ct开始commit跟踪一个链接，设置64002区，做nat设置snat为169.254.169.1

 cookie=0xdeff105, duration=447743.772s, table=5, n\_packets=0, n\_bytes=0, ip actions=ct(commit,table=2,zone=64001,nat)

#table 表5规则

#ip 匹配协议

#actions ct开始commit跟踪一个链接，复制数据跳转到表2，设置64001区，做nat