VXLAN第二层与不感知vlan的Linux桥接
VXLAN是一种覆盖网络，它通过现有的IP网络传输以太网流量，同时可以支持非常大数量的租户。它在RFC 7348中定义。每个覆盖网络被称为一个VXLAN段，并通过一个独特的24位段ID称为VXLAN网络标识符（VNI）进行标识。
VXLAN封装增加了50字节的开销，因此您需要将主机物理接口的MTU至少增加到1550。（或者在您的虚拟机内部将MTU减少到1450）
对于BUM流量（广播/未知单播流量，多播），我们有3种不同的VXLAN设置模式：多播，单播，BGP-EVPN。
组播模式
这个场景依赖于头端复制，这意味着，如果终端主机没有目的MAC地址的任何条目，它将向VXLAN网络中的其他设备/VTEPs发送ARP请求。这是通过将请求发送到VXLAN多播组来完成的，远程VTEPs将会收到这个包并相应地直接回应给发起请求的VTEP。
-
节点1
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.1
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-svcnodeip 225.20.1.1
vxlan-physdev eno1
auto vmbr2
iface vmbr2 inet manual
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-svcnodeip 225.20.1.1
vxlan-physdev eno1
auto vmbr3
iface vmbr3 inet manual
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
-
节点2
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.2
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-svcnodeip 225.20.1.1
vxlan-physdev eno1
auto vmbr2
iface vmbr2 inet manual
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-svcnodeip 225.20.1.1
vxlan-physdev eno1
auto vmbr3
iface vmbr3 inet manual
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
-
节点3
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.3
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-svcnodeip 225.20.1.1
vxlan-physdev eno1
auto vmbr2
iface vmbr2 inet manual
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-svcnodeip 225.20.1.1
vxlan-physdev eno1
auto vmbr3
iface vmbr3 inet manual
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
单播模式
我们可以通过将BUM帧的头部复制到静态配置的远程VTEPs列表来替换多播。VXLAN是在没有远程多播组的情况下定义的。相反，所有远程VTEPs都与全零地址关联：BUM帧将被复制到所有这些目的地。VXLAN设备仍将通过使用源地址学习自动学习远程地址。
-
节点1
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.1
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan_remoteip 192.168.0.2
vxlan_remoteip 192.168.0.3
auto vmbr2
iface vmbr2 inet manual
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
auto vxlan3
iface vxlan2 inet manual
vxlan-id 3
vxlan_remoteip 192.168.0.2
vxlan_remoteip 192.168.0.3
auto vmbr3
iface vmbr3 inet manual
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
-
节点2
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.2
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan_remoteip 192.168.0.1
vxlan_remoteip 192.168.0.3
auto vmbr2
iface vmbr2 inet manual
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
auto vxlan3
iface vxlan2 inet manual
vxlan-id 3
vxlan_remoteip 192.168.0.1
vxlan_remoteip 192.168.0.3
auto vmbr3
iface vmbr3 inet manual
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
-
节点3
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.3
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan_remoteip 192.168.0.2
vxlan_remoteip 192.168.0.3
auto vmbr2
iface vmbr2 inet manual
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
auto vxlan3
iface vxlan2 inet manual
vxlan-id 3
vxlan_remoteip 192.168.0.2
vxlan_remoteip 192.168.0.3
auto vmbr3
iface vmbr3 inet manual
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
BGP-EVPN
VTEPs通过BGP用于远程MAC地址的控制平面学习/分布，而不是数据平面学习。VTEPs有能力在VXLAN隧道上抑制ARP洪泛。
此处使用的控制平面是FRR，一种BGP路由软件。在Proxmox集群中的每个节点都与其他节点进行对等连接。对于更大的网络，或多个Proxmox集群，可以使用外部BGP路由反射服务器。
-
节点1
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.1
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet manual
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet manual
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
/etc/frr/frr.conf
router bgp 1234 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.2 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 ! address-family l2vpn evpn neighbor 192.168.0.2 activate neighbor 192.168.0.3 activate advertise-all-vni exit-address-family ! line vty !
-
节点2
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.2
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet manual
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet manual
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
/etc/frr/frr.conf
router bgp 1234 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.1 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 ! address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.3 activate advertise-all-vni exit-address-family ! line vty !
-
节点3
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.2
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.3
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet manual
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.3
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet manual
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
/etc/frr/frr.conf
router bgp 1234 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.1 remote-as 1234 neighbor 192.168.0.2 remote-as 1234 ! address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.2 activate advertise-all-vni exit-address-family ! line vty !
VXLAN三层路由与任播网关
根据这个需求，每个vmbr桥将成为虚拟机的网关。不同节点上的相同vmbr将拥有相同的IP地址和相同的MAC地址，以实现虚拟机的实时迁移和无网络中断。
VXLAN层三路由只能与FRR和非识别桥梁一起工作。（目前VLAN识别桥梁的支持存在缺陷）。
非对称模型
这是最简单的模式。要使其工作，所有的vxlan都需要在所有节点上定义。
不对称模型允许在VXLAN隧道入口处进行路由和桥接，但在出口处只允许桥接。这导致双向VXLAN流量在每个方向上（总是目的地VNI）通过路由基础设施时使用不同的VNI。
-
节点1
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.1
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet static
address 10.0.2.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
ip-forward on
ip6-forward on
arp-accept on
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet static
address 10.0.3.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
ip-forward on
ip6-forward on
arp-accept on
frr.conf
router bgp 1234 bgp router-id 192.168.0.1 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.2 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 ! address-family l2vpn evpn neighbor 192.168.0.2 activate neighbor 192.168.0.3 activate advertise-all-vni exit-address-family ! line vty !
-
节点2
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.2
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet static
address 10.0.2.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
ip-forward on
ip6-forward on
arp-accept on
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet static
address 10.0.3.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
ip-forward on
ip6-forward on
arp-accept on
frr.conf
router bgp 1234 bgp router-id 192.168.0.2 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.1 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 ! address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.3 activate advertise-all-vni exit-address-family ! line vty !
-
节点3
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.3
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.3
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet static
address 10.0.2.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
ip-forward on
ip6-forward on
arp-accept on
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.3
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet static
address 10.0.3.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
ip-forward on
ip6-forward on
arp-accept on
frr.conf
router bgp 1234 bgp router-id 192.168.0.3 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.1 remote-as 1234 neighbor 192.168.0.2 remote-as 1234 ! address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.2 activate advertise-all-vni exit-address-family ! line vty !
对称模型
使用这种模型，你不需要在所有节点上都有vxlan。这种模型也需要用来将流量路由到外部路由器。
对称模型在入口和出口叶节点上都配置了路由和桥接。这导致双向流量能够在同一个VNI上行驶，因此得名对称。然而，一个新的专用中继VNI被用于所有经路由的VXLAN流量，称为L3VNI。所有需要被路由的流量都将被路由到L3VNI上，通过第三层基础设施隧道传输，从L3VNI路由出来到适当的VLAN，最终桥接到目的地。
如果它们想要相互访问，L3VNI需要一个VRF，所以所有的vmbr桥都需要在VRF中。
-
节点1
auto vrf1
iface vrf1
vrf-table auto
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.1
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet static
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
address 10.0.2.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet static
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
address 10.0.3.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
iface vxlan4000 inet manual
vxlan-id 4000
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr4000
iface vmbr4000 inet manual
bridge_ports vxlan4000
bridge_stp off
bridge_fd 0
vrf vrf1
frr.conf
vrf vrf1 vni 4000 exit-vrf ! router bgp 1234 bgp router-id 192.168.0.1 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.2 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 ! address-family l2vpn evpn neighbor 192.168.0.2 activate neighbor 192.168.0.3 activate advertise-all-vni exit-address-family ! line vty !
-
节点2
auto vrf1
iface vrf1
vrf-table auto
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.2
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet static
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
address 10.0.2.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet static
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
address 10.0.3.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
iface vxlan4000 inet manual
vxlan-id 4000
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr4000
iface vmbr4000 inet manual
bridge_ports vxlan4000
bridge_stp off
bridge_fd 0
vrf vrf1
frr.conf
vrf vrf1 vni 4000 exit-vrf ! router bgp 1234 bgp router-id 192.168.0.2 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.1 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 ! address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.3 activate advertise-all-vni exit-address-family ! line vty !
-
节点3
auto vrf1
iface vrf1
vrf-table auto
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.3
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.3
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet static
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
address 10.0.2.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.3
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet static
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
address 10.0.3.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
iface vxlan4000 inet manual
vxlan-id 4000
vxlan-local-tunnelip 192.168.0.3
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr4000
iface vmbr4000 inet manual
bridge_ports vxlan4000
bridge_stp off
bridge_fd 0
vrf vrf1
frr.conf
vrf vrf1 vni 4000 exit-vrf ! router bgp 1234 bgp router-id 192.168.0.3 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.1 remote-as 1234 neighbor 192.168.0.2 remote-as 1234 ! address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.2 activate advertise-all-vni exit-address-family ! line vty !
VXLAN三层路由与任播网关的结合以及通过带有静态默认网关的外部路由器进行外部路由转发
路由到外部需要对称模型。
1个网关节点
在这个例子中，我们将使用只有1个proxmox节点作为出口网关。（节点1）这个节点在vrf1中宣布默认网关（默认来源）并将数据转发到它自己的默认网关（192.168.0.254）（路由器与节点1之间没有BGP）。
节点1
auto vrf1
iface vrf1
vrf-table auto
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.1
netmask 255.255.255.0
gateway 192.168.0.254
bridge_ports eno1
bridge_stp off
bridge_fd 0
ip-forward on
ip6-forward on
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet static
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
address 10.0.2.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet static
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
address 10.0.3.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
iface vxlan4000 inet manual
vxlan-id 4000
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr4000
iface vmbr4000 inet manual
bridge_ports vxlan4000
bridge_stp off
bridge_fd 0
vrf vrf1
frr.conf
vrf vrf1 vni 4000 exit-vrf ! router bgp 1234 bgp router-id 192.168.0.1 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.2 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 ! address-family ipv4 unicast import vrf vrf1 exit-address-family ! address-family ipv6 unicast import vrf vrf1 exit-address-family ! address-family l2vpn evpn neighbor 192.168.0.2 activate neighbor 192.168.0.3 activate advertise-all-vni exit-address-family ! router bgp 1234 vrf vrf1 ! address-family ipv4 unicast redistribute connected exit-address-family ! address-family ipv6 unicast redistribute connected exit-address-family ! address-family l2vpn evpn default-originate ipv4 default-originate ipv6 exit-address-family ! line vty !
-
节点2
auto vrf1
iface vrf1
vrf-table auto
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.2
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet static
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
address 10.0.2.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet static
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
address 10.0.3.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
iface vxlan4000 inet manual
vxlan-id 4000
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr4000
iface vmbr4000 inet manual
bridge_ports vxlan4000
bridge_stp off
bridge_fd 0
vrf vrf1
frr.conf
vrf vrf1 vni 4000 exit-vrf ! router bgp 1234 bgp router-id 192.168.0.2 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.1 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 ! address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.3 activate advertise-all-vni exit-address-family ! line vty !
-
节点3
auto vrf1
iface vrf1
vrf-table auto
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.3
netmask 255.255.255.0
bridge_ports eno1
bridge_stp off
bridge_fd 0
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.3
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet static
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
address 10.0.2.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.3
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet static
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
address 10.0.3.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
iface vxlan4000 inet manual
vxlan-id 4000
vxlan-local-tunnelip 192.168.0.3
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr4000
iface vmbr4000 inet manual
bridge_ports vxlan4000
bridge_stp off
bridge_fd 0
vrf vrf1
frr.conf
vrf vrf1 vni 4000 exit-vrf ! router bgp 1234 bgp router-id 192.168.0.3 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.1 remote-as 1234 neighbor 192.168.0.2 remote-as 1234 ! address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.2 activate advertise-all-vni exit-address-family ! line vty !
多个网关节点
在这个例子中，所有节点都将作为出口网关使用。（但如果你愿意，也只能使用2个节点）所有节点都有一个默认网关到外部路由器（192.168.0.254）（路由器与node1之间没有bgp）并在vrf中宣布这个默认网关（默认起源）外部路由器对所有proxmox节点都有ecmp路由。（平衡）。如果路由器将数据包发送到错误的节点（虚拟机不在这个节点上），该节点将通过vxlan将数据包路由到最终目的地。
如果您有多个网关节点，请禁用rp_filter，因为数据包可能从一个节点进入并从另一个节点发送出去。
sysctl.conf调优
net.ipv4.conf.default.rp_filter=0 net.ipv4.conf.all.rp_filter=0
节点1
auto vrf1
iface vrf1
vrf-table auto
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.1
netmask 255.255.255.0
gateway 192.168.0.254
bridge_ports eno1
bridge_stp off
bridge_fd 0
ip-forward on
ip6-forward on
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet static
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
address 10.0.2.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet static
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
address 10.0.3.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
iface vxlan4000 inet manual
vxlan-id 4000
vxlan-local-tunnelip 192.168.0.1
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr4000
iface vmbr4000 inet manual
bridge_ports vxlan4000
bridge_stp off
bridge_fd 0
vrf vrf1
frr.conf
vrf vrf1 vni 4000 exit-vrf ! router bgp 1234 bgp router-id 192.168.0.1 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.2 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 ! address-family ipv4 unicast import vrf vrf1 exit-address-family ! address-family ipv6 unicast import vrf vrf1 exit-address-family ! address-family l2vpn evpn neighbor 192.168.0.2 activate neighbor 192.168.0.3 activate advertise-all-vni exit-address-family ! router bgp 1234 vrf vrf1 ! address-family ipv4 unicast redistribute connected exit-address-family ! address-family ipv6 unicast redistribute connected exit-address-family ! address-family l2vpn evpn default-originate ipv4 default-originate ipv6 exit-address-family ! line vty !
-
节点2
auto vrf1
iface vrf1
vrf-table auto
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.2
netmask 255.255.255.0
gateway 192.168.0.254
bridge_ports eno1
bridge_stp off
bridge_fd 0
ip-forward on
ip6-forward on
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet static
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
address 10.0.2.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet static
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
address 10.0.3.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
iface vxlan4000 inet manual
vxlan-id 4000
vxlan-local-tunnelip 192.168.0.2
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr4000
iface vmbr4000 inet manual
bridge_ports vxlan4000
bridge_stp off
bridge_fd 0
vrf vrf1
frr.conf
vrf vrf1 vni 4000 exit-vrf ! router bgp 1234 bgp router-id 192.168.0.2 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.1 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 ! address-family ipv4 unicast import vrf vrf1 exit-address-family ! address-family ipv6 unicast import vrf vrf1 exit-address-family ! address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.3 activate advertise-all-vni exit-address-family ! address-family ipv4 unicast redistribute connected exit-address-family ! address-family ipv6 unicast redistribute connected exit-address-family ! address-family l2vpn evpn default-originate ipv4 default-originate ipv6 exit-address-family ! line vty !
-
节点3
auto vrf1
iface vrf1
vrf-table auto
auto eno1
iface eno1 inet manual
mtu 1550
auto vmbr0
iface vmbr0 inet static
address 192.168.0.3
netmask 255.255.255.0
gateway 192.168.0.254
bridge_ports eno1
bridge_stp off
bridge_fd 0
ip-forward on
ip6-forward on
auto vxlan2
iface vxlan2 inet manual
vxlan-id 2
vxlan-local-tunnelip 192.168.0.3
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr2
iface vmbr2 inet static
bridge_ports vxlan2
bridge_stp off
bridge_fd 0
address 10.0.2.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr2
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
auto vxlan3
iface vxlan3 inet manual
vxlan-id 3
vxlan-local-tunnelip 192.168.0.3
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr3
iface vmbr3 inet static
bridge_ports vxlan3
bridge_stp off
bridge_fd 0
address 10.0.3.254
netmask 255.255.255.0
hwaddress 44:39:39:FF:40:94 #must be same on each node vmbr3
vrf vrf1
ip-forward on
ip6-forward on
arp-accept on
#interconnect vxlan-vfr l3vni
auto vxlan4000
iface vxlan4000 inet manual
vxlan-id 4000
vxlan-local-tunnelip 192.168.0.3
bridge-learning off
bridge-arp-nd-suppress on
bridge-unicast-flood off
bridge-multicast-flood off
auto vmbr4000
iface vmbr4000 inet manual
bridge_ports vxlan4000
bridge_stp off
bridge_fd 0
vrf vrf1
frr.conf
vrf vrf1 vni 4000 exit-vrf ! router bgp 1234 bgp router-id 192.168.0.3 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.1 remote-as 1234 neighbor 192.168.0.2 remote-as 1234 ! address-family ipv4 unicast import vrf vrf1 exit-address-family ! address-family ipv6 unicast import vrf vrf1 exit-address-family ! address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.2 activate advertise-all-vni exit-address-family ! router bgp 1234 vrf vrf1 ! address-family ipv4 unicast redistribute connected exit-address-family ! address-family ipv6 unicast redistribute connected exit-address-family ! address-family l2vpn evpn default-originate ipv4 default-originate ipv6 exit-address-family ! line vty !
注意
如果您的外部路由器不支持 'ECMP静态路由' 来访问多个{pve}节点，您可以通过使用虚拟路由器冗余协议（VRRP）在proxmox节点上设置HA浮动vip。
在这个例子中，我们将在node1和node2上设置一个浮动IP 192.168.0.10。Node1是主节点，如果发生故障，会故障转移至node2。
当前设置需要 'vrrpd' 包（apt install vrrpd）。#TODO：使用frr的最新版本应该可以直接实现。
-
节点1
auto vmbr0
iface vmbr0 inet static
address 192.168.0.1
netmask 255.255.255.0
gateway 192.168.0.254
bridge_ports eno1
bridge_stp off
bridge_fd 0
vrrp-id 1
vrrp-priority 1
vrrp-virtual-ip 192.168.0.10
-
节点2
auto vmbr0
iface vmbr0 inet static
address 192.168.0.2
netmask 255.255.255.0
gateway 192.168.0.254
bridge_ports eno1
bridge_stp off
bridge_fd 0
vrrp-id 1
vrrp-priority 2
vrrp-virtual-ip 192.168.0.10
与上游BGP路由器连接的网关节点(们)
设置几乎与使用静态网关相同，但我们将连接到一个上游BGP路由器。
示例中，node1 作为 evpn-bgp 的网关（192.168.0.1），以及一个上游的 bgp 路由器（也运行 frr）192.168.0.254。
-
节点1
frr.conf
vrf vrf1 vni 4000 exit-vrf ! router bgp 1234 bgp router-id 192.168.0.1 no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.2 remote-as 1234 neighbor 192.168.0.3 remote-as 1234 neighbor 192.168.0.254 remote-as external ! address-family ipv4 unicast import vrf vrf1 neighbor 192.168.0.254 activate exit-address-family ! address-family ipv6 unicast import vrf vrf1 neighbor 192.168.0.254 activate exit-address-family ! address-family l2vpn evpn neighbor 192.168.0.1 activate neighbor 192.168.0.2 activate neighbor 192.168.0.254 activate advertise-all-vni exit-address-family ! router bgp 1234 vrf vrf1 ! address-family ipv4 unicast redistribute connected exit-address-family ! address-family ipv6 unicast redistribute connected exit-address-family ! address-family l2vpn evpn default-originate ipv4 default-originate ipv6 exit-address-family ! line vty !
-
BGP路由器
frr.conf
ip prefix-list NO32 seq 10 permit 0.0.0.0/0 ge 8 le 24 ip prefix-list NO32 seq 20 deny any ! router bgp 25253 bgp router-id 192.168.0.254 bgp bestpath as-path multipath-relax neighbor 192.168.0.1 remote-as external neighbor 192.168.0.1 capability extended-nexthop ! address-family ipv4 unicast neighbor 192.168.0.1 default-originate neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn exit-address-family ! address-family ipv6 unicast neighbor 192.168.0.1 default-originate neighbor 192.168.0.1 prefix-list NO32 in #don't import /32 route from evpn exit-address-family ! ! 路由反射器^^^^^^^^^^^^^^^^ 如果你有很多Proxmox节点，或者多个Proxmox集群，你可能会希望避免所有节点互相配对。为此，你可以创建专用的路由反射器（RR）服务器。由于RR是单点故障的风险点，强烈推荐至少有两台服务器充当RR，以实现冗余。 下面是使用'frr'的配置示例，包括`rrserver1 (192.168.0.200)`和`rrserver2 (192.168.0.201)`。 rrserver1
router bgp 1234 bgp router-id 192.168.0.200 bgp cluster-id 1.1.1.1 #cluster-id must be the same on each route reflector bgp log-neighbor-changes no bgp default ipv4-unicast neighbor fabric peer-group neighbor fabric remote-as 1234 neighbor fabric capability extended-nexthop neighbor fabric update-source 192.168.0.200 bgp listen range 192.168.0.0/24 peer-group fabric #allow any proxmoxnode client in the network range ! address-family l2vpn evpn neighbor fabric activate neighbor fabric route-reflector-client neighbor fabric allowas-in exit-address-family ! exit ! ---
rrserver2
router bgp 1234 bgp router-id 192.168.0.201 bgp cluster-id 1.1.1.1 bgp log-neighbor-changes no bgp default ipv4-unicast neighbor fabric peer-group neighbor fabric remote-as 1234 neighbor fabric capability extended-nexthop neighbor fabric update-source 192.168.0.201 bgp listen range 192.168.0.0/24 peer-group fabric ! address-family l2vpn evpn neighbor fabric activate neighbor fabric route-reflector-client neighbor fabric allowas-in exit-address-family ! exit ! --- Proxmox节点(们)
router bgp 1234 bgp router-id 192.168.0.x no bgp default ipv4-unicast coalesce-time 1000 neighbor 192.168.0.200 remote-as 1234 neighbor 192.168.0.201 remote-as 1234 ! address-family l2vpn evpn neighbor 192.168.0.200 activate neighbor 192.168.0.201 activate advertise-all-vni exit-address-family !