Metallb

Existing bugs for md5password on Ubuntu LXD images but debian works

TCP_MD5SIG

• https://github.com/metallb/metallb/issues/1125

2023/04/25 16:21:54.905 BGP: [KTTWK-0CPJ7][EC 100663303] sockopt_tcp_signature: setsockopt(22): Protocol not available
2023/04/25 16:21:54.905 BGP: [S3P4S-FRQ8G][EC 33554494] Unable to set TCP MD5 option on socket for peer 10.x.x.x (sock=22): Invalid argument

• https://bugs.launchpad.net/ubuntu/+source/linux-bluefield/+bug/1956238

Check kernel

grep TCP_MD5 /boot/config-*
CONFIG_TCP_MD5SIG is not set

lxc shell k8s3 -- cat /boot/config-5.4.0-1089-kvm | grep CONFIG_TCP_MD5SIG
# CONFIG_TCP_MD5SIG is not set

• 'n' - means it is not compiled at all;
• 'y' - means it is compiled inside the kernel binary file;
• 'm' - means it is compiled as a kernel module.

Enable? this doesn't work but you could compile your own https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel

vim /boot/config-5.4.0-1089-kvm
CONFIG_TCP_MD5SIG=y

Permissions maybe - CAP_NET_RAW to test - https://github.com/metallb/metallb/issues/762 - https://unofficial-kubernetes.readthedocs.io/en/latest/concepts/policy/container-capabilities/

other

• https://github.com/openwrt/packages/issues/16702
• https://bugs.launchpad.net/raspbian/+bug/1183546
• https://github.com/raspberrypi/linux/issues/302

This works

https://docs.openshift.com/container-platform/4.12/networking/metallb/metallb-configure-bgp-peers.html

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.9/config/manifests/metallb-frr.yaml
# kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/main/config/manifests/metallb-frr.yaml
# kubectl apply -f metallb-frr.yaml
kubectl apply -f addresspool.yaml
kubectl apply -f bgppeer.yaml
kubectl apply -f bgpadvertisement1.yaml

Get started with Microk8s

https://metallb.universe.tf/configuration/

If firewall/router in front you can route or dnat form public ip to metallb private address assignment

sudo ip route add 10.x.x.0/22 via 10.y.y.y (10.y.y.y is k8s instance ip address though usually you would use bgp)

• https://microk8s.io/docs/addon-metallb 10.x.x.0/22 is your ip range
• https://devopslearning.medium.com/metallb-load-balancer-for-bare-metal-kubernetes-43686aa0724f

BGP with frr password doesn't seem to work

frr

frr version 8.1
frr defaults traditional
hostname hm0101
log syslog informational
log file /var/log/frr/frr.log
service integrated-vtysh-config
!
ip router-id 10.x.x.r
!
router bgp 65551
 bgp log-neighbor-changes
 neighbor 10.x.x.x remote-as 65551
 ! neighbor 10.x.x.x password metallb
 neighbor 10.x.x.y remote-as 65551
 ! neighbor 10.x.x.y password metallb
 neighbor 10.x.x.z remote-as 65551
 ! neighbor 10.x.x.z password metallb
exit
!

metallb-system configMap config config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - # a test below is router/fw id
      peer-address: 10.x.x.r
      peer-asn: 65551
      my-asn: 65551
      # password: "metallb"
    address-pools:
    - name: my-ip-space
      protocol: bgp
      avoid-buggy-ips: true
      addresses:
      - 192.168.254.0/24

From Linux Router/GW/FW

ip route | grep bgp
sudo vtysh -c "show bgp sum"
sudo vtysh -c "show bgp neighbor"
sudo vtysh -c "show ip route"
sudo tcpdump -nnpi myint port 179
sudo tail -f /var/log/frr/frr.log

kubectl logs -f -l component=speaker -n metallb-system

Pod deployment with service for entry into cluster via bgp advertised ip

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
spec:
  selector:
    matchLabels:
      run: my-nginx
  replicas: 2
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx
        ports:
        - containerPort: 80
      tolerations:
      - key: "node.kubernetes.io/unreachable"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 10
      - key: "node.kubernetes.io/not-ready"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  labels:
    run: my-nginx
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    run: my-nginx
  type: LoadBalancer

Let's do a little loop and we can shutdown node it is on and watch routes change

kubectl get pods -o wide
shutdown your node
while true; do curl 192.168.254.4; sleep 2; done

Other options

• https://github.com/meta-magic/metallb-baremetal-example
• https://stackoverflow.com/questions/63974879/microk8s-metallb-ingress-nginx-how-to-route-external-traffic

https://v0-2-1--metallb.netlify.app/tutorial/

arp

https://faun.pub/metallb-configuration-in-minikube-to-enable-kubernetes-service-of-type-loadbalancer-9559739787df

https://link.medium.com/xgtCprRJ0lb

https://manintheit.org/en/posts/kubernetes/kubernetes-metallb-loadbalancer-with-bgp-mode/

https://cloud.redhat.com/blog/metallb-in-bgp-mode