Enable LoadBalancer services on cloudscale.ch

Steps to enable LoadBalancer services on an OpenShift 4 cluster on cloudscale.ch.

Starting situation

  • You already have an OpenShift 4 cluster on cloudscale.ch

  • You have admin-level access to the cluster

  • The cluster is using Cilium as the network plugin

  • You want to enable LoadBalancer services

Prerequisites

Prepare IP range

  1. Get a suitable public IP range. You can find all IP ranges owned by VSHN

    whois -i mnt-by MNT-VSHN -T inetnum -r

    Please coordinate with VSHN Corporate IT before using an IP range.

  2. Update RIPE DB to reflect the new usage of the IP range. Add the IP range with a descriptive name.

  3. Create a ticket with cloudscale.ch to make the range available as a Floating Network in the project of the cluster.

Configure Loadbalancers

Add the chosen IP range as a loadbalancer address pool in the APPUiO hieradata. Assuming you want to add the IP range 203.0.113.128/25 to the cluster c-cold-bush-7254:

export CLUSTER_ID="c-cold-bush-7254"
export IP_RANGE="203.0.113.128/25"

git checkout -b feat/lbaas/loadbalancer-ip/${CLUSTER_ID}

mkdir -p lbaas/${CLUSTER_ID}
touch lbaas/${CLUSTER_ID}/lb.yaml
yq eval -i ".\"profile_openshift4_gateway::loadbalancer_ip_address_pools\" += [\"${IP_RANGE}\"]" lbaas/${CLUSTER_ID}/lb.yaml

git add lbaas/
git commit -a -m "Add loadbalancer IP range ${IP_RANGE} to ${CLUSTER_ID}"
git push --set-upstream origin feat/lbaas/loadbalancer-ip/${CLUSTER_ID}

Finally review and merge the MR in the APPUiO hieradata.

Setup BGP speaker

The following steps need to be done manually on both loadbalancer VMs

  1. Install FRR

    apt install frr

  2. Enable bgp by setting bgpd=yes in /etc/frr/daemons

    ...
bgpd=yes
ospfd=no
ospf6d=no
ripd=no
ripngd=no
...

  3. Restart FRR

    systemctl restart frr

  4. Configure FRR

    Enter the FRR shell with

    vtysh

    Configure BGP neighbors

    conf t
router bgp 64512
  neighbor <ip-infra-node-0> remote-as 64512 (1)
  neighbor <ip-infra-node-1> remote-as 64512
  neighbor <ip-infra-node-2> remote-as 64512
  end
write (2)
    1 You can add any of the cluster node IPs. We propose to peer with each of the infrastructure nodes.
    2 You need to write the configuration to persist

Configure Cilium

  1. Add a ConfigMap called bgp-config in the namespace cilium. This ConfigMap contains MetalLB configuration.

    Here’s an example ConfigMap.

    apiVersion: v1
kind: ConfigMap
metadata:
  name: bgp-config
  namespace: cilium
data:
  config.yaml: |
    peers:
      - peer-address: 172.18.200.2 (1)
        peer-asn: 64512
        my-asn: 64512
      - peer-address: 172.18.200.3 (1)
        peer-asn: 64512
        my-asn: 64512
    address-pools:
      - name: default
        protocol: bgp
        addresses:
          - 203.0.113.128/25 (2)
    1 IP addresses of the loadbalancers. If necessary, adjust these to match your cluster’s private network CIDR.
    2 The public IP range you allocated for the cluster

  2. Update the Cilium configuration to enable BGP. Our Cilium setup is configured through the CiliumConfig CRD called cilium in namespace cilium. The CRD contains Helm values. Update the CRD spec with

    apiVersion: cilium.io/v1alpha1
kind: CiliumConfig
metadata:
  ...
  name: cilium
  namespace: cilium
  ...
spec:
  bgp:
    announce:
      loadbalancerIP: true
    enabled: true
  ...

Test LoadBalancer service

  1. Apply a LoadBalancer service and a deployment:

    apiVersion: v1
kind: Service
metadata:
  name: test-lb
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    svc: test-lb
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      svc: test-lb
  template:
    metadata:
      labels:
        svc: test-lb
    spec:
      containers:
      - name: web
        image: vshn/nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /
            port: 8080

  2. Observe that the Operator allocates an external IP for test-lb

    kubectl get svc

NAME        TYPE          CLUSTER-IP  EXTERNAL-IP   PORT(S)       AGE
test-lb     LoadBalancer  172.20.0.5  203.0.113.132 80:30724/TCP  10s

  3. Access the external IP

    curl 203.0.113.132

Check the NetworkPolicy in the target namespace. If the namespace doesn’t allow access from external nodes, everything will appear to work, but you won’t be able access the service from outside the cluster.

Troubleshoot

Check BGP peering

You can check if the BGP peering was successful by connecting to the loadbalancer VMs.

  1. Enter the FRR shell with

    vtysh

  2. Show BGP summary.

    show bgp summary

    This should show you something similar to

    BGP router identifier XXXX, local AS number 64512 vrf-id 0
BGP table version 6
RIB entries 5, using 920 bytes of memory
Peers 3, using 61 KiB of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
172.18.200.137  4      64512   11120   11117        0    0    0 3d20h37m            3
172.18.200.157  4      64512   11120   11117        0    0    0 3d20h37m            3
172.18.200.218  4      64512   11119   11116        0    0    0 3d20h37m            3

Total number of neighbors 3

  3. Show available routes

    show ip route

    This should include routes for the created LoadBalancer service.

If these checks look correct, the BGP setup works as expected. If you still can’t connect to the service, re-check the network policies and check if the Floating Network is assigned correctly.

If the neighbors or routes don’t show up correctly, follow the other troubleshooting steps.

Check BGP announcements

Next, check if Cilium sends out BGP announcements and whether they arrive at the loadbalancer VMs.

  1. Check if Cilium sends out BGP announcements. In one of the Cilium DaemonSet pods run

    tcpdump -n -i any tcp port 179

    If Cilium sends out announcements to the correct IPs, it’s most likely setup correctly. If it doesn’t, there is an issue with Cilium. One thing to consider is that Cilium doesn’t automatically pick up updates of the bgp-config ConfigMap. Make sure to restart the Cilium DaemonSet pods if you change the configuration.

  2. Check if any BGP announcements arrive and are accepted. On one of the loadbalancer VMs run

    tcpdump -n -i any tcp port 179

    There should be packets coming in from the cluster nodes and they should be answered.

    1. If no packets come in, check the connection between the cluster nodes and the loadbalancer VM.

    2. If packets come in but aren’t answered, the issue might be the firewall setup. Check if the BGP port is open with

      iptables -L

    3. If the firewall accepts BGP announcements, check the FRR configuration. In the FRR shell run

      show run

      It should show the current running configuration which should look similar to

      !
frr version 7.2.1
frr defaults traditional
hostname lb-1c
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
!
router bgp 64512
 neighbor 172.18.200.137 remote-as 64512
 neighbor 172.18.200.157 remote-as 64512
 neighbor 172.18.200.218 remote-as 64512
!
line vty
!
end