Update: 23/11/2018 - we now have a direct connection to our DDoS mitigation provider, so this article only applies in scenarios where the DDoS mitigation is active AND our direct connection to our provider is unavailable (e.g. due to maintenance or fault such as a fibre break), should this be the case we will publish information on our status website - https://cwcsstatus.co.uk 

Fault

Connections or file transfers to or from servers may fail or time out (hang) when DDoS mitigation is active.

Cause

Packets larger than 1476 bytes are too large to be transmitted through the tunnel used for the DDoS mitigation and Path MTU Discovery is disabled due to incorrect implementation of firewall filters for ICMP causing a "black hole connection".

Fix

Firewall administrators should allow ICMP IPv4 Fragmentation Required (type 3, code 4) and if using IPv6, allow ICMP Packet Too Big (type 2, code 0), this will re-enabled Path MTU Discovery which will allow the MTU to be automatically adjusted.

If it is not possible to adjust the firewall, you can reduce the MTU on your workstation and server.

Change MTU in Windows

  1. Click the Windows button on the task bar.
  2. Click All Programs.
  3. Click Accessories.
  4. Right-click on Command Prompt and click Run as administrator.
  5. If prompted click the Allow button.
  6. Once the Command Prompt window is open follow the steps below to change the MTU size:
  7. Type netsh interface ipv4 show subinterface
  8. Press Enter. You will see a list of network interfaces.
  9. Type netsh interface ipv4 set subinterface “Local Area Connection” mtu=1476 store=persistent
  10. You should replace Local Area Connection with the name that appeared in the “Interface” column from steps 1-3.
  11. Press Enter.
  12. Restart you computer and then test again.
  13. If you still have problems after modifying the MTU repeat the above steps – replacing the numbers 1476 with 1400.

Change MTU in Linux

  1. Login to the system via SSH as root user
  2. Identify the name of the interface (this will change depending on OS).
    You can use this command to retrieve the first network interface (this is normally the one that is internet facing):
    ip link | awk -F: '$0 !~ "lo|vir|wl|^[^0-9]"{print $2a;getline}'
  3. Run the following (change “eth0” with the appropriate interface name):
    ip link set eth0 mtu 1476
  4. If the ip link command is not available, you may need use to the deprecated net-tools instead:
    ifconfig eth0 mtu 1476

Note: these changes are not persistent across reboots. In order to make this persistent

Debian

In order to make the above change persistent across reboots, edit /etc/network/interfaces

Add mtu 1476 to the eth0 interface stanza:

auto eth0 iface eth0 inet static address 192.168.1.2 netmask 255.255.255.0 mtu 1476 

CentOS

Edit /etc/sysconfig/network-scripts/ifcfg-eth0

Add MTU=1476:

DEVICE=eth0 BOOTPROTO=static ONBOOT=yes IPADDR=192.168.1.2 NETMASK=255.255.255.0 MTU=1476

MSS Clamping

You can also reduce the TCP maximum segment size. TCP headers have a 20 byte overhead in addition to the GRE header and encapsulated IP header, so the largest MSS you could use is 1466 bytes. You can also try 1400 bytes.

First, show the current routing table:

root@server1 [~]# ip routedefault via 91.192.192.1 dev eth0 proto static metric 100
91.192.192.0/23 dev eth0 proto kernel scope link src 91.192.193.115
metric 100

 
 

Then use “ip route change“, copy and paste the default route from above, and add “advmss 1400” at the end, e.g.:

 

root@server1 [~]# ip route change default via 91.192.192.1 dev eth0 proto static metric 100 advmss 1400

 

The routing table should now be updated:

 

root@server1 [~]# ip route
default via 91.192.192.1 dev eth0 proto static metric 100 advmss 1400
91.192.192.0/23 dev eth0 proto kernel scope link src 91.192.193.115 metric 100

 

If reducing the MTU or MSS do not resolve the problem, contact technical support.

 

Technical Details

 

CWCS have DDoS protection of our network with a BGP DDoS mitigation solution. This uses virtual point-to-point links with Generic Routing Encapsulation (GRE) to exchange routing traffic between CWCS and the DDoS scrubbing centre. GRE is an industry standard protocol developed by Cisco to enable network operators to “tunnel” traffic over the internet and works by encapsulating network layer (internet protocol traffic) inside GRE packets.

 

CWCS has established BGP peering sessions across the GRE tunnels and we advertise our routes to them. When an attack is detected and CWCS have authorised mitigation, the DDoS scrubbing centre advertise upstream the /24 prefix (IP range) that includes the targeted IP address, which redirects the traffic to a scrubbing centre.

 

Cleaned traffic is forward back to CWCS routers via a GRE tunnel and then onwards to the original destination. Packets with a standard 1500 byte MTU are too large to be transmitted through the link. In IPv4 routers will attempt to fragment packets that are too large, however this can be undesirable as packet loss can result in excessive re-transmissions, so most TCP packets will be sent with the DF (Don’t Fragment) bit set. As the router is unable to fragment the packet, it sends an ICMP “Fragmentation Required” notification to the source IP address which advertises the maximum MTU of the link. The original packet is discarded.

 

Unfortunately, firewalls are often incorrectly configured to block the ICMP “Fragmentation Required” notifications needed for Path MTU Discovery to function. TCP connections can complete the three way hand shake, but then hang when data is transferred.