Squid and WCCP

By reuben, 21 November, 2005

This document is incomplete, I'm slowly getting to it, fixing and updating as I can.

Configuring squid and WCCP is not intuitive, but by following a set of detailed instructions it can be made to work reliably. There are many pitfalls and things to watch out for, so I thought I'd document them here to save people the grief that I went through when I set this up.

Requirements:

  • A recent squid. Preferably get the latest version you can (but not squid-3.0) from your distribution, or obtain it from http://www.squid-cache.org/
  • A Linux distribution, note that your config will be MUCH easier if you run a system or upgrade to a kernel later than 2.6.9 because WCCP support in the GRE module was integrated in about October 2004 into the mainline releases. You should be running the latest stable release anyway, for both performance and security reasons. This document assumes that you have already got your Squid cache up and running, and have tested it by manually configuring your web browser to use the cache. If it does not work, fix that problem first before embarking on your WCCP configuration. Note the Squid-2.6 supports WCCPv2, which will definitely be the best version to run when Squid-2.6 becomes stable.
  • If you cannot run a kernel as recent as above then you are strongly advised to upgrade your proxy box (you aren't running anything else on it that depends on old software, are you?), but as a last resort you may use the ip_wccp cache module found at http://www.squid-cache.org/WCCP-support/Linux/. Instructions for this are more complex, you need to build a sample kernel module (any module is OK) and get the command line options used to build it. Then compile the ip_wccp module and modprobe it as part of your bootup sequence.
  • iptables - if you are running your squid on non- port 80. Again, latest version is good, but the version here is less critical.
  • Cisco router/switch running at least IOS 12.0. Lower end router platforms (800, 1600, 1700) require an IP Plus featureset, but all others have WCCP in the base IP or IP/BROADBAND featureset (1800 and above). See the note below on which versions I have found work and don't work and require workarounds.

Time, patience, possibly a packet sniffer and a non-production network to test on.

Steps:

  • Load your router with a suitable version of IOS. See below for recommendations.
  • Configure WCCPv1 on your router (WCCPv2 if you are running Squid-2.6)
  • Create your GRE config or compile your ip_wccp module. DO NOT LOAD BOTH AT ONCE, you can only use one OR the other.
  • Configure Squid, editing your squid.conf
  • Configure your iptables rules if your squid is not listening on port 80
  • Test thoroughly before going live

 

Suggested versions of IOS:

  • 12.1: Latest GD release is fine.
  • 12.2: Latest GD release is fine.
  • 12.3: Latest LD/GD releases should be fine.
  • 12.3T:  12.3(7)T was the last known version in the 12.3T train which worked flawlessly.  12.3(8)T required an ACL to exclude the router from redirecting traffic back onto itself, 12.3(11)T did not work at all, 12.3(14)T was completely broken, and has various other problems.  DO NOT USE 12.3T for WCCP.  In fact, as 12.3T is now approaching "End of Engineering" and has been superseeded by the bugfix only train of 12.4, I would advise against loading 12.3T on *any* cisco device.
  • 12.4 almost works, but suffers from  mostly the same problem as 12.3T since 12.3T was merged to become 12.4.  12.4(7) seems to be fixed, but that version and 12.4(8) works as long as you are not using IOS firewall and doing ip inspection on the inbound interface where your WCCP cache engine is located.
  • 12.4T also has the same problem.  12.4(6)T onwards is OK but also suffers from the ip inspect issue mentioned above.  I currently have a cisco TAC case open on this, it results from a bug whereby TCP fragments appear to be dropped within the router.
  • 12.4(9)T2 seems to work well, it is the first release in a long time which reliably does WCCP.  If you can, either run this or 12.4(11)T or later.

 

The problem I was seeing in 12.4 and late 12.3T's is that the router sees the cache and attempts to forward, but cannot redirect WCCP traffic it due to a switching bug in the router software. The bug is that the router does not seem to be able to process switch a packet to a WCCP cache over the GRE tunnel, but if you establish the CEF entry first by pinging the remote site's IP address, it works. As any cisco gurus know, for an entry to be added to the CEF table, the first packet to that destination must be process switched. The symptoms you will see are that the cache is visible to the router and the WCCP works for a few seconds, then all redirected traffic starts timing out. I suggest you start with a KNOWN GOOD version of IOS before upgrading to later versions. WCCP is tricky enough without the uncertainty of whether the router software is properly functional.

IOS config:

WCCP can be applied either as the traffic comes into an interface (inbound) or out an interface (outbound). For performance reasons you will probably want to redirect inbound, but if you have tunnels and/or your router is routing across multiple interfaces you may wish to redirect the outbound traffic instead. Be warned that if you are using NAT you MUST use the inbound interface otherwise the router only sees the NATted IP address as the source of your clients. This is bad, because the router is also therefore unable to see your cache engine and it will redirect the cache engine requests back upon itself. You can of course have multiple redirect statements. Note that at this time, the latest stable releases of Squid do not support WCCPv2 so you must specify WCCPv1 (cisco devices default to V2).

If you are running Squid-2.6 you can and should configure WCCPv2 instead as this has performance benefits including better switching of traffic from the router to the cache. At initial release Squid-2.6 does not support more than one Linux cache engine due to some changes that were put into the Linux kernel which complicate communications between the router and cache, but this may be rectified in future releases.

Here is a basic cisco router IOS config:

ip cef
ip wccp version 1
ip wccp web-cache

interface FastEthernet0
   ip wccp web-cached redirect in
end

Create your GRE config or compile your ip_wccp module.  DO NOT DO BOTH, you can only use one OR the other.

If you are on a Redhat/Fedora distribution you may be able to get away with just this:

[root@tornado log]# cat /etc/sysconfig/network-scripts/ifcfg-gre0 
DEVICE=gre0
BOOTPROTO=static
IPADDR=172.16.1.6
NETMASK=255.255.255.252
ONBOOT=yes
IPV6INIT=no

That will bring up a GRE tunnel for you.  Just choose any address that you are not already using, it doesn't really matter what it is as you'll only be using the one address in that subnet.

Configure Squid, editing your squid.conf (Squid-2.5)

The relevant parts of your squid.conf are:

        http_port 3128
        httpd_accel_host virtual
        httpd_accel_port 80
        httpd_accel_with_proxy on
        httpd_accel_uses_host_header on
        wccp_router 192.168.0.1

If you are running Squid-2.6 or Squid-3.0 you simply need to put in:

        http_port 3128 transparent
        wccp2_router 192.168.0.1

You may need to add outgoing_address statements if you have multiple IP addresses on the box.

If you are running squid on port 80, you obviously will need to change this.  In that case you will also not need to create iptables rules in the next step.

Configure your iptables rules:

You will want iptables rules looking something like this:

iptables -A PREROUTING -s 192.168.0.0/255.255.0.0 -d ! 192.168.0.0/255.255.0.0 -i gre0 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.0.5:3128

In the above example -

  • 192.168.0.0/255.255.0.0 is my network
  • 192.168.0.5 is my cache traffic is coming in over the GRE interface gre0
  • I am redirecting TCP port 80 traffic
  • I'm DNATting it to my cache on 192.168.0.5 port 3128
  • -d ! 192.168.0.0/255.255.0.0 makes sure that I am not redirecting legitimate traffic destined for 192.168.0.5 (in my case I am running a web server on port 80, so I don't want port 80 traffic sourced from my LAN redirected as well)

If you are using iptables you will also want to turn on ip forwarding, so that your proxy 'routes' traffic between ports.

Do this by adding this line to /etc/sysctl.conf (again, Redhat/Fedora)

# Controls IP packet forwarding
net.ipv4.ip_forward = 1 
 

Test and troubleshoot.

This should now be functional.  You will need to give it a good work out and make sure that everything works OK.

Firstly, verify that your router is seeing the cache.  From enable mode:

router#show ip wccp
Global WCCP information:
    Router information:
        Router Identifier:                   10.0.0.1
        Protocol Version:                    1.0

    Service Identifier: web-cache
        Number of Cache Engines:             1
        Number of routers:                   1
        Total Packets Redirected:            5429
        Process:                             5308
        Fast:                                121
        CEF:                                 0
        Redirect access-list:                -none-
        Total Packets Denied Redirect:       0
        Total Packets Unassigned:            0
        Group access-list:                   -none-
        Total Messages Denied to Group:      0
        Total Authentication failures:       0
router#

Note how we see 1 router (itself) and 1 cache engine (your proxy).

If you do not see your cache engine then you need to check your squid.conf and also check your router config.  If they are OK then check that there is connectivity between them, and then that there is no firewalling going on.

You can also verify this by turning on debugging on your router:

The output should look like this, with one block of messages appearing every 10 or so seconds:

.Nov 21 15:04:26.650: WCCP-PKT: Received valid Here_I_Am packet from 192.168.0.5 w/rcvd_id 00007213
.Nov 21 15:04:26.650: WCCP-PKT: Sending I_See_You packet to 192.168.0.5 w/ rcvd_id 00007214
.Nov 21 15:04:37.098: WCCP-PKT: Received valid Here_I_Am packet from 192.168.0.5 w/rcvd_id 00007214
.Nov 21 15:04:37.098: WCCP-PKT: Sending I_See_You packet to 192.168.0.5 w/ rcvd_id 00007215

If not, your router is not seeing the cache.

Secondly, verify that your iptables rules are right:

[root@tornado log]# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       tcp  --  network.address.net/16 !network.address.net/16 tcp dpt:http to:192.168.0.5:3128

Thirdly, looking at your router now:

  • Turn off CEF
  • Try "ip wccp web-cache redirect out" on the outbound interface
  • Write an access list which excludes the proxy cache from redirection
  • Disable NAT if you possibly can, even to test.
  • Put the cache on a separate IP subnet/VLAN to your clients which you are redirecting
  • Ping the site first before you attempt to surf to it
  • Turn off any and all firewalling including ip inspection, between your cache and the WCCP router (including on the router itself)
  • Turn off any crypto

 

None of these steps should be necessary, but sometimes you need to try them to workaround undocumented bugs in IOS. If you do need any, then I suggest you log a bug with the Cisco TAC.

Note that whenever you change the IOS in your router you are advised to test WCCP in a lab first, or do your upgrade after hours when customer impact will be small. WCCP has a habit of breaking in very subtle ways.

Comments