From: "Brunelle, James (PCL)
Subject: Imperva WAF Upgrade at DCC (CHG0091517)
Date: June 20, 2018 at 6:53:45 PM PDT
Yesterday, we began work in implementing CHG0091517, to upgrade the physical Imperva gateway appliances at DCC from version 11.5 to version 13. We began by pulling one of the gateways out of datapath and performed the upgrade, which we believed to be successful. After placing the upgraded appliance back in datapath, we performed some basic testing to make sure the appliance was passing traffic. At the time, I was able to access multiple sites which were in-line with the web application firewall. While some misconfiguration for several of the HAL sites on the WAF prevented me from running our full battery of tests we usually run at PCL, I had seen enough data to suggest to me that the upgraded gateway was functioning correctly. It became quickly apparent the next morning that there were connectivity issues to these sites. As soon as it was clear that the issue was most likely the upgrade that was causing this issue, we pulled the upgraded gateway back out of datapath and connectivity was restored.
I spent most of today digging into this problem, and we believe we've identified what happened. The Imperva gateways at DCC are placed in-line with web traffic via the Gigamon appliances. The Gigamon adds a 2nd VLAN tag to the packets it sends through the in-line security tools (such as Imperva), pushing the maximum packet size above 1500 bytes. We identified this issue years ago when this was first set up, and we had to set the Imperva gateways network interfaces to accept Jumbo Frames (frames larger than the standard 1500 bytes). This change involved modifying network configuration scripts for each interface on the gateway. We checked to make sure the scripts were set to accept these larger packets after performing the upgrade. However, the setting does not appear to be taking effect on the latest version of Imperva, as the interfaces were set to only accept packets up to the standard 1500 bytes despite the larger value being specified in the configuration files. The result was that a lot of packets were being dropped, and caused the connectivity issues we experienced overnight into this morning.
I have opened a case with Imperva technical support regarding this issue and have been working with them this afternoon and this evening in troubleshooting the problem. I do not have a timeline on when this issue may be resolved or if there is a workaround for this. Obviously, we will not be moving forward with upgrading the other gateway which is currently in datapath until this issue has been resolved. We may want to keep the upgraded gateway out of datapath for the time being in order to perform troubleshooting on this issue, rather than reverting to v11.5 and putting it back in datapath. I will discuss this with our security operations folks tomorrow morning. For now, it remains out of datapath.
As for CHG0091517, I have marked this as "Completed, Unsuccessful". I have marked the P1 incident that was opened (INC0298214) as resolved, as the connectivity issues we experienced went away this morning once the failed gateway was pulled out of datapath. We will not attempt to put that gateway back in datapath without first contacting Anila Augustine first, making sure we have Anila's team standing by to perform their testing once the gateway is placed back in-line. This would also require a separate change be opened at this point since we are outside of our change window for CHG0091517.
