This is a bit of a tricky issue that we recently ran into in the office, so I thought I’d post it up here in case someone else runs into a similar problem.
On the evening before Australia day (January 26), and for the couple days after, our internet traffic did this:
The axis label only goes up to 2 digits but that’s actually close to 200gb used on Australia day – with no one in the office. Needless to say we went over our ~250gb monthly internode limit. They didn’t cap us for a few days, so we ended up using 460gb, and our internet only rolled over on the 19th!
I went straight to FAMS (Fortinets cloud based logging system) to see if I could find out who was the culprit, but FAMS showed a normal days internet usage, in fact the top 2 IP’s belong to unmetered internode sites.
But on the Fortigate you could clearly see the internet was being maxed right up until it was capped
and this one taken a day after we got capped:
So I was confused, the Fortigate knew about the traffic, but it wasn’t logging it in detail, so we couldn’t pinpoint who or what it was. Even torrenting would be hard to max out internet connection constantly.
Turns out the problem is to do with Adobe reader and the Fortigates HTTP virus scanning. The adobe reader auto updater detects there’s an update and sends a request to the server to download the file, as the file comes in, the Foritigate caches it ready to virus scan. At that point Adobe Reader thinks the download has failed, so it tries again – I believe it tries different servers/ports. The Fortigate deletes the cached file and a new one comes in. It does this in an infinite loop – maxing out the internet connection.
As the file never actually goes to the computer, the Fortigate never logs it as data against an IP. The only clue it left was that there were a lot of ‘requests’ to adobe.com, but I initially figured that would just be adobe readers around the office polling the server for the updates.
As soon as I turned off virus scanning of http downloads the problem was solved. Though I think I’ll just put a block on ardownload.adobe.com and turn virus scanning back on – we’d rather push out adobe updates manually anyway.
Hopefully someone else gets some help out of this, we would have been pulling our hair out for days if it weren’t for the helpful people at Overclockers Australia forums pointing me to a thread where someone had a similar issue!
Post a Comment