EdgeOS and Netflow

I’ve written a lot about getting stuff working on my Ubiquiti EdgeOS router. Recently, I got the idea in my head to enable netflow on the router to do some traffic analysis. My router does support exporting netflow data, so I thought it would be fairly simple to set up. In the end, it wasn’t too hard, but it did take some research and at least one dumb mistake.

Setting up netflow on the router wasn’t too hard at all. Below is the config I ultimately enabled:

system {
flow-accounting {
ingress-capture post-dnat
interface eth2
netflow {
enable-egress {
engine-id 2
}
engine-id 1
server 192.168.2.12 {
port 2055
}
timeout {
expiry-interval 60
flow-generic 60
icmp 60
max-active-life 604800
tcp-fin 60
tcp-generic 60
tcp-rst 60
udp 60
}
version 9
}
syslog-facility daemon
}
}

From the top, we are enabling flow-accounting under system. The ingress-capture post-dnat setting means that traffic will use my internal IP addresses as the destination for ingress traffic; if this is not set, then all inbound traffic appears to be destined for the public IP on my router, which isn’t helpful. Netflow is enabled on my WAN interface which is eth2. I also have egress netflow enabled, with an engine-id of 2; ingress has an engine-id of 1. These IDs are arbitrary, but they make it easier to parse what is ingress and what is egress later. Netflow will be sent to 192.168.2.12:2055, which is where I set up my collector.

The timeout settings are all set to 60 seconds, which is far lower than the defaults. Setting them lower will make the data less choppy, as the default generic flow has a timeout of an hour, so you may not get any data from a flow until it ends if it takes less than an hour, and then all of a sudden you see that you got a large chunk of data incoming/outgoing. Finally, the version of netflow exported is Cisco v9.

Although some of those settings required tweaking, particularly the post-dnat and the timeouts, but it wasn’t hard to set up at all. The first issue cropped up when it came time to find a collector for my CentOS server. The favorite on every list, ntop, is not free, so scratch that. I then turned to pmacct, specifically nfacctd as the collector, which I couldn’t get working for reasons which will soon become obvious. Sill stymied, I turned to NFDUMP and nfcapd, which once again would not work. I got very frustrated and set it aside for a couple of weeks, coming back to it later.

What frustrated me was the fact that I could see data packets arriving on port 2055 on my server. Doing a trusty tcpdump in very verbose mode showed netflow data arriving, and I even pulled out RFC3954 and groveled over the UDP payloads to validate that yes, this was in fact netflow v9 data in the packets. So then why weren’t any of the tools actually receiving data?

It was at this point that the “duh” moment hit. IPTABLES. I have IPTABLES as a host-based firewall on the CentOS server, and of course I did not have a rule allowing udp traffic to port 2055 through. Once I did this, I started getting data via nfcapd. I learned a very important lesson: tcpdump sees traffic before IPTABLES, so even if you see data arriving via tcpdump, that doesn’t mean it is passing your firewall. D’oh!

At this point I was finally receiving data, and as readers of this blog know, once I have data, I usually want to put it into Splunk. Tune in later for that misadventure!

Be Sociable, Share!

One Response

  1. You’re an AWESOME geek!
    Making me remember my wayward past and all of those crazy days trying to config firewalls to allow traffic to not only pass, but to fee logged and analyzed.