Hello Morten, this is how I approached your instructions:
$ timeout 3600 tcpdump -i lo udp port 2003 -w tcpdump-1hour.pcap $ grep --binary-files=text router01 tcpdump-1hour.pcap | grep --binary-files=text ae3.ifInOctets
This should give us about 12 entries because the interface is being checked every 5min. And we should see the counter rising with each entry. This is the output of of the grep command above: nav.devices.router1_domain_com.ports.ae3.ifInOctets 4267030379578 151981310 nav.devices.router1_domain_com.ports.ae3.ifInOctets 4272260055697 1519813402 nav.devices.router1_domain_com.ports.ae3.ifInOctets 4276300392448 151981370 nav.devices.router1_domain_com.ports.ae3.ifInOctets 4280165984433 1519814003 nav.devices.router1_domain_com.ports.ae3.ifInOctets 4283800404729 151981430 nav.devices.router1_domain_com.ports.ae3.ifInOctets 4287170549844 1519814603 nav.devices.router1_domain_com.ports.ae3.ifInOctets 4290575217783 151981490 nav.devices.router1_domain_com.ports.ae3.ifInOctets 4294304860274 1519815203 nav.devices.router1_domain_com.ports.ae3.ifInOctets 4298158132556 151981550 nav.devices.router1_domain_com.ports.ae3.ifInOctets 4302670400704 1519815803 nav.devices.router1_domain_com.ports.ae3.ifInOctets 4306587460714 1519816103 nav.devices.router1_domain_com.ports.ae3.ifInOctets 4310180259720 1519816403
12 entries, just as expected. Apparently everything is working fine, but the graph is still mostly gaps.
Best regards Karl
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ *From:* Morten Brekkevold [mailto:morten.brekkevold@uninett.no] *Sent:* Tuesday, Feb 27, 2018 10:35 AM CET *To:* Karl Gerhard *Cc:* nav-users@uninett.no *Subject:* Is that a healthy carbon cache?
On Mon, 26 Feb 2018 10:03:51 +0100 Karl Gerhard karl_gerh@gmx.at wrote:
apologies for the late reply, the flu season is terrible this year.
I hope this message finds you well :)
I have verified that the 5min jobs are running at the correct intervals and that we're using SNMP v2c everywhere. I checked everything that was suggested in "Debugging gaps in graphs" article and I'm looking for advice as to how to proceed further.
You seem to describe a situation where only some interfaces on some devices have gaps, and none of the issues described in the debugging guide apply.
At this point, I'd grab a swiss army knife: I'd use tcpdump and/or wireshark to verify a couple of things:
Does ipdevpoll actually consistently post the traffic counter values of these interfaces to the carbon backend? (UDP port 2003).
If not, does ipdevpoll consistently query and get a response for the SNMP traffic counters for these interfaces (UDP port 161).
Are you able to do this without further guidance?