Hello Morten,
 
this is how I approached your instructions:

$ timeout 3600 tcpdump -i lo udp port 2003 -w tcpdump-1hour.pcap
$ grep --binary-files=text router01 tcpdump-1hour.pcap | grep --binary-files=text ae3.ifInOctets

This should give us about 12 entries because the interface is being checked every 5min. And we should see the counter rising with each entry.
This is the output of of the grep command above:
nav.devices.router1_domain_com.ports.ae3.ifInOctets 4267030379578 151981310
nav.devices.router1_domain_com.ports.ae3.ifInOctets 4272260055697 1519813402
nav.devices.router1_domain_com.ports.ae3.ifInOctets 4276300392448 151981370
nav.devices.router1_domain_com.ports.ae3.ifInOctets 4280165984433 1519814003
nav.devices.router1_domain_com.ports.ae3.ifInOctets 4283800404729 151981430
nav.devices.router1_domain_com.ports.ae3.ifInOctets 4287170549844 1519814603
nav.devices.router1_domain_com.ports.ae3.ifInOctets 4290575217783 151981490
nav.devices.router1_domain_com.ports.ae3.ifInOctets 4294304860274 1519815203
nav.devices.router1_domain_com.ports.ae3.ifInOctets 4298158132556 151981550
nav.devices.router1_domain_com.ports.ae3.ifInOctets 4302670400704 1519815803
nav.devices.router1_domain_com.ports.ae3.ifInOctets 4306587460714 1519816103
nav.devices.router1_domain_com.ports.ae3.ifInOctets 4310180259720 1519816403

12 entries, just as expected. Apparently everything is working fine, but the graph is still mostly gaps.

Best regards
Karl

From: Morten Brekkevold [mailto:morten.brekkevold@uninett.no]
Sent: Tuesday, Feb 27, 2018 10:35 AM CET
To: Karl Gerhard
Cc: nav-users@uninett.no
Subject: Is that a healthy carbon cache?

On Mon, 26 Feb 2018 10:03:51 +0100 Karl Gerhard <karl_gerh@gmx.at> wrote:


apologies for the late reply, the flu season is terrible this year.

I hope this message finds you well :)


I have verified that the 5min jobs are running at the correct intervals and
that we're using SNMP v2c everywhere. I checked everything that was suggested
in "Debugging gaps in graphs" article and I'm looking for advice as to how to
proceed further.

You seem to describe a situation where only some interfaces on some
devices have gaps, and none of the issues described in the debugging
guide apply.

At this point, I'd grab a swiss army knife: I'd use tcpdump and/or
wireshark to verify a couple of things:

1. Does ipdevpoll actually consistently post the traffic counter values
   of these interfaces to the carbon backend? (UDP port 2003).

2. If not, does ipdevpoll consistently query and get a response for the
   SNMP traffic counters for these interfaces (UDP port 161).
   
Are you able to do this without further guidance?