Hello,
I tried to run graphite on another machine. I was able to redirect statistics collection, but I cannot see graph data:
I have made changes in etc/graphite.conf:
[carbon] # # NAV supports Carbon's UDP line receiver. Host and port information of # the backend can be configured in this section. # host = IP port = 2003
[graphiteweb] # # Where is the Graphite web interface located. NAV needs this to get graphs # and data from Graphite. # base=https://IP
When I first started NAV, it was configured to use the graphite on the same machine as NAV. It worked.
When I changed it and restarted NAV processes it sent statistics to the new IP (I confirmed it via graphite web fronted), but it still tries to access old graphiteweb for rendering graphs, not the one changed in grahpite.conf.
Can there be a problém with certificate? (it is self signed one)
Isn't it somehow cached in a database?
Brgds
Mat
-----Original Message----- From: nav-users-request@uninett.no [mailto:nav-users-request@uninett.no] On Behalf Of Martin.Jaburek@cah.cz Sent: Friday, March 28, 2014 1:38 PM To: Morten Brekkevold Cc: nav-users@uninett.no Subject: RE: Problem retrieving port statistics
I solved my problém with moving databases to memory - it reduces load on machine and now it is ok. But I am still experiencing gaps on some graphs. There is one device which suffers from it most - see attached graphs. (graphs are from 2 devices - one OK and the other with GAPS)
Gaps are only in port metrics graphs - they are at exactly the same time on all ports on given device.
Systém metrics are just fine on this device.
I checked the 5min graph of ipdevpoll systém metrics and it shows 100 at peak times, what I assume are only nearly 2 minutes.
I'm monitoring about 140 devices right now with nav.
I think that it is much correlated to my yesterday upgrade to 4.0.0 (from 4.0b5).
Since then I can also see attached errors. They were not there when I was on version 4.0b5.
It happens only on cisco catalyst 4500/6500 and one type of 3560E. (I have many 3750 but, there is no problém at all) and causes inventory jobs to fail.
Brgds
Mat
-----Original Message----- From: nav-users-request@uninett.no [mailto:nav-users-request@uninett.no] On Behalf Of Morten Brekkevold Sent: Friday, March 28, 2014 10:36 AM To: JABUREK Martin Cc: nav-users@uninett.no Subject: Re: Problem retrieving port statistics
On Thu, 27 Mar 2014 08:37:51 +0000 "Martin.Jaburek@cah.cz" Martin.Jaburek@cah.cz wrote:
Now I am able to collect all information needed.
But I still have a problém with collected data - there are very often gaps in graphs.
Which graphs have gaps? Port metrics? System metrics? All of them?
How many devices are you monitoring, and what does the runtime of the various ipdevpoll jobs look like?
If the 1minstats job is spending more than 1 minute to complete for any device, there may be gaps its graphs, 5 minutes for the 5minstats job.
-- Morten Brekkevold UNINETT
On Mon, 31 Mar 2014 07:01:09 +0000 "Martin.Jaburek@cah.cz" Martin.Jaburek@cah.cz wrote:
When I first started NAV, it was configured to use the graphite on the same machine as NAV. It worked.
When I changed it and restarted NAV processes it sent statistics to the new IP (I confirmed it via graphite web fronted), but it still tries to access old graphiteweb for rendering graphs, not the one changed in grahpite.conf.
You restarted NAV processes, but did you restart Apache (in which the NAV web UI lives)?
Can there be a problém with certificate? (it is self signed one)
If there was, it wouldn't cause NAV to get graphs from the old URL, it would cause an error. We haven't actually checked to see how accessing Graphite over HTTPS works, so it seems you may be a guinea pig here ;)
Hi,
I did restart the apache server. I have done some tcpdumping and find out very interesting thing:
- when I access ipdevinfo for some device it communicates with graphite server without any problém (Requests are truncated, I attached even some responses, the server serves them all) I think that this serves for availability and response info in status pane.
http://graphite:80/metrics/find?query=nav.devices.DEVICE.%2A http://graphite:80/metrics/find?query=nav.devices.DEVICE.cpu.%2A http://graphite:80/metrics/find?query=nav.devices.DEVICE.cpu.Linecard_slot_1... http://graphite:80/metrics/find?query=nav.devices.DEVICE.ipdevpoll.%2A ...
http://graphite/render?format=json&from=-1day&target=nav.devices.DEV... [{"target": "nav.devices.DEVICE.ping.packetLoss", "datapoints": [[0.0, 1396354560], [0.0, 1396354620], [0.0, 1396354680], [0.0, 1396354740], [0.0, 1396354800], ... http://graphite/render?format=json&from=-1week&target=nav.devices.DE... [{"target": "nav.devices.DEVICE.ping.packetLoss", "datapoints": [[0.0, 1395836400], [0.0, 1395836700], [0.0, 1395837000], [0.0, 1395837300], [0.0, 1395837600], ...
- since then it does not connect graphite server at all - when I click on Port Metrics -> some interface, I just got an error - Error loading image. I have done even some code crawling and found an actual requested link: /ipdevinfo/g/port/1/Octets/?timeframe=day
But I wasn't able to trace it any further. I expect, that NAV actually should proxy this request to graphite (thing which is broken). I'm no expert in python so I got lost there.
I hope this can help you pin down the problem.
I switched to port 80 on graphite side to make things easier.
Best regards
Mat -----Original Message----- From: Morten Brekkevold [mailto:morten.brekkevold@uninett.no] Sent: Wednesday, April 02, 2014 9:43 AM To: JABUREK Martin Cc: nav-users@uninett.no Subject: Re: Graphite on other machine
On Mon, 31 Mar 2014 07:01:09 +0000 "Martin.Jaburek@cah.cz" Martin.Jaburek@cah.cz wrote:
When I first started NAV, it was configured to use the graphite on the same machine as NAV. It worked.
When I changed it and restarted NAV processes it sent statistics to the new IP (I confirmed it via graphite web fronted), but it still tries to access old graphiteweb for rendering graphs, not the one changed in grahpite.conf.
You restarted NAV processes, but did you restart Apache (in which the NAV web UI lives)?
Can there be a problém with certificate? (it is self signed one)
If there was, it wouldn't cause NAV to get graphs from the old URL, it would cause an error. We haven't actually checked to see how accessing Graphite over HTTPS works, so it seems you may be a guinea pig here ;)
-- Morten Brekkevold UNINETT
On Wed, 2 Apr 2014 12:30:26 +0000 "Martin.Jaburek@cah.cz" Martin.Jaburek@cah.cz wrote:
- since then it does not connect graphite server at all - when I click
on Port Metrics -> some interface, I just got an error - Error loading image. I have done even some code crawling and found an actual requested link: /ipdevinfo/g/port/1/Octets/?timeframe=day
But I wasn't able to trace it any further. I expect, that NAV actually should proxy this request to graphite (thing which is broken). I'm no expert in python so I got lost there.
Yes, the link above should produce a redirect to a Graphite render proxy within NAV.
I hope this can help you pin down the problem.
I would want you to visit the abovementioned link with your browser to see what you get.
Hi,
I tried it and it really threw an exception. The problém was, that directive base in graphite.conf is expected to end with a slash. I was missing it. I think that it cause problém only when used with port number.
[graphiteweb] base=http://IP:80
should be [graphiteweb] base=http://IP:80/
Maybe it would be nice feature to sanitize these configuration variables so it would not be so sensitive about trailing slashes.
Thanks for your effort, this problém is solved.
Brgds
Mat -----Original Message----- From: Morten Brekkevold [mailto:morten.brekkevold@uninett.no] Sent: Thursday, April 03, 2014 10:23 AM To: JABUREK Martin Cc: nav-users@uninett.no Subject: Re: Graphite on other machine
On Wed, 2 Apr 2014 12:30:26 +0000 "Martin.Jaburek@cah.cz" Martin.Jaburek@cah.cz wrote:
- since then it does not connect graphite server at all - when I click
on Port Metrics -> some interface, I just got an error - Error loading image. I have done even some code crawling and found an actual requested link: /ipdevinfo/g/port/1/Octets/?timeframe=day
But I wasn't able to trace it any further. I expect, that NAV actually should proxy this request to graphite (thing which is broken). I'm no expert in python so I got lost there.
Yes, the link above should produce a redirect to a Graphite render proxy within NAV.
I hope this can help you pin down the problem.
I would want you to visit the abovementioned link with your browser to see what you get.
-- Morten Brekkevold UNINETT
On Thu, 3 Apr 2014 14:33:21 +0000 "Martin.Jaburek@cah.cz" Martin.Jaburek@cah.cz wrote:
I tried it and it really threw an exception. The problém was, that directive base in graphite.conf is expected to end with a slash. I was missing it. I think that it cause problém only when used with port number.
Ah, thanks, for the tip! Fixed: https://bugs.launchpad.net/nav/+bug/1303804
-
Martin.Jaburek＠cah.cz -
Morten Brekkevold