On Mon, 30 Jun 2014 11:28:09 +0200 Sigurd Mytting sigurd@mytting.no wrote:
Hi all!
Hi Sigurd,
I'm looking into doing an upgrade of our NAV installation, we're on 3.15 and it's time to move on to 4.
I'm looking for advice on hardware setup, filesystem layouts and so on.
Currently we have about 1500 devices, 99% routers and switches, in the database (mostly Catalysts ranging from 2940 to 3750, a few 6500 and Nexus 7k/5k, Juniper SRX/EX and a Juniper QFX, around 50 virtual Cisco ASA contexts as well).
Hardware is a single server, 2x dual core opteron 2.6GHz, 32GB ram and several raid1's (in hardware). I've put postgres on its own 15k RPM raid1, the same with /var/lib/nav and confined /var/log/nav to the OS-drives with its own partition.
Looks pretty good to me :)
I have been and will be using the excellent debian-packages from UNINETT for my installation.
I've done some tricks for optimizing disk performance, by using the "noop" disk scheduler I see a big difference in system sluggishness, even when I have loads around 30. This is mostly related to my hardware raid, the default debian scheduler does not give a good performance when operating on a hardware raid (same thing with SSDs).
It's always good to experiment with I/O schedulers, as the default CFQ scheduler isn't the best for the kind of loads on a typical NAV server.
I'm planning a new installation, this time on two servers, but I have yet to determine disk and raid layouts and what to put on each server.
Postgres away from the collector is rather given, I'm unfamiliar with graphite, could/should it be put on another server? (I might find a third server if its recommended.)
Our current template for customer deployments is a single Intel Xeon E5620 (Quad core @ 2.40GHz) based server, with 12GB RAM and 4x600GB 10K SAS disks in a RAID 10 configuration.
This has scaled well for all pre-4.0 deployments, except at NTNU, where we deployed two of these servers, separating PostgreSQL onto one of them.
Under NAV 4.0, more metrics are monitored, and as a result from monitoring ~40k ports / ~1k devices and storing metrics in Graphite, these two servers no longer scale very well for NTNU, even with Graphite load balanced across the two servers. Additionally, one other customer deployment also has scaling problems now.
Therefore, we are currently deploying dedicated Graphite servers to these two customers. This server features an SSD RAID for metric storage, and we will see how it fares in the coming days.
What will be the default datastores be? I'd like to set up separate partitions before installing the NAV-packages to make sure filesystems like /var don't run out of space when NAV logs grow, postgres goes wild and/or graphite stores what I'm actually asking it to store... What will typically be hit hard with random IO, sequential read/write and so on?
Any input would be apreciated.
Our template is to have separate partitions for /, /boot, /home, /usr, /usr/local, /var, /var/log and /var/lib/postgresql . We use LVM for flexibility in partition sizes.
Under NAV 4.0 on Debian, /var and /var/lib/postgresql will be hit the hardest, as Graphite metrics are stored in /var/lib/graphite/ - you should already know your PostgreSQL load :)
I can report back when we have more experience with an SSD-based setup, but my intuition is that you could probably get away with mounting /var/lib/graphite to an SSD-based RAID on an existing NAV server. Unfortunately, there are no free drive bays in our already deployed servers, which is why we are deploying a separate Graphite server to customers where things don't scale well.
If your existing server is running fine with both NAV 3.15 and PostgreSQL, I probably wouldn't deviate from that, but you might want to start thinking about a separate Graphite server with SSDs for NAV 4.0.