[ProgClub programming] Centralised logging

Jedd Rashbrooke jedd.rashbrooke at gmail.com
Tue Jun 11 23:21:46 AEST 2019


On Tue, 11 Jun 2019 at 20:25, John Elliot V | ProgClub <jj5 at progclub.org>
wrote:

> 1. Will a VM on my LAN do for the log repository?
>

 A VM will be fine, for varying values of cores & memory.


> 2. How much storage should I provision for the log repository?
>

 Use LVM (qv) and this isn't a decision you need to make before you know
how much store you need.  This is the joy of LVM.

 Don't fret about automating LVM (etc) up front - these are irregular tasks
that are difficult to automate, and the pay-off in doing so is vanishingly
small (refer the earlier, splendid xkcd about frequency / time spent / time
payoff).

I have around 20 to 70 hosts on my network in total, depending on how
> and when you want to count them. Should test systems send their logs to

the production log server, or should I provision a test log server too?
>

 Start by building a single logging server, and send everything to it.
Then after some weeks, review, and tune.  Easy.

 Note that logs won't give you performance metrics - both are (probably)
best served by dedicated systems.

 Elasticsearch can conceivably do both logs + metrics, but it's not
something I've played with.  I suggest Prometheus (as mentioned a few times
already) would be a low cost way for you to get into metrics, and it is
easy to then add instrumentation directly into your application layer as
well.  THere are agents (node-exporter) that you can run on your servers
and point at the prometheus server.

 For logs, Loki (Grafana) seems excessively difficult to get going for a
test system, so I'd defer that for now.

 I'd suggest Graylog on its own VM - it needs mongo (small instance) and
elasticsearch (relatively small) -- I run graylog with all its dependencies
on a single isolated VM with 3GB memory, and it's not seemingly constrained
by network, memory, or CPU.  Graylog is closest to Splunk in terms of UI &
tooling, and it's worth evaluating.

 I run Elasticsearch + Logstash + Kibana (mostly for netflows) on the same
VM that I run Prometheus and Zabbix and PostgreSQL - it's a 9GB VM with 4
vCores.

 Elasticsearch and Logstash are both JVM's and appreciate memory.
Prometheus is very lightweight in terms of memory and CPU, so could sit in
a much, much smaller instance (1GB, 2 core, would be plenty I suspect).

 Grafana (for visualisation) is not hungry either.

 Zabbix + PostgreSQL if you decide to head down that path will be slightly
hungrier than Prometheus, but for 70 servers you'll have trouble making it
struggle - this is tech that's been perfected over 15+ years to be rock
solid and very, very tight.


 TL;DR

 Spin up a VM for logging - install graylog as per instructions.  Point
some rsyslog stuff towards it.  Review the tutorials.

 Spin up a VM with Prometheus and Grafana - install node-exporter on one of
your other boxes, and point that towards your Prometheus server.

 Spin up a VM for Elasticsearch + Logstash + Kibana and set up Metricbeat
on one of your other boxes, and point that towards your Elasticsearch
server.

 Do all this in your test environment so you don't need to worry about
firewalls and user auth.

 j.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.progclub.org/pipermail/programming/attachments/20190611/94ebed23/attachment.html>


More information about the programming mailing list