Within this post we will be exploring the use of Telegraf, InfluxDB and Grafana to capture, store and visualize syslog messages sent by Junos.

Telegraf is an agent for collecting, processing, aggregating, and writing metrics. Its plugin-driven and has the concept of four plugin types:

  • Input Plugins - Collect metrics from the system, services, or 3rd party APIs
  • Processor Plugins - Transform, decorate, and/or filter metrics
  • Aggregator Plugins - Create aggregate metrics (e.g. mean, min, max, quantiles, etc.
  • Output Plugins - Write metrics to various destinations.

InfluxDB is an open source time series database that is designed to handle high write and query loads which provides an SQL-like query language called InfluxQL for interacting with data.

Grafana allows you to query, visualise and alert on metrics and logs due to the pluggable data source model which supports many popular time series databases like Graphite, Prometheus, Elasticsearch, OpenTSDB and InfluxDB.

Finally we will also be using Docker, a tool designed to make it easier to create, deploy and run applications by using containers.

Junos Configuration

Firstly we need to configure Junos to send Syslog messages to the host that will be running the TIG stack. The configuration looks like the following:

set system syslog host 192.168.35.100 any any
set system syslog host 192.168.35.100 port 6514
set system syslog host 192.168.35.100 source-address 192.168.3.254
set system syslog host 192.168.35.100 structured-data

After committing this we can verify the Docker host is receiving the messages by using TCP Dump sudo tcpdump -i ens32 udp port 6514.

You should see something like this in the output:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens32, link-type EN10MB (Ethernet), capture size 262144 bytes
10:15:01.454499 IP _gateway.syslog > dev.6514: SYSLOG auth.notice, length: 89
10:15:03.414488 IP _gateway.syslog > dev.6514: SYSLOG auth.debug, length: 128
10:15:03.457785 IP _gateway.syslog > dev.6514: SYSLOG auth.debug, length: 132

TIG Stack

Now we will get to the good stuff. Presuming you have Docker and docker-compose installed, if not select the links...

Telegraf

Firstly we need to make sure we have the correct configuration for Telegraf. For this we will use the below configuration in a file called telegraf.conf

[agent]
  ## Default data collection interval for all inputs
  interval = "10s"
  ## Rounds collection interval to 'interval'
  ## ie, if interval="10s" then always collect on :00, :10, :20, etc.
  round_interval = true

  ## Telegraf will send metrics to outputs in batches of at most
  ## metric_batch_size metrics.
  ## This controls the size of writes that Telegraf sends to output plugins.
  metric_batch_size = 1000

  ## Maximum number of unwritten metrics per output.  Increasing this value
  ## allows for longer periods of output downtime without dropping metrics at the
  ## cost of higher maximum memory usage.
  metric_buffer_limit = 10000

  ## Collection jitter is used to jitter the collection by a random amount.
  ## Each plugin will sleep for a random time within jitter before collecting.
  ## This can be used to avoid many plugins querying things like sysfs at the
  ## same time, which can have a measurable effect on the system.
  collection_jitter = "0s"

  ## Default flushing interval for all outputs. Maximum flush_interval will be
  ## flush_interval + flush_jitter
  flush_interval = "10s"
  ## Jitter the flush interval by a random amount. This is primarily to avoid
  ## large write spikes for users running a large number of telegraf instances.
  ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
  flush_jitter = "0s"

  ## By default or when set to "0s", precision will be set to the same
  ## timestamp order as the collection interval, with the maximum being 1s.
  ##   ie, when interval = "10s", precision will be "1s"
  ##       when interval = "250ms", precision will be "1ms"
  ## Precision will NOT be used for service inputs. It is up to each individual
  ## service input to set the timestamp at the appropriate precision.
  ## Valid time units are "ns", "us" (or "µs"), "ms", "s".
  precision = ""

  ## Override default hostname, if empty use os.Hostname()
  hostname = "syslog_test"
  ## If set to true, do no set the "host" tag in the telegraf agent.
  omit_hostname = false


###############################################################################
#                            OUTPUT PLUGINS                                   #
###############################################################################

# Configuration for sending metrics to InfluxDB
[[outputs.influxdb]]
  urls = ["http://influxdb:8086"]
  database = "telegraf"


###############################################################################
#                            INPUT PLUGINS                                    #
###############################################################################

# # Accepts syslog messages following RFC5424 format with transports as per RFC5426, RFC5425, or RFC6587
 [[inputs.syslog]]
   server = "udp://:6514"

This defines the telegraf configuration to look and for a database called telegraf, if it not there by default it will create the database for you. However, the main bits we are looking at are the Input and Output Plugins.

Output Plugin

Here we define the url for the Influx Database in this instance we can give it the name and port of InfluxDB as this will be defined in the docker-compose plugin. We can then select the database we are using, again as mentioned above it will create the database if it does not already exist by default.

Input Plugin

As we varified that the host is receiving the Syslog messages with TCPDump we can use the input plugin inputs.syslog and tell it to use UDP port 6514. This is done in the following line: server = "udp://:6514".

Once this file is created save it in your local directory.

Docker

Firstly we need to create some folders within our directory so that we can have persistant storage for the containers mainly InfluxDB and Grafana as we dont want data to delete when the container is deleted.

axians@alex-jb:~/docker/TIG$ pwd
    /home/axians/docker/TIG
axians@alex-jb:~/docker/TIG$ mkdir grafana_data
axians@alex-jb:~/docker/TIG$ mkdir influxdb
axians@alex-jb:~/docker/TIG$ id -u
    1000

These commands show that I'm creating the grafana_data and influxdb folders and showing my Linux user ID. Keep this in mind we will need it.

Docker Compose File

We now need to create a docker-compose.yml file to bring all of the environment up at once.

We will use the following file to accomplish this:

version: "3.7"
services:
  influxdb:
    image: influxdb:latest
    container_name: influxdb
    ports:
      - "8083:8083"
      - "8086:8086"
      - "8090:8090"
    volumes:
      - /home/axians/docker/TIG/influxdb:/var/lib/influxdb

  telegraf:
    image: telegraf:latest
    container_name: telegraf
    ports:
      - "6514:6514/udp"
    links:
      - influxdb
    volumes:
      - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=pass
    depends_on:
      - influxdb
    user: "1000"
    ports:
      - "3000:3000"
    volumes:
      - /home/axians/docker/TIG/grafana_data:/var/lib/grafana

This is a lot of information so I will go through it in small steps.

Firstly the top two lines show what version of docker-compose we are using and then defining the services. The services definition contains the configuration that is applies to each container started for that service. Compose-File

This will start up all three services with all the required ports being mapped to the container. We select the images we need such as influxdb:latest, telegraf:latest and grafana/grafana:latest. Some containers such as the Grafana instance may require environment variables for example GF_SECURITY_ADMIN_PASSWORD sets the admin password. Ports allow us to expose the container ports to the host system so that we can externally access the container such as http://<host-ip>:3000. As you can see in the Grafana configuration we use the user ID collected earlier to make sure that there are no permission errors when the container uses the folder for persistant storage. Finally we map the volumes so that InfluxDB and Grafana can store the configuration and not have it erased when the container is removed.

Running the Containers

Finally lets run the containers by using the command docker-compose up

You should get an output similar to the following:

axians@alex-jb:~/docker/TIG$ docker-compose up
Creating network "tig_default" with the default driver
Pulling influxdb (influxdb:latest)...
latest: Pulling from library/influxdb
146bd6a88618: Pull complete
9935d0c62ace: Pull complete
db0efb86e806: Pull complete
5dd32e36b488: Pull complete
750868d0ab2b: Pull complete
f4d98645d729: Pull complete
c8bd5f153b8d: Pull complete
f458001f5cb1: Pull complete
Digest: sha256:eae897c8ebf85ac3e2bdff8ba053d40a3df7598c41f4b63d42faf2603e2eef74
Status: Downloaded newer image for influxdb:latest
Pulling telegraf (telegraf:latest)...
latest: Pulling from library/telegraf
146bd6a88618: Already exists
9935d0c62ace: Already exists
db0efb86e806: Already exists
b857ffe0c422: Pull complete
02db9717e0bc: Pull complete
b22cca777282: Pull complete
6d2c93a41774: Pull complete
Digest: sha256:91e0465070b0cb088ae52d5609fb202e330db36e2e5b53f7f749a8f68d089a61
Status: Downloaded newer image for telegraf:latest
Pulling grafana (grafana/grafana:latest)...
latest: Pulling from grafana/grafana
89d9c30c1d48: Pull complete
92c128799d27: Pull complete
fa1904dc426e: Pull complete
0bc30826133d: Pull complete
a086b998918c: Pull complete
3e65953c80f4: Pull complete
c8acf10409a4: Pull complete
deff1c4eb3ee: Pull complete
Digest: sha256:5c2fc6c625d8d5aa44926a9bc7d02ce91ff85d1769ed2378006caed378e9fb4a
Status: Downloaded newer image for grafana/grafana:latest
Creating influxdb ... done
Creating grafana  ... done
Creating telegraf ... done
Attaching to influxdb, telegraf, grafanaaxians@alex-jb:~/docker/TIG$ docker-compose up
Creating network "tig_default" with the default driver
Pulling influxdb (influxdb:latest)...
latest: Pulling from library/influxdb
146bd6a88618: Pull complete
9935d0c62ace: Pull complete
db0efb86e806: Pull complete
5dd32e36b488: Pull complete
750868d0ab2b: Pull complete
f4d98645d729: Pull complete
c8bd5f153b8d: Pull complete
f458001f5cb1: Pull complete
Digest: sha256:eae897c8ebf85ac3e2bdff8ba053d40a3df7598c41f4b63d42faf2603e2eef74
Status: Downloaded newer image for influxdb:latest
Pulling telegraf (telegraf:latest)...
latest: Pulling from library/telegraf
146bd6a88618: Already exists
9935d0c62ace: Already exists
db0efb86e806: Already exists
b857ffe0c422: Pull complete
02db9717e0bc: Pull complete
b22cca777282: Pull complete
6d2c93a41774: Pull complete
Digest: sha256:91e0465070b0cb088ae52d5609fb202e330db36e2e5b53f7f749a8f68d089a61
Status: Downloaded newer image for telegraf:latest
Pulling grafana (grafana/grafana:latest)...
latest: Pulling from grafana/grafana
89d9c30c1d48: Pull complete
92c128799d27: Pull complete
fa1904dc426e: Pull complete
0bc30826133d: Pull complete
a086b998918c: Pull complete
3e65953c80f4: Pull complete
c8acf10409a4: Pull complete
deff1c4eb3ee: Pull complete
Digest: sha256:5c2fc6c625d8d5aa44926a9bc7d02ce91ff85d1769ed2378006caed378e9fb4a
Status: Downloaded newer image for grafana/grafana:latest
Creating influxdb ... done
Creating grafana  ... done
Creating telegraf ... done
Attaching to influxdb, telegraf, grafana

To run headless hit ctrl+c and use the command docker-compose up -d to run it detached.

^CGracefully stopping... (press Ctrl+C again to force)
Stopping telegraf ... done
Stopping grafana  ... done
Stopping influxdb ... done
axians@alex-jb:~/docker/TIG$ docker-compose up -d
Starting influxdb ... done
Starting telegraf ... done
Starting grafana  ... done
axians@alex-jb:~/docker/TIG$ 

We can verify the containers using docker ps -a:

axians@alex-jb:~/docker/TIG$ docker ps -a
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                                    NAMES
a53c6e685ba2        telegraf:latest          "/entrypoint.sh tele…"   2 minutes ago       Up 54 seconds       8092/udp, 0.0.0.0:6514->6514/udp, 8125/udp, 8094/tcp                     telegraf
e1ac38412c85        grafana/grafana:latest   "/run.sh"                2 minutes ago       Up 53 seconds       0.0.0.0:3000->3000/tcp                                                   grafana
f2fe38289536        influxdb:latest          "/entrypoint.sh infl…"   2 minutes ago       Up 55 seconds       0.0.0.0:8083->8083/tcp, 0.0.0.0:8086->8086/tcp, 0.0.0.0:8090->8090/tcp   influxdb
axians@alex-jb:~/docker/TIG$

Configuring Grafana

To get to the Grafana dashboard navigate to the host IP address and port 3000 in my case this is http://192.168.35.100:3000.

1

To login use the credentials:

username: admin
password: pass

We now need to add the InfluxDB container as a datasource this can be done by clicking "Add Data Source" on the Home Dashboard page as you just login. Or navigating to Configuration>Data Sources>Add Data Source.

2

Select InfluxDB.

3

Add the following configuration:

3-1

url: http://influxdb:8086
database: telegraf

Click save and test and you should get the green notifications.

4

We can then add a new dashboard and display the logs we have been collecting by navigating to the plus icon and selecting dashboard. We can then add a visualization.

Select Table.

5

Now we can show the logs using the table by creating a query. Mine is very simple but you can play around as much as you like to make it look fancy.

6

We can also filter out specific severity by doing the following:

7

Thats all. If you would like to see this check out my Github here...