Installation of Loki and Promtail on RHEL 8.3

Installation of Loki and Promtail on RHEL 8.3

March 03, 2021 ( last updated : March 03, 2021 )
rhel grafana loki logging


Installation of Loki and Promtail on RHEL 8.3

Overview & Architecture

I’ve been looking for a “light weight” method of getting aggregated logging across the home lab for a while. Loki has been interesting since its release, so I decided to have a play with it. I’ve been using Grafana heavily for years, so it seemed like the logical choice, as it is much lower on resources than Graylog, the other platform I was investigating.

The plan is to setup a single node in the lab as a log server, it has two disks associated with it - a boot device and a data device. The data device is a separate LV and will be configured to store logs on disk.

Note: This is a home lab installation, and is not configured with production grade security.

Loki Installation

Sources: https://grafana.com/docs/loki/latest/installation/local/

# Install the requirements
dnf install nano wget unzip policycoreutils setroubleshoot

# Create a directory for Loki to run from 
mkdir -v /opt/loki

# Create a user to run Loki as
useradd -s /sbin/nologin -d /opt/loki loki

# Provide access to the above directory to this user.
chown loki:loki /opt/loki

# Open the default loki firewall port
firewall-cmd --zone=public --add-port=3100/tcp
firewall-cmd --zone=public --add-port=3100/tcp --permanent

Download the Loki binaries and their respective configuration. There is a bug in the documentation that suggests the configuration should be pulled from the master branch, this is incorrect. The default configuration files should come from the same release branch as the binaries that are used.

cd /opt/loki

wget https://github.com/grafana/loki/releases/download/v2.1.0/loki-linux-amd64.zip
wget https://github.com/grafana/loki/releases/download/v2.1.0/promtail-linux-amd64.zip

unzip loki-linux-amd64.zip
unzip loki-linux-amd64.zip
rm *.zip

# Loki default configuration
wget https://raw.githubusercontent.com/grafana/loki/v2.1.0/cmd/loki/loki-local-config.yaml

# Promtail default configuration
wget https://raw.githubusercontent.com/grafana/loki/v2.1.0/cmd/promtail/promtail-local-config.yaml

At this point you should be able to run the example configuration and access it at http://{host_ip}:3100/metrics. By default Loki will listen on all interfaces, and with the firewall rules already open this will be accessible from your network. ./loki-linux-amd64 -config.file=loki-local-config.yaml

This should “work” in the most basic sense of the word. Depending on which user you are testing as you may encounter permissions issues. These can be rectified by ensuring the temporary directory exists and has the correct permissions. I have included the recursive flag on the chown in case this directory has been created already by running loki as root. mkdir -pv /tmp/loki chown -R loki:loki /tmp/loki

Whilst we are at it, lets create a file system location for log retention. I’m opting to use the file system backend at this time with a separate LV mounted at /data/, so I have used: mkdir -pv /data/loki chown loki:loki /data/loki

Once these were sorted I was able run Loki as the correct user with: sudo -u loki loki-linux-amd64 -config.file=loki-local-config.yaml

At this point I was able to direct the Grafana Loki data source to my Loki server and add the data source without errors.

The final step is to adjust the configuration for local storage, more details can be found here: https://grafana.com/docs/loki/latest/configuration/examples/#complete-local-config. I’m not yet 100% convinced that I have the best configuration, but this seems to work.

#cat loki-local-config.yaml 

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
	store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
	prefix: index_
        period: 168h

storage_config:
  boltdb_shipper:
    active_index_directory: /data/loki/boltdb-shipper-active
    cache_location: /data/loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /data/loki/chunks

compactor:
  working_directory: /tmp/loki/boltdb-shipper-compactor
  shared_store: filesystem

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

ruler:
  storage:
    type: local
    local:
      directory: /data/loki/rules
  rule_path: /tmp/loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true

Promtail Configuration

Promtail is the tool that is used to ingest logs to Loki.

Sources:

Edit the default configuration file using your editor of personal or religous choice: nano /opt/loki/promtail-local-config.yaml As I particularly wish to use this for syslog aggregation, I have replaced the default job with the below configuration:

  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:20514
      idle_timeout: 60s
      label_structured_data: yes
	  use_incoming_timestamp: yes
      labels:
        job: "syslog"
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'
      - source_labels: ['__syslog_message_severity']
        target_label: level
      - source_labels: ['__syslog_message_app_name']
        target_label: app_name
      - source_labels: [__syslog_message_facility]
        target_label: facility

This is the extent of default configuration required for using promtail.

NOTE: THE PORT OF 20514 IS NOT ARBITRARY! This port is part of the default SELinux rsyslog configuration that allows TCP communication. Another port can be used, but this requires further modification to the SELinux configuration. See https://support.logz.io/hc/en-us/articles/209486429-Troubleshooting-Rsyslog-SELinux-configuration for details on this process.

Add a firewall configuration to allow connections on this port.

firewall-cmd --zone=public --add-port=20514/tcp
firewall-cmd --zone=public --add-port=20514/tcp --permanent

Host Configuration

Edit the local rsyslog configuration: nano /etc/rsyslog.conf Append the following to the end of the file: action(type="omfwd" target="127.0.0.1" protocol="tcp" port="20514" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted") Save the file and exit.

At this point, you should be able to start Loki with: sudo -u loki loki-linux-amd64 -config.file=loki-local-config.yaml Then start promtail with the following: sudo -u loki ./promtail-linux-amd64 -config.file=promtail-local-config.yaml Finally, restart rsyslog with: sudo systemctl restart rsyslog

Voila! If all has gone according to plan, you should now have syslog entries in Loki. This can be easily tested with Grafana.

Configuring rsyslog as UDP Listener

Some legacy devices don’t provide much configurability in their remote syslog setup and so I would like my local rsyslog instance to also be able to listen on UDP 514 in order to forward to Loki.

Sources: https://rainer.gerhards.net/2019/10/rsyslog-configure-syslog-udp-reception.html

Modify the following comment lines to enable reception:

# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")

Add a firewall configuration to allow connections on this port.

firewall-cmd --zone=public --add-port=514/udp
firewall-cmd --zone=public --add-port=514/udp --permanent

Running as a Service

Loki

Create the following file: sudo nano /etc/systemd/system/loki.service

[Unit]
Description=Grafana Loki service
After=network.target

[Service]
Type=simple
User=loki
ExecStart=/opt/loki/loki-linux-amd64 -config.file /opt/loki/loki-local-config.yaml

[Install]
WantedBy=multi-user.target

Promtail

Create the following file: sudo nano /etc/systemd/system/promtail.service

[Unit]
Description=Grafana Promtail service
After=network.target

[Service]
Type=simple
User=loki
ExecStart=/opt/loki/promtail-linux-amd64 -config.file /opt/loki/promtail-local-config.yaml

[Install]
WantedBy=multi-user.target

Start the Services

systemctl enable --now loki
systemctl enable --now promtail

This will likely fail due to a SELinux issue. The below type enforcement file should fix the identified issues.

Note: This was generated by setting permissive mode to be on setenforce 0 and then running grep loki /var/log/audit/audit.log | audit2allow -m loki > loki.te

# cat loki.te 

module loki 1.0;

require {
	type user_home_t;
	type unreserved_port_t;
	type init_t;
	type unlabeled_t;
	class file { execute execute_no_trans map open read };
	class tcp_socket name_connect;
	class dir add_name;
}

#============= init_t ==============
allow init_t unlabeled_t:dir add_name;

#!!!! This avc is allowed in the current policy
allow init_t unreserved_port_t:tcp_socket name_connect;

#!!!! This avc is allowed in the current policy
allow init_t user_home_t:file { execute execute_no_trans map open read };


This file can be converted into a module with:

checkmodule -M -m -o loki.mod loki.te

Before compiling as a policy package:

semodule_package -o loki.pp -m loki.mod

And installing:

semodule -i loki.pp

At this point, this can be tidied up with:

rm loki.mod loki.pp loki.te

Assuming you carried out the earlier service enabling, a restart of the services should yield success:

systemctl restart loki
systemctl restart promtail

Originally published March 03, 2021 (Updated March 03, 2021)

Related posts :