Installation of Loki and Promtail on RHEL 8.3

Installation of Loki and Promtail on RHEL 8.3

March 03, 2021 ( last updated : March 03, 2021 )
rhel grafana loki logging

Installation of Loki and Promtail on RHEL 8.3

Overview & Architecture

I’ve been looking for a “light weight” method of getting aggregated logging across the home lab for a while. Loki has been interesting since its release, so I decided to have a play with it. I’ve been using Grafana heavily for years, so it seemed like the logical choice, as it is much lower on resources than Graylog, the other platform I was investigating.

The plan is to setup a single node in the lab as a log server, it has two disks associated with it - a boot device and a data device. The data device is a separate LV and will be configured to store logs on disk.

Note: This is a home lab installation, and is not configured with production grade security.

Loki Installation


# Install the requirements
dnf install nano wget unzip policycoreutils setroubleshoot

# Create a directory for Loki to run from 
mkdir -v /opt/loki

# Create a user to run Loki as
useradd -s /sbin/nologin -d /opt/loki loki

# Provide access to the above directory to this user.
chown loki:loki /opt/loki

# Open the default loki firewall port
firewall-cmd --zone=public --add-port=3100/tcp
firewall-cmd --zone=public --add-port=3100/tcp --permanent

Download the Loki binaries and their respective configuration. There is a bug in the documentation that suggests the configuration should be pulled from the master branch, this is incorrect. The default configuration files should come from the same release branch as the binaries that are used.

cd /opt/loki


rm *.zip

# Loki default configuration

# Promtail default configuration

At this point you should be able to run the example configuration and access it at http://{host_ip}:3100/metrics. By default Loki will listen on all interfaces, and with the firewall rules already open this will be accessible from your network. ./loki-linux-amd64 -config.file=loki-local-config.yaml

This should “work” in the most basic sense of the word. Depending on which user you are testing as you may encounter permissions issues. These can be rectified by ensuring the temporary directory exists and has the correct permissions. I have included the recursive flag on the chown in case this directory has been created already by running loki as root. mkdir -pv /tmp/loki chown -R loki:loki /tmp/loki

Whilst we are at it, lets create a file system location for log retention. I’m opting to use the file system backend at this time with a separate LV mounted at /data/, so I have used: mkdir -pv /data/loki chown loki:loki /data/loki

Once these were sorted I was able run Loki as the correct user with: sudo -u loki loki-linux-amd64 -config.file=loki-local-config.yaml

At this point I was able to direct the Grafana Loki data source to my Loki server and add the data source without errors.

The final step is to adjust the configuration for local storage, more details can be found here: I’m not yet 100% convinced that I have the best configuration, but this seems to work.

#cat loki-local-config.yaml 

auth_enabled: false

  http_listen_port: 3100

	store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled

    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
	prefix: index_
        period: 168h

    active_index_directory: /data/loki/boltdb-shipper-active
    cache_location: /data/loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
    directory: /data/loki/chunks

  working_directory: /tmp/loki/boltdb-shipper-compactor
  shared_store: filesystem

  reject_old_samples: true
  reject_old_samples_max_age: 168h

  max_look_back_period: 0s

  retention_deletes_enabled: false
  retention_period: 0s

    type: local
      directory: /data/loki/rules
  rule_path: /tmp/loki/rules-temp
  alertmanager_url: http://localhost:9093
      store: inmemory
  enable_api: true

Promtail Configuration

Promtail is the tool that is used to ingest logs to Loki.


Edit the default configuration file using your editor of personal or religous choice: nano /opt/loki/promtail-local-config.yaml As I particularly wish to use this for syslog aggregation, I have replaced the default job with the below configuration:

  - job_name: syslog
      idle_timeout: 60s
      label_structured_data: yes
	  use_incoming_timestamp: yes
        job: "syslog"
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'
      - source_labels: ['__syslog_message_severity']
        target_label: level
      - source_labels: ['__syslog_message_app_name']
        target_label: app_name
      - source_labels: [__syslog_message_facility]
        target_label: facility

This is the extent of default configuration required for using promtail.

NOTE: THE PORT OF 20514 IS NOT ARBITRARY! This port is part of the default SELinux rsyslog configuration that allows TCP communication. Another port can be used, but this requires further modification to the SELinux configuration. See for details on this process.

Add a firewall configuration to allow connections on this port.

firewall-cmd --zone=public --add-port=20514/tcp
firewall-cmd --zone=public --add-port=20514/tcp --permanent

Host Configuration

Edit the local rsyslog configuration: nano /etc/rsyslog.conf Append the following to the end of the file: action(type="omfwd" target="" protocol="tcp" port="20514" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted") Save the file and exit.

At this point, you should be able to start Loki with: sudo -u loki loki-linux-amd64 -config.file=loki-local-config.yaml Then start promtail with the following: sudo -u loki ./promtail-linux-amd64 -config.file=promtail-local-config.yaml Finally, restart rsyslog with: sudo systemctl restart rsyslog

Voila! If all has gone according to plan, you should now have syslog entries in Loki. This can be easily tested with Grafana.

Configuring rsyslog as UDP Listener

Some legacy devices don’t provide much configurability in their remote syslog setup and so I would like my local rsyslog instance to also be able to listen on UDP 514 in order to forward to Loki.


Modify the following comment lines to enable reception:

# Provides UDP syslog reception
# for parameters see
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")

Add a firewall configuration to allow connections on this port.

firewall-cmd --zone=public --add-port=514/udp
firewall-cmd --zone=public --add-port=514/udp --permanent

Running as a Service


Create the following file: sudo nano /etc/systemd/system/loki.service

Description=Grafana Loki service

ExecStart=/opt/loki/loki-linux-amd64 -config.file /opt/loki/loki-local-config.yaml



Create the following file: sudo nano /etc/systemd/system/promtail.service

Description=Grafana Promtail service

ExecStart=/opt/loki/promtail-linux-amd64 -config.file /opt/loki/promtail-local-config.yaml


Start the Services

systemctl enable --now loki
systemctl enable --now promtail

This will likely fail due to a SELinux issue. The below type enforcement file should fix the identified issues.

Note: This was generated by setting permissive mode to be on setenforce 0 and then running grep loki /var/log/audit/audit.log | audit2allow -m loki > loki.te

# cat loki.te 

module loki 1.0;

require {
	type user_home_t;
	type unreserved_port_t;
	type init_t;
	type unlabeled_t;
	class file { execute execute_no_trans map open read };
	class tcp_socket name_connect;
	class dir add_name;

#============= init_t ==============
allow init_t unlabeled_t:dir add_name;

#!!!! This avc is allowed in the current policy
allow init_t unreserved_port_t:tcp_socket name_connect;

#!!!! This avc is allowed in the current policy
allow init_t user_home_t:file { execute execute_no_trans map open read };

This file can be converted into a module with:

checkmodule -M -m -o loki.mod loki.te

Before compiling as a policy package:

semodule_package -o loki.pp -m loki.mod

And installing:

semodule -i loki.pp

At this point, this can be tidied up with:

rm loki.mod loki.pp loki.te

Assuming you carried out the earlier service enabling, a restart of the services should yield success:

systemctl restart loki
systemctl restart promtail

Originally published March 03, 2021 (Updated March 03, 2021)

Related posts :