rsyslog - Practical guide (Part 2)

rsyslog to ingest custom log file

Part 2

Here we learn to ingest custom application logs i.e. httpd logs are send to rsyslog server from the rsyslog client that runs httpd service

On client rsyslog node
cat /etc/rsyslog.conf

# MODULES

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal

# Add imfile module
$ModLoad imfile

# Send httpd message - START
## httpd Error log
$InputFileName /var/log/httpd/error_log
$InputFileTag httpd-error
$InputFileStateFile httpd-error
$InputFileSeverity error
$InputFileFacility ip-172-31-39-238.us-east-2.compute.internal
$InputRunFileMonitor

## httpd Access  log
$InputFileName /var/log/httpd/access_log
$InputFileTag httpd-access
$InputFileStateFile httpd-access
$InputFileSeverity info
$InputFileFacility ip-172-31-39-238.us-east-2.compute.internal
$InputRunFileMonitor
# Send httpd message - END

# Set interval to poll events, default is 10sec
$InputFilePollInterval 10

# ### begin forwarding rule ###
*.* @172.31.41.229:514
# ### end of the forwarding rule ###

On server rsyslog node
cat /etc/rsyslog.conf

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

#### RULES ####
# Sending httpd - START
$template HTTPDFile,"/var/log/httpd_logs/%HOSTNAME%/httpd.error"
*.*     -?HTTPDFile
# Sending httpd - END

Verification

Run the below on rsyslog server

$ sudo ls -ltr /var/log/httpd_logs/ip-172-31-39-238.us-east-2.compute.internal/httpd.error
-rw------- 1 root root 1875 Aug 12 01:27 /var/log/httpd_logs/ip-172-31-39-238.us-east-2.compute.internal/httpd.error