rsyslog - Practical guide (Part 2)

rsyslog to ingest custom log file

Part 2

Here we learn to ingest custom application logs i.e. httpd logs are send to rsyslog server from the rsyslog client that runs httpd service

On client rsyslog node
cat /etc/rsyslog.conf

# MODULES

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal

# Add imfile module
$ModLoad imfile

# Send httpd message - START
## httpd Error log
$InputFileName /var/log/httpd/error_log
$InputFileTag httpd-error
$InputFileStateFile httpd-error
$InputFileSeverity error
$InputFileFacility ip-172-31-39-238.us-east-2.compute.internal
$InputRunFileMonitor

## httpd Access  log
$InputFileName /var/log/httpd/access_log
$InputFileTag httpd-access
$InputFileStateFile httpd-access
$InputFileSeverity info
$InputFileFacility ip-172-31-39-238.us-east-2.compute.internal
$InputRunFileMonitor
# Send httpd message - END

# Set interval to poll events, default is 10sec
$InputFilePollInterval 10

# ### begin forwarding rule ###
*.* @172.31.41.229:514
# ### end of the forwarding rule ###

On server rsyslog node
cat /etc/rsyslog.conf

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

#### RULES ####
# Sending httpd - START
$template HTTPDFile,"/var/log/httpd_logs/%HOSTNAME%/httpd.error"
*.*     -?HTTPDFile
# Sending httpd - END

Verification

Run the below on rsyslog server

$ sudo ls -ltr /var/log/httpd_logs/ip-172-31-39-238.us-east-2.compute.internal/httpd.error
-rw------- 1 root root 1875 Aug 12 01:27 /var/log/httpd_logs/ip-172-31-39-238.us-east-2.compute.internal/httpd.error

rsyslog - Practical guide (Part 1)

 rsyslog or rocket syslog as the name suggest, its a rocket for syslog traditional services that takes care of over million message forwarding in a second

Here I would show two practical use cases in 2 part series. The setup is done on AWS EC2 instances, lies within the same subnet.

Part 1 

Send client host's syslog facility like kernel, mail, auth, etc in summary syslog facility logs

For the security group, following was enabled:

i.e. UDP/514 within the same security group

+ Install rsyslog service on both server and client

sudo yum update && yum install rsyslog

+ Start and enable rsyslog service

sudo systemctl start rsyslog
sudo systemctl enable rsyslog
sudo systemctl status rsyslog

+ edit /etc/rsyslog.conf (the below is added to the default file)

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Sending syslogfacility - START
$template DynamicFile,"/var/log/loghost/%HOSTNAME%/%syslogfacility-text%.log"
*.*    -?DynamicFile
# Sending syslogfacility - END

+ sudo  systemctl restart rsyslog

Note: At this point you may see the system facility log files are generated for the localhost ip

sudo ls -ltr /var/log/loghost/ip-172-31-41-229

total 64
-rw------- 1 root root   560 Aug 11 11:20 kern.log
-rw------- 1 root root   151 Aug 11 12:04 auth.log
-rw------- 1 root root   630 Aug 11 12:21 mail.log
-rw------- 1 root root   319 Aug 11 12:30 syslog.log
-rw------- 1 root root  2779 Aug 11 12:41 user.log
-rw------- 1 root root  1434 Aug 11 12:50 cron.log
-rw------- 1 root root 13433 Aug 11 12:52 daemon.log
-rw------- 1 root root 21022 Aug 11 12:53 authpriv.log

This confirms the syslog is configured to listen to the traffic

+ enable rsyslog on the client host

*.* @172.31.41.229:514

+ restart rsyslog service

sudo systemctl restart rsyslog

Now check on rsyslog server if it has start receiving system log message from client ip

$ sudo ls -ltr /var/log/loghost
total 0
drwx------ 2 root root 142 Aug 11 12:17 ip-172-31-41-229
drwx------ 2 root root  76 Aug 11 12:41 ip-172-31-39-238

$ sudo ls -ltr /var/log/loghost/ip-172-31-39-238
total 20
-rw------- 1 root root  322 Aug 11 12:41 user.log
-rw------- 1 root root  240 Aug 11 12:50 cron.log
-rw------- 1 root root  647 Aug 11 12:56 authpriv.log
-rw------- 1 root root 4238 Aug 11 12:56 daemon.log

Troubleshooting rsyslog

+ install firewall-cmd to enable firewall on rsyslog server

sudo yum install firewalld

sudo systemctl start firewalld

sudo systemctl enable firewalld

sudo systemctl status firewalld

+ enable port udp/514 on the host on rsyslog server

sudo firewall-cmd --add-port=514/udp --permanent

sudo firewall-cmd --reload

+ verify the changes

$ sudo iptables -S

.....

.....

-A IN_public_allow -p udp -m udp --dport 514 -m conntrack --ctstate NEW -j ACCEPT