We assume that you have a splunk enteprise server installed and the Splunk Add-On for Unix addon downloaded and installed on the server side.
We now go ahead and install the same on an ubuntu 18.4.0 forwarder.
Upload the same package you used on your server for the installation onto the splunk forwarder. At the time of writing this file is splunk-add-on-for-unix-and-linux_602.tgz
Untar the file to a location of your choice:
tar -xvzf splunk-add-on-for-unix-and-linux_602.tgz
Copy the Splunk_TA_nix directory and its contents across to the splunk addons directory:
cp -R /app/images/splunk_linux/Splunk_TA_nix /opt/splunkforwarder/etc/apps
The default configuration file for the Splunk Add-On for Unix addon has all stanzas disabled. Edit the /opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf configuration file and change the disabled = 1 sections to disabled = 0 at the stanzas you would like to get covered. We disabled the ps and top sections at the test environment as they were generating way too much traffic. We used the following inputs.conf:
Copyright (C) 2019 Splunk Inc. All Rights Reserved. [script://./bin/vmstat.sh] interval = 60 sourcetype = vmstat source = vmstat disabled = 0 [script://./bin/iostat.sh] interval = 60 sourcetype = iostat source = iostat disabled = 0 [script://./bin/nfsiostat.sh] interval = 60 sourcetype = nfsiostat source = nfsiostat disabled = 0 [script://./bin/ps.sh] interval = 30 sourcetype = ps source = ps disabled = 1 [script://./bin/top.sh] interval = 60 sourcetype = top source = top disabled = 1 [script://./bin/netstat.sh] interval = 60 sourcetype = netstat source = netstat disabled = 0 [script://./bin/bandwidth.sh] interval = 60 sourcetype = bandwidth source = bandwidth disabled = 0 [script://./bin/protocol.sh] interval = 60 sourcetype = protocol source = protocol disabled = 0 [script://./bin/openPorts.sh] interval = 300 sourcetype = openPorts source = openPorts disabled = 0 [script://./bin/time.sh] interval = 21600 sourcetype = time source = time disabled = 0 [script://./bin/lsof.sh] interval = 600 sourcetype = lsof source = lsof disabled = 0 [script://./bin/df.sh] interval = 300 sourcetype = df source = df disabled = 0 Shows current user sessions [script://./bin/who.sh] sourcetype = who source = who interval = 150 disabled = 0 Lists users who could login (i.e., they are assigned a login shell) [script://./bin/usersWithLoginPrivs.sh] sourcetype = usersWithLoginPrivs source = usersWithLoginPrivs interval = 3600 disabled = 0 Shows last login time for users who have ever logged in [script://./bin/lastlog.sh] sourcetype = lastlog source = lastlog interval = 300 disabled = 0 Shows stats per link-level Etherner interface (simply, NIC) [script://./bin/interfaces.sh] sourcetype = interfaces source = interfaces interval = 60 disabled = 0 Shows stats per CPU (useful for SMP machines) [script://./bin/cpu.sh] sourcetype = cpu source = cpu interval = 30 disabled = 0 This script reads the auditd logs translated with ausearch [script://./bin/rlog.sh] sourcetype = auditd source = auditd interval = 60 disabled = 0 Run package management tool collect installed packages [script://./bin/package.sh] sourcetype = package source = package interval = 3600 disabled = 0 [script://./bin/hardware.sh] sourcetype = hardware source = hardware interval = 36000 disabled = 0 [monitor:///Library/Logs] disabled = 1 [monitor:///var/log] whitelist=(.log|log$|messages|secure|auth|mesg$|cron$|acpid$|.out) blacklist=(lastlog|anaconda.syslog) disabled = 1 [monitor:///var/adm] whitelist=(.log|log$|messages) disabled = 0 [monitor:///etc] whitelist=(.conf|.cfg|config$|.ini|.init|.cf|.cnf|shrc$|^ifcfg|.profile|.rc|.rules|.tab|tab$|.login|policy$) disabled = 1 bash history [monitor:///root/.bash_history] disabled = true sourcetype = bash_history [monitor:///home/*/.bash_history] disabled = true sourcetype = bash_history Added for ES support Note that because the UNIX app uses a single script to retrieve information from multiple OS flavors, and is intended to run on Universal Forwarders, it is not possible to differentiate between OS flavors by assigning different sourcetypes for each OS flavor (e.g. Linux:SSHDConfig), as was the practice in the older deployment-apps included with ES. Instead, sourcetypes are prefixed with the generic "Unix". May require Splunk forwarder to run as root on some platforms. [script://./bin/openPortsEnhanced.sh] disabled = true interval = 3600 source = Unix:ListeningPorts sourcetype = Unix:ListeningPorts [script://./bin/passwd.sh] disabled = true interval = 3600 source = Unix:UserAccounts sourcetype = Unix:UserAccounts Only applicable to Linux [script://./bin/selinuxChecker.sh] disabled = true interval = 3600 source = Linux:SELinuxConfig sourcetype = Linux:SELinuxConfig Currently only supports SunOS, Linux, OSX. May require Splunk forwarder to run as root on some platforms. [script://./bin/service.sh] disabled = true interval = 3600 source = Unix:Service sourcetype = Unix:Service Currently only supports SunOS, Linux, OSX. May require Splunk forwarder to run as root on some platforms. [script://./bin/sshdChecker.sh] disabled = true interval = 3600 source = Unix:SSHDConfig sourcetype = Unix:SSHDConfig Currently only supports Linux, OSX. May require Splunk forwarder to run as root on some platforms. [script://./bin/update.sh] disabled = true interval = 86400 source = Unix:Update sourcetype = Unix:Update [script://./bin/uptime.sh] disabled = true interval = 86400 source = Unix:Uptime sourcetype = Unix:Uptime [script://./bin/version.sh] disabled = true interval = 86400 source = Unix:Version sourcetype = Unix:Version This script may need to be modified to point to the VSFTPD configuration file. [script://./bin/vsftpdChecker.sh] disabled = true interval = 86400 source = Unix:VSFTPDConfig sourcetype = Unix:VSFTPDConfig
The last step is to restart the splunk forwarder:
/opt/splunkforwarder/bin/splunk restart
Now verify if the changes took place by running:
/opt/splunkforwarder/bin/splunk cmd btool inputs listYou should see all the Linux OS related monitoring options listed. Just like this:
[SSL] _rcvbuf = 1572864 allowSslRenegotiation = true cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ecdhCurves = prime256v1, secp384r1, secp521r1 host = zds.ztacs.com index = default sslQuietShutdown = false sslVersions = tls1.2 [batch:///opt/splunkforwarder/var/run/splunk/search_telemetry/*search_telemetry.json] _rcvbuf = 1572864 crcSalt = host = zds.ztacs.com index = _introspection log_on_completion = 0 move_policy = sinkhole sourcetype = search_telemetry [batch:///opt/splunkforwarder/var/spool/splunk] _rcvbuf = 1572864 crcSalt = host = zds.ztacs.com index = default move_policy = sinkhole [batch:///opt/splunkforwarder/var/spool/splunk/…stash_new] _rcvbuf = 1572864 crcSalt = host = zds.ztacs.com index = default move_policy = sinkhole queue = stashparsing sourcetype = stash_new [blacklist:/opt/splunkforwarder/etc/auth] _rcvbuf = 1572864 host = zds.ztacs.com index = default [blacklist:/opt/splunkforwarder/etc/passwd] _rcvbuf = 1572864 host = zds.ztacs.com index = default [fschange:/opt/splunkforwarder/etc] _rcvbuf = 1572864 delayInMills = 100 filesPerDelay = 10 followLinks = false fullEvent = false hashMaxSize = -1 host = zds.ztacs.com index = default pollPeriod = 600 recurse = true sendEventMaxSize = -1 signedaudit = true [http] _rcvbuf = 1572864 allowSslCompression = true allowSslRenegotiation = true dedicatedIoThreads = 2 disabled = 1 enableSSL = 1 host = zds.ztacs.com index = default maxSockets = 0 maxThreads = 0 port = 8088 sslVersions = *,-ssl2 useDeploymentServer = 0 [monitor:///Library/Logs] _rcvbuf = 1572864 disabled = 1 host = zds.ztacs.com index = default [monitor:///etc] _rcvbuf = 1572864 disabled = 1 host = zds.ztacs.com index = default whitelist = (.conf|.cfg|config$|.ini|.init|.cf|.cnf|shrc$|^ifcfg|.profile|.rc|.rules|.tab|tab$|.login|policy$) [monitor:///home/*/.bash_history] _rcvbuf = 1572864 disabled = true host = zds.ztacs.com index = default sourcetype = bash_history [monitor:///opt/splunkforwarder/etc/splunk.version] _TCP_ROUTING = * _rcvbuf = 1572864 host = zds.ztacs.com index = _internal sourcetype = splunk_version [monitor:///opt/splunkforwarder/var/log/splunk] _rcvbuf = 1572864 host = zds.ztacs.com index = _internal [monitor:///opt/splunkforwarder/var/log/splunk/license_usage_summary.log] _rcvbuf = 1572864 host = zds.ztacs.com index = _telemetry [monitor:///opt/splunkforwarder/var/log/splunk/metrics.log] _TCP_ROUTING = * _rcvbuf = 1572864 host = zds.ztacs.com index = _internal [monitor:///opt/splunkforwarder/var/log/splunk/splunkd.log] _TCP_ROUTING = * _rcvbuf = 1572864 host = zds.ztacs.com index = _internal [monitor:///opt/splunkforwarder/var/log/watchdog/watchdog.log*] _rcvbuf = 1572864 host = zds.ztacs.com index = _internal [monitor:///root/.bash_history] _rcvbuf = 1572864 disabled = true host = zds.ztacs.com index = default sourcetype = bash_history [monitor:///var/adm] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default whitelist = (.log|log$|messages) [monitor:///var/log] _rcvbuf = 1572864 blacklist = (lastlog|anaconda.syslog) disabled = 1 host = zds.ztacs.com index = default whitelist = (.log|log$|messages|secure|auth|mesg$|cron$|acpid$|.out) [monitor:///var/log/apache2/zds_access.log] _rcvbuf = 1572864 disabled = false host = zds.ztacs.com index = default sourcetype = access_log [monitor:///var/log/syslog] _rcvbuf = 1572864 disabled = false host = zds.ztacs.com index = remotelogs sourcetype = linux_logs [script] _rcvbuf = 1572864 host = zds.ztacs.com index = default interval = 60.0 start_by_shell = true [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/bandwidth.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 60 source = bandwidth sourcetype = bandwidth [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/cpu.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 30 source = cpu sourcetype = cpu [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/df.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 300 source = df sourcetype = df [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/hardware.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 36000 source = hardware sourcetype = hardware [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/interfaces.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 60 source = interfaces sourcetype = interfaces [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/iostat.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 60 source = iostat sourcetype = iostat [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lastlog.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 300 source = lastlog sourcetype = lastlog [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/lsof.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 600 source = lsof sourcetype = lsof [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/netstat.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 60 source = netstat sourcetype = netstat [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/nfsiostat.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 60 source = nfsiostat sourcetype = nfsiostat [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/openPorts.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 300 source = openPorts sourcetype = openPorts [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/openPortsEnhanced.sh] _rcvbuf = 1572864 disabled = true host = zds.ztacs.com index = default interval = 3600 source = Unix:ListeningPorts sourcetype = Unix:ListeningPorts [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/package.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 3600 source = package sourcetype = package [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/passwd.sh] _rcvbuf = 1572864 disabled = true host = zds.ztacs.com index = default interval = 3600 source = Unix:UserAccounts sourcetype = Unix:UserAccounts [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/protocol.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 60 source = protocol sourcetype = protocol [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/ps.sh] _rcvbuf = 1572864 disabled = 1 host = zds.ztacs.com index = default interval = 30 source = ps sourcetype = ps [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/rlog.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 60 source = auditd sourcetype = auditd [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/selinuxChecker.sh] _rcvbuf = 1572864 disabled = true host = zds.ztacs.com index = default interval = 3600 source = Linux:SELinuxConfig sourcetype = Linux:SELinuxConfig [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/service.sh] _rcvbuf = 1572864 disabled = true host = zds.ztacs.com index = default interval = 3600 source = Unix:Service sourcetype = Unix:Service [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/sshdChecker.sh] _rcvbuf = 1572864 disabled = true host = zds.ztacs.com index = default interval = 3600 source = Unix:SSHDConfig sourcetype = Unix:SSHDConfig [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/time.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 21600 source = time sourcetype = time [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/top.sh] _rcvbuf = 1572864 disabled = 1 host = zds.ztacs.com index = default interval = 60 source = top sourcetype = top [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/update.sh] _rcvbuf = 1572864 disabled = true host = zds.ztacs.com index = default interval = 86400 source = Unix:Update sourcetype = Unix:Update [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/uptime.sh] _rcvbuf = 1572864 disabled = true host = zds.ztacs.com index = default interval = 86400 source = Unix:Uptime sourcetype = Unix:Uptime [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/usersWithLoginPrivs.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 3600 source = usersWithLoginPrivs sourcetype = usersWithLoginPrivs [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/version.sh] _rcvbuf = 1572864 disabled = true host = zds.ztacs.com index = default interval = 86400 source = Unix:Version sourcetype = Unix:Version [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/vmstat.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 60 source = vmstat sourcetype = vmstat [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/vsftpdChecker.sh] _rcvbuf = 1572864 disabled = true host = zds.ztacs.com index = default interval = 86400 source = Unix:VSFTPDConfig sourcetype = Unix:VSFTPDConfig [script:///opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin/who.sh] _rcvbuf = 1572864 disabled = 0 host = zds.ztacs.com index = default interval = 150 source = who sourcetype = who [splunktcp] _rcvbuf = 1572864 acceptFrom = * connection_host = ip host = zds.ztacs.com index = default route = has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue [tcp] _rcvbuf = 1572864 acceptFrom = * connection_host = dns host = zds.ztacs.com index = default [udp] _rcvbuf = 1572864 connection_host = ip host = zds.ztacs.com index = default
Once you open the splunk console and go to Search and Reporting, filter for the hostname of your forwarder then click on sourcetype on the left hand side. You should see data already flowing across just like this:
