How to slash your Splunk bill with Syslog Watcher

Splunk is priced on how much you index per day. A firewall heartbeat every few seconds is cheap to store on a Windows disk and expensive once Splunk indexes it. To lower the bill, keep the full stream in Syslog Watcher and forward a filtered copy.

What you are paying for

Every message that arrives at Splunk counts toward the daily ingest license. The volume usually comes from traffic nobody searches in Splunk:

  • Heartbeats and health checks, every few seconds
  • Informational and debug messages left on after a change
  • The same non-critical event, repeated by one device
  • Duplicates of the same message

Those messages are still worth having when you are tracing a fault a week later. Indexing them is a poor use of the license. Local storage is the inexpensive copy. Splunk is the copy you pay for, because that is what the correlation rules search.

Keep the message, then decide

Two filters in Syslog Watcher do different jobs. Mixing them up is how a “savings” project deletes the evidence.

  • Incoming filter, on the collector, discards a message before it is stored. You cannot search it later. Use it only for traffic you are sure you will never want.
  • Forward to Syslog Server, a feature on the Server, reads syslog messages that are already in storage and sends the ones that match. The rest stay on the Windows server. Splunk never sees them, so they never touch the ingest license.

The bill goes down because of the second filter. The first one only saves disk space.

The collector writes messages to storage first. The syslog forwarding feature reads that storage afterward. If Splunk is down, or the feature is suspended, the collector keeps writing. You can catch up, or skip ahead, when you resume the feature. The settings are in Forward to Syslog Server.

Forward a filtered copy

You need a server that is already collecting. Splunk must already be listening for syslog, over UDP, TCP, or TLS, on a host the Windows server can reach.

  1. In Manager, add a feature of type Forward to Syslog Server.
  2. Set Target server and Target port to the Splunk syslog input. Set Protocol to the transport that input expects. Turn on certificate verification for TLS when the receiver has a real certificate. Verification is off by default so a lab with a self-signed certificate still connects.
  3. Set the feature filter to the messages Splunk should index. A blank filter forwards everything, which does not change the bill.
  4. Leave the message as it was received unless Splunk’s input requires a rebuilt RFC 5424 message. By default, the feature does not rewrite the message.
  5. Start syslog forwarding. On the Server tab, the Features table lists Messages for it: how many messages the feature has consumed. With a filter in place, that count stays well below what the collector stored. If it matches the collector, the filter is not removing anything.

What to forward, and what to leave

Forward the messages a person or a correlation search will actually open:

  • Authentication failures, configuration changes, and VPN session events
  • Severity 0–3 from devices whose vendors use those numbers for real faults
  • The specific text your detections already look for

Leave on the syslog server:

  • Severity 6 and 7 from chatty gear, once you have confirmed those devices do not hide faults at informational
  • Heartbeat and “status ok” text
  • Debug left enabled on a switch, router, or firewall

Vendors do not agree on severity. Read a sample on the Last 15 Minutes tab before you drop a whole severity. A Cisco %LINK-3-UPDOWN is not the same event as another vendor’s severity 3.

Start with the one device that dominates the Splunk volume. Compare a day of that device in Syslog Watcher with a day of ingest from that source in Splunk after the filter. Then add the next device.

A measured cut

Since deploying Syslog Watcher, we cut our Splunk ingestion by 65% without losing visibility on critical events—and trimmed our annual license cost.

IT Manager, Swisscard AECS GmbH

The local copy is what makes that possible. If Splunk does not have a message, the Windows syslog server still has the log, for as long as your storage retention keeps it. A second copy of the archive can go to the cloud: How to collect syslog messages to cloud storage (S3).

Download Syslog Watcher to try the filter on your own noisiest source. What is a syslog server? is the shorter explanation of collector and store.