rsyslog

rsyslog: why disk-assisted queues keep a file open

From time to time, someone asks why rsyslog disk-assisted queues keep one file open until shutdown. So it probably is time to elaborate a bit about it.

Let’s start with explaining what can be seen: if a disk-assisted queue is configured, rsyslog will normally use the in-memory queue. It will open a disk queue only if there is good reason to do so (because it severely hurts performance). The prime reason to go to disk is when the in memory queue’s configured size has been exhausted. Once this happens, rsyslog begins to create spool files, numbered consequtively. This should normally happen in cases where e.g. a remote target is temporarily not reachable or a database engine is responding extremely slow.  Once the situation has been cleared, rsyslog clears out the spool files. However, one spool file always is kept until you shut rsyslog down. Note well that when the disk queue is idle, all messages are processed even though the the physical spool file still contains some already-processed data (impstats will show you the exact details).

This is expected behaviour, and there is good reason for it. This is what happens technically:

A DA queue is actually “two queues in one”: It is a regular in-memory queue, which has a (second) helper disk queue in case it neeeds it. As long as operations run smoothly, the disk queue is never used. When the system starts up, only the in-memory queue is started. Startup of the disk queue is deferred until it is actually needed (as in most cases it will never be needed).

When the in-memory queue runs out of space, it starts that Disk queue, which than allocates its queue file. In order to reclaim space, not a single file is written but a series of files, where old files are deleted when they are processed and new files are created on an as-needed basis. Initially, there is only one file, which is read and written. And if the queue is empty, this single file still exists, because it is the representation of a disk queue (like the in-memory mapping tables for memory queues always exist, which just cannot be seen by the user).

So what happens in the above case is that the disk queue is created, put into action (the part where it writes and deletes files) and is then becoming idle and empty. At that stage, it will keep its single queue file, which holds the queue’s necessary mappings.

One may now ask “why not shut down the disk queue if no longer needed”? The short answer is that we anticpiate that it will be re-used and thus we do not make the effort to shut it down and restart when the need again arises. Let me elaborate: experience tells that when a system needs the disk queue once, it is highly likely to need it again in the future. The reason for disk queues to kick into action are often cyclic, like schedule restarts of remote systems or database engines (as part of a backup process, for example). So we assume if we used it once, we will most probably need it again and so keep it ready. This also helps reduce potential message loss during the switchover process to disk – in extreme cases this can happen if there is high traffic load and slim in-memory queues (remember that starting up a disk queue needs comparativley long).

The longer answer is that in the past rsyslog tried to shut down disk queues to “save” ressources. Experience told us that this “saving” often resulted in resource wasting. Also, the need to synchronize disk queue shutdown with the main queue was highly complex (just think about cases where we shut it down at the same time the in-memory queue actually begins to need it again!). This caused quite some overhead even when the disk queue was not used (again, this is the case most of the time – if properly sized). An indication of this effect probably is that we could remove roughly 20% of rsyslog’s queue code when we removed the “disk queue shutdown” feature!

Bottom line: keeping the disk queue active once it has been activated is highly desirable and as such seeing a queue file around until shutdown is absolutely correct. Many users will even never see this spool file, because there are no exceptional circumstances that force the queue to go to disk. So they may be surprised if it happens every now and then.

Side-Note: if disk queue files are created without a traget going offline, one should consider increasing the in-memory queue size, as disk queues are obviouly much less efficient than memory queues.

 

Queues are used to pass content, mostly syslog messages, between components of rsyslog. With queues, rsyslog is capable of processing multiple messages simultaneously and to apply several actions to a single message at once. The data flow inside rsyslog can be illustrated as follows:

Figure 20.1. Message Flow in Rsyslog

Whenever rsyslog receives a message, it passes this message to the preprocessor and then places it into the main message queue. Messages wait there to be dequeued and passed to the rule processor.

The rule processor is a parsing and filtering engine. Here, the rules defined in /etc/rsyslog.conf are applied. Based on these rules, the rule processor evaluates which actions are to be performed. Each action has its own action queue. Messages are passed through this queue to the respective action processor which creates the final output. Note that at this point, several actions can run simultaneously on one message. For this purpose, a message is duplicated and passed to multiple action processors.

Only one queue per action is possible. Depending on configuration, the messages can be sent right to the action processor without action queuing. This is the behavior of direct queues (see below). In case the output action fails, the action processor notifies the action queue, which then takes an unprocessed element back and after some time interval, the action is attempted again.

To sum up, there are two positions where queues stand in rsyslog: either in front of the rule processor as a single main message queue or in front of various types of output actions as action queues. Queues provide two main advantages that both lead to increased performance of message processing:

• they serve as buffers that decouple producers and consumers in the structure of rsyslog
• they allow for parallelization of actions performed on messages

Apart from this, queues can be configured with several directives to provide optimal performance for your system. These configuration options are covered in the following sections.

WARNING

If an output plug-in is unable to deliver a message, it is stored in the preceding message queue. If the queue fills, the inputs block until it is no longer full. This will prevent new messages from being logged via the blocked queue. In the absence of separate action queues this can have severe consequences, such as preventing SSH logging, which in turn can prevent SSH access. Therefore it is advised to use dedicated action queues for outputs which are forwarded over a network or to a database.

20.4.1. Defining Queues

Based on where the messages are stored, there are several types of queues: direct, in-memory, disk, anddisk-assisted in-memory queues that are most widely used. You can choose one of these types for the main message queue and also for action queues. Add the following into /etc/rsyslog.conf:

$objectQueueType queue_type

Here, you can apply the setting for the main message queue (replace object with MainMsg) or for an action queue (replace object with Action). Replace queue_type with one of direct, linkedlist or fixedarray (which are in-memory queues), or disk.

The default setting for a main message queue is the FixedArray queue with a limit of 10,000 messages. Action queues are by default set as Direct queues.

⁠Direct Queues

For many simple operations, such as when writing output to a local file, building a queue in front of an action is not needed. To avoid queuing, use:

$objectQueueType Direct

Replace object with MainMsg or with Action to use this option to the main message queue or for an action queue respectively. With direct queue, messages are passed directly and immediately from the producer to the consumer.

⁠Disk Queues

Disk queues store messages strictly on a hard drive, which makes them highly reliable but also the slowest of all possible queuing modes. This mode can be used to prevent the loss of highly important log data. However, disk queues are not recommended in most use cases. To set a disk queue, type the following into /etc/rsyslog.conf:

$objectQueueType Disk

Replace object with MainMsg or with Action to use this option to the main message queue or for an action queue respectively. Disk queues are written in parts, with a default size 10 Mb. This default size can be modified with the following configuration directive:

$objectQueueMaxFileSize size

where size represents the specified size of disk queue part. The defined size limit is not restrictive,rsyslog always writes one complete queue entry, even if it violates the size limit. Each part of a disk queue matches with an individual file. The naming directive for these files looks as follows:

$objectQueueFilename name

This sets a name prefix for the file followed by a 7-digit number starting at one and incremented for each file.

⁠In-memory Queues

With in-memory queue, the enqueued messages are held in memory which makes the process very fast. The queued data is lost if the computer is power cycled or shut down. However, you can use the $ActionQueueSaveOnShutdown setting to save the data before shutdown. There are two types of in-memory queues:

• FixedArray queue — the default mode for the main message queue, with a limit of 10,000 elements. This type of queue uses a fixed, pre-allocated array that holds pointers to queue elements. Due to these pointers, even if the queue is empty a certain amount of memory is consumed. However, FixedArray offers the best run time performance and is optimal when you expect a relatively low number of queued messages and high performance.
• LinkedList queue — here, all structures are dynamically allocated in a linked list, thus the memory is allocated only when needed. LinkedList queues handle occasional message bursts very well.

In general, use LinkedList queues when in doubt. Compared to FixedArray, it consumes less memory and lowers the processing overhead.

Use the following syntax to configure in-memory queues:

$objectQueueType LinkedList

$objectQueueType FixedArray

Replace object with MainMsg or with Action to use this option to the main message queue or for an action queue respectively.

⁠Disk-Assisted In-memory Queues

Both disk and in-memory queues have their advantages and rsyslog lets you combine them in disk-assisted in-memory queues. To do so, configure a normal in-memory queue and then add the $objectQueueFileName directive to define a file name for disk assistance. This queue then becomesdisk-assisted, which means it couples an in-memory queue with a disk queue to work in tandem.

The disk queue is activated if the in-memory queue is full or needs to persist after shutdown. With a disk-assisted queue, you can set both disk-specific and in-memory specific configuration parameters. This type of queue is probably the most commonly used, it is especially useful for potentially long-running and unreliable actions.

To specify the functioning of a disk-assisted in-memory queue, use the so-called watermarks:

$objectQueueHighWatermark number

$objectQueueLowWatermark number

Replace object with MainMsg or with Action to use this option to the main message queue or for an action queue respectively. Replace number with a number of enqueued messages. When an in-memory queue reaches the number defined by the high watermark, it starts writing messages to disk and continues until the in-memory queue size drops to the number defined with the low watermark. Correctly set watermarks minimize unnecessary disk writes, but also leave memory space for message bursts since writing to disk files is rather lengthy. Therefore, the high watermark must be lower than the whole queue capacity set with $objectQueueSize. The difference between the high watermark and the overall queue size is a spare memory buffer reserved for message bursts. On the other hand, setting the high watermark too low will turn on disk assistance unnecessarily often.

⁠

Example 20.12. Reliable Forwarding of Log Messages to a Server

Rsyslog is often used to maintain a centralized logging system, where log messages are forwarded to a server over the network. To avoid message loss when the server is not available, it is advisable to configure an action queue for the forwarding action. This way, messages that failed to be sent are stored locally until the server is reachable again. Note that such queues are not configurable for connections using the UDP protocol.

Procedure 20.1. Forwarding To a Single Server

Suppose the task is to forward log messages from the system to a server with host name example.com, and to configure an action queue to buffer the messages in case of a server outage. To do so, perform the following steps:

• Use the following configuration in /etc/rsyslog.conf or create a file with the following content in the /etc/rsyslog.d/ directory:

  $ActionQueueType LinkedList
  $ActionQueueFileName example_fwd
  $ActionResumeRetryCount -1
  $ActionQueueSaveOnShutdown on
  *.*       @@example.com:6514

  Where:

  • $ActionQueueType enables a LinkedList in-memory queue,
  • $ActionFileName defines a disk storage, in this case the backup files are created in the /var/lib/rsyslog/ directory with the example_fwd prefix,
  • the $ActionResumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if server is not responding,
  • enabled $ActionQueueSaveOnShutdown saves in-memory data if rsyslog shuts down,
  • the last line forwards all received messages to the logging server, port specification is optional.
  With the above configuration, rsyslog keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

Procedure 20.2. Forwarding To Multiple Servers

The process of forwarding log messages to multiple servers is similar to the previous procedure:

• Each destination server requires a separate forwarding rule, action queue specification, and backup file on disk. For example, use the following configuration in /etc/rsyslog.conf or create a file with the following content in the /etc/rsyslog.d/ directory:

  $ActionQueueType LinkedList
  $ActionQueueFileName example_fwd1
  $ActionResumeRetryCount -1
  $ActionQueueSaveOnShutdown on
  *.*       @@example1.com
  
  $ActionQueueType LinkedList
  $ActionQueueFileName example_fwd2 
  $ActionResumeRetryCount -1
  $ActionQueueSaveOnShutdown on
  *.*       @@example2.com

20.4.2. Creating a New Directory for rsyslog Log Files

Rsyslog runs as the syslogd daemon and is managed by SELinux. Therefore all files to which rsyslog is required to write to, must have the appropriate SELinux file context.

Procedure 20.3. Creating a New Working Directory

1. If required to use a different directory to store working files, create a directory as follows:

   ~]# mkdir /rsyslog

2. Install utilities to manage SELinux policy:

   ~]# yum install policycoreutils-python

3. Set the SELinux directory context type to be the same as the /var/lib/rsyslog/ directory:

   ~]# semanage fcontext -a -t syslogd_var_lib_t /rsyslog

4. Apply the SELinux context:

   ~]# restorecon -R -v /rsyslog
   restorecon reset /rsyslog context unconfined_u:object_r:default_t:s0->unconfined_u:object_r:syslogd_var_lib_t:s0

5. If required, check the SELinux context as follows:

   ~]# ls -Zd /rsyslog
   drwxr-xr-x. root root system_u:object_r:syslogd_var_lib_t:s0   /rsyslog

6. Create subdirectories as required. For example:

   ~]# mkdir /rsyslog/work/

   The subdirectories will be created with the same SELinux context as the parent directory.

7. Add the following line in /etc/rsyslog.conf immediately before it is required to take effect:

   $WorkDirectory /rsyslog/work

   This setting will remain in effect until the next WorkDirectory directive is encountered while parsing the configuration files.

20.4.3. Managing Queues

All types of queues can be further configured to match your requirements. You can use several directives to modify both action queues and the main message queue. Currently, there are more than 20 queue parameters available, see Section 20.12, “Online Documentation”. Some of these settings are used commonly, others, such as worker thread management, provide closer control over the queue behavior and are reserved for advanced users. With advanced settings, you can optimize rsyslog‘s performance, schedule queuing, or modify the behavior of a queue on system shutdown.

⁠Limiting Queue Size

You can limit the number of messages that queue can contain with the following setting:

$objectQueueHighWatermark number

Replace object with MainMsg or with Action to use this option to the main message queue or for an action queue respectively. Replace number with a number of enqueued messages. You can set the queue size only as the number of messages, not as their actual memory size. The default queue size is 10,000 messages for the main message queue and ruleset queues, and 1000 for action queues.

Disk assisted queues are unlimited by default and can not be restricted with this directive, but you can reserve them physical disk space in bytes with the following settings:

$objectQueueMaxDiscSpace number

Replace object with MainMsg or with Action. When the size limit specified by number is hit, messages are discarded until sufficient amount of space is freed by dequeued messages.

⁠Discarding Messages

When a queue reaches a certain number of messages, you can discard less important messages in order to save space in the queue for entries of higher priority. The threshold that launches the discarding process can be set with the so-called discard mark:

$objectQueueDiscardMark number

Replace object with MainMsg or with Action to use this option to the main message queue or for an action queue respectively. Here, number stands for a number of messages that have to be in the queue to start the discarding process. To define which messages to discard, use:

$objectQueueDiscardSeverity priority

Replace priority with one of the following keywords (or with a number): debug (7), info (6), notice (5), warning (4), err (3), crit (2), alert (1), and emerg (0). With this setting, both newly incoming and already queued messages with lower than defined priority are erased from the queue immediately after the discard mark is reached.

⁠Using Timeframes

You can configure rsyslog to process queues during a specific time period. With this option you can, for example, transfer some processing into off-peak hours. To define a time frame, use the following syntax:

$objectQueueDequeueTimeBegin hour

$objectQueueDequeueTimeEnd hour

With hour you can specify hours that bound your time frame. Use the 24-hour format without minutes.

⁠Configuring Worker Threads

A worker thread performs a specified action on the enqueued message. For example, in the main message queue, a worker task is to apply filter logic to each incoming message and enqueue them to the relevant action queues. When a message arrives, a worker thread is started automatically. When the number of messages reaches a certain number, another worker thread is turned on. To specify this number, use:

$objectQueueWorkerThreadMinimumMessages number

Replace number with a number of messages that will trigger a supplemental worker thread. For example, with number set to 100, a new worker thread is started when more than 100 messages arrive. When more than 200 messages arrive, the third worker thread starts and so on. However, too many working threads running in parallel becomes ineffective, so you can limit the maximum number of them by using:

$objectQueueWorkerThreads number

where number stands for a maximum number of working threads that can run in parallel. For the main message queue, the default limit is 1 thread. Once a working thread has been started, it keeps running until an inactivity timeout appears. To set the length of timeout, type:

$objectQueueWorkerTimeoutThreadShutdown time

Replace time with the duration set in milliseconds. Without this setting, a zero timeout is applied and a worker thread is terminated immediately when it runs out of messages. If you specify time as -1, no thread will be closed.

⁠Batch Dequeuing

To increase performance, you can configure rsyslog to dequeue multiple messages at once. To set the upper limit for such dequeueing, use:

$objectQueueDequeueBatchSize number

Replace number with the maximum number of messages that can be dequeued at once. Note that a higher setting combined with a higher number of permitted working threads results in greater memory consumption.

⁠Terminating Queues

When terminating a queue that still contains messages, you can try to minimize the data loss by specifying a time interval for worker threads to finish the queue processing:

$objectQueueTimeoutShutdown time

Specify time in milliseconds. If after that period there are still some enqueued messages, workers finish the current data element and then terminate. Unprocessed messages are therefore lost. Another time interval can be set for workers to finish the final element:

$objectQueueTimeoutActionCompletion time

In case this timeout expires, any remaining workers are shut down. To save data at shutdown, use:

$objectQueueTimeoutSaveOnShutdown time

If set, all queue elements are saved to disk before rsyslog terminates.

20.4.4. Using the New Syntax for rsyslog queues

In the new syntax available in rsyslog 7, queues are defined inside the action() object that can be used both separately or inside a ruleset in /etc/rsyslog.conf. The format of an action queue is as follows:

action(type="action_type" queue.size="queue_size" queue.type="queue_type" queue.filename="file_name")

Replace action_type with the name of the module that is to perform the action and replace queue_sizewith a maximum number of messages the queue can contain. For queue_type, choose disk or select from one of the in-memory queues: direct, linkedlist or fixedarray. For file_name specify only a file name, not a path. Note that if creating a new directory to hold log files, the SELinux context must be set. See Section 20.4.2, “Creating a New Directory for rsyslog Log Files” for an example.

⁠

Example 20.13. Defining an Action Queue

To configure the output action with an asynchronous linked-list based action queue which can hold a maximum of 10,000 messages, enter a command as follows:

action(type="omfile" queue.size="10000" queue.type="linkedlist" queue.filename="logfile")

The rsyslog 7 syntax for a direct action queues is as follows:

*.* action(type="omfile" file="/var/lib/rsyslog/log_file
     )

The rsyslog 7 syntax for an action queue with multiple parameters can be written as follows:

*.* action(type="omfile"
              queue.filename="log_file"
              queue.type="linkedlist"
              queue.size="10000"
     )

The default work directory, or the last work directory to be set, will be used. If required to use a different work directory, add a line as follows before the action queue:

global(workDirectory="/directory")

⁠

Example 20.14. Forwarding To a Single Server Using the New Syntax

The following example is based on the procedure Procedure 20.1, “Forwarding To a Single Server” in order to show the difference between the traditional sysntax and the rsyslog 7 syntax. The omfwd plug-in is used to provide forwarding over UDP or TCP. The default is UDP. As the plug-in is built in it does not have to be loaded.

Use the following configuration in /etc/rsyslog.conf or create a file with the following content in the /etc/rsyslog.d/ directory:

*.* action(type="omfwd"
      queue.type="linkedlist"
      queue.filename="example_fwd"
      action.resumeRetryCount="-1"
      queue.saveOnShutdown="on"
      target="example.com" port="6514" protocol="tcp"
     )

Where:

• queue.type="linkedlist" enables a LinkedList in-memory queue,
• queue.filename defines a disk storage. The backup files are created with the example_fwd prefix, in the working directory specified by the preceding global workDirectory directive,
• the action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if server is not responding,
• enabled queue.saveOnShutdown="on" saves in-memory data if rsyslog shuts down,
• the last line forwards all received messages to the logging server, port specification is optional.

GNU Automake

Reference