Forwarding Issues:
Issue:
Heavy forwarder Paused Data Flow with Warning: "The TCP output processor has paused the data flow".
WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest="xxx.xxx.xxx.xxx" inside output group default-autolb-group from host_src="xxx" has been blocked for blocked_seconds="10." This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
Root cause:
Indexing queue is spiking up to 100% frequently. The indexer stopped all listening ports when it was not able to write data to the hot bucket.
WARN TcpInputProc - Stopping all listening ports. Queues blocked for more than 300 seconds.
Solution:
Ensure that the disk subsystem of the indexer can handle a minimum of 800 average Input/Output Operations Per Second (IOPS) & Restart the indexer instance.
File name: $SPLUNK_HOME/etc/system/local/server.conf
[queue=indexQueue]
maxSize = 4096KB