Forwarding Issues:
Issue: Forwarder stopped sending data (after upgrading from 8.x to 9.0.x)
ERROR TcpOutputQ Unexpected event
Root cause:
1. Mostly after upgrading from 8.2.x to 9.0.x, you can see this errors.
2. If useACK set to true and batch mode is ON (default) with Splunk 9.0, there is a possibility of hitting the error "Unexpected event ID.".
07-21-2023 11:49:18.729 +0200 ERROR TcpOutputQ [1215026 TcpOutEloop] - Unexpected event>
07-21-2023 11:49:18.729 +0200 ERROR TcpOutputQ [1215026 TcpOutEloop] - Unexpected event>
07-21-2023 11:49:18.730 +0200 ERROR TcpOutputQ [1215026 TcpOutEloop] - Unexpected event>
Solution:
Workaround: Either set useACK="false" or autoBatch="false.
Issue is fixed by 9.0.3 patch.
Note: After 9.0.3 upgrade, you will still see benign “Unexpected event ID” log message. However, there should not be following log messages.
"Invalid ACK received from indexer" (or) "Got unexpected ACK with eventid"