Data Ingestion Issues
Issue:
- Both internal logs and data from UF get delayed for about 30 seconds. (Event Indexing delay)
- Difference between _time and _indextime is about 30 seconds.
Root Cause:
- UF processes a larger number of files than it typically does.
- Increased size of the fishbucket & the processing of fishbucket.
- UF spent a lot of time traversing the fish bucket in checkpoint() routine, which caused a TCP sending issue.
Solution:
- Tune the parameter "file_tracking_db_threshold_mb" of the [inputproc] stanza to a lower value in limits.conf
file name: /opt/splunk/etc/system/local/limits.conf
[inputproc]
file_tracking_db_threshold_mb = 2