Blog categorized as Splunk Troubleshooting Use Cases
Issue:
Latest bundle from Deployment Server is not reflected in the forwarder.
Root cause:
On the Forwarder side, the conf files & few scripts were kept open in an editor. So forwarder was not able to overwrite those open files.
Solution:
Close all the opened files from the forwarder side &...
14.06.24 07:23 PM - Comment(s)
Issue:
Timestamp is not extracted properly for the summary index (JSON data copied using collect command)
index=web_idx ......
| collect index="web_summary_idx"
Root cause:
Default sourcetype of the copied events is "stash". It may not be able to recognize the timestamps i...
14.06.24 07:23 PM - Comment(s)
Issue:
Log file monitored is enabled, but data is not getting indexed
Root cause:
Another file also has the first 256 characters same.
Note: If multiple files, have the first 256 same, then Splunk will consider only one file.
Solution:
Add “crcSalt=<text>“ under the respective stanz...
14.06.24 07:23 PM - Comment(s)
Issue:
Sourcetype is not working for CSV data, when it is added to Indexer.
Root cause:
For Structured data, the parsing happens at Universal Forwarder(UF) itself, so the sourcetype should be configured at UF itself.
Solution:
Configure the sourcetype in Universal Forwarder for CSV Data alone...
14.06.24 07:23 PM - Comment(s)
Issue:
Monitored file suddenly stops being monitored/indexing.
Root cause:
File growth rate is higher than indexing or forwarding rate. (maxKBps value is too low)
[thruput]
maxKBps = 384
maxKBps = 384
Solution:
Change the maxKBps value to a higher value(increase the maxKBps value upto 500) & R...
14.06.24 07:23 PM - Comment(s)
Issue: CSV Event Data is extracting twice
Root Cause: Sourcetype is having improper configuration (e.g: Here, INDEXED_EXTRACTIONS & KV_MODE have the same value)
Solution: Make sure any one of the property is having none (INDEXED_EXTRACTIONS & KV_MODE)
Root Cause: Sourcetype is having improper configuration (e.g: Here, INDEXED_EXTRACTIONS & KV_MODE have the same value)
Solution: Make sure any one of the property is having none (INDEXED_EXTRACTIONS & KV_MODE)
14.06.24 07:02 PM - Comment(s)
Issue: Query is not working for a particular users, but for others
Root Cause: Users missing access to the extracted field (e.g: in this case, the startup_code field is extracted using props)
Solution: Enable access to the respective users & validate it again with users
Root Cause: Users missing access to the extracted field (e.g: in this case, the startup_code field is extracted using props)
Solution: Enable access to the respective users & validate it again with users
14.06.24 07:00 PM - Comment(s)
Issue: Data from particular forwarder has +3 hrs difference for all events
Root cause: NTP Protocol Sync was not configured for that Forwarder.
Solution: Configure NTP Protocol sync on that particular server & restart the forwarder.
Root cause: NTP Protocol Sync was not configured for that Forwarder.
Solution: Configure NTP Protocol sync on that particular server & restart the forwarder.
14.06.24 06:58 PM - Comment(s)
Issue-9: Users were onboarded to security groups, but not able to
login to Splunk.
Root Cause: The new security group was not configured/mapped with none of the roles in Splunk.
Solution: Configure the security group & map it with the correct role..
login to Splunk.
Root Cause: The new security group was not configured/mapped with none of the roles in Splunk.
Solution: Configure the security group & map it with the correct role..
14.06.24 06:43 PM - Comment(s)
Issue-8: Indexed data (more than 35 days) is removed even before the retention period (90 days)
Root Cause: The max size of an index (maxTotalSizeMB) is reached before the retention period in seconds (frozenTimePeriodInSecs).
Solution: Increase the maxTotalSizeMB to a big number, say 100 GB, based on ...
Root Cause: The max size of an index (maxTotalSizeMB) is reached before the retention period in seconds (frozenTimePeriodInSecs).
Solution: Increase the maxTotalSizeMB to a big number, say 100 GB, based on ...
14.06.24 06:42 PM - Comment(s)
Issue-7:
Latest entries in lookups were not detected by queries
Root Cause:
Root Cause:
Lookup with the same name present in another app.
Solution:
Solution:
Remove the extra copy from the other app
14.06.24 06:40 PM - Comment(s)
Issue:
Dashboard changes are not reflecting in all the Search heads
Root Cause:
Root Cause:
Previous changes of the dashboard were available in the local folder of all the search heads.
Solution:
Solution:
Remove the local folder copy from all the search heads & do a debug refresh.
Then disable edit acces...
14.06.24 06:38 PM - Comment(s)
Issue-5:
Json Data - Events are truncated, only half of the event is indexed
Root Cause:
Root Cause:
Event size is too big - more than 10,000 bytes
Solution:
Solution:
Configure “TRUNCATE” property in the sourcetype, as shown below in props.conf
[yoursourcetype]
TRUNCATE = 99999
[yoursourcetype]
TRUNCATE = 99999
13.06.24 10:53 PM - Comment(s)
Scenario-2: Event timestamp & _time field have a difference, which is the same for all events
Root Cause: Data is coming from a different time zone, but the Forwarder is configured with the UTC timezone. So that difference in the timezones reflected during the search
Solution: Configure the Time z...
Root Cause: Data is coming from a different time zone, but the Forwarder is configured with the UTC timezone. So that difference in the timezones reflected during the search
Solution: Configure the Time z...
13.06.24 10:52 PM - Comment(s)
Issue:
Event timestamp & _time field do not match
Scenario-1:
Scenario-1:
All events are showing the same Timestamp (current timestamp)
Root Cause:
Root Cause:
Event timestamp is not in the standard format.
Solution:
Solution:
Configure your custom timestamp format in the sourcetype, as shown below in props.conf file
13.06.24 10:46 PM - Comment(s)
Scenario-6: Except 1 source other sources are sending logs from same forwarder.
Root Cause: Data input was disabled during previous deployment.
Solution: Enable the data input & check if the data starts flowing for the respective source.
Root Cause: Data input was disabled during previous deployment.
Solution: Enable the data input & check if the data starts flowing for the respective source.
13.06.24 10:46 PM - Comment(s)
Issue: Forwarder is not sending data
Scenario-5: Forwarder error logs say, Indexers are not reachable
Root Cause:
9997 port is not open in Security Groups inbound & outbound rules.
Solution:
Enable 9997 port in the outbound of the Forwarder & inbound of the Indexer
Root Cause:
9997 port is not open in Security Groups inbound & outbound rules.
Solution:
Enable 9997 port in the outbound of the Forwarder & inbound of the Indexer
13.06.24 10:43 PM - Comment(s)
Data Forwarding Issues:
Issue: Forwarder is not sending data.
Scenario 3: None of the logs are available in Indexer for last few hours, even Indexer’s internal logs.
Root Cause:
The Indexer Disk size was full.
Solution:
Increase the Indexer Disk/Storage size
(or)
Remove unwanted data from the Indexer, Like ...
13.06.24 10:26 PM - Comment(s)
Data Forwarding Issues:
Issue: Forwarder is not sending data
Scenario-4: Intermittent data flow -But the source is producing the data live
Root Cause:
Sudden burst of incoming data, leads to stall the forwarder queue
Solution:
This is expected in few scenarios where the data sources are too many. One way ...
13.06.24 10:22 PM - Comment(s)
Data Forwarding Issues:
Issue: Forwarder is not sending data
Scenario 2: Forwarder is running & was sending data earlier, has all the access
Root Cause:
Forwarder had indexer IP in outputs.conf, the corresponding indexer was down
Solution:
Bring up the Indexer
(or)
Enable Indexer...
13.06.24 10:22 PM - Comment(s)
Categories
Tags
- Data Replication Issues
- Data Frowarding Issues
- App Deployment Issues
- Indexers down 3-member cluster
- Two Indexers down 3-member cluster
- All Indexers down 3-member cluster
- Search Heads down 3-member cluster
- Two Search heads down 3-member cluster
- All Search Heads down 3-member cluster
- Deployer is down
- Monitoring Console down
- Deployment Server down
- Universal Forwarder down
- License Server/Manager down
- Decide number Search Heads & Indexers
- how to choose forwarder (UF or HF)
- Intermediate Forwarder (IF))
- Licence Forwarders
- can't use single instance with huge size instead separatly
- How splunk stores Indexes
- Possible open flat files in Notepad++
- Possible rename index
- clean index splunk instances & indexer cluster
- Migrate index 1 splunk server to another splunk server
- Backup splunk configuration/data
- upgrade splunk enterprise
- upgrade splunk enterprise which clustered
- Upgrade the Splunk Universal Forwarder
- Deploy apps to search head clusters
- Deploy Apps indexer Cluster
- Connect Forwarders to indexer cluster
- Difference between Heavy forwarders & HTTP Event collector
- Cluster Master is down Then need a cluster master
- colocation of splunk components
- meant colocation splunk components
- Deployment server to distribute apps to search head cluster & indexer
- reduce licence in splunk
- why need license master/server
- Replication Factor lower than search factor
- Timestamp Issues
- Event Truncation Issues
- Retention Plicy Issues
- SAML Issues
- Parsing Issues
- File Monitoring Issue
- Configuration Issue
- Summary Index Issues
- Deployment Issues