Murugan
Blog by Murugan
Issue-8: Indexed data (more than 35 days) is removed even before the retention period (90 days)
Root Cause: The max size of an index (maxTotalSizeMB) is reached before the retention period in seconds (frozenTimePeriodInSecs).
Solution: Increase the maxTotalSizeMB to a big number, say 100 GB, based on ...
Root Cause: The max size of an index (maxTotalSizeMB) is reached before the retention period in seconds (frozenTimePeriodInSecs).
Solution: Increase the maxTotalSizeMB to a big number, say 100 GB, based on ...
14.06.24 06:42 PM - Comment(s)
Issue-7:
Latest entries in lookups were not detected by queries
Root Cause:
Root Cause:
Lookup with the same name present in another app.
Solution:
Solution:
Remove the extra copy from the other app
14.06.24 06:40 PM - Comment(s)
Issue:
Dashboard changes are not reflecting in all the Search heads
Root Cause:
Root Cause:
Previous changes of the dashboard were available in the local folder of all the search heads.
Solution:
Solution:
Remove the local folder copy from all the search heads & do a debug refresh.
Then disable edit acces...
14.06.24 06:38 PM - Comment(s)
Issue-5:
Json Data - Events are truncated, only half of the event is indexed
Root Cause:
Root Cause:
Event size is too big - more than 10,000 bytes
Solution:
Solution:
Configure “TRUNCATE” property in the sourcetype, as shown below in props.conf
[yoursourcetype]
TRUNCATE = 99999
[yoursourcetype]
TRUNCATE = 99999
13.06.24 10:53 PM - Comment(s)
Scenario-2: Event timestamp & _time field have a difference, which is the same for all events
Root Cause: Data is coming from a different time zone, but the Forwarder is configured with the UTC timezone. So that difference in the timezones reflected during the search
Solution: Configure the Time z...
Root Cause: Data is coming from a different time zone, but the Forwarder is configured with the UTC timezone. So that difference in the timezones reflected during the search
Solution: Configure the Time z...
13.06.24 10:52 PM - Comment(s)
Issue:
Event timestamp & _time field do not match
Scenario-1:
Scenario-1:
All events are showing the same Timestamp (current timestamp)
Root Cause:
Root Cause:
Event timestamp is not in the standard format.
Solution:
Solution:
Configure your custom timestamp format in the sourcetype, as shown below in props.conf file
13.06.24 10:46 PM - Comment(s)
Scenario-6: Except 1 source other sources are sending logs from same forwarder.
Root Cause: Data input was disabled during previous deployment.
Solution: Enable the data input & check if the data starts flowing for the respective source.
Root Cause: Data input was disabled during previous deployment.
Solution: Enable the data input & check if the data starts flowing for the respective source.
13.06.24 10:46 PM - Comment(s)
Issue: Forwarder is not sending data
Scenario-5: Forwarder error logs say, Indexers are not reachable
Root Cause:
9997 port is not open in Security Groups inbound & outbound rules.
Solution:
Enable 9997 port in the outbound of the Forwarder & inbound of the Indexer
Root Cause:
9997 port is not open in Security Groups inbound & outbound rules.
Solution:
Enable 9997 port in the outbound of the Forwarder & inbound of the Indexer
13.06.24 10:43 PM - Comment(s)
Data Forwarding Issues:
Issue: Forwarder is not sending data.
Scenario 3: None of the logs are available in Indexer for last few hours, even Indexer’s internal logs.
Root Cause:
The Indexer Disk size was full.
Solution:
Increase the Indexer Disk/Storage size
(or)
Remove unwanted data from the Indexer, Like ...
13.06.24 10:26 PM - Comment(s)
Data Forwarding Issues:
Issue: Forwarder is not sending data
Scenario 2: Forwarder is running & was sending data earlier, has all the access
Root Cause:
Forwarder had indexer IP in outputs.conf, the corresponding indexer was down
Solution:
Bring up the Indexer
(or)
Enable Indexer...
13.06.24 10:22 PM - Comment(s)
Data Forwarding Issues:
Issue: Forwarder is not sending data
Scenario-4: Intermittent data flow -But the source is producing the data live
Root Cause:
Sudden burst of incoming data, leads to stall the forwarder queue
Solution:
This is expected in few scenarios where the data sources are too many. One way ...
13.06.24 10:22 PM - Comment(s)
Data Forwarding Issues
Issue-3:
The Forwarder is not sending data
Scenario-1:
No logs are sent by forwarder, even internal logs
Root Cause:
The forwarder is stopped due to a source server restart done by the Application team
Solution:
Enable boot start for forwarder, this will make sure t...
13.06.24 10:15 PM - Comment(s)
Categories
Tags
- Data Replication Issues
- Data Frowarding Issues
- App Deployment Issues
- Indexers down 3-member cluster
- Two Indexers down 3-member cluster
- All Indexers down 3-member cluster
- Search Heads down 3-member cluster
- Two Search heads down 3-member cluster
- All Search Heads down 3-member cluster
- Deployer is down
- Monitoring Console down
- Deployment Server down
- Universal Forwarder down
- License Server/Manager down
- Decide number Search Heads & Indexers
- how to choose forwarder (UF or HF)
- Intermediate Forwarder (IF))
- Licence Forwarders
- can't use single instance with huge size instead separatly
- How splunk stores Indexes
- Possible open flat files in Notepad++
- Possible rename index
- clean index splunk instances & indexer cluster
- Migrate index 1 splunk server to another splunk server
- Backup splunk configuration/data
- upgrade splunk enterprise
- upgrade splunk enterprise which clustered
- Upgrade the Splunk Universal Forwarder
- Deploy apps to search head clusters
- Deploy Apps indexer Cluster
- Connect Forwarders to indexer cluster
- Difference between Heavy forwarders & HTTP Event collector
- Cluster Master is down Then need a cluster master
- colocation of splunk components
- meant colocation splunk components
- Deployment server to distribute apps to search head cluster & indexer
- reduce licence in splunk
- why need license master/server
- Replication Factor lower than search factor
- Timestamp Issues
- Event Truncation Issues
- Retention Plicy Issues
- SAML Issues
- Parsing Issues
- File Monitoring Issue
- Configuration Issue
- Summary Index Issues
- Deployment Issues