Zivaro Blog

Splunk Product Timestamp Issue Solution

What is the issue? Splunk has identified a time-sensitive issue that affects all current versions of Splunk Enterprise, Splunk Light, and Splunk Cloud. This issue has potential significant impact on data ingestion – including causing inaccurate, unsearchable, or prematurely-deleted data. Beginning January 1, 2020, un-patched Splunk platform instances will be unable to recognize timestamps from […]

What is the issue?2019-Splunk-Corp-Logo-w-tag-K-Horizontal

Splunk has identified a time-sensitive issue that affects all current versions of Splunk Enterprise, Splunk Light, and Splunk Cloud. This issue has potential significant impact on data ingestion – including causing inaccurate, unsearchable, or prematurely-deleted data.

Beginning January 1, 2020, un-patched Splunk platform instances will be unable to recognize timestamps from events where the date contains a two-digit year. This means data that meets these criteria will be indexed with incorrect timestamps.

Beginning September 13, 2020 at 12:26:39 PM Coordinated Universal Time (UTC), un-patched Splunk platform instances will be unable to recognize timestamps from events with dates that are based on Unix time, due to incorrect parsing of timestamp data.

There is no method to correct the timestamps after the Splunk platform has ingested the data when the problem starts. If you ingest data with an un-patched Splunk platform instance beginning on January 1, 2020, you must patch the instance and re-ingest that data for its timestamps to be correct.

What is the Impact?

This issue affects ALL un-patched Splunk platform instance types, on any operating system:

  • Splunk Cloud
  • Splunk Light
  • Splunk Enterprise
    • Indexers, clustered or not
    • Heavy forwarders
    • Search heads, clustered or not
    • Search head deployers
    • Deployment servers
    • Cluster masters
    • License masters
  • Splunk universal forwarders, under the following known conditions:
    • When they have been configured to process structured data, such as CSV, XML, and JSON files, using the INDEXED_EXTRACTIONS setting in props.conf
    • When they have been configured to process data locally, using the force_local_processing setting in props.conf
    • When they have been configured with a monitor input, and that input subsequently encounters an unknown file type

Patched Versions of Splunk Enterprise not affected by this issue:

·      6.6.12.1

·      7.2.9.1

·      7.0.13.1

·      7.3.3

·      7.1.10

·      8.0.1

 

What is the Solution?

Resolution of this issue involves updating the datetime.xml file on impacted Splunk instances. Updating of this file can be achieved via one of the following methods:

  • Deploy a Splunk configuration app via the customer’s current app deployment mechanism (most likely the Splunk Deployment Server) to temporarily modify the datetime.xml file to avoid the stated impact. This app is freely available for download directly from Splunk.
  • Download and deploy an updated version of the datetime.xml that contains necessary modifications to address and mitigate impact.
  • Upgrade impacted Splunk instances to a patched version of Splunk Enterprise/Splunk Universal Forwarder (see versions above)
  • Manually modify the datetime.xml file on impacted Splunk instances

Full details on this issue and possible resolutions can be found at:
https://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/FixDatetimexml2020

Zivaro Recommended Actions

Zivaro is recommending the following actions for its Splunk customers. Please note, specific actions to resolve this issue depend on the architecture of the customer’s Splunk environment.

Single-instance (“All-in-one”) Splunk Environment:

Customers with a single-instance Splunk environment have environments with all or most Splunk roles contained in one single Splunk Enterprise instance. For these customers, Zivaro recommends the following actions:

  1. Download the modified/patched version of the datetime.xml file from Splunk. The file can be downloaded here:
    https://download.splunk.com/products/ingest2020/datetime.zip
  2. Unpack/Unizp the downloaded file. It will create a directory named ‘cfg’. Inside of the ‘cfg’ directory is the modified datetime.xml file.
  3. Using your operating system management utilities, copy the modified datetime.xml file to the $SPLUNK_HOME/etc directory. ‘$SPLUNK_HOME’ is the installation path of the Splunk application on your system – for example C:\Program Files\Splunk or /opt/splunk.
  4. If you are using Splunk Universal Forwarders to collect data from hosts, and you manage your Unviersal Forwarders via the Splunk Deployment Server, perform steps 5 through 9.
  5. Download the Splunk archive file containing apps containing the resolution for the timestamp interpretation issue and extract it. The archive file can be downloaded here:
    https://download.splunk.com/products/ingest2020/apps_date_patch_props_v2.zip
  6. The resulting directory will contain two apps:
    1. all_date_patch_props
    2. idxc_date_patch_props
  7. Copy the all_date_patch_props directory and all of its contents into the $SPLUNK_HOME/etc/deployment-apps directory
  8. Within the Splunk Deployment Server UI, either identify or create a new server class that contains all Unviersal Forwarders. If you need to create a new server class, give it a meaningful name such as “All_Forwarders_Datetime_Fix”. Use a wildcard (‘*’ character) to have the server class match all deployment clients.
  9. Assign the all_date_patch_props app to your server class. Be sure to enable the ‘Restart splunkd’ option for this app within the Deployment Server UI. This will ensure that each Universal Forwarder restarts its splunkd process upon receiving the all_date_patch_props app.

Distributed and Clustered Splunk Environments:

  1. Download the Splunk archive file containing apps containing the resolution for the timestamp interpretation issue and extract it. The archive file can be downloaded here:
    https://download.splunk.com/products/ingest2020/apps_date_patch_props_v2.zip
  2. The resulting directory will contain two apps:
    1. all_date_patch_props
    2. idxc_date_patch_props
  3. Place the “all_date_patch_props” app in the appropriate filesystem location on the following Splunk server roles:
    1. Deployment Server – the app should be placed in the $SPLUNK_HOME/etc/deployment-apps directory
    2. Search Head Cluster Deployer (if using Search Head Clustering) – the app should be placed in the $SPLUNK_HOME/etc/shcluster/apps directory
    3. Any non-clustered Splunk search head or indexer in your environment that is not managed by the Deployment Servrer – the app should be placed in $SPLUNK_HOME/etc/apps
  4. Place the “idxc_date_pach_props” app in the appropriate filesystem location on the following Splunk server roles:
    1. Indexer Cluster Master Node – the app should be placed in $SPLUNK_HOME/etc/master-apps directory
  5. On the Splunk Deployment Server, do the following:
    1. Within the Splunk Deployment Server UI, either identify or create a new server class that contains all Splunk Deployment Clients that connect to the Deployment Server. If you need to create a new server class, give it a meaningful name such as “All_Deployment_Clients_Datetime_Fix”. Use a wildcard (‘*’ character) to have the server class match all deployment clients.
    2. Assign the all_date_patch_props app to your server class. Be sure to enable the ‘Restart splunkd’ option for this app within the Deployment Server UI. This will ensure that each Universal Forwarder restarts its splunkd process upon receiving the all_date_patch_props app.
  6. If the environment leverages Splunk Search Head Clustering, complete the following:
    1. On the Search Head Deployer, perform a configuration bundle push using the “splunk apply shcluster-bundle” command. Full instructions are available here: https://docs.splunk.com/Documentation/Splunk/latest/DistSearch/PropagateSHCconfigurationchanges#Deploy_a_configuration_bundle
  7. If the environment leverages Splunk Indexer Clustering, complete the following:
    1. On the Master Node, validate and distribute the configuration bundle to the indexer peers. Full instructions are available here: https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Updatepeerconfigurations#Distribute_the_configuration_bundle

If you have questions, need assistance or would like additional consultation, please contact one of Zivaro’s Splunk Consultants listed below:

Scott DeMoss, Consulting Sales Engineer – sdemoss@zivaro.com

Chris Greenwood, Sales Engineer – cgreenwood@zivaro.com

 

3900 E Mexico Avenue, Suite 1000,
Denver, CO 80210