About Us

Citrus Consulting Services is the Consulting and the Transformation Services arm of Redington Gulf.

Sunday – Thursday: 9:00AM–6:00PM (Sales), Sunday – Saturday: 24×7 / 365 (Support) E.O#3, Ground Floor, Building 01 Dubai Internet City, P.O Box 501 761 Dubai, UAE (+971) 04 516 1500
(+966) 11 462 5323
info@citrusconsulting.com

Citrus Consulting Services

ITSI Version upgrade from 4.4.x to 4.7.x

Planning to upgrade your ITSI to mitigate the products End of Support? Are you facing KVStore issues during upgrade? If yes, this guide will provide you information on how to perform an ITSI upgrade with the KVStore failure.

This guide takes example of ITSI upgrade from version 4.4.x to 4.7.x. In this example, KVStore is rolling back to v4.6.0 from v4.7.2 whereas the updated_itsi_kvstore_version at 4.6.0.

Pre-upgrade actions on the SHC:

  • Increased kvstore max_size_per_batch_save_mb = 750MB
  • Assessed the KVStore and SHC health and performance baselines

Upgrade steps:

  1. Check the search head cluster status.
    Command: splunk show shcluster-status
  2. Disable the Rules Engine.
  3. Extract the ITSI installation package on deployer.
    Directory: /opt/splunk/etc/shcluster/apps/
  4. Take backup of /opt/splunk/etc/apps/SA-ITOA/lib/itsi/upgrade/migration_handlers_4_7_0.py
  5. Edit the original file.
    File: /opt/splunk/etc/apps/SA-ITOA/lib/itsi/upgrade/migration_handlers_4_7_0.py
    and go to line 236,
    dataSources = gt_definition.get(‘dataSources’)
    edit that line such that it becomes:
    dataSources = gt_definition.get(‘dataSources’, {})
  6. Save the changes in file.
  7. Run the following command from SH from CLI
    Command:
    curl -k -u admin -X POST https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_migration_queue –data ‘{

        “skip_pre_checks”: [“KVSIZECHK”, “RECHK”],

        “is_migration_triggered”: true,

        “skip_local_failure”: true,

    }’ \

–header “Content-Type: application/json”

  1. Get the status again making sure the migration is running now:

Command: curl -k -u admin -X GET https://localhost:8089/servicesNS/nobody/SA-ITOA/migration/info

  1. Let’s go to the Splunk web UI, but don’t go to ITSI app.

Instead, we go to the main Search app and let’s run this search manually:

index=_internal “itsi.migration” UI source=”*/splunk/var/log/splunk/itsi_migration_queue.log” sourcetype=itsi_internal_log

  1. When you see this message with the above search:

Data transformation has completed, saving back to KV store…

Then do another search:

index=_internal “itsi.migration” source=”*/splunk/var/log/splunk/itsi_migration_queue.log” sourcetype=itsi_internal_log

When you see the following message (the log will show 4.7.1, not 4.7.2):

Migration bulk save to KV store successfully completed, updating the version to 4.7.1.

  1. On the master node, place a copy of SA-IndexCreation in $SPLUNK_HOME/etc/master-apps/
  2. Use splunk apply cluster-bundle to push Index Creation app to peer nodes.
  3. Upgrade SA-ITSI-Licensechecker and SA-UserAccess on license master.
  4. The Splunk IT Service Intelligence upgrade process is now complete. Objects disabled during the upgrade process are automatically reenabled. The ITSI shows the following message: IT Service Intelligence upgrade has completed successfully.

Priyanka is a technology consultant with a demonstrated history of working in the operations and service industry. A Splunk certified core consultant skilled in Splunk, Linux, DevOps and managing client applications. Priyanka inherently understands that the customer is the single most valuable asset an organization can have, and is driven by the unrelenting pursuit of customer-driven focus, ideals and user experience. She has experience with customers across multiple industries and domains and often aided with expediting their journey to Digital Transformation.

Post a Comment

eighteen − 3 =