Wednesday, August 19, 2015

Isilon InsightIQ code upgrade from 3.0 to 3.2



Direct Upgrade from 3.0 to 3.2 is not supported. code needs to be upgraded from 3.0 to 3.1 and from 3.1 to 3.2
Upgrade from 3.0 to 3.1
Install the dependencies
1) Unzip the file
tar xvfz iiq_3.1_upgrade_dependencies.tar.gz
2) Install the offline depencies
./install_dependencies
3) Run the upgrade skipping the process off connecting to centos as depencies are already installed
sudo yum upgrade --noplugins --disablerepo= */home/administrator/isilon-insightiq-3.1.0.0078-1.x86_64.rpm
Upgrade from 3.1 to 3.2
Upgrading from 3.1 to 3.2 EMC provided code in .sh format,(previous codes are in .rpm format)
One of the issue we experienced using winscp to copy the file to InsightIQ VM is, md5sum values are getting
modified while on the copy process, which causing the upgrade to fail throwing the errors like invalid checksum
To copy the file, use either ftp or copy the file to Isilon cluster and then to InsightIQ using scp protocol
1) Run upgrade sudo sh install-insightiq-3.2.1.0001.sh
2) Run datastore upgrade iiq_datastore_upgrade

Saturday, August 8, 2015

Isilon : Rolling upgrade failure

While doing code upgrade, there are cases where network interruption cause the loss on connection to the Cluster.  If we are in middle of the code upgrade, there is no way reconnecting to the cluster take us to the step where the upgrade left.

In these cases, user needs verify the list of nodes upgraded completely, kill the upgrade process, and restart the upgrade, which skip the upgrade for nodes that completed and proceeds with nodes on previous versions.

Below are the troubleshooting commands could help during this scenarios.

isi update --rolling --manual    -  Initiates rolling upgrade, manual option ask the user for confirmation before rebooting every node
isi update --check-only  Pre upgrade health check
cd /var/log  
ls -l update*
cat update_handler_2015-07-23_14:57:10.txt  log file
isi_for_array -s ps awux | grep update                list the running update process
isi_for_array -s killall -9 update                             Kill the current running update process
isi_for_array date                                                     dispalys date of the cluster
isi_readonly
isi readonly
isi auth error 54
isi_for_array -n2 killall -9 isi_upgrade_d           kill the upgrade process
isi_for_array -s ps awux | grep isi_for_array    
ssh isceist01-2
isi_for_array -s uname -a | awk  '{print $4}'
isi update --rolling --manual
isi_for_array -s ps awux | grep upgrade_d
isi_for_array -s killall -9 isi_upgrade_d             Kill the upgrade process
isi_for_array -s ps awux | grep upgrade_d      list the current running processes by upgrade filter
isi update --rolling --manual                               Initiate rolling upgrade