Isilon code upgrade (Rolling/Simultaneous)

Restart dnsiq service before starting upgrade process. Below are the steps.

Restarting dnsiq is one of the best practice before performing node reboots

# isi_for_array -s 'ps awux|grep dnsiq_d|grep -v grep'

isi_for_array -s 'ps awux|grep dnsiq_d|grep -v grep'

root    1560  0.0  0.5 54740  7064  ??  Ss    1:03PM   0:00.78 /usr/sbin/isi_dnsiq_d

Check for the status 'Ss' is normal (sleep) - 'Is' was Idle, this would be something to look for.

WCPU @ 0:00.78 is very little, but the process just started, check to see if it differs from the dnsiq daemons from other nodes that do not serve smartconnect requests.

To restart the process (if this was from node 1 - specified with the -x 1 to exclude that node from the command):

# isi_for_array -x 1 killall isi_dnsiq_d

Implementation Steps:

1) Verify Health status of Isilon cluster

isi status -v

Resolve errors and warning if any exists.

2) Restart the cluster before performing the upgrade. Reboot One node at a time. Restarting the cluster prior to performing the upgrade flushes the caches, frees memory, clears unused connections, and allows you to find and address issues that could impact the upgrade. 

3) Verify available space on the cluster is greater than 10% and the available space on each node is greater than 5%. 

4) Verify hardware status with isi_for_array -s "isi_hw_status". 

5) Resolve errors and outstanding events

view events: isi events list

6) verify boot drive status

isi_for_array -s "gmirror status |grep -i degraded'

7) verify data devices status

isi devices |grep -v healthy

8) Collect cluster logs

isi_gather_info

8) Upload code on to Isilon Using WINSCP to /ifs/data 

9) Open a secure shell (SSH) connection to any node in the cluster and log in using the

root account

10) In the OneFS command-line interface, run the following command, specifying the

installation image file name:

md5 

The command returns an MD5 checksum value.

11) . Compare the MD5 checksum value recorded from the EMC Online Support site against

the MD5 checksum generated from the command-line interface.

12) Perform pre upgrade health check Using command isi update --check only

The system returns a list of any warnings or errors that are present, and then the

following prompt appears:

Please specify the image to update:

13) At the prompt, type the absolute path or URL to the image location and then press

ENTER.

14) Disable CMP and TPSTAT Pods

Rolling

15) Run upgrade isi update --rolling --manual

Simultaneous

15) Run upgrade isi update

16) Enable CMP and TPSTAT Pods

15) Verify version

Validation steps:

Post upgrade steps

1. Check the new version number of the cluster:

uname -a

2. View the status of the cluster and make sure all your nodes are operational:

isi status -D -w

3. Ping all of the cluster's internal and external interfaces to verify network connectivity

and to help verify that SmartConnect works correctly.

4. Review the list of events and address any critical events:

isi events list -w

5. Check the status of jobs and resume the jobs that you paused for the upgrade:

isi job status view

6. Verify your network interfaces:

isi networks list interfaces

7. Verify your subnets:

isi networks list subnets --verbose

8. Verify your pools:

isi networks list pools --verbose

9. Review the cluster's other log files to check for stray problems:

cat /var/log/messages

 10.Review the list of SyncIQ jobs:

isi sync jobs list

11.Check the SyncIQ job reports:

isi sync reports list

12.Review the list of your scheduled snapshots:

isi snapshot schedules list

13.Check the cluster's input and output; type Ctrl-C when you are done:

isi statistics system --nodes --top

14.Check the devices in the nodes to validate the status of your drives:

isi_for_array -s "isi devices | grep -iv healthy"

15.Check your global SMB settings:

isi smb settings global view

16.Check the status of the firmware to ensure that the firmware is consistent across

nodes:

isi firmware status

17.Make sure that all your licenses carried over and remain up to date:

isi license

18.Check the status of your authentication providers to make sure they remain active:

isi auth status --verbose