Scripts for running commands accross multiple - Linux

for i in `cat delete_share`; do echo $i; server_export server_2 -unexport -perm $i; done

for i in `cat delete_share`; do echo $i; server_umount server_2 -perm /$i

for i in `cat delete_mountpoint`; do echo $i; server_mountpoint server_2 -delete /$i; done

for i in `cat delete_fs`; do echo $i; nas_fs -delete $i

Isilon - Sync list performance issue

Performance issue may related to synciq reports cluster is retaining.  The recommended amount of reports should be below 8K to have expected WebUI and CLI performance regarding sync policy management.

Total number of reports generated by synciq -

# find /ifs/.ifsvar/modules/tsm/sched/reports -name "report-[0-9]*.gc" |wc -l

determine how many reports are older than a certain period so we can clean these up..

 # find /ifs/.ifsvar/modules/tsm/sched/reports -name "report-[0-9]*.gc" -Btime +1w|wc -l

adjust  to +1d for older than a day, or +2w for older than two weeks.. etc..

command to remove SyncIQ reports that are older than a week:

 find /ifs/.ifsvar/modules/tsm/sched/reports -name "report-[0-9]*.gc" -Btime +1w -exec rm -f {} \;

$ grep "scheduler.schedule" siq-policies.gc|cut -d "|" -f 2|grep -v '""'|sed 's/"//g'|sort |uniq -c

      2 Every 1 days every 15 minutes from 00:00 to 23:59

      1 Every 1 days every 1 minutes from 00:00 to 23:59

      1 Every 1 weeks on Sunday at 1:00 AM

      4 when-source-modified

To modify maximum number of reports for every policy use following commands

To change the max reports: isi sync policy modify <policyname> --max_reports=200

Example: isi sync policy modify test_sync_prod --max_reports=200

After max number is set, response from sync list is very faster.

Isilon - Patch Install

Please  verify Readme file that comes with patch every time because patch procedures are different for every type.


1. Open an SSH connection on any node in the cluster and log in using the
    "root" account.

2. Copy the patch-xxxxx file to the /ifs/data directory on the cluster.

3. Run the following command to change to the /ifs/data directory:
 
   cd /ifs/data

4. To extract the patch file, run the following command:

   tar -zxvf patch-xxxxx.tgz

5. To install this patch, run the following command:

   isi pkg install patch-xxxx.tar

6. To verify that this patch is installed, run the following command:
 
   isi pkg info

7. Verify that patch-135046 appears in the list of installed packages.

********************************************************************************

PERFORMING A ROLLING REBOOT

After the patch is installed, manually reboot each node in succession.

1. Open an SSH connection on any node in the cluster and log in using the "root"
   account.

2. Shut down the first node in the cluster by running the following command:

   shutdown -r now
 
3. To monitor the progress of the reboot, run the following command:

   isi status
 
4. Wait for the node to successfully reboot.

5. Repeat steps 2 - 4 for each remaining node.

********************************************************************************
REMOVING THIS PATCH

If you need to remove this patch, complete the steps below.

IMPORTANT!
Read INSTALLATION/REMOVAL IMPACTS before performing this procedure.

1. To delete this patch, run the following command:

   isi pkg delete patch-xxxx

2. To verify that this patch was removed, run the following command:

   isi pkg info

3. Verify that patch-xxxx does not appear in the list of installed packages.

Isilon : Enable and Disable Isilon jobs

Disable Isilon running jobs

1. Open an SSH connection on any node in the cluster and log on using the "root" account.

2. Run the following command to disable the Collect job:

isi job types modify collect --enabled false

3. When asked if you are sure you want to modify the job, type yes.

4. Run the following command to disable the MultiScan job:

isi job types modify multiscan --enabled false

5. When asked if you are sure you want to modify the job, type yes

Enable Isilon jobs

1. Open an SSH connection on any node in the cluster and log on using the "root" account.

2. Run the following command to enable the Collect job:

isi job types modify collect --enabled true

3. When asked if you are sure you want to modify the job, type yes.

4. Run the following command to enable the MultiScan job:

isi job types modify multiscan --enabled true

5. When asked if you are sure you want to modify the job, type yes

Isilon : Restart DNS service

Restarting dnsiq is one of the best practice before performing node reboots

# isi_for_array -s 'ps awux|grep dnsiq_d|grep -v grep'

isi_for_array -s 'ps awux|grep dnsiq_d|grep -v grep'

root    1560  0.0  0.5 54740  7064  ??  Ss    1:03PM   0:00.78 /usr/sbin/isi_dnsiq_d

Check for the status 'Ss' is normal (sleep) - 'Is' was Idle, this would be something to look for.

WCPU @ 0:00.78 is very little, but the process just started, check to see if it differs from the dnsiq daemons from other nodes that do not serve smartconnect requests.

To restart the process (if this was from node 1 - specified with the -x 1 to exclude that node from the command):

# isi_for_array -x 1 killall isi_dnsiq_d

Once the process is killed, it will be automatically started by Master Control Process (MCP).

Isilon code upgrade (Rolling/Simultaneous)

Restart dnsiq service before starting upgrade process. Below are the steps.

Restarting dnsiq is one of the best practice before performing node reboots

# isi_for_array -s 'ps awux|grep dnsiq_d|grep -v grep'

isi_for_array -s 'ps awux|grep dnsiq_d|grep -v grep'

root    1560  0.0  0.5 54740  7064  ??  Ss    1:03PM   0:00.78 /usr/sbin/isi_dnsiq_d

Check for the status 'Ss' is normal (sleep) - 'Is' was Idle, this would be something to look for.

WCPU @ 0:00.78 is very little, but the process just started, check to see if it differs from the dnsiq daemons from other nodes that do not serve smartconnect requests.

To restart the process (if this was from node 1 - specified with the -x 1 to exclude that node from the command):

# isi_for_array -x 1 killall isi_dnsiq_d

Implementation Steps:

1) Verify Health status of Isilon cluster

isi status -v

Resolve errors and warning if any exists.

2) Restart the cluster before performing the upgrade. Reboot One node at a time. Restarting the cluster prior to performing the upgrade flushes the caches, frees memory, clears unused connections, and allows you to find and address issues that could impact the upgrade. 

3) Verify available space on the cluster is greater than 10% and the available space on each node is greater than 5%. 

4) Verify hardware status with isi_for_array -s "isi_hw_status". 

5) Resolve errors and outstanding events

view events: isi events list

6) verify boot drive status

isi_for_array -s "gmirror status |grep -i degraded'

7) verify data devices status

isi devices |grep -v healthy

8) Collect cluster logs

isi_gather_info

8) Upload code on to Isilon Using WINSCP to /ifs/data 

9) Open a secure shell (SSH) connection to any node in the cluster and log in using the

root account

10) In the OneFS command-line interface, run the following command, specifying the

installation image file name:

md5 

The command returns an MD5 checksum value.

11) . Compare the MD5 checksum value recorded from the EMC Online Support site against

the MD5 checksum generated from the command-line interface.

12) Perform pre upgrade health check Using command isi update --check only

The system returns a list of any warnings or errors that are present, and then the

following prompt appears:

Please specify the image to update:

13) At the prompt, type the absolute path or URL to the image location and then press

ENTER.

14) Disable CMP and TPSTAT Pods

Rolling

15) Run upgrade isi update --rolling --manual

Simultaneous

15) Run upgrade isi update

16) Enable CMP and TPSTAT Pods

15) Verify version

Validation steps:

Post upgrade steps

1. Check the new version number of the cluster:

uname -a

2. View the status of the cluster and make sure all your nodes are operational:

isi status -D -w

3. Ping all of the cluster's internal and external interfaces to verify network connectivity

and to help verify that SmartConnect works correctly.

4. Review the list of events and address any critical events:

isi events list -w

5. Check the status of jobs and resume the jobs that you paused for the upgrade:

isi job status view

6. Verify your network interfaces:

isi networks list interfaces

7. Verify your subnets:

isi networks list subnets --verbose

8. Verify your pools:

isi networks list pools --verbose

9. Review the cluster's other log files to check for stray problems:

cat /var/log/messages

 10.Review the list of SyncIQ jobs:

isi sync jobs list

11.Check the SyncIQ job reports:

isi sync reports list

12.Review the list of your scheduled snapshots:

isi snapshot schedules list

13.Check the cluster's input and output; type Ctrl-C when you are done:

isi statistics system --nodes --top

14.Check the devices in the nodes to validate the status of your drives:

isi_for_array -s "isi devices | grep -iv healthy"

15.Check your global SMB settings:

isi smb settings global view

16.Check the status of the firmware to ensure that the firmware is consistent across

nodes:

isi firmware status

17.Make sure that all your licenses carried over and remain up to date:

isi license

18.Check the status of your authentication providers to make sure they remain active:

isi auth status --verbose