Isilon : Sync IQ scheduler memory leak issue

Current Isilon versions  7.* have a memory leak issue which causes the sync scheduler to run out of it's allocated 512 max memory and go into hung state. This state will stops all jobs from initializing weather incremental or full. Current code doesn't trigger any alerts during this outage until some one manually verify.

To avoid enter into the outage situation follow the below steps:


Isilon has developed a script for monitoring and trigger email alerts once sync scheduler memory utilization reaches certain threshold, so that sync process can be restarted before go into the hung state.

Below are the commands to verify the memory usage manually.

# isi_for_array -s ps awxu | grep isi_migr_sched | grep -v grep |awk '{print $1 $6}'    This command give the current memory usage across all nodes in the cluster

For example, if we want to be notified when memory reaches 470 MB, script is available with EMC support. edit the threshold values to 470 MB from the script.

Once we receive the email,  run the following commands to reset the memory.

isi sync settings modify --service=off

isi sync settings modify --service=on

This command will reset the memory value to around 76 MB

Note: Script from Isilon has to be executed every time the node gets rebooted.

** Permanent fix will be expected to be on Riptide version (8.0) which is expected in Q4