StorageGrid notes

Day 1 Training Balancing Clinet access 1) No load balancing 2) DNS Round Robin 3) Connection Load balancer (CLB) service 4) Load balacer service. ( Recommended) Archive Node - VM -> About to extinct Optimal Storage Nodes Administrative Domain Controller ( ADC) Gateway ADC SSM LDR DMV Query ADC ADC returns list Gateway talk to LDR on node based on first one in the list When writes comes in 1) Goes to optimal 2) copy to second optimal 3) send ack when read request 1) Query the node 2) Query Cassandra database 3) Find which node has the data 4) send databack to node received request and send to user 5) LDR finds the node SG6060, SGF6060 SAS All Flash ILM protection policy rules Erasure coding Data pieces Parity Pieces Order of sequence 1) Dual commit on write 2) Ack 3) run ILM policy 4) ILM policy is long term protection ILM policy is the protection policy Workflow during object replication ILM engine in the LDR service evaluates ILM policy rules and determines that an object should be replicated 1) ILM engine sends a replicaiton request to the optimal Dest SN 2) Destination Storage Node LDR retrives object from SN 3) Destination storage node LDR writes to object storage 4) Destation storage node LDR sends object metadata to the DDS service 5) DDS service replicates the metadata and CMN service - runs on primary. configuration management while pimary is down -> you can not make any config changes, upgrades Chapter 2 Storage GRID Grid Manager 1) StorageGrid Topology tree 1) GRID Health 2) Information lifecycle management(ILM) activity 3) client activity Grid administrators use Grid manager to create 1) Storage tenant accounts, 2) manage ILM policiies and rules 3) configure grid nodes and services 4) perform maintenance Grid Topology Tree 1) Grid Site Grid Node Node services Service components Analyzing storage node SSM service components Storage Node Server status Monitor (SSM) service state number of threads CPU Load amount of memory consumed by the service Link Cost Cost of communication between data center sites ADC uses link cost to determine the Grid node to retrivet he object 0 - 100 Object Transformation Grid Options 1) compressed (LZW algorithm) default off 2) encrypted ( AES-128 or AES-256) default off 3) Segmented 4) object hasing by default SHA-1 5) prevent client modify ( default off) x-amz-server-side-encryption in the HTTP header to enable encryption per object Transfermation Option: Segmentation single control block identifier ( CBID) object container segment container that lists the header information of all segments as content default max segment size is 1 GB StorageGRID object durability options Dual commitment Stored object hashing prevent client modify Dual commit Stored Object hashing Fingerprinting is used to protect the integrity of stored objects object hash information stored in content management database (CMDB) Distributed Data Store(DDS) service ILM evaluation ILM Policy Object Ingest Prevent client modify : is a system wide setting StorageGRID Administrators root account Configuring Identity Federation enable identity federation 2 certificates. one for grid management interface Storage nodes and API gateway nodes Obtaining the StorageGRID CA Certificate Day 2 Training Storage Tenant Administration: create Tenant as management unit Tenants created based on management Metadata Grid admin create tenants Tenant admin create buckets Bucket contains data volumes under the nodes are filesystems part of physical Tenant account based on entities YOu can setup access between the buckets. relationship can be setup 1) Creating a Tenant account 1) Tenant 2) Create Allow platform services is disabled Tenant authentication local user root account for Tenant Grid admin knows tenant admin creds Tenant admin can manage and change password, Tenant admin can block Grid admin from managing AS grid admin, password can be modified for Tenant Grid admin configures access to the bucket Once you login in to tenant admin, you can configure Identiy federation Grid admin knows Teanant admin password Tenant manager webpage To long into tenant manager webpage add account ID in URL ex: https:///?accountid= URL for Tenant manager can be accessed from StorageGRID webpage root accounts are at Grid level root accounts are at Tenant level as well Tenant Manager Dashboard depends on Quota utilization will be displayed S3 policy - Allows Group of users access / manage S3 buckets in specific Tenant you can create multiple groups to manage different set of buckets to manage Group Policy: { "Statement": [ { "Effect": "Allow", "Action": "s3:*" , "Resrouce": "arn:aws:s3:::*" } } } S3 Access Keys each user of an S3 tenant account must have an access key to store and retrieve objects Grid admins cannot create bucket Only Tenant admins create bucket Quota is set at Tenant level not bucket level S3 API only Tenant admin manage buckets Access key is like username password compliance can be done at Grid level or Bucket level no Tenant level Access keys Control access to the bucket Connect to Tenant manager Create Access Keys and set expiration time Access Key ID: Secret Access Key: either create 1 key for multiple buckets or create 1 key for each bucket Bucket access control: path-style URL: You will never have to change certificate. path style URL requests do not include the bucket name in the domain name virtua-hosted-syle URL requests include the bucket name in the domain name http://bucket_name.host_name.domain_name buckets and objects are resources that are accessed by using a unique resource identifier Cloud Mirror Replication Endpoints Universal resource identifier ( URI) Destination host and port For a StorageGRID destination API Gateway node for storage node port 8082 https://dc1-g1.dem.netapp.com:8082 URN: destination S3 bucket For AWS as destination arn:aws:s3:::bucket_name for StorageGRID as destination urn:sgws:s3:::Bucket_name Controlling access to buckets and Objects Bucket policies: are configured using the S3 REST API control access from specific users and groups to a bucket and to the objects in the bucket apply to only one bucket but possibly to multiple users and groups Group Policies: are configured using tenant Manager or the tenant managment API Give the group members access to speicific resources Apply to only one group but possibly to mulitple buckets Bucket configuration options: Creating compliant buckets: U will need to enable at Grid level to enable at bucket lelve Bucket details: Name Region compliance enable compliance check box retention period after retention period. Legal hold : if you check, data becomes undeletable, non modifiable when you uncheck data can be deletable as per retention policy consistency level: Default consistency level default update database Grid wide 1 copy on B and 1 copy on C database knows where data is then ILM policy kicks in dual commit. , ack, and then ILM rule when update comes in, it creates new object, does not modify existing Object consistency is perfromed eventually for Strong site: increase database replicaiton with in site for Strong Global: increase database replication across GRID which could lead to poor performance Last access time update: default disable Plastform services: compute network storage lambda serverless compute Notifications: OCR Example Day 3 Training ILM policies rules Grid manager defines protection Grid manager talk to tenants and how it want to protect Grid manager only one that makes rules ILM rules ILM rules ->. Filter and type of protoection Filter -> what you want to protect Protection -> 1 copy in A and 1 copy in B rule1.tenant1.bucket1. 1 copy in A and 1 copy in B ILM Policy collection of rules policies are prioritized Rule1 rule2 order of rule matters Filter -> Filter based on anything protection -> how many copies and where Filters identify which rule applies to an object basic rules advanced rules 11.5 -> 11.3 -> one tenant per rule Object Last Access Time updates Advanced Filter: Metadata Type Ingest Time Last access time Key Object Size (MB) user metadata Location Constraint Object Tag ( S3 only) - recommended with S3 Key-value pair key:value are defined by application owner storage admin usse key value but not create key vaulue Rule put object in Storage pool Storage pool is collection of nodes with similar attributes Site A: A-Cap-Pool A-perf-Pool Stoage poolA Storage PoolB Storage poolC Storage Grades: default 0 performance 1 capacity 2 secure 3 If you do not assign grade default 0, all are treate dsame capex - Capital expenditure opex - operating expenditure 2 different budgets Erasure Coding: regionally distributed erasure coding (6+3+ 6 data 3 parity 1 GB = 1.5 gb stored (erasure coding) 1 gb = 3 gb stored (multiple copies) erasure coding drawback is latency re-assemble packets copy packets from remote site during the write you will not experice any latency for read you will experience any latency Each site will have thier own gateway Gateway node for each site -> 2 of them per site ILM Policy creation: Define Storage Grades (optional) Assign Storage grades to Storage Nodes Configure Storage Pools Define S3 Regions Create ILM Rules Configure the proposed ILM policy Activate the ILM policy ILM rule object placement one copy in DC1 one copy in DC2 one copy in DC3 ILM Rule ingest Behaviour 11.5 balanced is default Strict, Balanced and dual commit After policy is created, Add rules to it always test your policy Use extreme caution when modifying ILM policies and rules Always simulate and validate a proposed ILM policy before activating the policy when a new ILM policy is activated, ILM policy rules are appllied Any time new rule is made, simulate and test Verify Object Placement Lookup section helps to troubleshoot performance issues it finds where the object is located rebuild objects Object metadata lookup. Object. <>. Lookup Day 4 Training Monitoring Unknown - most severe current alarms DCM Alarm Class Types There are 3 classes of alarms Default alarms Global custom alarms Custom alarms Node level alarms Custom Grid level Grid Manager Attribute Charts Grafana Auto support Audit logs - gathered by Admin node not very human readable netapp has tools that are more readable command line tool that can be readable Off, Error, Normal, Debug (Trace logging) audit logs are stored in /var/local/audit/export Audit-expalin tool for readable audit-explain audit.log audit-explain 2019-08-12.txt.gz audit node can make the audit log directory accessible to client hosts To share the autdit log files run CIFS utility start CIFS configuration utility : config_cifs.rb For NFS config_nfs.rb add-audit-share add-ip-to-share validate-config Monitoring to Stop and Start service you need to run from CLI storagegrid-status Server manager to stop, start, restart services Stopping and Starting all storageGRID Node services Stop all node services /etc/init.d/servermanager stop Start. /etc/init.d/servermanager start restart all node services /etc/init.d/servermanager restart Physical appliances are extremely robust For one particular service service status service start service stop force anode service to stop sv -w force-stop < service_name> restart. service restart Maintenance procedures: 1) system expansion 2) decomissioning of nodes 3) Node recovery recovery package needed to recover any node sgws-recovery-package-id-revision.zip must download and create recovery package upon any changes on topology store copies of recovery package in 2 safe, secure locations download recovery package->. maintenance ->> recovery package enter provisioning passphrase and start download configuration management node ( CMN service) Grid Tasks status Storage Grid System expansions: compute /var/local/rangedb/ 0 CMDB/ object storage 1. object storage 2 object storage N. object storage Storage Node can have as many as 16 object datastores ( rangedb) To add an object store complete following steps add_rangedbs.rb to discover and format the new object stores Adding Nodes one node will be added at a time Upgrading StorageGRID software create recovery package before upgrade create recovery package after upgrade Install Hotfixes:

NetApp - FlexVol

Flex Volumes

1. data volumes
2. basic building block of data management
3. snapshots are taken at volume level
4. specific to a data SVM

CLI commands

mounting on client

• mkdir /mnt/vol1
• mount -t nfs <netappinterface>:/vol1 /mnt/vol1

Creating volume

• volume create -vserver demo -volume vol10 -aggregate aggr4 -size 50MB
• volume mount -vserver demo -volume vol10 -junction-path /vol4
• volume modify -vserver demo -volume vol10 -policy default

• volume create -vserver demo -volume vol50 -aggregate aggr20 -size 500MB -policy default -junction-path /vol50  --> all pieces in on command

Isilon Tasks

Isilon Tasks

Initial Installations

Capacity Expansions

Adding new capacity

Configuring Network

Creating Static, Dynamic pools

Create NFS shares

Create SMB shares

Create Quotas

Create SyncIQ Policies

Create SnapshotIQ

Configure SyncIQ failover and Failback

Dedupe

Simultaneous code upgrade

Rolling code upgrade

Node upgrades

disk upgrades

InsightIQ code upgrade

InsightIQ Virtual appliance deployment

InsightIQ Linux installation

Migrations

rsync

isi_vol_copy

isi_vol_copy_vnx

EMCOPY

Pure Storage

Dashboard

Hardware

M10 - 30TB
M20 - 250 TB
M50 - 500 TB
M70 - 1.5 PB
X70 - 1.1. PB

Operating System

Version: 3.0


Health status
latency
IOPS
Bandwidth

Storage

1) Create host - host1
2) Add FC ports
3) create volumes
4) create voluems
5) create clone -> copy snapshot to volume


Replications

1) Create Protection group
2) add members to the group
3) add ESX cluster
4) add targets  - > add pure storage array targets
5) define replication policy

SRM
site recovery manager to connect Pure storage to vmware

Analysis  for graphs

System

alerts configration, SNMP, system time, directory service,

Host Connections

Plugins - vCenter plug ins

VSS - application consistent snapshots for the windows



Pure storage architecture



Monitoring of the Pure storage through  PURE1 Cloud ( similar to ESRS)


Pure 3.0 features


Non-disruptive - >  software updates
                                Capacity expansion
                                performance expansions


ZeroSnap using vmware xCopy feature


More security through  Always on Encryption  AES

VAAI xcopy
VAAI thin prov
vSphere plugin

Support model - PureCloud

IBM XIV run guide

IBM XIV

Models:

GUI Tools: XIV storage management



Tasks: 

Provision Storage

1) Add Hosts
2) Add HBA ports to Hosts
3) Verify Host connectivity



Use snapshots

1) right click on data volume
2) Create snapshot
     snapshot is the pointers to original 1MB data blocks
3) Snapshot restore, copying the snapshot pointers to original data

Mirror -  Synchronous, Asynchronous

Monitor and view snapshots

Nutanix Hardware

Version list

NX-8000
NX-6000
NX-3000
NX-1000

Dell EMC XC series
Lenevo  HX
IBM CS
 

VEDA - helpful links

These are just helpful links. Credits to the owner

Step by step process

http://www.virtualizetips.com/2010/06/28/install-vsphere-esx-4-0-with-eda-deployment-appliance/

https://www.experts-exchange.com/articles/2369/Installing-ESX-through-EDA-multi-VMware-installation.html

PowerCLI - Mount NFS datastores

Get-Cluster | Get-VMHost | New-Datastore -Nfs -Name <name>  -Path <NFS share path> -NfsHost <NFS server>

Get-VMHost | New-Datastore -nfs -Name <>  -path <> -NfsHost <>

Expect script to connect to the switches

Credits to the owner.

Original link reference:

https://supportforums.cisco.com/discussion/11553001/script-automate-tasks
http://www.corecoding.com/cisco-expect-script_c32.html

MDS switch migrations

MDS switch migration from 9148 switches to 9396 switches


1) Verify configuration on old switches

      show run
      show port-channel database
      show flogi database
      show interface brief

2)  Take backup from Fabric Interconnects
 
3)   Log into old MDS switch A / B side

4)  Turn off port-channel on MDS switch

     config t
     interface port-channel 11
     np interface po 11

5)  Once port-channel interface is turned off, remove the cables from old switch, and Connect cables to old MDS switches

       Configure port-channel on New MDS switches
     
        config t
        interface fc1/1
        switchport description
        port-license acquire
        channel-group <> force
        no shutdown

        repeat the config on remaining ports

         Interface port-channel<>
         Channel mode active
         switchport mode F
         switchport description <>
         switchport rate-mode dedicated
 
        Add ports to portchannel

        config t
        interface po <>
        interface fc 1/1-8


Once connections are enabled,  Disable the port channel  on Other side of switch and enable the port channel.


Repeat the same config on the B side




Additional commands:

disable port-channel
no interface port-channel <>

config t
inteface fc1/15
channel-group 15
no channel-group 15

Restart Insight IQ service

Restarting InsightIQ service

Monitoring halted : OperationalError
Fatal: Pipe() failed: Too many open files in system
InsightIQ cannot resolve this error automatically

InsightIQ service needs to be restarted to resolve this issue.

iiq_restart

Isilon InsightIQ code upgrade from 3.0 to 3.2

Direct Upgrade from 3.0 to 3.2 is not supported. code needs to be upgraded from 3.0 to 3.1 and from 3.1 to 3.2

Upgrade from 3.0 to 3.1

Install the dependencies

1) Unzip the file

tar xvfz iiq_3.1_upgrade_dependencies.tar.gz

2) Install the offline depencies

./install_dependencies

3) Run the upgrade skipping the process off connecting to centos as depencies are already installed

sudo yum upgrade --noplugins --disablerepo= */home/administrator/isilon-insightiq-3.1.0.0078-1.x86_64.rpm

Upgrade from 3.1 to 3.2

Upgrading from 3.1 to 3.2 EMC provided code in .sh format,(previous codes are in .rpm format)

One of the issue we experienced using winscp to copy the file to InsightIQ VM is, md5sum values are getting

modified while on the copy process, which causing the upgrade to fail throwing the errors like invalid checksum

To copy the file, use either ftp or copy the file to Isilon cluster and then to InsightIQ using scp protocol

1) Run upgrade sudo sh install-insightiq-3.2.1.0001.sh

2) Run datastore upgrade iiq_datastore_upgrade

Isilon : Rolling upgrade failure

While doing code upgrade, there are cases where network interruption cause the loss on connection to the Cluster.  If we are in middle of the code upgrade, there is no way reconnecting to the cluster take us to the step where the upgrade left.

In these cases, user needs verify the list of nodes upgraded completely, kill the upgrade process, and restart the upgrade, which skip the upgrade for nodes that completed and proceeds with nodes on previous versions.

Below are the troubleshooting commands could help during this scenarios.

isi update --rolling --manual    -  Initiates rolling upgrade, manual option ask the user for confirmation before rebooting every node

isi update --check-only  Pre upgrade health check

cd /var/log  

ls -l update*

cat update_handler_2015-07-23_14:57:10.txt  log file

isi_for_array -s ps awux | grep update                list the running update process

isi_for_array -s killall -9 update                             Kill the current running update process

isi_for_array date                                                     dispalys date of the cluster

isi_readonly

isi readonly

isi auth error 54

isi_for_array -n2 killall -9 isi_upgrade_d           kill the upgrade process

isi_for_array -s ps awux | grep isi_for_array    

ssh isceist01-2

isi_for_array -s uname -a | awk  '{print $4}'

isi update --rolling --manual

isi_for_array -s ps awux | grep upgrade_d

isi_for_array -s killall -9 isi_upgrade_d             Kill the upgrade process

isi_for_array -s ps awux | grep upgrade_d      list the current running processes by upgrade filter

isi update --rolling --manual                               Initiate rolling upgrade 

LDAP Performance Troubleshooting - Isilon

Recently we were complained about authentication issue, where some of the users are unable to login to the Isilon cluster, where login requests are getting time out. This is kind of issue, where some users are able to login to every Isilon individual node with IP address as well as SSIP and remaining DNS addresses. On the other side few users are able to login to couple of nodes and while others not.

No changes performed on either side of Isilon cluster as well as LDAP server.  Below are the troubleshooting steps performing to figure out the location and root cause of the issue.

1) Login to each and individual nodes separately with IP address, SSIP, and pool DNS addresses.

2) Tried with different LDAP users by all step one ways

3) Checked with LDAP team whether they are receiving the LDAP requests, where by verifying the logs as well as Splunk log repositories.

4) Restarted authentication services

5) Verified if any changes performed on Isilon cluster as well as LDAP server by checking the time when issue was started.

6) Verified if whether configuration on Isilon cluster is reflecting accross all nodes, since login to some of them working fine while others not.

7) Listed all the Physical components where the request and response flows in the network. Like

Isilon -> Nexus 5K -> Nexus 7K -> F 5 load balancers -> Nexus 7K -> Fabric Interconnect - > ESXi hosts  -> Virtual LDAP machines and vice versa response from LDAP virtual machines to Isilon clusters

8) Captured network traces on Isilon cluster as well as on Nexus 7K switch while doing couple of tests to see the flow of LDAP request

I will provide the commands for taking TCP dumps on Isilon cluster and troubleshooting helpful commands at the end.

9) Used wireshark to verify the tcpdump pcap captures.

10) Once pcaps are opened in wireshark, filter the frames by decode to LDAP to minimize the output to LDAP frames 

11) Decode can be performed to any type like TCP, UDP, LDAP to minimize the output to our preferred format for ease of troubleshooting.

Other ways to filter is 

HTTPS is eq to "ABC"

LDAP eq to "ABC"

Where ABC is user id or any filter

right click on any frame and see the TCP flow to check complete flow happend during particular session.  

Red color code indicates the request from Isilon cluster, and green represent the response from Server.

12) We opened both captures performed at Isilon cluster as well as 7K and compared same session from both pcaps.

The 7K switch showed it received the responses from LDAP servers but has lot of retransmissions and frames in red color.  where Isilon pcaps missed all the responses.  It just waited for 100 seconds before sending unbind request and received response and successfully closed the connection.

13) F5 engineer verified and confirmed all packets are being placed on the wire which going to 7K switch

14) That way isolated, Isilon cluster and LDAP server, F5 from the issue list as both are trying to communicate, Since response reached all the way back to 7K which left 2 devices on the network

7K and 5K switches.

15) Logged in to 7K switch and started shutting off one port at a time, connecting to 5K switch and tested the logins to Isilon cluster

16) Tested with 3 ports, the logins started working after shutting down the 4 th port.

17) Verified the configuration for the port interface and found it has some CRC errors and others which eating the packets.


Below are the commands which helps towards the troubleshooting process. Use as required.

Collecting tcp dumps.

tcpdump -i vlan0 -s 0 -w /ifs/data/Isilon_Support/node2/ssh_login.pcap host 10.10.10.10 

tcpdump -i vlan0 -s 0 -w /ifs/data/Isilon_Support/node2/node2_ldap.pcap host 10.10.10.10

tcpdump -i vlan0 -s 0 -w /ifs/data/Isilon_Support/node2/node2_mapping.pcap &

isi_for_array 'tcpdump -s 0 -i lagg0 -w /ifs/data/Isilon_Support/$(date +%m%d%Y)/`hostname`.$(date +%m%d%Y_%H%M%S).lagg0.cap &'

Verify active connections on Isilon cluster

isi_for_array netstat -an|grep 10.10.10.10

isi_for_array ifconfig |grep 10.10.10.10

Other commands

tail -f /var/log/lsassd.log       Authentication log file

ps aux |grep lsass                  Current running processes

ifcofnig -a

ls -lrth

isi auth ldap list                     List ldap servers configured on Isilon clsuter

isi auth mapping token --user=abce --zone=1    Verify mapping information for LDAP user

isi auth mapping token --user=abcd

isi auth mapping flush                                         Flush the cache

isi auth mapping flush --all                                 Flush the cache

isi_for_array -n3,4,5 isi auth mapping token --user=abcd

isi_for_array -n3 isi auth mapping token --user=abcd

ldapsearch -H ldap://10.10.10.10 -b 'ou=enterprise,o=abc,c=us' 

ldapsearch -H ldap://10.10.10.10 -x -b "" -s base objectclass="*"  supportedControl

ldapsearch -H 10.10.10.10 -x -b "" -s base objectclass="*"  supportedControl

ldapsearch -H 10.10.10.10 -x -b ,ou=enterprise,o=abc,c=xyz

ldapsearch -H 10.10.10.10 -x -b ou=enterprise,o=abc,c=xyz

ldapsearch -H ldap://10.10.10.10 -x -b ou=enterprise,o=abc,c=xyz

ldapsearch -H ldap://10.10.10.10:2389 -x -b ou=enterprise,o=abc,c=xyz

/usr/likewise/bin/lwsm list

ldapsearch -H ldap://10.10.10.10:2389 -x -b ou=enterprise,o=abc,c=xyz

ldapsearch -x LLL -H ldap://10.10.10.10:2389  -b 'ou=enterprise,o=abc,c=xyz' -D abcd

date; isi auth mapping token --user=abcd

ls -l /ifs/data/Isilon_Support/node2/node2_mapping.pcap

ls -lh /ifs/data/Isilon_Support/node2/node2_mapping.pcap

ping -c 1000 10.10.10.10 -W 1

ping -c 1000 -W 1 10.10.10.10 

ping -c 1000 10.10.10.10 

isi services -a

isi_for_array "ps auxww | grep lsass | grep -v grep"

ldapsearch -x -h abc.xyz.com -p 2389 -D "abcd" -W -b "" -s base "objectclass =*"

ldapsearch -x -h abc.xyz.com -p 2389 -D "abcd" -W -b "" -s base "objectclass=*"

isi_for_array "ps auxww | grep lsass | grep -v grep"

isi_for_array "isi_classic auth ads cache flush --all"    Flush the cache

isi_for_array "isi_classic auth mapping flush --all"      Flush the cache

isi_for_array "killall lsassd -9"                                     Kill the lsassd authentication deamon, which whill be automatically restarted by MCP master control process

ldapsearch -h abc.xyz.ldap.com -p 2389 -D "uid=abc,ou=def,ou=enterprise,o=hij,c=abd" -W  -b "ou=enterprise,o=hij,c=abd" 

ldapsearch -h abc.xyz.ldapserver.com -p 2389 -D "uid=abc,ou=def,ou=enterprise,o=hij,c=abd" -W  -b "ou=enterprise,o=hij,c=abd"  

isi auth ldap list -v

 ldapsearch -h abc.xyz.ldapserver.com -p 2389 -D "uid=abc,ou=def,ou=enterprise,o=hij,c=abd" -W  -b "ou=enterprise,o=hij,c=abd"  "(&(objectClass=posixAccount)(uidNumber=1234))"

 ldapsearch -h abc.xyz.com -p 2389 -D "uid=abc,ou=def,ou=enterprise,o=hij,c=abd" -W  -b "ou=enterprise,o=hij,c=abd"  "(&(objectClass=posixAccount)(uidNumber=1234))"

isi auth mapping dump| less

isi auth mapping dump| wc -l

isi auth mapping token --user=abc

isi auth mapping token --uid 1234

less /var/log/messages

tail /var/log/messages

isi auth log-level

isi auth log-level --set=debug                Set the log level to debug

isi auth log-level --set=warning             Set the log level to warning

ping -c 10 abc.xyz.ldap.com

traceroute abc.xyz.ldapserver.com

isi auth status

isi status

isi auth ldap view Primary

less /var/log/lsassd.log

isi auth mapping token abcdef

isi auth users view abcdef

less /var/log/lwiod.log

less /var/log/messages

less /var/log/lsassd.log

cd /etc/openldap

ls

less ldap.conf

less ldap.conf.default

less /ifs/.ifsvar/main_config_changes.log

less /var/log/lsassd.log

isi_for_array -s isi auth ldap.conf

isi auth status

isi auth ldap view --provider-name=Primary | grep "Group Filter:" | grep "User Filter:"

isi auth ldap view --provider-name=Primary 

isi_for_array -s isi auth ldap view --provider-name=Primary | grep "User Filter:"

isi_for_array -s isi auth ldap view --provider-name=Primary | grep "Group Filter:"

isi_for_array -s isi auth ldap view --provider-name=Primary | grep "User Domain:"

isi_for_array -s /usr/likewise/bin/lwsm list 

isi_for_array -s ps awux | grep lw

ifconfig

isi zone zones list

isi zone zones view system

isi_for_array -s isi zone zones view system

isi networks list pools

isi networks list pools -v

exit

isi status

isi networks list pools

isi networks list pools --name=pool1

mkdir /ifs/data/Isilon_Support/$(date +%m%d%Y)

isi_for_array 'tcpdump -s 0 -i lagg0 -w /ifs/data/Isilon_Support/$(date +%m%d%Y)/`hostname`.$(date +%m%d%Y_%H%M%S).lagg0.cap &'

Isilon : Sync IQ scheduler memory leak issue

Current Isilon versions  7.* have a memory leak issue which causes the sync scheduler to run out of it's allocated 512 max memory and go into hung state. This state will stops all jobs from initializing weather incremental or full. Current code doesn't trigger any alerts during this outage until some one manually verify.

To avoid enter into the outage situation follow the below steps:


Isilon has developed a script for monitoring and trigger email alerts once sync scheduler memory utilization reaches certain threshold, so that sync process can be restarted before go into the hung state.

Below are the commands to verify the memory usage manually.

# isi_for_array -s ps awxu | grep isi_migr_sched | grep -v grep |awk '{print $1 $6}'    This command give the current memory usage across all nodes in the cluster

For example, if we want to be notified when memory reaches 470 MB, script is available with EMC support. edit the threshold values to 470 MB from the script.

Once we receive the email,  run the following commands to reset the memory.

isi sync settings modify --service=off

isi sync settings modify --service=on

This command will reset the memory value to around 76 MB

Note: Script from Isilon has to be executed every time the node gets rebooted.

** Permanent fix will be expected to be on Riptide version (8.0) which is expected in Q4

VMAX : Storage provision for boot LUNs

Most of the clients prefer to create boot luns as Thick.  Below provide steps to create LUNs from thick storage (Disk groups).

1) Verify Disk Group information

symdisk list -dskgrp_summary

Verify available space on disk group.

2) Create Thick LUN

There are 2 ways to create LUNs. We can use existing LUN which already in use as reference to create new LUNs with same confirguration

symconfigure -sid 123 -cmd "configure 2 devices copying dev 1234 overriding config=vdev;" preview/prepare/commit -nop -v

1234 is existing LUN. I am copying same configuration and creating 2 new LUNs

OR

symconfigure  -sid 123 -cmd "create dev count=6, size=2322 cal, emulation=FBA, data_member_count=3, config=RAID-5, disk_group=1;" commit -nop 

3) Create Storage group and add dev to group

symaccess -sid 123 create -name Boot_LUN -type stor -dev 1234;

4) Create Port group

symaccess -sid 845 create -name Boot_LUN -type port -dirport 3E:0, 13E:0, 4E:0, 14E:0

5) Create Initiator group

Creating Child and Parent Storage groups allows the user to use same child group nested under multiple parent groups

     a) Create Child group and set flags

               Create group and add host wwn's to child group

    

          symaccess -sid 123 create -name IG_Child -type init -wwn    20000012345678  ;

          symaccess -sid 123 -name IG_Child -type init add -wwn   200000123456787

            Adding flags C,SPC2  & consistent lun

           symaccess -sid 123 -name IG_Child -type init set ig_flags on C,SPC2 -enable ;

           symaccess -sid 123 -name IG_Child -type init set  consistent_lun on ;

   B)  Create Parent and set flags

            symaccess -sid 123 create -name IG_Parent -type init ;

            Enable C, SPC2

            symaccess -sid 123 -name IG_Parent -type init set ig_flags on C,SPC2 -enable ;

            Enable Consistent LUN

            symaccess -sid 845 -name IG_Parent -type init set consistent_lun on ;

     C) Add child IG groups to Parent IG Groups

            symaccess -sid 845 add -name IG_Parent -type init -ig IG_Child;

6)  Create Masking View

       Create Masking view  symaccess -sid 123 create view -name Boot_LUN  -sg Boot_LUN -pg Boot_LUN -ig IG_Parent -lun 0 ;

Isilon - Clear CE log database

Some times log files gets filled up which avoids Isilon cluster from sending alerts to either emails or call homes.  Performing the following commands will free up the logs database and start sends the alerts again

There is one more case where we need to reset the CE log database. Some times quieting the old alerts throws the error saying "event database not accessible"  or  while doing pre-health check during code upgrades the output shows "Health check returns with warning message saying event database is not accessible".  This can be resolved by clearing and restarting the CE log services and databases.

You can run all commands at once or One at a time if want to.

Create <SR number> directory under Isilon_support directory to store logs for further analysis or troubleshooting

mkdir -p /ifs/.ifsvar/db/celog /ifs/data/Isilon_Support/sandbox /ifs/data/Isilon_Support/celog_backups ;

mkdir /ifs/data/Isilon_Support/<SR Number> ;

isi_for_array -sX 'gcore -c /ifs/data/Isilon_Support/<SR Number>/$(hostname)_$(date +"%Y-%m-%dT%H.%M.%S")_isi_celog_monitor.core $(pgrep isi_celog_monitor)' ;

isi_for_array -sX 'gcore -c /ifs/data/Isilon_Support/<SR Number>/$(hostname)_$(date +"%Y-%m-%dT%H.%M.%S")_isi_celog_coalescer.core $(pgrep isi_celog_coalescer)' ;

isi_for_array -sX 'gcore -c /ifs/data/Isilon_Support/<SR Number>/$(hostname)_$(date +"%Y-%m-%dT%H.%M.%S")_isi_celog_notification.core $(pgrep isi_celog_notifi)' ;sleep 120; 

isi services -a celog_coalescer disable ;

isi services -a celog_monitor disable ;

isi services -a celog_notification disable ;

isi_for_array -sX 'pkill isi_celog_';

mv -vf /ifs/.ifsvar/db/celog/* /ifs/data/Isilon_Support/celog_backups/ ;

isi_for_array -sX 'rm -f /var/db/celog/*.db' ;

isi_for_array -sX 'rm -f /var/db/celog_master/*.db' ;

rm -f /ifs/.ifsvar/db/celog/*.db ;

isi services -a celog_coalescer enable ;

isi services -a celog_monitor enable ;

isi services -a celog_notification enable ;

          isi services -a | grep celog

Isilon - Useful links

Below links are just for reference. Credits to the authors

Isilon integration with Avamar : https://splitbrained.wordpress.com/2014/02/19/isilon-avamar-ndmp/

Create Multi access zones on Isilon: https://storagenerd.wordpress.com/2013/02/01/how-to-setup-access-zones-for-multiple-active-directory-domains-isilon-7/

Multi access zone video demonstration: https://www.youtube.com/watch?v=hF3W8o-n-Oo
                                                                 https://www.youtube.com/watch?v=R6XRJSp3mj4

Isilon - Measuring cluster latency

CPU:  isi statistics system --nodes -top
NET : isi statistics protocol --top --orderby=timeavg
           ping/iperf
DISK: isi statistics drive -nall -top --long
MEMORY: isi statistics query -nall -stats=node.memory.used

Isilon: measuring IOPS for drive

Recommended max IOPS rates for Isilon drives are
100 for Sata drives
200 for SAS drives
Stech Mach8 SSD drives:2600
Hitachi SSD drive: 4800

measuring iops per drive
requires root access
isi statistics query --stats=node.disk.xfers.rate.<drive #>
isi statistics query --nodes=all --stats=node.disk.xfers.rate.sum   for all nodes

Measuring latency
isi statistics -nall --top --long

Disk latency
7200 RPM SATA = 8-10 ms
10000 RPM SAS = 3 ms

Infiniband latency: ~.050 milliseconds


Measuring CPU performance under load
isi statistics protocl
isi statistics protocol --orderby-TimeAvg --top
isi statistics system --top  to see greater details about cpu processing
Too see load averages
sysctl vm.loadavg
sysctl vm.uptime

To display performance information for all nodes by drive:
isi statistics drive --nodes=all --orderby=timeinqueue

isi statistics client --remote-name=<IP_address>