Isilon - Log analysis

CELOG coalescer Log file raw data
Each node has a set of logs
cluster wide log files    eg: lsas, snapshot, dedupe
/var is unique and individual for every nodes
/var/log either 500 MB or 2 GB depending on node version
if /var/log partition reaches 95 percent full, node gets rebooted every 30 seconds

Log file locations
/var/log on each node
ls /var/log
find /var/log -name "*celog*.log" -print

Log collection isi_gather_info
Logs from specific node isi_gather_info -n <node #>
isi_gather_info -f /var/crash -s 'isi_hw_status -i'


From GUI

Clustermanagement - > diagnostics -> gather

Generic logs:  eg: /var/log/messages
process specific logs eg;  /var/log/lsassd.log  any kind of authentication goes through lsassd.log
/var/log/isi_celog_coalescer.log

Log gather structure:
Isilon-1
Isilon-2
Isilon-3
local  logs that are generic
base level files  like any specific switches used


Isilon-1
varlog.tar
isi_hangdump.tar


isi_gather_info -noupload
isi_gather_info --noupload --group fs --nologs   Log Group --group fs example


Commands for log file filtering
ls
less
grep
cat

common useful options

ls -l
grep -v <expression> file
less -d <file>
cat -n <files>
cd ; ls
ls | less
ls > files.txt
ls>>files.txt

The grep utility
grep -v Diskless /tmp/isistats.txt |grep SSD

ls -l
wc -l
tail, head and grep
grep and cut
sort and uniq

ls -l isi_job_history
wc -l isi_job_history

Narrow scope
tail isi_job_history |head -l
grep ^03 isi_job_history | wc -l


Extract suspected relevant data
grep  ^03 isi_job_history |cut -d\ f4 | cut -d\[ -f1 |sort | uniq -c
find . -name <filename> -print
grep -i error local/messages  |grep -iv cleared | cut -d: -f2- |less