Those "creates" should show up in  your  isi statistics client & heat outputs.

Client statistics will help breaking it down per user and PC,

while heat statistics can go down do directory and file level.

I'd guess some app/download/update is just running wild,

possibly for multiple users.

hth

-- Peter

Your statistics monitoring looks great.  From the description it looks like the network load is heavy, but not abnormally so.  The couple other areas to check on status would be the current job engine status and tasks, any other modules running such as a SyncIQ job, Snapshot job or other operation?  How is the storage capacity looking, and balanced across the nodes? 

If there's nothing else seen there, I might browse into the log files and take a look at the messages log to see if there's anything unusual there - such as a drive stall or other behavior not obvious from the statistics.  /var/log/messages on any node.  Not saying there should be something in there, just worth a look in case.

Hi Sebastian, thanks for your reply.  One thing to consider are the recommendations on page 11 of the IIQ 3.1 Installation Guide, w.r.t. configuring the name or IP.  As an alternative, you can specify the SIP (SmartConnect IP) instead of the SC zonename.  This reduces the load on your DNS server as it avoids name resolution for each query.

In addition, for your SIQ jobs, have you considered using the 'continuous' policy for certain directories?  If you are running policy every 5 minutes, it might be easier to just set continuous mode.  This avoids running the SIQ policy with nothing to do (i.e. no changes over the last 5-minute interval.)  Depends on your own preference and RPO, really.  Remember each SIQ job will invoke snapshot on the source cluster, so with your 5 minute policy you are accumulating 12x24 = 360 snaps per day.  That's quite a bit more than 3 :-)

Cheers

Rob

Thank you for the input.

I am going to monitor the background jobs too.

We got 3 snapshots a day and a syncIQ job every 5 minutes.

Maybe we should monitor isi_job_d and isi_migrate ?

Also i am going to monitor the client stats as Rob wrote.

Do you think an interval of -i 1 is to heavy ?

InsightIQ is no option because it is spamming our DNS servers with queries.

We have tried to get it up and running in our enviroment by configuring the smartconnect-zone of the clusters which results in heavy DNS traffic :-( .

Configuring a node IP is not the best option because when the specific node is down there is no reporting :-(

Another problem is that a max. of 8 clusters could be monitored by one insightIQ instance.

We have a much higher cluster count.

/Sebastian

Hi Rob,

i will hold the SIP solution in mind. It i not that easy to set it up in our network. But never say never :-)

Right now i have to figure out a way without IIQ.

During the mirgration projekt to isilon the emc² solution architect mentioned that a synciq job every 5 mins produces heavy workload. But thats a detail we are not able to Change.

How could i monitore the synciq Impact - isi_migrate job on every node ?

The continous-mode is a very interesting detail you wrote about. I will check it out if this is a way for us with onefs 7.0.2.7.

/Sebastian

Hi Sebastian,

Sorry to hear it's difficult to setup an IP address in your network.

Anyway, the best way to measure node resource consumption is what you are already doing - isi statistics ...

Continuous mode SIQ is a feature in OneFS 7.1 and newer.

But remember the key architectural point - OneFS is designed to scale out.  If your current cluster is reaching saturation of resources, due to workload growth to support a growing business, it's time to scale it out.

Cheers

Rob

Very useful post.

Thanks a lot.

Sebastian, did you find out where the 10000+ creates/sec come from?

That's actually a pretty high rate for 6 x X200 cluster if metadata

is on SATA drives, so I wondered whether the files get created at all...

If the files are created and kept, you might run into some

surprise when crawling through the filesystem at a later point.

BUT if the files are NOT actually created

(say due to permission or path error),

then the create ops DO show up in client/protocol statistics,

but NOT in filesystem "heat" statistics -- pretty easy

to check this way. Such finding would indicate a useless

operation storm from the clients, and would be pretty

much in line with the observed HIGH CPU load

while disk IOPS rate stays MEDIUM.

Curious

-- Peter

HI Peter,

thx for that info.

I tried

isi statistics Client --nodes=all --protocols=smb2,lsass_out --classes=create,delete,file_state --oderby=Ops

for some reason i get an error wehn i try to add lsass_in (invalid statistics key:'node.clientstats.proto.lasass_in'

i compared the above Output with

isi statistics heat --nodes=all --top i only see the classes other and namespace_read.

This is a indicator that the clients are spamming the System with create Ops which will never be processed in the filesystem ?

/Sebastian