I have a question regarding NFSv3 and rebooting Isilon nodes; currently we are using NFS3 to mount several shares from an Isilon cluster to more than one RHEL6.2 box. For various reasons I've had to reboot a node from time to time to address a problem in our environment. Based on historical conversations with Isilon, it is my understanding that a node can be safely rebooted (i.e. shutdown -r now) such at the IP addresses allocated in a SmartConnect pool will get redistributed to other nodes in the cluster. I guess what I'm saying is that it is my understanding that if NFSv3 is being used, rebooting a cluster should be transparent to an NFS client. Is this an accurate assumption? I have taken a look at some log data from a client in our environment around the time when I rebooted node, and I am receiving the following log data. It looks time that it is taking upwards of 15 minutes for a the failover to occur. Have others seen this behavior? It appears when I reboot a node that is participating in SmartConnect, I get the following information in /var/log/messages. Is this to be expected? It looks like the server is taking up to 15 minutes to recover, which appears to be occurring naturally. I appreciate any insight somebody could provide why this would occur. ARP cache timeout on the router possibly?
Jun 14 11:15:53 ln01 kernel: nfs: server someserver.uchicago.edu not responding, still trying
Jun 14 11:15:59 ln01 kernel: nfs: server someserver.uchicago.edu not responding, still trying
Jun 14 11:16:08 ln01 kernel: nfs: server someserver.uchicago.edu not responding, still trying
Jun 14 11:16:39 ln01 kernel: nfs: server someserver.uchicago.edu not responding, still trying
Jun 14 11:17:25 ln01 kernel: nfs: server someserver.uchicago.edu not responding, still trying
Jun 14 11:18:38 ln01 kernel: nfs: server someserver.uchicago.edu not responding, still trying
Jun 14 11:21:54 ln01 kernel: nfs: server someserver.uchicago.edu not responding, still trying
Jun 14 11:22:06 ln01 kernel: nfs: server someserver.uchicago.edu not responding, still trying
Jun 14 11:22:12 ln01 kernel: nfs: server someserver.uchicago.edu not responding, still trying
Jun 14 11:23:07 ln01 kernel: nfs: server someserver.uchicago.edu not responding, still trying
Jun 14 11:24:46 ln01 kernel: nfs: server someserver.uchicago.edu not responding, still trying
Jun 14 11:25:45 ln01 kernel: nfs: server someserver.uchicago.edu not responding, still trying
Jun 14 11:27:08 ln01 kernel: nfs: server someserver.uchicago.edu not responding, still trying
Jun 14 11:28:24 ln01 kernel: nfs: server someserver.uchicago.edu OK
Jun 14 11:28:24 ln01 kernel: nfs: server someserver.uchicago.edu OK
Jun 14 11:28:24 ln01 kernel: nfs: server someserver.uchicago.edu OK
Jun 14 11:28:24 ln01 kernel: nfs: server someserver.uchicago.edu OK
Jun 14 11:28:24 ln01 kernel: nfs: server someserver.uchicago.edu OK
Jun 14 11:28:24 ln01 kernel: nfs: server someserver.uchicago.edu OK
Jun 14 11:28:24 ln01 kernel: nfs: server someserver.uchicago.edu OK
Jun 14 11:28:24 ln01 kernel: nfs: server someserver.uchicago.edu OK
Jun 14 11:28:24 ln01 kernel: nfs: server someserver.uchicago.edu OK
Jun 14 11:28:24 ln01 kernel: nfs: server someserver.uchicago.edu OK
Jun 14 11:28:24 ln01 kernel: nfs: server someserver.uchicago.edu OK
Jun 14 11:28:24 ln01 kernel: nfs: server someserver.uchicago.edu OK
Jun 14 11:28:24 ln01 kernel: nfs: server someserver.uchicago.edu OK
15 minutes appear way too long. What happens if you remove an active node's interfaces from a pool manually, so that SmartConnect assigns the affected IPs to other nodes' interfaces? By this, no "failure detection" on the cluster's side is needed and the transition should take place within seconds. If the clients still see 15 minutes outage, check the network setup including ARP caching.
I appreciate your reply. I will have to test this in a maintenance window at this point. I am not 100% certain regarding the ARP settings on our layer 3 device, however I am fairly confident that there is no persistent ARP caching involved. I opened a separate ticket on Isilon's forum to get an answer to this question (with discipline I could test this using two separate servers in our environment, altnough I do not have the bandwith to complete this at the particular moment).
Your response is appreciated. I verified that the allocation method for our NFS pool is set to "Dynamic" (it was previously set to this, and I did not have to change this value).
I've experienced this same issue and it does seem like an ARP cache issue, because it eventually resolves itself. I'm wondering if the Isilon interfaces aren't notifying the switch that the addresses have moved.
Peter_Sero
6 Operator
•
1169 Posts
1092
0
Posted June 15th, 2013 08:00
15 minutes appear way too long. What happens if you remove an active node's interfaces from a pool manually, so that SmartConnect assigns the affected IPs to other nodes' interfaces? By this, no "failure detection" on the cluster's side is needed and the transition should take place within seconds. If the clients still see 15 minutes outage, check the network setup including ARP caching.
Peter