I'm a little confused here and need some help.
I have the following VP FS which is reporting fully allocated:
id = 27
name = FC_R5_iSCSI_2
acl = 0
in_use = True
type = uxfs
worm = off
volume = v116
pool = clar_r5_performance
member_of = root_avm_fs_group_3
rw_servers= server_2
ro_servers=
rw_vdms =
ro_vdms =
auto_ext = hwm=90%,max_size=2189672M (reached),virtual_provision=yes
The following two VP LUNs were carved from it:
Logical Unit 2 on target ranchns40_iscsi:
(Production) fsid=27 size=512000MB alloc=55435MB virtually_provisioned read-write
path=/FC_iSCSI_3/fs27_T1_LUN2_APM00082700374_0000/fs27_T1_LUN2_APM00082700374_0000 (snapped)
replication=source
max_extension_size=1585151MB
Healthy
Logical Unit 4 on target ranchns40_iscsi:
(Production) fsid=27 size=1536000MB alloc=785138MB virtually_provisioned read-write
path=/FC_iSCSI_3/fs27_T1_LUN4_APM00082700374_0000/fs27_T1_LUN4_APM00082700374_0000 (snapped)
replication=source
max_extension_size=561151MB
Healthy
If I am correct these LUNs are actually only taking up about 840.5 GB of space together.
So I'm assuming the remaining space is being taken up by the snaps?
These LUNs were being replicated but the replication halted because I believe the snaps ran out of space on the FS.
However, if I look at the details of the replication there are no source snaps listed.
Here are the details on the replication:
ID = fs27_T1_LUN2_APM00082700374_0000_fs23_T1_LUN2_APM00082602822_0000
Name = lun2_replication
Source Status = Critical 8865448248: The replication session encountered an error that halted progress.
Network Status = OK
Destination Status = OK
Last Sync Time = Sun May 03 07:46:40 CDT 2009
Type = iscsiLun
Celerra Network Server = opns40
Dart Interconnect = ranchns40_opns40
Peer Dart Interconnect = opns40_ranchns40
Replication Role = source
Source Target = iqn.1992-05.com.emc:apm000827003740000-1
Source LUN = 2
Source Data Mover = server_2
Source Interface = 10.1.11.62
Source Control Port = 0
Source Current Data Port = 0
Destination Target = iqn.1992-05.com.emc:apm000826028220000-1
Destination LUN = 2
Destination Data Mover = server_2
Destination Interface = 10.1.11.68
Destination Control Port = 5085
Destination Data Port = 8888
Max Out of Sync Time (minutes) = 10
Application Data =
Next Transfer Size (Kb) = 0
Latest Snap on Source =
Latest Snap on Destination = fs23_T1_LUN2_APM00082602822_0000.ckpt000_50059989011613
Current Transfer Size (KB) = 0
Current Transfer Remain (KB) = 0
Estimated Completion Time =
Current Transfer is Full Copy = No
Current Transfer Rate (KB/s) = 0
Current Read Rate (KB/s) = 0
Current Write Rate (KB/s) = 0
Previous Transfer Rate (KB/s) = 0
Previous Read Rate (KB/s) = 0
Previous Write Rate (KB/s) = 0
Average Transfer Rate (KB/s) = 0
Average Read Rate (KB/s) = 0
Average Write Rate (KB/s) = 0
ID = fs27_T1_LUN4_APM00082700374_0000_fs23_T1_LUN4_APM00082602822_0000
Name = lun4_replication
Source Status = Critical 8865448248: The replication session encountered an error that halted progress.
Network Status = OK
Destination Status = Critical 8865448248: The replication session encountered an error that halted progress.
Last Sync Time = Sun May 03 07:18:04 CDT 2009
Type = iscsiLun
Celerra Network Server = opns40
Dart Interconnect = ranchns40_opns40
Peer Dart Interconnect = opns40_ranchns40
Replication Role = source
Source Target = iqn.1992-05.com.emc:apm000827003740000-1
Source LUN = 4
Source Data Mover = server_2
Source Interface = 10.1.11.62
Source Control Port = 0
Source Current Data Port = 0
Destination Target = iqn.1992-05.com.emc:apm000826028220000-1
Destination LUN = 4
Destination Data Mover = server_2
Destination Interface = 10.1.11.68
Destination Control Port = 5085
Destination Data Port = 8888
Max Out of Sync Time (minutes) = 10
Application Data =
Next Transfer Size (Kb) = 0
Latest Snap on Source =
Latest Snap on Destination = fs23_T1_LUN4_APM00082602822_0000.ckpt000_24524733573638270
Current Transfer Size (KB) = 0
Current Transfer Remain (KB) = 0
Estimated Completion Time =
Current Transfer is Full Copy = No
Current Transfer Rate (KB/s) = 0
Current Read Rate (KB/s) = 15348
Current Write Rate (KB/s) = 17957
Previous Transfer Rate (KB/s) = 0
Previous Read Rate (KB/s) = 0
Previous Write Rate (KB/s) = 0
Average Transfer Rate (KB/s) = 0
Average Read Rate (KB/s) = 0
Average Write Rate (KB/s) = 0
yes, since the ISCSI LUNs are VP they only take up the space mentioned in the alloc= column their next snap wont reserve space - however the transfer will need as much space as the difference between the last two snaps
Take a look at the GUI in the properties of the ICSI LUN - what does the value for "Space required for next snap" say ?
Are the destination LUNs VP as well ?
I think it would be best to open a support case for this
The error message you are getting isnt very specific:
Brief_Description = The replication session encountered an error that halted progress.
Full_Description = The replication session has entered an inactive state as a result of encountering an error condition. Persistent error conditions will need to be corrected before the replication can continue.
Recommended_Action = Verify network connectivity using the nas_cel -interconnect -validate interconnect_name command. Verify that both source and destination targets are accessible, stopped, and in the correct mode (RW/RO). Restart the replication session. For unrecoverable errors, it may be necessary to delete and recreate the replication session.
I don't see "Space required for next snap" but there is a Snap=yes. From the CLI, I see two snaps on each of the two source LUNs in question. Should there be only one snap on each source LUN? How can I see how much space these snaps are taking up? Should I just delete the existing snaps and restart the replication?
These Celerras are connected to each other via cross-over cable so I can't imagine it being a network issue although they did have a power outage recently. But if they have to do a full sync that shouldn't be a problem at this point.
Here is some more info on the FS:
[nasadmin@ranchns40 ~]$ server_df server_2 server_2 : Filesystem kbytes used avail capacity Mounted on FC_R5_iSCSI_2 2208010112 2011256840 196753272 91% /FC_iSCSI_3
Another question, can I convert a VP FS/LUN to a dense fully allocated FS/LUN? I haven't overallocated the storage. The two LUNs are 2TB exactly but that doesn't leave much space for snaps. I think I would have rather used dense FS/LUN on source and VP on destination but I didn't realize I could do that when I set this up.
I don't see "Space required for next snap" but there is a Snap=yes.
In Celerra Manager just right-click on the LUN and select Properties to get to the iSCSI LUN Properties page which looks like:
Data Mover: server_2 LUN: 0 Target: Target1 File System: fs_iscsi Size: 2.0 GB Read Only: No Storage Required for Next Snap: 64.0 MB Has Snaps: No -------------------------------------------------------------------------------- Target Properties iSCSI Qualified Target Name: iqn.1992-05.com.emc:ck2000732005870000-3 Network Portals: 192.168.99.123 --------------------------------------------------------------------------------
From the CLI, I see two snaps on each of the two source LUNs in question. Should there be only one snap on each source LUN?
no, replication does use two snaps on both src and dst
How can I see how much space these snaps are taking up?
server_iscsi -snap -info should tell you
Should I just delete the existing snaps and restart the replication?
you can try that if you dont mind the time and traffic of the baseline copy
Another question, can I convert a VP FS/LUN to a dense fully allocated FS/LUN?
for the file system it doesnt matter and you should be able to change it from the GUI (unless there is a dens LUN on it) a VP file system is just like a regular one plus automatic extension and that we tell clients the max size instead of the current size if they ask
for the ISCSI LUN there isnt a command to change the LUN from VP to dense or vice versa if you have the space you could use a local or remote replication to replicate into a new LUN of the desired type I wouldnt advise to change it into a dense LUN though unless you have lots of free space in the file system since dense LUNs need a space guarantee of up to 100% when creating a snap
If you are uneasy about the PLU running out of space then you could just fill up your PLU from a client (large file fill its full then delete) *before* you put snaps and replication on it. Then for the Celerra's view these blocks are used and will stay allocated - similar to a dense LUN
In Celerra Manager just right-click on the LUN and select Properties to get to the iSCSI LUN Properties page which looks like:
Data Mover: server_2 LUN: 0 Target: Target1 File System: fs_iscsi Size: 2.0 GB Read Only: No Storage Required for Next Snap: 64.0 MB Has Snaps: No ----------------------------------------------------- -------------------------- Target Properties iSCSI Qualified Target Name: iqn.1992-05.com.emc:ck2000732005870000-3 Network Portals: 192.168.99.123 ------------------------------------------------------
This is what I see: Data Mover: server_2 LUN: 2 Target: ranchns40_iscsi File System: fc_r5_iscsi_2 Size: 500.0 GB Read Only: No Virtually Provisioned: Yes Storage Used: 65.2 GB Has Snaps: Yes
I'm assuming that the snap will only use what the storage used lists. To make sure I understand, one snap will be 65.2 and the second snap will only be the amount of changed data since the last update, correct?
How can I see how much space these snaps are taking
up?
server_iscsi -snap -info should tell you
This is all that command gives me: [nasadmin@ranchns40 ~]$ server_iscsi server_2 -snap -info fs27_T1_LUN2_APM00082700374_0000.ckpt000_6079582181288721 server_2: Snap Name = fs27_T1_LUN2_APM00082700374_0000.ckpt000_6079582181288721 Lun Number = 2 Target = ranchns40_iscsi Promoted Number = Promoted Initiator = Application Label = ReplicationV2 LUN Size MB = 512000 Create Time = Sun May 03 07:55:00 CDT 2009 Application Data = (non-printable characters)
I don't see "Space required for next snap" but there is a Snap=yes.
In Celerra Manager just right-click on the LUN and select Properties to get to the iSCSI LUN Properties page which looks like:
Data Mover: server_2 LUN: 0 Target: Target1 File System: fs_iscsi Size: 2.0 GB Read Only: No Storage Required for Next Snap: 64.0 MB Has Snaps: No
sorry - I just realized that we show "Space Required for Next Snap" for fully provisioned LUNs because this is actually the space that the system will reserve as space guarantee when you create the next snap since snaps for VP LUNs dont reserve any space we say "Storage Used" instead
I'm assuming that the snap will only use what the storage used lists.
To make sure I understand, one snap will be 65.2 and the second snap will only be the amount of changed data since the last update, correct?
I know its kinda confusing but it really depends on if the PLU is VP or FP
You also need to differentiate between space reservation (gets taken immediately when a snap is created) and allocation (allocated block-by-block when changes to the blocks are done)
For snaps on FP LUNs as much space as there is allocated in the PLU or the previous snap gets reserved. Later on when the blocks are needed they are allocated from that reserve.
hard to explain verbally - lets assume you start with a 100MB FP LUN - this means you have guaranteed to the client that it can always write 100MB worth of blocks then you use 20MB and they get allocated when you create a snap these 20MB are now frozen - so in order to still guarantee the full 100MB the system has to reserve another 20MB to back-fill for the "frozen" ones then you write another 30MB and the next snap needs to reserve 30MB to back-fill so that there are always 100MB worth of writeable blocks available
with VP LUNs we dont reserve anything up-front - neither for the PLU nor for any snaps so if you create a LUN or a snap only a few kiloBytes for metadata info gets allocated the rest is pay-as-you-go allocation That means that you need to have enough free space or fs extension so that the system never runs ouf of blocks - or it would have to tell "I/O Error" to the client
I have been dealing with all of these same issues and confusion.
Here is my replication......
[nasadmin@emcnas02_cs0 ~]$ nas_replicate -info reptest ID = fs84_T2_LUN0_APM00045101404_0000_fs586_T8_LUN0_APM00032001391_0000 Name = reptest Source Status = OK Network Status = OK Destination Status = OK Last Sync Time = Wed Jun 17 14:04:50 CDT 2009 Type = iscsiLun Celerra Network Server = emcnas_80 Dart Interconnect = 137_82 Peer Dart Interconnect = 82_137 Replication Role = source Source Target = iqn.1992-05.com.emc:apm000451014040000-2 Source LUN = 0 Source Data Mover = server_2 Source Interface = 192.168.1.137 Source Control Port = 0 Source Current Data Port = 0 Destination Target = iqn.1992-05.com.emc:apm000320013910000-8 Destination LUN = 0 Destination Data Mover = server_2 Destination Interface = 192.168.1.82 Destination Control Port = 5085 Destination Data Port = 8888 Max Out of Sync Time (minutes) = 60 Application Data = Next Transfer Size (KB) = 0 Latest Snap on Source = fs84_T2_LUN0_APM00045101404_0000.ckpt000_11637356203898016 Latest Snap on Destination = fs586_T8_LUN0_APM00032001391_0000.ckpt000_8207042717688740 Current Transfer Size (KB) = 0 Current Transfer Remain (KB) = 0 Estimated Completion Time = Current Transfer is Full Copy = No Current Transfer Rate (KB/s) = 0 Current Read Rate (KB/s) = 0 Current Write Rate (KB/s) = 0 Previous Transfer Rate (KB/s) = 68 Previous Read Rate (KB/s) = 27036 Previous Write Rate (KB/s) = 0 Average Transfer Rate (KB/s) = 54 Average Read Rate (KB/s) = 15380 Average Write Rate (KB/s) = 0
I have 3GB worth of data on this LUN on a Win2k3 box. How is alloc=0MB????
Here is the df....
[nasadmin@emcnas02_cs0 ~]$ server_df server_2 reptest server_2 : Filesystem kbytes used avail capacity Mounted on reptest 20166984 736 20166248 0% /reptest
How can server_df only show 736K used?
Then I added another 1 to 1.5GB or so. Still in the GUI it shows <1MB Used for this LUN.
Data Mover: server_2
LUN: 0
Target: iscsi_test
File System: reptest
Replication: Source
Size: 14.9 GB
Read Only: Yes
Virtually Provisioned: Yes
Storage Used: <1 MB
Has Snaps: Yes
AND...... Here is the new nas_replicate output.
[nasadmin@emcnas02_cs0 ~]$ nas_replicate -info reptest ID = fs84_T2_LUN0_APM00045101404_0000_fs586_T8_LUN0_APM00032001391_0000 Name = reptest Source Status = OK Network Status = OK Destination Status = OK Last Sync Time = Wed Jun 17 14:04:50 CDT 2009 Type = iscsiLun Celerra Network Server = emcnas_80 Dart Interconnect = 137_82 Peer Dart Interconnect = 82_137 Replication Role = source Source Target = iqn.1992-05.com.emc:apm000451014040000-2 Source LUN = 0 Source Data Mover = server_2 Source Interface = 192.168.1.137 Source Control Port = 0 Source Current Data Port = 0 Destination Target = iqn.1992-05.com.emc:apm000320013910000-8 Destination LUN = 0 Destination Data Mover = server_2 Destination Interface = 192.168.1.82 Destination Control Port = 5085 Destination Data Port = 8888 Max Out of Sync Time (minutes) = 60 Application Data = Next Transfer Size (KB) = 0 Latest Snap on Source = fs84_T2_LUN0_APM00045101404_0000.ckpt000_11637356203898016 Latest Snap on Destination = fs586_T8_LUN0_APM00032001391_0000.ckpt000_8207042717688740 Current Transfer Size (KB) = 0 Current Transfer Remain (KB) = 0 Estimated Completion Time = Current Transfer is Full Copy = No Current Transfer Rate (KB/s) = 0 Current Read Rate (KB/s) = 0 Current Write Rate (KB/s) = 0 Previous Transfer Rate (KB/s) = 68 Previous Read Rate (KB/s) = 27036 Previous Write Rate (KB/s) = 0 Average Transfer Rate (KB/s) = 54 Average Read Rate (KB/s) = 15380 Average Write Rate (KB/s) = 0
How can the next transfer size be 0??????????????????
I do have a case opened with support. Rainer you seem to have some good insight here. Anything you can add to this?
Rainer_EMC
6 Operator
•
8645 Posts
552
1
Posted June 9th, 2009 09:00
their next snap wont reserve space - however the transfer will need as much space as the difference between the last two snaps
Take a look at the GUI in the properties of the ICSI LUN - what does the value for "Space required for next snap" say ?
Are the destination LUNs VP as well ?
I think it would be best to open a support case for this
The error message you are getting isnt very specific:
[nasadmin@NS20 ~]$ nas_message -info 8865448248
MessageID = 8865448248
BaseID = 312
Severity = CRITICAL
Component = DART
Facility = REP
Type = STATUS
Brief_Description = The replication session encountered an error that halted progress.
Full_Description = The replication session has entered an inactive state as a result of encountering an error condition. Persistent error conditions will need to be corrected before the replication can continue.
Recommended_Action = Verify network connectivity using the nas_cel -interconnect -validate interconnect_name command. Verify that both source and destination targets are accessible, stopped, and in the correct mode (RW/RO). Restart the replication session. For unrecoverable errors, it may be necessary to delete and recreate the replication session.