UNSOLVED

aaronj1

updated

17 years ago

A

aaronj1

35 Posts

0

1842

June 9th, 2009 08:00

Virtually provisioned LUNs & replication

I'm a little confused here and need some help.
I have the following VP FS which is reporting fully allocated:

id = 27
name = FC_R5_iSCSI_2
acl = 0
in_use = True
type = uxfs
worm = off
volume = v116
pool = clar_r5_performance
member_of = root_avm_fs_group_3
rw_servers= server_2
ro_servers=
rw_vdms =
ro_vdms =
auto_ext = hwm=90%,max_size=2189672M (reached),virtual_provision=yes

The following two VP LUNs were carved from it:
Logical Unit 2 on target ranchns40_iscsi:
(Production) fsid=27 size=512000MB alloc=55435MB virtually_provisioned read-write
path=/FC_iSCSI_3/fs27_T1_LUN2_APM00082700374_0000/fs27_T1_LUN2_APM00082700374_0000 (snapped)
replication=source
max_extension_size=1585151MB
Healthy
Logical Unit 4 on target ranchns40_iscsi:
(Production) fsid=27 size=1536000MB alloc=785138MB virtually_provisioned read-write
path=/FC_iSCSI_3/fs27_T1_LUN4_APM00082700374_0000/fs27_T1_LUN4_APM00082700374_0000 (snapped)
replication=source
max_extension_size=561151MB
Healthy

If I am correct these LUNs are actually only taking up about 840.5 GB of space together.
So I'm assuming the remaining space is being taken up by the snaps?
These LUNs were being replicated but the replication halted because I believe the snaps ran out of space on the FS.
However, if I look at the details of the replication there are no source snaps listed.

Here are the details on the replication:
ID = fs27_T1_LUN2_APM00082700374_0000_fs23_T1_LUN2_APM00082602822_0000
Name = lun2_replication
Source Status = Critical 8865448248: The replication session encountered an error that halted progress.
Network Status = OK
Destination Status = OK
Last Sync Time = Sun May 03 07:46:40 CDT 2009
Type = iscsiLun
Celerra Network Server = opns40
Dart Interconnect = ranchns40_opns40
Peer Dart Interconnect = opns40_ranchns40
Replication Role = source
Source Target = iqn.1992-05.com.emc:apm000827003740000-1
Source LUN = 2
Source Data Mover = server_2
Source Interface = 10.1.11.62
Source Control Port = 0
Source Current Data Port = 0
Destination Target = iqn.1992-05.com.emc:apm000826028220000-1
Destination LUN = 2
Destination Data Mover = server_2
Destination Interface = 10.1.11.68
Destination Control Port = 5085
Destination Data Port = 8888
Max Out of Sync Time (minutes) = 10
Application Data =
Next Transfer Size (Kb) = 0
Latest Snap on Source =
Latest Snap on Destination = fs23_T1_LUN2_APM00082602822_0000.ckpt000_50059989011613
Current Transfer Size (KB) = 0
Current Transfer Remain (KB) = 0
Estimated Completion Time =
Current Transfer is Full Copy = No
Current Transfer Rate (KB/s) = 0
Current Read Rate (KB/s) = 0
Current Write Rate (KB/s) = 0
Previous Transfer Rate (KB/s) = 0
Previous Read Rate (KB/s) = 0
Previous Write Rate (KB/s) = 0
Average Transfer Rate (KB/s) = 0
Average Read Rate (KB/s) = 0
Average Write Rate (KB/s) = 0

ID = fs27_T1_LUN4_APM00082700374_0000_fs23_T1_LUN4_APM00082602822_0000
Name = lun4_replication
Source Status = Critical 8865448248: The replication session encountered an error that halted progress.
Network Status = OK
Destination Status = Critical 8865448248: The replication session encountered an error that halted progress.
Last Sync Time = Sun May 03 07:18:04 CDT 2009
Type = iscsiLun
Celerra Network Server = opns40
Dart Interconnect = ranchns40_opns40
Peer Dart Interconnect = opns40_ranchns40
Replication Role = source
Source Target = iqn.1992-05.com.emc:apm000827003740000-1
Source LUN = 4
Source Data Mover = server_2
Source Interface = 10.1.11.62
Source Control Port = 0
Source Current Data Port = 0
Destination Target = iqn.1992-05.com.emc:apm000826028220000-1
Destination LUN = 4
Destination Data Mover = server_2
Destination Interface = 10.1.11.68
Destination Control Port = 5085
Destination Data Port = 8888
Max Out of Sync Time (minutes) = 10
Application Data =
Next Transfer Size (Kb) = 0
Latest Snap on Source =
Latest Snap on Destination = fs23_T1_LUN4_APM00082602822_0000.ckpt000_24524733573638270
Current Transfer Size (KB) = 0
Current Transfer Remain (KB) = 0
Estimated Completion Time =
Current Transfer is Full Copy = No
Current Transfer Rate (KB/s) = 0
Current Read Rate (KB/s) = 15348
Current Write Rate (KB/s) = 17957
Previous Transfer Rate (KB/s) = 0
Previous Read Rate (KB/s) = 0
Previous Write Rate (KB/s) = 0
Average Transfer Rate (KB/s) = 0
Average Read Rate (KB/s) = 0
Average Write Rate (KB/s) = 0
  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    552

    1

    Posted June 9th, 2009 09:00

    yes, since the ISCSI LUNs are VP they only take up the space mentioned in the alloc= column
    their next snap wont reserve space - however the transfer will need as much space as the difference between the last two snaps

    Take a look at the GUI in the properties of the ICSI LUN - what does the value for "Space required for next snap" say ?

    Are the destination LUNs VP as well ?


    I think it would be best to open a support case for this

    The error message you are getting isnt very specific:

    [nasadmin@NS20 ~]$ nas_message -info 8865448248
    MessageID = 8865448248
    BaseID = 312
    Severity = CRITICAL
    Component = DART
    Facility = REP
    Type = STATUS

    Brief_Description = The replication session encountered an error that halted progress.

    Full_Description = The replication session has entered an inactive state as a result of encountering an error condition. Persistent error conditions will need to be corrected before the replication can continue.

    Recommended_Action = Verify network connectivity using the nas_cel -interconnect -validate interconnect_name command. Verify that both source and destination targets are accessible, stopped, and in the correct mode (RW/RO). Restart the replication session. For unrecoverable errors, it may be necessary to delete and recreate the replication session.
  • aaronj1

    35 Posts

    552

    0

    Posted June 9th, 2009 09:00

    I don't see "Space required for next snap" but there is a Snap=yes.
    From the CLI, I see two snaps on each of the two source LUNs in question.
    Should there be only one snap on each source LUN?
    How can I see how much space these snaps are taking up?
    Should I just delete the existing snaps and restart the replication?

    These Celerras are connected to each other via cross-over cable so I can't imagine it being a network issue although they did have a power outage recently.
    But if they have to do a full sync that shouldn't be a problem at this point.

    Here is some more info on the FS:

    [nasadmin@ranchns40 ~]$ server_df server_2
    server_2 :
    Filesystem kbytes used avail capacity Mounted on
    FC_R5_iSCSI_2 2208010112 2011256840 196753272 91% /FC_iSCSI_3

    Another question, can I convert a VP FS/LUN to a dense fully allocated FS/LUN?
    I haven't overallocated the storage. The two LUNs are 2TB exactly but that doesn't leave much space for snaps.
    I think I would have rather used dense FS/LUN on source and VP on destination but I didn't realize I could do that when I set this up.

    Thanks!
    Aaron
  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    552

    0

    Posted June 9th, 2009 10:00

    I don't see "Space required for next snap" but there is a Snap=yes.


    In Celerra Manager just right-click on the LUN and select Properties to get to the iSCSI LUN Properties page which looks like:

    Data Mover: server_2
    LUN: 0
    Target: Target1
    File System: fs_iscsi
    Size: 2.0 GB
    Read Only: No
    Storage Required for Next Snap: 64.0 MB
    Has Snaps: No
    --------------------------------------------------------------------------------
    Target Properties
    iSCSI Qualified Target Name: iqn.1992-05.com.emc:ck2000732005870000-3
    Network Portals: 192.168.99.123
    --------------------------------------------------------------------------------

    From the CLI, I see two snaps on each of the two source LUNs in question.
    Should there be only one snap on each source LUN?


    no, replication does use two snaps on both src and dst

    How can I see how much space these snaps are taking up?


    server_iscsi -snap -info should tell you

    Should I just delete the existing snaps and restart the replication?


    you can try that if you dont mind the time and traffic of the baseline copy


    Another question, can I convert a VP FS/LUN to a dense fully allocated FS/LUN?


    for the file system it doesnt matter and you should be able to change it from the GUI (unless there is a dens LUN on it)
    a VP file system is just like a regular one plus automatic extension and that we tell clients the max size instead of the current size if they ask

    for the ISCSI LUN there isnt a command to change the LUN from VP to dense or vice versa
    if you have the space you could use a local or remote replication to replicate into a new LUN of the desired type
    I wouldnt advise to change it into a dense LUN though unless you have lots of free space in the file system since dense LUNs need a space guarantee of up to 100% when creating a snap

    If you are uneasy about the PLU running out of space then you could just fill up your PLU from a client (large file fill its full then delete) *before* you put snaps and replication on it.
    Then for the Celerra's view these blocks are used and will stay allocated - similar to a dense LUN
  • aaronj1

    35 Posts

    552

    0

    Posted June 9th, 2009 14:00

    I don't suppose it's possible to place the iSCSI LUN snaps on a different file system is it?
  • aaronj1

    35 Posts

    552

    0

    Posted June 9th, 2009 14:00

    I don't see "Space required for next snap" but

    there is a Snap=yes.

    In Celerra Manager just right-click on the LUN and
    select Properties to get to the iSCSI LUN Properties
    page which looks like:

    Data Mover: server_2
    LUN: 0
    Target: Target1
    File System: fs_iscsi
    Size: 2.0 GB
    Read Only: No
    Storage Required for Next Snap: 64.0 MB
    Has Snaps: No
    -----------------------------------------------------
    --------------------------
    Target Properties
    iSCSI Qualified Target Name:
    iqn.1992-05.com.emc:ck2000732005870000-3
    Network Portals: 192.168.99.123
    ------------------------------------------------------


    This is what I see:
    Data Mover: server_2
    LUN: 2
    Target: ranchns40_iscsi
    File System: fc_r5_iscsi_2
    Size: 500.0 GB
    Read Only: No
    Virtually Provisioned: Yes
    Storage Used: 65.2 GB
    Has Snaps: Yes

    I'm assuming that the snap will only use what the storage used lists.
    To make sure I understand, one snap will be 65.2 and the second snap will only be the amount of changed data since the last update, correct?

    How can I see how much space these snaps are taking

    up?

    server_iscsi -snap -info should tell you


    This is all that command gives me:
    [nasadmin@ranchns40 ~]$ server_iscsi server_2 -snap -info fs27_T1_LUN2_APM00082700374_0000.ckpt000_6079582181288721
    server_2:
    Snap Name = fs27_T1_LUN2_APM00082700374_0000.ckpt000_6079582181288721
    Lun Number = 2
    Target = ranchns40_iscsi
    Promoted Number =
    Promoted Initiator =
    Application Label = ReplicationV2
    LUN Size MB = 512000
    Create Time = Sun May 03 07:55:00 CDT 2009
    Application Data = (non-printable characters)

    I
  • Peter_EMC

    674 Posts

    552

    0

    Posted June 9th, 2009 22:00

    No, iSCSI Snaps and the PLU have to be in the same FS.
    If you need more space in your FS you can extend it (up to 16 TB)
  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    552

    0

    Posted June 10th, 2009 11:00

    How can I see how much space these snaps are taking up?


    server_iscsi -snap -info should tell you


    sorry - I was a bit quick there

    after some examination it seems to be difficult to tell in the current code

    What I found out is that you can tell how much space the latest/newest is using by looking at the alloc= value of the primary LUN

    For older ones I dont know

    I would suggest to open a Product Enhancement Request for better ISCSI snapshot space reporting
  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    552

    0

    Posted June 10th, 2009 11:00

    I don't see "Space required for next snap" but there is a Snap=yes.


    In Celerra Manager just right-click on the LUN and
    select Properties to get to the iSCSI LUN Properties
    page which looks like:

    Data Mover: server_2
    LUN: 0
    Target: Target1
    File System: fs_iscsi
    Size: 2.0 GB
    Read Only: No
    Storage Required for Next Snap: 64.0 MB
    Has Snaps: No


    sorry - I just realized that we show "Space Required for Next Snap" for fully provisioned LUNs because this is actually
    the space that the system will reserve as space guarantee when you create the next snap
    since snaps for VP LUNs dont reserve any space we say "Storage Used" instead
  • Rainer_EMC

    6 Operator

    •

    8645 Posts

    552

    0

    Posted June 10th, 2009 11:00

    I'm assuming that the snap will only use what the storage used lists.


    To make sure I understand, one snap will be 65.2 and the second snap will only be the amount of changed
    data since the last update, correct?


    I know its kinda confusing but it really depends on if the PLU is VP or FP

    You also need to differentiate between space reservation (gets taken immediately when a snap is created) and allocation (allocated block-by-block when changes to the blocks are done)

    For snaps on FP LUNs as much space as there is allocated in the PLU or the previous snap gets reserved. Later on when the blocks are needed they are allocated from that reserve.

    hard to explain verbally - lets assume you start with a 100MB FP LUN - this means you have guaranteed to the client that it can always write 100MB worth of blocks
    then you use 20MB and they get allocated
    when you create a snap these 20MB are now frozen - so in order to still guarantee the full 100MB the system has to reserve another 20MB to back-fill for the "frozen" ones
    then you write another 30MB and the next snap needs to reserve 30MB to back-fill so that there are always 100MB worth of writeable blocks available

    with VP LUNs we dont reserve anything up-front - neither for the PLU nor for any snaps
    so if you create a LUN or a snap only a few kiloBytes for metadata info gets allocated
    the rest is pay-as-you-go allocation
    That means that you need to have enough free space or fs extension so that the system never runs ouf of blocks - or it would have to tell "I/O Error" to the client

    hope that clarifies more than it confuses
  • mcartmill

    2 Posts

    186

    0

    Posted June 17th, 2009 12:00

    I have been dealing with all of these same issues and confusion.


    Here is my replication......

    [nasadmin@emcnas02_cs0 ~]$ nas_replicate -info reptest
    ID = fs84_T2_LUN0_APM00045101404_0000_fs586_T8_LUN0_APM00032001391_0000
    Name = reptest
    Source Status = OK
    Network Status = OK
    Destination Status = OK
    Last Sync Time = Wed Jun 17 14:04:50 CDT 2009
    Type = iscsiLun
    Celerra Network Server = emcnas_80
    Dart Interconnect = 137_82
    Peer Dart Interconnect = 82_137
    Replication Role = source
    Source Target = iqn.1992-05.com.emc:apm000451014040000-2
    Source LUN = 0
    Source Data Mover = server_2
    Source Interface = 192.168.1.137
    Source Control Port = 0
    Source Current Data Port = 0
    Destination Target = iqn.1992-05.com.emc:apm000320013910000-8
    Destination LUN = 0
    Destination Data Mover = server_2
    Destination Interface = 192.168.1.82
    Destination Control Port = 5085
    Destination Data Port = 8888
    Max Out of Sync Time (minutes) = 60
    Application Data =
    Next Transfer Size (KB) = 0
    Latest Snap on Source = fs84_T2_LUN0_APM00045101404_0000.ckpt000_11637356203898016
    Latest Snap on Destination = fs586_T8_LUN0_APM00032001391_0000.ckpt000_8207042717688740
    Current Transfer Size (KB) = 0
    Current Transfer Remain (KB) = 0
    Estimated Completion Time =
    Current Transfer is Full Copy = No
    Current Transfer Rate (KB/s) = 0
    Current Read Rate (KB/s) = 0
    Current Write Rate (KB/s) = 0
    Previous Transfer Rate (KB/s) = 68
    Previous Read Rate (KB/s) = 27036
    Previous Write Rate (KB/s) = 0
    Average Transfer Rate (KB/s) = 54
    Average Read Rate (KB/s) = 15380
    Average Write Rate (KB/s) = 0

    Here is my VP iSCSI LUN info.......

    [nasadmin@emcnas02_cs0 ~]$ server_iscsi server_2 -lun -info 0 -target iscsi_test
    server_2 :
    Logical Unit 0 on target iscsi_test:
    (Production) fsid=84 size=15240MB alloc=0MB virtually_provisioned read-only
    path=/reptest/fs84_T2_LUN0_APM00045101404_0000/fs84_T2_LUN0_APM00045101404_0000 (snapped, extended)
    replication=source
    max_extension_size=2081911MB
    Healthy

    I have 3GB worth of data on this LUN on a Win2k3 box. How is alloc=0MB????


    Here is the df....

    [nasadmin@emcnas02_cs0 ~]$ server_df server_2 reptest
    server_2 :
    Filesystem kbytes used avail capacity Mounted on
    reptest 20166984 736 20166248 0% /reptest

    How can server_df only show 736K used?

    Then I added another 1 to 1.5GB or so. Still in the GUI it shows <1MB Used for this LUN.

    Data Mover: server_2

    LUN: 0

    Target: iscsi_test

    File System: reptest

    Replication: Source

    Size: 14.9 GB

    Read Only: Yes

    Virtually Provisioned: Yes

    Storage Used: <1 MB

    Has Snaps: Yes


    AND...... Here is the new nas_replicate output.


    [nasadmin@emcnas02_cs0 ~]$ nas_replicate -info reptest
    ID = fs84_T2_LUN0_APM00045101404_0000_fs586_T8_LUN0_APM00032001391_0000
    Name = reptest
    Source Status = OK
    Network Status = OK
    Destination Status = OK
    Last Sync Time = Wed Jun 17 14:04:50 CDT 2009
    Type = iscsiLun
    Celerra Network Server = emcnas_80
    Dart Interconnect = 137_82
    Peer Dart Interconnect = 82_137
    Replication Role = source
    Source Target = iqn.1992-05.com.emc:apm000451014040000-2
    Source LUN = 0
    Source Data Mover = server_2
    Source Interface = 192.168.1.137
    Source Control Port = 0
    Source Current Data Port = 0
    Destination Target = iqn.1992-05.com.emc:apm000320013910000-8
    Destination LUN = 0
    Destination Data Mover = server_2
    Destination Interface = 192.168.1.82
    Destination Control Port = 5085
    Destination Data Port = 8888
    Max Out of Sync Time (minutes) = 60
    Application Data =
    Next Transfer Size (KB) = 0
    Latest Snap on Source = fs84_T2_LUN0_APM00045101404_0000.ckpt000_11637356203898016
    Latest Snap on Destination = fs586_T8_LUN0_APM00032001391_0000.ckpt000_8207042717688740
    Current Transfer Size (KB) = 0
    Current Transfer Remain (KB) = 0
    Estimated Completion Time =
    Current Transfer is Full Copy = No
    Current Transfer Rate (KB/s) = 0
    Current Read Rate (KB/s) = 0
    Current Write Rate (KB/s) = 0
    Previous Transfer Rate (KB/s) = 68
    Previous Read Rate (KB/s) = 27036
    Previous Write Rate (KB/s) = 0
    Average Transfer Rate (KB/s) = 54
    Average Read Rate (KB/s) = 15380
    Average Write Rate (KB/s) = 0


    How can the next transfer size be 0??????????????????


    I do have a case opened with support. Rainer you seem to have some good insight here. Anything you can add to this?

    TIA