UNSOLVED

borjacc

updated

11 years ago

B

borjacc

12 Posts

0

1937

December 22nd, 2015 12:00

Celerra NS20 can't access some cifs folders

Some friend called me to help with a celerra NS20. About 1TB data is not accesible and is very worried.

This morning 1 drive faulted in group raid with two other disks already faulted, so now, there are 3 disk faulted of 5. But, i'm not sure that this is related to the problem, because i understand that with two disk faulted this raid group was innacessible since 2014 when other disk of the group become faulted.

Over this raid there are two LUNs and about 1TB data space. There is only one storage group forme by all LUNs of the system.

I connected to navisphere and in alert panel i read the log for the disk faulted but no other events shown. Since 2013 there is no events related to luns become offline.

Some folder are accesible and some others not. But i dont know how can i see the mapping between luns and data or how to try to mount the folder. Also, dont know how can i access data from navicli or control station.

I really appreciatte any help.

Thanks.

  • dynamox

    11 Legend

    •

    20419 Posts

    •

    87439 Points

    711

    0

    Posted December 22nd, 2015 19:00

    if file system is corrupted, i am amazed datamover has not paniced.   Have you connected to the control station and start investigating while file systems are there, which are still mounted.

  • borjacc

    12 Posts

    711

    0

    Posted December 22nd, 2015 23:00

    I connected to control station but can't found where filesystems are mounted.

  • dynamox

    11 Legend

    •

    20419 Posts

    •

    87439 Points

    711

    0

    Posted December 23rd, 2015 06:00

    what commands did you use to check ?

    nas_fs -l

    server_df server_2

    server_mount server_2

  • borjacc

    12 Posts

    711

    0

    Posted December 28th, 2015 04:00

    The first command shows me a list of directories, i only recognized two of them, i think they are the two main storage filesystems. One can be accesed, the other not.

    Second command gives me an error 4000.

    Is there a way of extract raw data of LUNs online? There are some programs that analyze data trying to find files like excel or doc file types.

    Thank you. 

  • borjacc

    12 Posts

    711

    0

    Posted January 12th, 2016 01:00

    I managed to run the commands correctly.

    With nas_fs -l i can see the two main filesystems DATOS and ARCHIVO and both shows that are in use.

    With server_df server_2 i can see that only ARCHIVO filesystem is mounted with root_fs _common and root_fs_2.

    server_mount server_2 tries to mount the 4 filesystems, but in DATOS can't do it:

    DATOS on /DATOS uxfs,perm,rw, .

    I also tried to mount this filesystem with: server_mount server_2 DATOS /DATOS and get: error 4020 failed to complete the command.

    Any ideas?

    Thnaks

  • dynamox

    11 Legend

    •

    20419 Posts

    •

    87439 Points

    711

    0

    Posted January 12th, 2016 11:00

    please post output from the commands.

    Thanks

  • borjacc

    12 Posts

    711

    0

    Posted January 12th, 2016 12:00

    Sure,

    [root@NASxxxxx ~]# /nas/bin/nas_fs -l

    id      inuse type acl   volume    name                server

    1         n    1   0     10        root_fs_1

    2         y    1   0     12        root_fs_2           1

    3         n    1   0     14        root_fs_3

    4         n    1   0     16        root_fs_4

    5         n    1   0     18        root_fs_5

    6         n    1   0     20        root_fs_6

    7         n    1   0     22        root_fs_7

    8         n    1   0     24        root_fs_8

    9         n    1   0     26        root_fs_9

    10        n    1   0     28        root_fs_10

    11        n    1   0     30        root_fs_11

    12        n    1   0     32        root_fs_12

    13        n    1   0     34        root_fs_13

    14        n    1   0     36        root_fs_14

    15        n    1   0     38        root_fs_15

    16        y    1   0     40        root_fs_common      1

    17        n    5   0     73        root_fs_ufslog

    19        n    5   0     77        root_fs_d3

    20        n    5   0     78        root_fs_d4

    21        n    5   0     79        root_fs_d5

    22        n    5   0     80        root_fs_d6

    27        y    1   0     97        DATOS               1

    44        y    1   0     348       ARCHIVO             1

    46        n    5   0     366       root_panic_reserve

    ---------------------------------------------------------------------------------------------------------------

    ]# /nas/bin/server_df server_2

    server_2 :

    Filesystem          kbytes         used        avail capacity Mounted on

    ARCHIVO         3788103392   3677132224    110971168   97%    /ARCHIVO

    root_fs_common       13624         5280         8344   39%    /.etc_common

    root_fs_2           231944        87832       144112   38%    /

    ---------------------------------------------------------------------------------------------------------------------

    # /nas/bin/nas_fs -info DATOS

    id        = 27

    name      = DATOS

    acl       = 0

    in_use    = True

    type      = uxfs

    worm      = off

    volume    = v97

    pool      = xxxxxxx

    member_of = root_avm_fs_group_27

    rw_servers= server_2

    ro_servers=

    rw_vdms   =

    ro_vdms   =

    auto_ext  = hwm=90%,max_size=5400000M (reached),virtual_provision=yes

    deduplication   = unavailable

    stor_devs = CK200074400335-0011,CK200074400335-0012,CK200074400335-0013,CK200074400335-0010,CK200074400335-0015,CK200074400335-0016,CK200074400335-0014,CK200074400335-0017,CK200074400335-0019,CK200074400335-0018

    disks     = d8,d9,d10,d7,d11,d12,d13,d14,d15,d16

    disk=d8    stor_dev=CK200074400335-0011 addr=c16t1l1        server=server_2

    disk=d8    stor_dev=CK200074400335-0011 addr=c0t1l1         server=server_2

    disk=d9    stor_dev=CK200074400335-0012 addr=c0t1l2         server=server_2

    disk=d9    stor_dev=CK200074400335-0012 addr=c16t1l2        server=server_2

    disk=d10   stor_dev=CK200074400335-0013 addr=c16t1l3        server=server_2

    disk=d10   stor_dev=CK200074400335-0013 addr=c0t1l3         server=server_2

    disk=d7    stor_dev=CK200074400335-0010 addr=c0t1l0         server=server_2

    disk=d7    stor_dev=CK200074400335-0010 addr=c16t1l0        server=server_2

    disk=d11   stor_dev=CK200074400335-0015 addr=c0t1l5         server=server_2

    disk=d11   stor_dev=CK200074400335-0015 addr=c16t1l5        server=server_2

    disk=d12   stor_dev=CK200074400335-0016 addr=c0t1l6         server=server_2

    disk=d12   stor_dev=CK200074400335-0016 addr=c16t1l6        server=server_2

    disk=d13   stor_dev=CK200074400335-0014 addr=c16t1l4        server=server_2

    disk=d13   stor_dev=CK200074400335-0014 addr=c0t1l4         server=server_2

    disk=d14   stor_dev=CK200074400335-0017 addr=c16t1l7        server=server_2

    disk=d14   stor_dev=CK200074400335-0017 addr=c0t1l7         server=server_2

    disk=d15   stor_dev=CK200074400335-0019 addr=c0t1l9         server=server_2

    disk=d15   stor_dev=CK200074400335-0019 addr=c16t1l9        server=server_2

    disk=d16   stor_dev=CK200074400335-0018 addr=c16t1l8        server=server_2

    disk=d16   stor_dev=CK200074400335-0018 addr=c0t1l8         server=server_2

    ------------------------------------------------------------------------------------------------------------------------------------------------

    Where disk d12 and d14 are de LUNs affected by the faulted RG.

    # /nas/bin/server_mount server_2 DATOS /DATOS

    server_2 :

    Error 4020: server_2 : failed to complete command

    ------------------------------------------------------------------------------------------------------------------------------------------------

    # /nas/bin/nas_fsck -start DATOS -monitor

    Error 5: Input/output error

    Thank you!

  • kelleg

    6 Operator

    •

    4537 Posts

    711

    0

    Posted January 12th, 2016 14:00

    I think the main issue is to try to repair the faulted disks. If the raid group was a Raid 6 which can withstand a two disk failure and still function and a third disk fails, then the raid group will be faulted and access disabled to the LUNs from the raid group.

    The way to fix this is to replace the last disk that failed (the 3rd one) first and let the raid group rebuild, then replace the second disk that failed, let raid group rebuild, the replace the first disk that failed. This is only safe way to fix this issue, removing or moving around any of the faulted disks could result in the correct disk getting into the wrong slot. It's very important that if you do remove a disk, always return the disk back to the same slot before during any other work.

    The raid group should be OK and the LUNs should come back on-line once you replace the disks.

    glen