UNSOLVED

bogdan_badea

updated

13 years ago

0

6814

March 17th, 2014 04:00

Problem backing up Hyper-V CSV running on Win 2012

Hello,

I am having some difficulties backing up a two node Hyper-V cluster running on Win 2012.

NMM version is  2.4.1

Server version is 8.0.3.0 build 329 (both support 2012 I believe, as per compatibility matrix)

Backup command :

nsrsnap_vss_save.exe -s svhostname

or

nsrsnap_vss_save.exe

Snapsots enabled:

Rollover ALL

1 per day, retain1

Client direct is disabled. I attempted to back up a single VM off a physical node and job fails with message below :

APPLICATIONS:\Microsoft Hyper-V\CAROM2: NSRSNAPCK::success

APPLICATIONS:\Microsoft Hyper-V\CAROM2: Backup of [APPLICATIONS:\Microsoft Hyper-V\CAROM2] failed

Worker Module for Microsoft Applications (x64)

50423:nsrsnap_vss_save:nsrsnap_vss_save: Unknown Application Information parameter: DIRECT_ACCESS, may not be supported

50411:nsrsnap_vss_save:nsrsnap_vss_save: Data mover host name not specified. Assuming local host: srv-h2.hq.cre.ro

89199:nsrsnap_vss_save:nsrsnap_vss_save: Backup is level FULL. Set GLR compatibility to YES.

Microsoft DiskPart version 6.2.9200

Copyright (C) 1999-2012 Microsoft Corporation.

On computer: SRV-H2

Automatic mounting of new volumes enabled.

47363:nsrsnap_vss_save: nsrpsd service is running on SRV-H2.hq.cre.ro.

NMM .. created RM deposit path C:\Windows\TEMP\RMDir_1712923183.

82624:nsrsnap_vss_save:Save set name APPLICATIONS:\Microsoft Hyper-V\CAROM2 is valid.

63335:nsrsnap_vss_save:NMM backup failed to complete successfully.

90563:nsrsnap_vss_save:NMM .. If a Hyper-V GLR virtual machine is currently mounted, please unmount it before retrying the backup

Internal error.

68151 1395050976 2 0 0 23820 8904 0 srv-h2.hq.cre.ro nsrsnap_vss_save NSR warning 39 nsrsnap_vss_save: Exiting with failure. 0

Additionally, when running nsrsnap_vss_save -s -? the application crashes on the Hyper-V node side. Event Viewer error output is below :

Faulting application name: nsrsnap_vss_save.exe, version: 2.4.1.92, time stamp: 0x50bf3b7f

Faulting module name: nsrsnap_vss_save.exe, version: 2.4.1.92, time stamp: 0x50bf3b7f

Exception code: 0xc0000005

Fault offset: 0x000000000016de5d

Faulting process id: 0x21d0

Faulting application start time: 0x01cf41bbe425c5c1

Faulting application path: C:\Program Files\EMC NetWorker\nsr\bin\nsrsnap_vss_save.exe

Faulting module path: C:\Program Files\EMC NetWorker\nsr\bin\nsrsnap_vss_save.exe

Report Id: 357af1da-adaf-11e3-9408-bc305bf4c992

Faulting package full name:

Faulting package-relative application ID:

Has anyone seen this kind of error before ?

Thanks,

Bogdan

  • crazyrov

    6 Operator

    •

    1319 Posts

    1573

    0

    Posted March 17th, 2014 05:00

    Hi Bogdan

         When you run "nsrsnap_vss_save -v -?", what output do you get ? Also, please have "Client Dierct" unchecked  on the client properties to get a clearer error for the failure.

  • 1573

    0

    Posted March 17th, 2014 06:00

    Hello,

    Client Direct is disabled in client properties( and yes the Networker server name is netbackup, long story:))

    I attempted to run nsrsnap_vss_save with verbose attribute and it worked. Please see output below :

    C:\Windows\system32>nsrsnap_vss_save -v -s netbackup -?

    86745:nsrsnap_vss_save:Version information for C:\Program Files\EMC NetWorker\ns

    r\bin\nsrsnap_vss_save.exe:

            Original file name: nsrsnap_vss_save.exe

            Version: 2.4.1.92

            Comments: NetWorker Module for Microsoft Applications (x64)

    NMM .. changing backup name SRV-H2 to primary name srv-h2.hq.cre.ro for multi NI

    C client.

    51340:nsrsnap_vss_save:jobhandler(290): Created Job ID: 712877

            Save command: nsrsnap_vss_save.exe -z srv-h2.hq.cre.ro -?

    51340:nsrsnap_vss_save:jobhandler(290): Created Job ID: 712879

            Save command: nsrsnap_vss_save.exe -z srv-h2.hq.cre.ro -?

    NMM : saveset on srv-h2.hq.cre.ro :

    "APPLICATIONS:\Microsoft Hyper-V"

    "APPLICATIONS:\Microsoft Hyper-V\CAROM2"                   CSV

    "APPLICATIONS:\Microsoft Hyper-V\Host Component"                nonCSV

    NMM : saveset on srv-h1.hq.cre.ro :

    NMM :"APPLICATIONS:\Microsoft Hyper-V"

    "APPLICATIONS:\Microsoft Hyper-V\Host Component"                nonCSV

    "APPLICATIONS:\Microsoft Hyper-V\ssas2k5"                   CSV

    "APPLICATIONS:\Microsoft Hyper-V\Windows2003"                nonCSV

    68150:nsrsnap_vss_save:nsrsnap_vss_save: Exiting with success.



    Thanks,


    Bogdan

  • crazyrov

    6 Operator

    •

    1319 Posts

    1573

    0

    Posted March 17th, 2014 06:00

    I should have told the other way around, the system does not know what client direct is, to remove the error you will have to enable client direct so that it is not sent along with the backup command.

    Also, is the target device ready and available ? Please try a filesystem backup to test this, as i believe this has something to do with the destination. Just a hunch !

  • CarlosRojas

    1 Rookie

    •

    1693 Posts

    1573

    0

    Posted March 19th, 2014 01:00

    Hi Bogdan,

    As per error message:

    If a Hyper-V GLR virtual machine is currently mounted, please unmount it before retrying the backup

    There is a chance that at some point you attempted a GLR operation and either failed, didn't complete or didn't clean up the TMP files created for the GLR, so please make sure there is no trace f any previous GLR operation.

    As for the crash of nsrsnap_vss_save, I would recommend you to double check the configuration and, if everything is OK then please raise a case with support.

    Thank you,

    Carlos

  • 1573

    0

    Posted March 20th, 2014 09:00

    Hello,

    I attempted a test in a non production environment. Can you please tell me whether I am makingsome configuration mistake.

    Environment is as follows:

    Server: 8.0.3 running on Win 2008 R2

    Client: NMM 2.4.1, 8.0.3 client on Win 2012 Standard

    If I use a snapshot disabled backup group and I uninstall the NMM, I am able to run client file level backups

    However, if I install NMM and configure the group and client resource for NMM snapshot backups, the backup fails with the error below :

    C:\PerfLogs: Backup of failed

    92347:nsrsnap:***************** Snapshot Management operation has started, logging to "C:\Program Files\EMC NetWorker\nsr\logs\nwsnap.raw" *****************[nsrsnap.c 610] Calling nsrrm_host

    [nsrsnap.c 907] Calling get_pool_name..

    52051:nsrsnap:Printing savecmd=nsrsnap_vss_save.exe after parsing

    79549:nsrsnap:[nsrsnap.c 2037] Invoking command: "nsrsnap_vss_save.exe -A  optype=conventional  -A "NSR_SNAP_TYPE=VSS"  -A "NSR_VSS_FORCE_SYSTEM_PROVIDER=yes"  -A "NSR_IGNORE_MISSING_SYSTEM_FILES=yes"  -A NSR_PARENT_JOBID=235  -c WIN-OEC5SBJQVJJ -g "test2" -LL -m WIN-OEC5SBJQVJJ -s testewin2008cet -a "DIRECT_ACCESS=No" -t 1395328648 -o RENAMED_DIRECTORIES:index_lookup=on;BACKUPTIME:lookup_range=1395328648:1395328648; -l incr -q -W 78  -A snap_sessionid=1395330115 "[nsrsnap.c 2047] Calling start_savecmd..

    C:\PerfLogs

    [nsrsnap.c 2060] Calling snap_wait..

    [nsrsnap.c 2081] Calling parse_fs_logs_and_prefix..

    86745:nsrsnap_vss_save:Version information for C:\Program Files\EMC NetWorker\nsr\bin\nsrsnap_vss_save.exe:

      Original file name: nsrsnap_vss_save.exe

      Version: 2.4.1.122

      Comments: NetWorker Module for Microsoft Applications (x64)

    50423:nsrsnap_vss_save:nsrsnap_vss_save: Unknown Application Information parameter: DIRECT_ACCESS, may not be supported

    50411:nsrsnap_vss_save:nsrsnap_vss_save: Data mover host name not specified. Assuming local host: win-oec5sbjqvjj

    47363:nsrsnap_vss_save: nsrpsd service is running on WIN-OEC5SBJQVJJ.

    63335:nsrsnap_vss_save:NMM backup failed to complete successfully.

    Internal error.

    68151 1395330115 2 0 0 744 864 0 win-oec5sbjqvjj nsrsnap_vss_save NSR warning 39 nsrsnap_vss_save: Exiting with failure. 0

    Configuration:

    - snapshot enabled group

    - backup command : nsrsnap_vss_save.exe

    - application information :

    NSR_SNAP_TYPE=VSS

    NSR_VSS_FORCE_SYSTEM_PROVIDER=yes

    NSR_IGNORE_MISSING_SYSTEM_FILES=yes

    Am I missing anything ? How should I configure the Remote access field ? The NMM 2.4.1 Application Guide says to leave it empty

    Thanks

  • 1573

    0

    Posted March 20th, 2014 09:00

    Hi Carlos,

    I attempted to back up at file system level in a test environment, but this is still failing.

    I also checked the Compatibility Matrix and Win 2012 is supported by NMM 2.4.1.

    Thanks

  • CarlosRojas

    1 Rookie

    •

    1693 Posts

    1573

    0

    Posted March 20th, 2014 09:00

    Bogdan,

    Do you mean that even file system level backup is failing? If so, what is the error message?

    Thank you,

    Carlos

  • 1573

    0

    Posted March 21st, 2014 01:00

    Hi Carlos,

    Please see below scenario where I used NMM for file level backups:

    I attempted a test in a non production environment. Can you please tell me whether I am makingsome configuration mistake.

    Environment is as follows:

    Server: 8.0.3 running on Win 2008 R2

    Client: NMM 2.4.1, 8.0.3 client on Win 2012 Standard

    If I use a snapshot disabled backup group and I uninstall the NMM, I am able to run client file level backups

    However, if I install NMM and configure the group and client resource for NMM snapshot backups, the backup fails with the error below :

    C:\PerfLogs: Backup of failed

    92347:nsrsnap:***************** Snapshot Management operation has started, logging to "C:\Program Files\EMC NetWorker\nsr\logs\nwsnap.raw" *****************[nsrsnap.c 610] Calling nsrrm_host

    [nsrsnap.c 907] Calling get_pool_name..

    52051:nsrsnap:Printing savecmd=nsrsnap_vss_save.exe after parsing

    79549:nsrsnap:[nsrsnap.c 2037] Invoking command: "nsrsnap_vss_save.exe -A  optype=conventional  -A "NSR_SNAP_TYPE=VSS"  -A "NSR_VSS_FORCE_SYSTEM_PROVIDER=yes"  -A "NSR_IGNORE_MISSING_SYSTEM_FILES=yes"  -A NSR_PARENT_JOBID=235  -c WIN-OEC5SBJQVJJ -g "test2" -LL -m WIN-OEC5SBJQVJJ -s testewin2008cet -a "DIRECT_ACCESS=No" -t 1395328648 -o RENAMED_DIRECTORIES:index_lookup=on;BACKUPTIME:lookup_range=1395328648:1395328648; -l incr -q -W 78  -A snap_sessionid=1395330115 "[nsrsnap.c 2047] Calling start_savecmd..

    C:\PerfLogs

    [nsrsnap.c 2060] Calling snap_wait..

    [nsrsnap.c 2081] Calling parse_fs_logs_and_prefix..

    86745:nsrsnap_vss_save:Version information for C:\Program Files\EMC NetWorker\nsr\bin\nsrsnap_vss_save.exe:

      Original file name: nsrsnap_vss_save.exe

      Version: 2.4.1.122

      Comments: NetWorker Module for Microsoft Applications (x64)

    50423:nsrsnap_vss_save:nsrsnap_vss_save: Unknown Application Information parameter: DIRECT_ACCESS, may not be supported

    50411:nsrsnap_vss_save:nsrsnap_vss_save: Data mover host name not specified. Assuming local host: win-oec5sbjqvjj

    47363:nsrsnap_vss_save: nsrpsd service is running on WIN-OEC5SBJQVJJ.

    63335:nsrsnap_vss_save:NMM backup failed to complete successfully.

    Internal error.

    68151 1395330115 2 0 0 744 864 0 win-oec5sbjqvjj nsrsnap_vss_save NSR warning 39 nsrsnap_vss_save: Exiting with failure. 0

    Configuration:

    - snapshot enabled group

    - backup command : nsrsnap_vss_save.exe

    - application information :

    NSR_SNAP_TYPE=VSS

    NSR_VSS_FORCE_SYSTEM_PROVIDER=yes

    NSR_IGNORE_MISSING_SYSTEM_FILES=yes

    Am I missing anything ? How should I configure the Remote access field ? The NMM 2.4.1 Application Guide says to leave it empty

    Thanks

  • CarlosRojas

    1 Rookie

    •

    1693 Posts

    1573

    0

    Posted March 21st, 2014 01:00

    Hi Bogdan,

    Ensure that the version of Integration Components running inside the virtual machine is the same as the version of Hyper-V on the host.

    Check the output of the following command (run the command from the physical node that is currently hosting the application server.):

    nsrsnap_vss_save -s -?

    Backup command should be nsrsnap_vss_save (no -s required).


    Make sure Integration Services is installed on each virtual machine.

    Hyper-V CSV distributed backup supports only conventional backup from a temporary shadow copy (rollover). Proxy host backup and instant backup using persistent point-in-time shadow copies are not supported. In other words, the snapshot policy should indicate to take 1 snapshot, backup all, and retain 0.

    Set the save set attribute “Applications: \Microsoft Hyper-V\” in the client resource for the cluster virtual server to back up all the CSV VMs in the cluster.


    Add NSR_FEDERATED_BACKUP=yes to the client resource.

    Are you using any HW provider or just Microsoft Shadow Copy provider?

    What is the saveset you are specifying? Are all VM's CSV?

    Have you tried to use the client wizard to create the resource for the Virtual Name client?

    Are you backing up physical nodes or the virtual?

    Thank you,
    Carlos

  • 187

    0

    Posted March 21st, 2014 02:00

    Hi Carlos,

    Integration Components are running the same version on Hyper-V nodes and VM

    nsrsnap_vss_save -s - command provides the correct savesets

    Integration Services is also installed.

    Save set used is Applications: \Microsoft Hyper-V\vm_name. I got the save set from the output of nsrsnap_vss_save -s

    NSR_FEDERATED_BACKUP=yes - I did not use this in the application information field, as I configured only the physical node instances for backup. As far as I know if the physical instance is

    configured, it alone can back up VMs residing on that particular node

    MS Shadow Copy is used

    All VMs are CSV

    I cannot create the Hyper-v cluster virtual client name using the client wizard, this is why I attempted to back up a VM using the physical node instance, so I am backing up the physical nodes

    Thanks,

    Bogdan