UNSOLVED

sjperson

updated

16 years ago

S

sjperson

7 Posts

0

1359

December 8th, 2010 08:00

Tapes marked 'full' with lots of space left

Hello,

For some reason, most of the media in our library gets marked as 'full' well before it should.  Often times, media with less than 1MB of data is marked 'full'.    Because of this, we're burning through tapes at a pace we can't keep up with while sending lots of nearly-empty tapes offsite for storage.

Here's the setup.

Server:

     V440 with Solaris 10

     SUNWebss 7.4.5,REV=726

Library:

     StorageTek L700 attached via SCSI

     16 x SDLT320 drives attached via SCSI

Clients:

     Mix of 7.4 and 7.5

     Mix of Solaris 8 through 10

Help!

Thanks.

Steve Person

Systems/Network Administrator

Oracle America

Hillsboro, Oregon

  • DavidHampson-rY

    1 Rookie

    •

    294 Posts

    521

    1

    Posted December 8th, 2010 09:00

    Steve, do you get any error messages before the tapes are marked as full?  My guess would be that you have a faulty device somewhere, I would suggest looking at several of the tapes that were marked as full and seeing if there is any pattern to which drive(s) they were using.

  • sjperson

    7 Posts

    521

    0

    Posted December 8th, 2010 10:00

    Thanks David.  It looks like that's the problem.  All of my drives have at least a few 'Hard Error' messages recorded.  This has been going on for a long time but seems to be getting worse.  I'll get the service folks out to look at the drives.  Is there a way to change the behavior of EBS such that it doesn't mark the media full on a hard error?

    Thanks.

    --Steve

    server: /nsr/logs $ for drive in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
    > do print -n "/dev/rmt/${drive}cbn: "
    > grep "Hard error" messages|grep /dev/rmt/${drive}cbn|wc -l
    > done
    /dev/rmt/0cbn:       21
    /dev/rmt/1cbn:       20
    /dev/rmt/2cbn:       54
    /dev/rmt/3cbn:       21
    /dev/rmt/4cbn:       30
    /dev/rmt/5cbn:       21
    /dev/rmt/6cbn:       26
    /dev/rmt/7cbn:       11
    /dev/rmt/8cbn:       12
    /dev/rmt/9cbn:        8
    /dev/rmt/10cbn:        9
    /dev/rmt/11cbn:        4
    /dev/rmt/12cbn:       35
    /dev/rmt/13cbn:       27
    /dev/rmt/14cbn:       26
    /dev/rmt/15cbn:       25

  • ble1

    6 Operator

    •

    14354 Posts

    •

    56186 Points

    521

    0

    Posted December 8th, 2010 10:00

    No, this is designed way and it should stay like that.  What you get is an I/O error from device which indicates bad drive or bad tape.  NW can't know what it is and it simply marks tape full and it is up to backup admin to investigate this further.  I somehow doubt that you can have so many hard errors on all devices unless something really basic is wrong.  All drives on library?  I doubt...unless really heavily used and in desperate need for cleaning.  HBA/SCSI card or cable on server? More probably even I doubt all drives go over same card.  Whatever it is, you should be able to see traces of it in system logs too.

  • DavidHampson-rY

    1 Rookie

    •

    294 Posts

    521

    0

    Posted December 8th, 2010 11:00

    I'd echo Hrvoje on this; your problem is distributed across all your drives and it is improbable that all drives would fail simultaneously; it is also unlikely that so many drives would share connectivity so problem is more fundamental.  I once saw a similar problem albeit in a smaller environment; the customer installed their tape library in the office next to the shredder and their tapes got paper dust in them and became useless.  I'm not necessarily saying you will have the same problem, but you need to take a broad overview of your system and find out if there is something that is out of the ordinary that maybe affecting things.  Your message did imply that though some tapes fail other tapes are successful - do you have any approximate percentage figures as to how many tapes are affected by the problem?  Have you checked whether the affected tapes are all from the same batch?

  • sjperson

    7 Posts

    521

    0

    Posted December 8th, 2010 12:00

    Thanks guys.  The drives are distributed across 8 2-port SCSI cards so I doubt I/O is the issue.  I don't have a history on the media itself as I didn't procure it.  This has been happening for quite a while and I'm just getting around to addressing it.  It's possible that it could be a combination of a couple bad drives and multiple bad pieces of media.   I've looked through the logs for messages around the same time as the media became marked as full and see a variety of errors.  Some of them have 'drive status is UNRECOVERED READ ERROR' and some have 'TapeAlert flag "Hard error" has been posted for /dev/rmt/Xcbn'.  I've got a hardware ticket open, so hopefully our support folks can test the drives for issues.  I'm not sure how to go about testing the media itself other than to keep a close eye on individual tapes once the drives are in a known-good state.

  • DavidHampson-rY

    1 Rookie

    •

    294 Posts

    521

    0

    Posted December 9th, 2010 04:00

    I would suggest taking one of the pieces of media with only 1MB of data on it, label it to a test pool and carry out some tests on it, see if it is consistent in only backing up a small amount of data in different drives.  If it does appear to be the media have your tape vendor run some tests on it.