UNSOLVED
Z
Z_A_P_a0ce3f
1 Rookie
•
3 Posts
0
303
November 23rd, 2023 15:12
ME4 scheduled snapshots issue
Hello,
We have an issue with a ME4 system and scheduled snapshots. There are a number of volumes and most of them are configured for multiple snapshots.
For some time now not all scheduled snapshots can be executed. The error logged is:
Event:
Scheduler: Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large. (task: Vol0011-xxx, task type: TakeSnapshot)
EVENT ID:#A60954
EVENT CODE:362
EVENT SEVERITY:Error
EVENT TIME:2023-11-21 02:31:07
Midplane Serial Number:xxx
System Name:ME4024
Bundle A Version:GT280R011-01
MC A Version:GXM280R013-30
SC A Version:GTS280R10-29
Bundle B Version:GT280R011-01
MC B Version:GXM280R013-30
SC B Version:GTS280R10-29
Additional Information:unknown
Recommended Action:unknown
The system has plenty of storage space left. If we reduce the number of stored snapshots (manual deletion), the error does not occur for a while. We know that the snapshot space can be too small.
But too big?
Thanks.
0
0
Posted November 24th, 2023 11:28
Please see below for the list of vols, snapshots and corresponding tasks.
When currently trying to snap a 5tb vol, the operation fails with error "... the total snapshot space is too large". When snapping a 1tb vol, we can do so 4 times in a row but a 5th snap fails with the same error.
To us it very much looks like the total space for snapshots is somehow limited. Is it?
The only info we can find is this:
https://www.dell.com/support/manuals/en-us/powervault-me4024/me4_series_ag_pub/creating-snapshots?guid=guid-ef9f7a11-49db-46a0-a8b9-65837c61c791&lang=en-us
But the large-pools option is disabled on our system.
Is the any other snapshot (storage) limit?
Thanks# show volumes type base
Pool Name Total Size Alloc Size Class Type Large Virtual Extents Health Reason Action
----------------------------------------------------------------------------------------------------------------
A Vol0001 1024.0GiB 210.5GiB Virtual base Disabled OK
A Vol0002 100.0GiB 0iB Virtual base Disabled OK
A Vol0002-xxx 25.0TiB 7776.9GiB Virtual base Disabled OK
A Vol0002-user 20.0TiB 7948.4GiB Virtual base Disabled OK
A Vol0003 4096.0GiB 130.1GiB Virtual base Disabled OK
A Vol0004-xxx 1024.0GiB 376.9GiB Virtual base Disabled OK
A Vol0005-xxx 50.0TiB 39.3TiB Virtual base Disabled OK
A Vol0006-xxx 1024.0GiB 460.0MiB Virtual base Disabled OK
A Vol0007-xxx 5120.0GiB 1394.0GiB Virtual base Disabled OK
A Vol0008-xxx 1024.0GiB 112.0MiB Virtual base Disabled OK
A Vol0009-xxx 5120.0GiB 1223.9GiB Virtual base Disabled OK
A Vol0010-xxx 50.0TiB 5801.1GiB Virtual base Disabled OK
A Vol0011-xxx 25.0TiB 278.2GiB Virtual base Disabled OK
A Vol0020-xxx 8192.0GiB 0iB Virtual base Disabled OK
A Vol0021-xxx 16.0TiB 46.2GiB Virtual base Disabled OK
A Vol_tmp_01 100.0GiB 9.2GiB Virtual base Disabled OK
A Vol_tmp_02 100.0GiB 24.2GiB Virtual base Disabled OK
A Vol_tmp_04 10.0GiB 0iB Virtual base Disabled OK
A vol_tmp_03 100.0GiB 41.4GiB Virtual base Disabled OK
----------------------------------------------------------------------------------------------------------------
# show schedules
Name Specification Status Next Time Task To Run Error Message
---------------------------------------------------------------------------------------------------------------------------------------------------------
Vol0001SS001 Start 2020-06-05 01:00:00, Every 1 Days Ready 2023-11-25 01:00:00 Vol0001TS001
Vol0001SS002 Start 2020-09-27 02:00:00, Every 1 Days, Only Any Sunday Of Month Ready 2023-11-26 02:00:00 Vol0001TS002
Vol0001SS003 Start 2021-05-01 02:30:00, Every 1 Days, Only 1st Day Of Month Ready 2023-12-01 02:30:00 Vol0001TS003
Vol0002-xxxSS001 Start 2023-07-13 01:30:00, Every 1 Days Ready 2023-11-25 01:30:00 Vol0002-xxxTS001
Vol0002-xxxSS002 Start 2023-07-16 02:00:00, Every 1 Days, Only Any Sunday Of Month Ready 2023-11-26 02:00:00 Vol0002-xxxTS002
Vol0002-xxxSS003 Start 2023-08-01 02:30:00, Every 1 Days, Only 1st Day Of Month Ready 2023-12-01 02:30:00 Vol0002-xxxTS003
Vol0002-userSS001 Start 2023-06-07 01:00:00, Every 1 Days Ready 2023-11-25 01:00:00 Vol0002-userTS001
Vol0002-userSS002 Start 2023-06-11 02:00:00, Every 1 Days, Only Any Sunday Of Year Ready 2023-11-26 02:00:00 Vol0002-userTS002
Vol0002-userSS003 Start 2023-07-01 02:30:00, Every 1 Days, Only 1st Day Of Month Ready 2023-12-01 02:30:00 Vol0002-userTS003
Vol0003SS001 Start 2021-04-11 01:00:00, Every 1 Days Ready 2023-11-25 01:00:00 Vol0003TS001
Vol0003SS002 Start 2021-04-11 02:00:00, Every 1 Days, Only Any Sunday Of Year Ready 2023-11-26 02:00:00 Vol0003TS002
Vol0004-xxxSS001 Start 2020-08-17 23:30:00, Every 1 Days Ready 2023-11-24 23:30:00 Vol0004-xxxTS001
Vol0005-xxxSS001 Start 2020-08-20 02:30:00, Every 1 Days Ready 2023-11-25 02:30:00 Vol0005-xxxTS001
Vol0007-xxxSS001 Start 2020-08-20 02:30:00, Every 1 Days Ready 2023-11-25 02:30:00 Vol0007-xxxTS001
Vol0008-xxxSS001 Start 2023-08-30 02:30:00, Every 1 Days Ready 2023-11-25 02:30:00 Vol0008-xxxTS001
Vol0009-xxxSS001 Start 2021-11-30 02:30:00, Every 1 Days Ready 2023-11-25 02:30:00 Vol0009-xxxTS001
Vol0010-xxxSS001 Start 2021-11-30 02:30:00, Every 1 Days Ready 2023-11-25 02:30:00 Vol0010-xxxTS001
Vol0011-xxxSS001 Start 2023-08-30 02:30:00, Every 1 Days Ready 2023-11-25 02:30:00 Vol0011-xxxTS001
---------------------------------------------------------------------------------------------------------------------------------------------------------
# show tasks
Name Type Status State Error Message
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Vol0001TS001 TakeSnapshot Ready Start
Vol0001TS002 TakeSnapshot Ready Start
Vol0001TS003 TakeSnapshot Ready Start
Vol0002xxxTS001 TakeSnapshot Ready Start
Vol0002xxxTS002 TakeSnapshot Error CreateSnap Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
Vol0002xxxTS003 TakeSnapshot Error CreateSnap Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
Vol0002-userTS001 TakeSnapshot Error CreateSnap Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
Vol0002-userTS002 TakeSnapshot Error CreateSnap Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
Vol0002-userTS003 TakeSnapshot Error CreateSnap Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
Vol0003TS001 TakeSnapshot Ready Start
Vol0003TS002 TakeSnapshot Ready Start
Vol0004-xxxTS001 TakeSnapshot Ready Start
Vol0005-xxxTS001 TakeSnapshot Ready Start
Vol0007-xxxTS001 TakeSnapshot Ready Start
Vol0008-xxxTS001 TakeSnapshot Ready Start
Vol0009-xxxTS001 TakeSnapshot Ready Start
Vol0010-xxxTS001 TakeSnapshot Error CreateSnap Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
Vol0011-xxxTS001 TakeSnapshot Error CreateSnap Unable to create the snapshot. - The operation cannot be completed because the total snapshot space is too large.
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------0
November 24th, 2023 12:50
Hi, I've read this article. PowerVault ME4: What is Overcommitment and how does it work? | Dell Dominica The ME4 system can only store snapshots up to half of its total space. If this limit is reached, no new snapshots can be created and an error will show up. Just a brainstorming to avoid the snapshot space limit, you can either delete old snapshots, schedule fewer snapshots, or enable a larger pool option.(which will increase the limit to 80% of the total capacity of the system)
0
0
Posted November 24th, 2023 15:00
@Dell-ErmanO
Thanks for your answer and the link. It's an interesting reading and brings some light into the storage architecture of the ME4.
Although I can't find any information about a 50% snapshot limit in the document, I don't think that's our problem. Looks like our current snapshot size is way smaller.# show pools
Name Class Blocksize Total Size Avail Snap Size OverCommit Disk Groups Volumes Low Thresh Mid Thresh High Thresh Sec Fmt Health
---------------------------------------------------------------------------------------------------------------------------------------------
A Virtual 512 258.4TiB 192.2TiB 2278.5GiB Enabled 2 114 50.00 % 75.00 % 99.92 % 512e OK
---------------------------------------------------------------------------------------------------------------------------------------------But what about "... the FW supports a max of 1 PiB of Metadata Index."? Is it possible that we are hitting the metadata limit of the system?
0
November 24th, 2023 15:28
hmm when you mentioned it, I thought about whether this could be metadata. But I'm not sure, we might be missing something. Now if so you think you're reaching the metadata limit, review and reduce your volumes and snapshots if possible to free up metadata space might be helpful. However let's see how our community contributes.
DELL-Young E
Moderator
•
5421 Posts
•
37 Points
0
0
Posted November 23rd, 2023 23:25
Hello thanks for choosing Dell.
This schedule- What is the schedule setup? How often?
And this happens all of a sudden? It has been no problem till now?
Respectfully,
Social Media and Communities Professional
Dell Technologies | Enterprise Support Services
#IWork4Dell
Did I answer your query? Please click on ‘Mark as Accepted Answer’. ‘Thumbs up’ the posts you like!