symmetrix (dmx4) metavolume destruction & unmasking from the command line

problem: you read the previous post where i detailed how you do all kinds of cool stuff from the command line like identify free devices for use, create some files, run some symconfigure commands, and then finally put a lun out there on the wire so a host can use that bad boy! but now, the host is no longer using that metavolume we created and you need to return the metavolume back to the free pool of devices from which to choose. so, how is that accomplished?

okay, so if you remember from our last post (which was waaaaaay back a few months ago…it has been very busy at work!), we went all the way to the masking out of the device on the selected fas so that the server could see the device.  this new scenario will address how we clean up our previous allocation. before starting, this post assumes a number of things:

first, you need to ensure that the devices in question are no longer visible to your server/host.

second, you want to make sure you don’t need any of the data that is on the devices you are working with. by very sure, I mean that once we are done with the devices any/all data that was on them will be long gone and there is no way to get it back! remember, we are breaking down metavolumes so the data will definitely be gone! okay, on to the good stuff!

the first thing we want to do is to check each fa to see that our two devices (we are going to be working with devices 0400 and 0478 – these are different from the ones we created in the previous post, but they will still serve our purpose) are masked out to the server hbas in question:

[root@san]:[3.00.15]:[18:49:40]:[~]:[6842]#: symmaskdb database list Symmetrix ID : 000xxxxxxx51 Database Type : Type6 Last updated at : 09:27:48 AM on Mon Dec 28,2009 Director Identification : FA-8A Director Port : 0 User-generated Identifier Type Node Name Port Name Devices —————- —– ——————————— ——— 10000000c94bxxxx Fibre 10000000c94bxxxx 10000000c94bxxxx 0400 0478

as you can see, both devices (and these are 3TB metavolumes!) are present in the symmetrix masking database and masked out so that only the server with wwn 10000000c94bxxxx can see both devices.

the next step is to “unmask” the devices so that the server with the aforementioned hba/wwn can no longer see the device. this can be done by running the following command:

[root@san]#: symmask -sid 51 -wwn 10000000c94bxxxx -dir all -p 0 remove devs 0400,0478

now, a couple of quick notes are in order. as with most things there are many different ways to accomplish a task (especially one that has anything remotely close to do with unix!). this step, which i have separated from an upcoming task (the actual removal of the devices from the fa – which is different than what we just did above) because this was how i was shown how to do this. i will also cover the other method below which basically combines the removal of the device from the fa with the unmasking of the device from the symmetrix device masking database. also, you will notice that i used “-dir all”, but i only showed you output indicating that the device was on a single fa. in reality, it is on 8 fas but in order to keep things somewhat manageable i am only showing you the output for a single fa (all 8 would make this post a monster to get through!). but, for the sake of argument, let’s say that the two devices were only on a single fa or that you only wanted to remove it from a single fa…what command would you use? the command would be:

[root@san]#: symmask -sid 51 -wwn 10000000c94bxxxx -dir 8a -p 0 remove devs 0400,0478

now, you would obviously not remove the device from a single fa if it was on 8 of them with the intention of breaking down the metavolume and returning the devices to the free pool, but i want to make sure you know you could, if you wanted to, just remove the device from a single fa if that was what you were trying to do. so, how do we confirm that the device has actually been removed? we could look at the server, but we might have changed the zoning on our brocade/cisco fiber channel switches. i always like to make sure the commands i run have the expected outcome. to check, you would run the following command:

[root@san]:[3.00.15]:[18:49:40]:[~]:[6842]#: symmaskdb database list Symmetrix ID : 000xxxxxxx51 Database Type : Type6 Last updated at : 09:27:48 AM on Mon Dec 28,2009 Director Identification : FA-8A Director Port : 0 User-generated Identifier Type Node Name Port Name Devices —————- —– ——————————— ——— 10000000c94bxxxx Fibre 10000000c94bxxxx 10000000c94bxxxx None

as you can now see, the hba/wwn in the symmetrix masking database shows that it can see “None”. you should check to make sure that, if you have the device on multiple fas, each fa that your hba/wwn has access to and where this device was masked out shows up with “None”.

now that we have “unmasked” the devices and made it so that the devices are no longer visible to the server with the hba/wwn listed above we want to remove these two devices off of the actual fas where they were placed. if you remember from the previous post on this topic, the symcfg command will now be used. here is the command we run to see at which address locations we initially placed the devices:

[root@san]#: symcfg -sid 51 -fa 8a -p 0 list -address Symmetrix ID: 000xxxxxxx51 Director Device Name Attr Address———————- —————————– —- ————– Ident Symbolic Port Sym Physical VBUS TID LUN —— ——– —- —- ———————– —- — — FA-8A 08A 0 0022 Not Visible VCM 0 00 000 0053 Not Visible (M) 0 00 001 0152 Not Visible (M) 0 00 002 0206 Not Visible (M) 0 00 003 0218 Not Visible (M) 0 00 004 0400 Not Visible (M) 0 00 050 0478 Not Visible (M) 0 00 051 Total —- Mapped Devices: 37 Including Metamembers: 1304 Available Addresses: 2792 (s) Legend for Available address: (s): The Available Addresses for a director are shared among its ports (shared)

we can now see that the two devices (0400 and 0478) were placed at addresses 050 and 051 respectively. once we confirm each of the fas where the devices reside, we can construct the text file to execute their removal.

before we do that though, we will need to change the devices from their “Ready” state to one of “Not Ready” or else we will be unable to remove them from the fas in question. while i don’t officially know the reason for this, it is a good safeguard from having someone accidentally remove a device from an fa when that device is actually in use! to prove that you cannot remove a device in a “Ready” state, i actually attempted to remove both device 0400 and 0478 and got the following error:

[root@san]#: symconfigure -f dev_3tb_meta_unmap.cmd commit Establishing a configuration change session……………Established. Processing symmetrix 000xxxxxxx51 Performing Access checks…………………………….Allowed. Checking Device Reservations…………………………Allowed. Definition 1 is in error: Unmap change: Device must be write-disabled or not ready Device 0400 generated the failure Terminating the configuration change session…………..Done. The configuration change session has failed.

at this point make extra sure that you want to remove these devices! so the next question is how do you change the state of a metavolume (or a hypervolume) from “Ready” (which we will see shows up as “RW” in the output of the symdev command) to “Not Ready”? there are two ways that i know of, but one is far more preferable than the other. the one i won’t cover here (because it is a bit more tricky) is the creation of a symdg group with the devices in question.

the method i will cover, and the one you will probably want to use in your travels, is the command line method. first, in order to check to see if your device is in the “RW” or “Ready” state you can issue the following commands:

[root@san]#: symdev -sid 51 list | grep 0400 0400 Not Visible ***:* 16D:D3 RAID-6 N/Grp’d (M) RW 3150000 [root@san]#: symdev -sid 51 list | grep 0478 0478 Not Visible ***:* 01D:D0 RAID-6 N/Grp’d (M) RW 3150000

as you can see, each of our devices is listed as “RW” so we will need to change them to “Not Ready” or “unwriteable” (yeah, i know it isn’t a word but it fits). to do this we will want to issue the following commands:

[root@san]#: symdev -sid 51 not_ready 0400 Execute a ‘Not Ready’ Device operation for device ‘0400’ (y/[n]) ? y ‘Not Ready’ Device operation successfully completed for the device. [root@san]#: symdev -sid 51 not_ready 0478 Execute a ‘Not Ready’ Device operation for device ‘0478’ (y/[n]) ? y ‘Not Ready’ Device operation successfully completed for the device.

again, we can check to see that the devices have changed from the “RW” state to the “Not Ready” state by running the following command:

[root@san]#: symdev -sid 51 list | grep 0400 0400 Not Visible ***:* 16D:D3 RAID-6 N/Grp’d (M) NR 3150000 [root@san]#: symdev -sid 51 list | grep 0478 0478 Not Visible ***:* 01D:D0 RAID-6 N/Grp’d (M) NR 3150000

the “RW” from before has now changed to “NR” for “Not Ready”. at this point we are now ready to actually remove the devices from the fas! in order to do that, we can reuse and simply edit the file we created to map the devices (i personally create a new file all together for continuity purposes). the file will look like the following:

[root@san]#: cat dev_3tb_meta_unmap.cmd ## ## Created on 12.28.09 by tpbonfi for server DEV ## unmap dev 0400 from dir 8a:0; unmap dev 0400 from dir 9b:0; unmap dev 0400 from dir 8d:0; unmap dev 0400 from dir 9d:0; unmap dev 0478 from dir 8a:0; unmap dev 0478 from dir 9b:0; unmap dev 0478 from dir 8d:0; unmap dev 0478 from dir 9d:0; ## ## End of file…

again, and i can’t stress this enough, it is a GREAT idea to place comments in your files with the change control number, project code, and/or any notes that will serve you in the future should you need to troubleshoot or go back and try to figure out why or what you were doing on that day. also, my example shows some additional fas.

next, you will execute the symconfigure command with your newly created file name as the argument to the “-f” option. i always run through all three phases (preview, prepare, and commit) just to be certain. here is what that looks like:

[root@san]#: symconfigure -f dev_3tb_meta_unmap.cmd preview Establishing a configuration change session……………Established. Processing symmetrix 000xxxxxxx51 Performing Access checks…………………………….Allowed. Checking Device Reservations…………………………Allowed. Submitting configuration changes……………………..Submitted Locking devices…………………………………….Locked. Validating configuration changes……………………..Validated. Closing configuration change request………………….Closed. Terminating the configuration change session…………..Done. The configuration change session has completed successfully. [root@san]#: symconfigure -f dev_3tb_meta_unmap.cmd prepare Establishing a configuration change session……………Established. Processing symmetrix 000xxxxxxx51 Performing Access checks…………………………….Allowed. Checking Device Reservations…………………………Allowed. Submitting configuration changes……………………..Submitted Locking devices…………………………………….Locked. Validating configuration changes……………………..Validated. Initiating PREPARE of configuration changes……………Prepared. Closing configuration change request………………….Closed. Terminating the configuration change session…………..Done. The configuration change session has completed successfully. [root@san]#: symconfigure -f dev_3tb_meta_unmap.cmd commit Establishing a configuration change session……………Established. Processing symmetrix 000xxxxxxx51 Performing Access checks…………………………….Allowed. Checking Device Reservations…………………………Allowed. Submitting configuration changes……………………..Submitted Locking devices…………………………………….Locked. Validating configuration changes……………………..Validated. Initiating PREPARE of configuration changes……………Prepared. Initiating COMMIT of configuration changes…………….Queued. COMMIT requesting required resources………………….Obtained. Step 005 of 013 steps……………………………….Executing. Step 013 of 013 steps……………………………….Executing. Local: COMMIT……………………………………..Done. Terminating the configuration change session…………..Done. The configuration change session has successfully completed.

now, to confirm that they have been removed from the fas on to which they were placed for use, just run the following command:

[root@san]#: symcfg -sid 51 -fa 8a -p 0 list -address Symmetrix ID: 000xxxxxxx51 Director Device Name Attr Address ———————- —————————– —- ————– Ident Symbolic Port Sym Physical VBUS TID LUN —— ——– —- —- ———————– —- — — FA-8A 08A 0 0022 Not Visible VCM 0 00 000 0053 Not Visible (M) 0 00 001 0152 Not Visible (M) 0 00 002 0206 Not Visible (M) 0 00 003 0218 Not Visible (M) 0 00 004 Total —- Mapped Devices: 35 Including Metamembers: 1064 Available Addresses: 3032 (s) Legend for Available address: (s): The Available Addresses for a director are shared among its ports (shared)

one additional check to run (and one that is HIGHLY recommended) is to run the symdev command with the -noport option. you might be asking why this is so highly recommended. the reason for this final check is to make sure you removed the devices in question from all the ports you had originally placed them on…maybe you overlooked a single port by accident or because you are not getting enough sleep again, you can never be to certain, so invest 15 seconds to save yourself what could end up being much more work. to do this i use the following command:

[root@san]#: symdev -sid 51 list -noport | more Symmetrix ID: 000xxxxxxx51 Device Name Directors Device ————————— ————- ————————————- Cap Sym Physical SA DA :IT Config Attribute Sts (MB) ————————— ————- ————————————- 03FA Not Visible ???:? 01B:C3 RAID-6 N/Grp’d RW 26250 03FB Not Visible ???:? 01A:CA RAID-6 N/Grp’d RW 26250 03FC Not Visible ???:? 01D:D2 RAID-6 N/Grp’d RW 26250 03FD Not Visible ???:? 16C:D6 RAID-6 N/Grp’d RW 26250 03FE Not Visible ???:? 16C:D8 RAID-6 N/Grp’d RW 26250 03FF Not Visible ???:? 16C:D4 RAID-6 N/Grp’d RW 26250 0400 Not Visible ???:? 16D:D3 RAID-6 N/Grp’d (M) NR 3150000 0478 Not Visible ???:? 01D:D0 RAID-6 N/Grp’d (M) NR 3150000 04F0 Not Visible ???:? 01C:C0 RAID-6 N/Grp’d RW 26250

you can see that both of the devices in question now show back up in the output and notice that the “NR” is still there (we will change this back as part of the next few steps).

so, at this point we have successfully unmasked the device from the hba/wwn of the server that was using it, changed the device state to “Not Ready” and removed the device from the fas on to which it had been placed for use. next will be the total destruction and annihilation of the metavolume back into the many tiny hypervolumes that currently comprise it…240 hypers in fact!

what we are going to do first, though, is to change the metavolumes to the “Ready” or “RW” state. we can do this and still destroy (or, using the official command term, dissolve the devices). i have never tried to dissolve metavolumes without first changing them to “RW” status, but it can safely be assumed that it will work just the same (i feel a follow-on post coming!).

*update: it doesn’t work the same if you leave them in the ‘NR’ status! check out my post here!

here are the commands to change the two metavolumes to “RW” status:

[root@san]#: symdev -sid 51 ready 0400 Execute a ‘Ready’ Device operation for device ‘0400’ (y/[n]) ? y ‘Ready’ Device operation successfully completed for the device. [root@san]#: symdev -sid 51 ready 0478 Execute a ‘Ready’ Device operation for device ‘0478’ (y/[n]) ? y ‘Ready’ Device operation successfully completed for the device.

again, a quick check with the symdev command just to be sure:

[root@san]#: symdev -sid 51 list -noport | more Symmetrix ID: 000xxxxxxx51 Device Name Directors Device ————————— ————- ————————————- Cap Sym Physical SA DA :IT Config Attribute Sts (MB) ————————— ————- ————————————- 03FA Not Visible ???:? 01B:C3 RAID-6 N/Grp’d RW 26250 03FB Not Visible ???:? 01A:CA RAID-6 N/Grp’d RW 26250 03FC Not Visible ???:? 01D:D2 RAID-6 N/Grp’d RW 26250 03FD Not Visible ???:? 16C:D6 RAID-6 N/Grp’d RW 26250 03FE Not Visible ???:? 16C:D8 RAID-6 N/Grp’d RW 26250 03FF Not Visible ???:? 16C:D4 RAID-6 N/Grp’d RW 26250 0400 Not Visible ???:? 16D:D3 RAID-6 N/Grp’d (M) RW 3150000 0478 Not Visible ???:? 01D:D0 RAID-6 N/Grp’d (M) RW 3150000 04F0 Not Visible ???:? 01C:C0 RAID-6 N/Grp’d RW 26250

okay, we are ready to dissolve the metavolumes! as you might have guessed since the creation of the metavolumes was done with a file, so too will the destruction of them be done. below is the text file you would create to dissolve the metavolumes:

## ## File to dissolve the two (2) metavolumes created for the DEV ## test between 12.22.2009 and 12.28.2009 ## dissolve meta dev 0400; dissolve meta dev 0478; ## ## End of file…

the final step will be to execute the symconfigure command once again using the name of our file as the argument to the “-f” command line option (and, as always, i run it through the preview/prepare/commit cycles just to be sure). here is what that will look like:

[root@san]#: symconfigure -f dev_dissolve_3tb_metaone.cmd preview Establishing a configuration change session……………Established. Processing symmetrix 000xxxxxxx51 Performing Access checks…………………………….Allowed. Checking Device Reservations…………………………Allowed. Submitting configuration changes……………………..Submitted Locking devices…………………………………….Locked. Validating configuration changes……………………..Validated. Closing configuration change request………………….Closed. Terminating the configuration change session…………..Done. The configuration change session has completed successfully. [root@san]#: symconfigure -f dev_dissolve_3tb_metaone.cmd prepare Establishing a configuration change session……………Established. Processing symmetrix 000xxxxxxx51 Performing Access checks…………………………….Allowed. Checking Device Reservations…………………………Allowed. Submitting configuration changes……………………..Submitted Locking devices…………………………………….Locked. Validating configuration changes……………………..Validated. Initiating PREPARE of configuration changes……………Prepared. Closing configuration change request………………….Closed. Terminating the configuration change session…………..Done. The configuration change session has completed successfully. [root@san]#: symconfigure -f dev_dissolve_3tb_metaone.cmd commit Establishing a configuration change session……………Established. Processing symmetrix 000xxxxxxx51 Performing Access checks…………………………….Allowed. Checking Device Reservations…………………………Allowed. Submitting configuration changes……………………..Submitted Locking devices…………………………………….Locked. Validating configuration changes……………………..Validated. Initiating PREPARE of configuration changes……………Prepared. Initiating COMMIT of configuration changes…………….Queued. COMMIT requesting required resources………………….Obtained. Step 003 of 008 steps……………………………….Executing. Local: COMMIT……………………………………..Done. Terminating the configuration change session…………..Done. The configuration change session has successfully completed.

you guessed it…one final check! i use the symdev command with the “-noport” option to confirm that my 240 hypervolumes are now back in the free pool of storage. here is what that looks like:

[root@san]#: symdev -sid 51 list -noport | more Symmetrix ID: 000xxxxxxx51 Device Name Directors Device ————————— ————- ————————————- Cap Sym Physical SA DA :IT Config Attribute Sts (MB) ————————— ————- ————————————- 03FA Not Visible ???:? 01B:C3 RAID-6 N/Grp’d RW 26250 03FB Not Visible ???:? 01A:CA RAID-6 N/Grp’d RW 26250 03FC Not Visible ???:? 01D:D2 RAID-6 N/Grp’d RW 26250 03FD Not Visible ???:? 16C:D6 RAID-6 N/Grp’d RW 26250 03FE Not Visible ???:? 16C:D8 RAID-6 N/Grp’d RW 26250 03FF Not Visible ???:? 16C:D4 RAID-6 N/Grp’d RW 26250 0400 Not Visible ???:? 16D:D3 RAID-6 N/Grp’d RW 26250 0401 Not Visible ???:? 01D:DA RAID-6 N/Grp’d RW 26250 0402 Not Visible ???:? 01A:C2 RAID-6 N/Grp’d RW 26250 0403 Not Visible ???:? 01A:C6 RAID-6 N/Grp’d RW 26250 0404 Not Visible ???:? 01A:C8 RAID-6 N/Grp’d RW 26250 0405 Not Visible ???:? 01A:C4 RAID-6 N/Grp’d RW 26250 0406 Not Visible ???:? 16A:C3 RAID-6 N/Grp’d RW 26250 0407 Not Visible ???:? 16B:CA RAID-6 N/Grp’d RW 26250 0408 Not Visible ???:? 16C:D2 RAID-6 N/Grp’d RW 26250 0409 Not Visible ???:? 01D:D6 RAID-6 N/Grp’d RW 26250

super! we are now finished with our unmasking and breaking down of an emc metavolume…two in fact!

okay, i hope that this helps someone out there on the wire and best of luck from the dark side…

devnull