symmetrix (dmx4) metavolume creation & masking from the command line

problem: you don’t have access to emc’s symmetrix management console (smc) or the ecc suite of tools and you need to create your metavolume from the command line to include masking, fa mapping placement, and a handy way to keep track of all the metavolumes you are creating and mapping.  or, the guys you work with are terrified that you might learn what they know and won’t tell you anything!

to be honest, i have only used the smc a few times and ecc even fewer.  i love the command line.  i was born and raised on it (thank you again freddy mac!).  i am sure it is some kind of control issue that could be linked to my childhood, but that is probably a topic covered under another blog somewhere so i will just leave that alone for now.  i really need to thank rick naquin for sharing with me some steps i forgot from back in the day when i was at usi.  i have taken those steps and have chuch’ed them up a bit with some new information that i have recently collected.  so, here are the steps to create a metavolume from the command line and, of course, here are the caveats for this to work:

1. the host you are using (we use a rhel5 box) must have fiber channel connectivity/visibility to the symmetrix array in question.  you will also need to have emc’s solutions enabler software loaded on the host (we are using v7.0 which i believe is the latest and greatest).  you can check by running the following command on your rhel5 box:

[root@sanadmin]#: rpm -qa | grep -i symcli
symcli-datacore-V7.0.0-0
symcli-symrecover-V7.0.0-0
symcli-storbase-V7.0.0-0
symcli-core-V7.0.0-0
symcli-symcli-V7.0.0-0
symcli-datastorbase-V7.0.0-0
symcli-storfull-V7.0.0-0
symcli-srmbase-V7.0.0-0
symcli-star_perl-V7.0.0-0

2. if you have the software installed, you can confirm that you have visibility back to the array by running the following command (and you should see something similar to the following):

[root@sanadmin]:[3.00.15]:[12:37:53]:[~]:[4573]#: syminq Device              Product                   Device —————- ————————— ——————— Name      Type   Vendor    ID           Rev  Ser Num      Cap (KB) —————- ————————— ——————— /dev/sda         Dell      VIRTUAL DISK 1028 Dell Int    142577664 /dev/sdb  GK     EMC       SYMMETRIX    5773 5100022000       2880 /dev/sdc  GK     EMC       SYMMETRIX    5773 5100022000       2880 /dev/sdd  GK     EMC       SYMMETRIX    5773 5100022000       2880 /dev/sde  GK     EMC       SYMMETRIX    5773 5100022000       2880 /dev/sdf  GK     EMC       SYMMETRIX    5773 5100022000       2880 /dev/sdg  GK     EMC       SYMMETRIX    5773 5100022000       2880 /dev/sdh  GK     EMC       SYMMETRIX    5773 5100022000       2880 /dev/sdi  GK     EMC       SYMMETRIX    5773 5100022000       2880 /dev/dm-0        Dell      VIRTUAL DISK 1028 Dell Int    142577664 /dev/dm-1        Dell      VIRTUAL DISK 1028 Dell Int    142577664

those tiny devices with the “gk” next to them are your gatekeeper devices.  if you don’t see any of those, none of the commands that follow (well, the symconfigure commands anyway) will not work as they require communication with a gatekeeper device to function properly…or at all.

3. if you have visibility to more than one symmetrix disk array, you will need to know the serial number or at least the last two digits of the serial number.  you can actually figure the last two digits of the serial number out by looking over the output of the syminq command above.  taking a look at one of the lines from above we see:

/dev/sdb  GK     EMC       SYMMETRIX    5773      5100022000       2880

if you look at the sixth column over (right after the microcode software version listing of 5773) you will see a long number that might appear to be random, but actually if you look at the first two digits they will correspond to the last two digits of your symmetrix array.  for us, they are 51 and that is what i will use in the examples that follow.

4. all of your brocade/cisco zoning is done for the host you to which you want to allocate the storage.  i say brocade/cisco because, let’s be honest, is anyone really using anything else with their symmetrix array?

5. you will notice that my command line prompt changes from time to time in the output below.  i had to make some minor modifications to it so that the commands i am running can be seen…those are far more important than the time, version of bash, and current working directory!

those are all the caveats i can think of for now so let’s move on to the actual steps involved with creating our metavolume, selecting the devices we are going to use, masking them out to the host, and getting ourselves up and running with some emc storage!

1. picking the devices: this is really the first question i asked when i was first exposed to the command line back in the day – with no gui to see what is there, how will i know what devices i am going to pick to make my metavolume.  or more importantly, how do i ensure that i don’t pick a device that is already in use!  this first step revolves around the “symdev” command and its options.  probably the most important task to attend to first is to refresh/re-read the current devices on the symmetrix with which you are working.  yeah, you might think that nothing has changed, but trust me, things to change and especially if you are on a team where multiple team members have access to your array.  to refresh everything just enter the following command:

[root@sanadmin]:[3.00.15]:[15:47:55]:[~]:[4622]#: symcfg discover This operation may take up to a few minutes. Please be patient…

you always want to be working with the latest information, so the symcfg discover command is important.  to see a list of the devices that are available for use, but not assigned/mapped out to an fa you can run the following command:

# symdev -sid 51 list -noport

(you can pipe it to more if you want to walk through the file and this output is extremely truncated at the end for the sake of brevity…and sanity!  when you run the symdev list command without using the noport option, you will get a listing of every single device in the array…not what we want here)

Symmetrix ID: 000190103951 Device Name           Directors                  Device ————————— ————- ————————————- Cap Sym  Physical               SA :P DA :IT  Config        Attribute    Sts   (MB) ————————— ————- ————————————- 03E2 Not Visible            ???:? 01A:D3  RAID-6        N/Grp’d      RW   26250 03E3 Not Visible            ???:? 01B:DA  RAID-6        N/Grp’d      RW   26250 03E4 Not Visible            ???:? 01C:C2  RAID-6        N/Grp’d      RW   26250 03E5 Not Visible            ???:? 16D:C6  RAID-6        N/Grp’d      RW   26250 03E6 Not Visible            ???:? 16D:C8  RAID-6        N/Grp’d      RW   26250

so, as you can see, i have a number of RAID-6 devices above that i can use: 03e2, 03e3, 03e4, 03e5, and 03e6.  we will go ahead and say that i am going to create a monster metavolume using four of these guys (the first four).

2. now that we know that we are using hypervolumes 03e2, 03e3, 03e4, and 03e5 we will use a method that i love and will credit to steven fletcher and wes “clinton” garner.  they came up with a simple methodology that will allow you to keep track of all the changes/configurations/mappings by creating configuration files based on the client/hostname and whether or not the file is being used as a metavolume creation file or a metavolume mapping file.  using the scenario i have here, we will say that i am creating this metavolume for the server “sanadmin”.  so, for the filename i will use something very descriptive such as the following:

sanadmin_logspace_082009_meta.cmd

basically, the file you create with your favorite text editor (mine is vi) has the hostname of the server for which you are creating the metavolume, followed by the reason – in our case i am using the space for log files or something similar – the date the file was created (which should probably be the day you ran the symconfigure command using the file), the fact that it is a metavolume (which might be considered redundant, but hey, why not), and then finally, the fact that this is a “cmd” or command file for creating this metavolume.  the “cmd” portion of the file name will make more sense once i talk about the mapping file.

again, what i love about this method is that you can then go back and look to see what you did, for which server, and on which date.  i have been in some other shops where they just reuse the same file over and over again and i have also seen that type of methodology lead to pandemonium.  the syntax for the file we would create using the scenario we have above will be the following:

form meta from dev 03e2 config=striped, stripe_size=1920; add dev 03e3:03e5 to meta 03e2;

be sure you use the meta head device number twice in the syntax…for us it will be device 03e2.  now, if you are creating more than one metavolume (which you can do in the same file) you would simply follow the two lines above with the next two lines indicating which devices you are using and following the syntax above.

3. now that we have the devices selected and our metavolume command file created and ready we will use the symconfigure command to first test the syntax of the file and then to commit the changes for processing by the symmetrix array.  now, you don’t have to run the “preview” option with the symconfigure command if you don’t want to, but why not use a little caution?  the best analogy i can give you for not first running the “preview” option is the following:  very rarely do i first close my eyes and then run across a busy freeway to get to the other side…so why not take a few minutes and “look both ways” before you commit the changes.  the preview option will check the syntax of your file and i can honestly say that this option has saved me from disaster about a dozen times for just as many reasons.  here is the output and the commands you would run to create the metavolume:

[root@sanadmin]:[~]:[4577]#: symconfigure -f sanadmin_logspace_082009_meta.cmd preview Establishing a configuration change session……………Established. Processing symmetrix 000190103951 Performing Access checks…………………………….Allowed. Checking Device Reservations…………………………Allowed. Submitting configuration changes……………………..Submitted Locking devices…………………………………….Locked. Validating configuration changes……………………..Validated. Closing configuration change request………………….Closed. Terminating the configuration change session…………..Done. The configuration change session has completed successfully. [root@sanadmin]:[~]:[4578]#: symconfigure -f sanadmin_logspace_082009_meta.cmd prepare Establishing a configuration change session……………Established. Processing symmetrix 000190103951 Performing Access checks…………………………….Allowed. Checking Device Reservations…………………………Allowed. Submitting configuration changes……………………..Submitted Locking devices…………………………………….Locked. Validating configuration changes……………………..Validated. Initiating PREPARE of configuration changes……………Prepared. Closing configuration change request………………….Closed. Terminating the configuration change session…………..Done. The configuration change session has completed successfully. [root@sanadmin]:[~]:[4579]#: symconfigure -f sanadmin_logspace_082009_meta.cmd commit Establishing a configuration change session……………Established. Processing symmetrix 000190103951 Performing Access checks…………………………….Allowed. Checking Device Reservations…………………………Allowed. Submitting configuration changes……………………..Submitted Locking devices…………………………………….Locked. Validating configuration changes……………………..Validated. Initiating PREPARE of configuration changes……………Prepared. Initiating COMMIT of configuration changes…………….Queued. COMMIT requesting required resources………………….Obtained. Step 002 of 008 steps……………………………….Executing. Local:  COMMIT……………………………………..Done. Terminating the configuration change session…………..Done. The configuration change session has successfully completed.

i forgot to mention the “prepare” option which i also run for completeness and to make sure there are no issues.

4. okay, so the session is complete but how can i tell if my device was actually created?  we go back to the symdev command we ran back in step #1.  you should expect to see something similar to the following:

[root@sanadmin]:[3.00.15]:[13:22:32]:[~]:[4580]#: symdev -sid 51 list -noport | more Symmetrix ID: 000190103951 Device Name           Directors                  Device ————————— ————- ————————————- Cap Sym  Physical               SA DA :IT  Config        Attribute    Sts   (MB) ————————— ————- ————————————- 03E2 Not Visible            ???:? 01A:D3  RAID-6        N/Grp’d  (M) RW  105000 03E6 Not Visible            ???:? 16D:C8  RAID-6        N/Grp’d      RW   26250

as you can see, the meta head 03e2 is still listed and it now has the designation “(M)” next to its RW status.  you can also see that the size has now changed as well and the devices that comprise the metavolume (with the exception of the meta head) are no longer listed.  the output skips from 03e2 to 03e6.  if you wanted to see the actual members listed out you would run the command “symdev -sid 51 list | more” and if we did that we would see the following:

03E2 Not Visible            ???:? 01A:D3  RAID-6        N/Grp’d  (M) RW  105000 03E3 Not Visible            ???:? 01B:DA  RAID-6        N/Grp’d  (m) RW       – 03E4 Not Visible            ???:? 01C:C2  RAID-6        N/Grp’d  (m) RW       – 03E5 Not Visible            ???:? 16D:C6  RAID-6        N/Grp’d  (m) RW       – 03E6 Not Visible            ???:? 16D:C8  RAID-6        N/Grp’d      RW   26250

the meta members are designated with a “(m)” and follow the meta head.  so, now that we see that our metavolume has been created, we can confirm the meta head is device 03e2, and the size is now ~105gb total.  we can now prepare the map file which will allow us to map the device to the front end host ports (i commonly call these fas and can’t recall hearing them called anything else) on which we want to provide access.

7. similar to step #1 (where we needed to figure out what devices we could safely use) we now need to figure out where to place the metavolume we have created.  we need to figure out on which fas and at which address location.  i will even show a preview example of what to expect when you attempt to place the device at a location that is already being used by another device (something you want to avoid doing!).  to see the current address mappings on each fa you would run the following command:

[root@sanadmin]:[~]:[4591]#: symcfg -sid 51 -fa 8a -p 0 list -address Symmetrix ID: 000190103951 Director                 Device Name          Attr     Address ———————- —————————– —- ————– Ident   Symbolic  Port Sym   Physical                     VBUS  TID  LUN ——  ——–  —- —-  ———————–      —-  —  — FA-8A   08A       0    0022  Not Visible             VCM     0   00  000 0053  Not Visible             (M)     0   00  001 0152  Not Visible             (M)     0   00  002 0206  Not Visible             (M)     0   00  003 0218  Not Visible             (M)     0   00  004 022A  Not Visible             (M)     0   00  005 0301  Not Visible             (M)     0   00  006 0325  Not Visible             (M)     0   00  007 0349  Not Visible             (M)     0   00  008 036D  Not Visible             (M)     0   00  009 0391  Not Visible             (M)     0   00  00A 03B5  Not Visible             (M)     0   00  00B 03D9  Not Visible                     0   00  00C 03DA  Not Visible                     0   00  00D 03DB  Not Visible                     0   00  00E 03DC  Not Visible                     0   00  00F 0543  Not Visible             (M)     0   00  010 056B  Not Visible                     0   00  011 056C  Not Visible                     0   00  012 056D  Not Visible             (M)     0   00  020 057F  Not Visible             (M)     0   00  021 0593  Not Visible             (M)     0   00  022 05BB  Not Visible                     0   00  023 05BC  Not Visible                     0   00  024 0591  Not Visible                     0   00  025 0592  Not Visible                     0   00  026 03DD  Not Visible             (M)     0   00  027 0023  Not Visible                     0   00  0F8 0024  Not Visible                     0   00  0F9 0025  Not Visible                     0   00  0FA 0026  Not Visible                     0   00  0FB 0027  Not Visible                     0   00  0FC 0028  Not Visible                     0   00  0FD Total                  —- Mapped Devices:          33 Including Metamembers: 1040 Available Addresses:   3056 (s) Legend for Available address: (s): The Available Addresses for a director are shared among its ports (shared)

from the output above we can see in the first column that the actual device id is listed and the last column over is the address mapping on that specifc fa and of course, in vintage emc parlance, in hex format looking down the list i have been placing the majority of these devices in order and it appears that 027 is the last address i have used.  the next available address is 028 and so i will now run the command from above on each fa to ensure that the address 028 is available on each of the fas.  another best practice i picked up along the way…when you are putting devices out there on the fas it makes your work so much easier come troubleshooting time if you place the device at the same address location on each fa.  trust me, having had to troubleshoot scenarios where that is not the case is a nightmare from both a host and storage perspective.

i confirmed that 028 is available so we are now on to the next step which is creating the map file and mapping the new device to the fas we have selected to use (i will use 8 in my example).

6. again, i have to give props to steven and wes for the methodology here.  just like the meta command file that we created in step #2 above, we will create a metavolume map file that will be used to map our device out to the fas.  with the file we will also have an audit trail so we can always go back and reference the file to see what it was we did or what we thought we did on that day.  the file name will be the following (i spiced it up by adding my initials for even more specificity should you have a team with multiple members):

sanadmin_logspace_082009_tpb_map.cmd

keeping with the best practice of being as descriptive as possible we have our file name and now we will enter the following syntax into the file to map the device out to the fas (i will map this device to 8 of the ports we are using on both 8a and 9a and all on port #0):

[root@sanadmin]:[~]:[4602]#: cat sanadmin_logspace_082009_tpb_map.cmd ## ## This file is my test file… ## map dev 03e2 to dir 8a:0 target=0, lun=28; map dev 03e2 to dir 8b:0 target=0, lun=28; map dev 03e2 to dir 8c:0 target=0, lun=28; map dev 03e2 to dir 8d:0 target=0, lun=28; map dev 03e2 to dir 9a:0 target=0, lun=28; map dev 03e2 to dir 9b:0 target=0, lun=28; map dev 03e2 to dir 9c:0 target=0, lun=28; map dev 03e2 to dir 9d:0 target=0, lun=28;

the above is important because as you can see, you can place comments into your files as well.  this is another great opportunity to maybe put in the change control number you are working under, the system poc information, or really anything else that will aid you in the future.  now, that we have seen the syntax of the file, we can run the symconfigure commands below to check and commit our changes:

[root@sanadmin]:[4598]#: symconfigure -f sanadmin_logspace_082009_tpb_map.cmd preview Establishing a configuration change session……………Established. Processing symmetrix 000190103951 Performing Access checks…………………………….Allowed. Checking Device Reservations…………………………Allowed. Submitting configuration changes……………………..Submitted Locking devices…………………………………….Locked. Validating configuration changes……………………..Validated. Closing configuration change request………………….Closed. Terminating the configuration change session…………..Done. The configuration change session has completed successfully. [root@sanadmin]:[4599]#: symconfigure -f sanadmin_logspace_082009_tpb_map.cmd prepare Establishing a configuration change session……………Established. Processing symmetrix 000190103951 Performing Access checks…………………………….Allowed. Checking Device Reservations…………………………Allowed. Submitting configuration changes……………………..Submitted Locking devices…………………………………….Locked. Validating configuration changes……………………..Validated. Initiating PREPARE of configuration changes……………Prepared. Closing configuration change request………………….Closed. Terminating the configuration change session…………..Done. The configuration change session has completed successfully. [root@sanadmin]:[4600]#: symconfigure -f sanadmin_logspace_082009_tpb_map.cmd commit Establishing a configuration change session……………Established. Processing symmetrix 000190103951 Performing Access checks…………………………….Allowed. Checking Device Reservations…………………………Allowed. Submitting configuration changes……………………..Submitted Locking devices…………………………………….Locked. Validating configuration changes……………………..Validated. Initiating PREPARE of configuration changes……………Prepared. Initiating COMMIT of configuration changes…………….Queued. COMMIT requesting required resources………………….Obtained. Step 005 of 013 steps……………………………….Executing. Step 011 of 013 steps……………………………….Executing. Local:  COMMIT……………………………………..Done. Terminating the configuration change session…………..Done. The configuration change session has successfully completed.

we can now confirm that our device is at the correct address space on the fa by running the previous symcfg command:

[root@sanadmin]:[~]:[4601]#: symcfg -sid 51 -fa 8a -p 0 list -address Symmetrix ID: 000190103951 Director                 Device Name          Attr     Address ———————- —————————– —- ————– Ident   Symbolic  Port Sym   Physical                     VBUS  TID  LUN ——  ——–  —- —-  ———————–      —-  —  — FA-8A   08A       0    0022  Not Visible             VCM     0   00  000 0053  Not Visible             (M)     0   00  001 0152  Not Visible             (M)     0   00  002 0206  Not Visible             (M)     0   00  003 0218  Not Visible             (M)     0   00  004 022A  Not Visible             (M)     0   00  005 0301  Not Visible             (M)     0   00  006 0325  Not Visible             (M)     0   00  007 0349  Not Visible             (M)     0   00  008 036D  Not Visible             (M)     0   00  009 0391  Not Visible             (M)     0   00  00A 03B5  Not Visible             (M)     0   00  00B 03D9  Not Visible                     0   00  00C 03DA  Not Visible                     0   00  00D 03DB  Not Visible                     0   00  00E 03DC  Not Visible                     0   00  00F 0543  Not Visible             (M)     0   00  010 056B  Not Visible                     0   00  011 056C  Not Visible                     0   00  012 056D  Not Visible             (M)     0   00  020 057F  Not Visible             (M)     0   00  021 0593  Not Visible             (M)     0   00  022 05BB  Not Visible                     0   00  023 05BC  Not Visible                     0   00  024 0591  Not Visible                     0   00  025 0592  Not Visible                     0   00  026 03DD  Not Visible             (M)     0   00  027 03E2  Not Visible             (M)     0   00  028 0023  Not Visible                     0   00  0F8 0024  Not Visible                     0   00  0F9 0025  Not Visible                     0   00  0FA 0026  Not Visible                     0   00  0FB 0027  Not Visible                     0   00  0FC 0028  Not Visible                     0   00  0FD Total                  —- Mapped Devices:          34 Including Metamembers: 1044 Available Addresses:   3052 (s) Legend for Available address: (s): The Available Addresses for a director are shared among its ports (shared)

you can also run the symdev list command with the -noport option again to confirm that metavolume 03e2 no longer shows up as available…not a bad idea!  so, where does that leave us?  we have created a new device/metavolume, placed it out on the fas we want to access it on (and this is probably based on your zoning in caveat #4 above), confirmed that it is out there, have superb records of what we have done up to this point in our configuration files, and we are now ready to mask the device so that only the hbas in our host system can see the metavolume.

7. in order to make the device visible to the host (again, note caveat #4…zoning would be a completely different post ) we need to mask the device using the symmask and symmaskdb commands.  first things first, let’s run a command to see if the wwn of the hba on our host is logged in (i have truncated this substantially for brevity…but you will still get the idea from the output below):

[root@sanadmin]:[3.00.15]:[15:30:24]:[~]:[4607]#: symmask list logins Symmetrix ID            : 000190103951 Director Identification : FA-8A Director Port           : 0 User-generated                      Logged On Identifier       Type  Node Name        Port Name        FCID   In     Fabric —————- —– ——————————— —— —— —— 10000000c94cd8xx Fibre 10000000c94cd8xx 10000000c94cd8xx 031f00 Yes    Yes 2101001b322095xx Fibre 2101001b322095xx 2101001b322095xx 030f00 Yes    Yes

from this, we can see that the wwn of our host has logged in to fa 8a on the symmetrix (the host i am using for this post is the second one listed – ending in 95xx).  if you don’t see your wwn there, a very good place to start troubleshooting is by looking at your zoning and making sure you don’t have any mistakes there.  if the zoning is correct, you should be logged in to your fa and ready to roll.

8. once you see that your hba/wwn is logged in to the symmetrix, you can now mask out access to the metavolume you created for that specific host hba (and if you had more than one hba you would follow the same procedure for that hba as well):

[root@sanadminrh]#: symmask -sid 51 -wwn 2101001b322095xx -dir 8a -p 0 add devs 03e2

you will do this for each fa (in our example i would run it for 8a/8b/8c/8d and 9a/9b/9c/9d).  you can add “-noprompt” at the end to avoid seeing a message telling you that you have already masked out this device to an fa on subsequent symmask commands if you would like as well.

9. we now have our device masked out to our host wwn on the fas of our choice.  in order to confirm that they are out there you will run the following command again:

[root@sanadmin]:[3.00.15]:[15:44:17]:[~]:[4620]#: symmaskdb database list Symmetrix ID            : 000190103951 Database Type           : Type6 Last updated at         : 11:40:57 AM on Thu Aug 20,2009 Director Identification : FA-8A Director Port           : 0 User-generated Identifier        Type   Node Name        Port Name         Devices —————-  —–  ———————————  ——— 2100001b3200b6xx  Fibre  2100001b3200b6f6 2100001b3200b6xx  0023:0026 0206 0218 0543 2101001b322095xx  Fibre  2101001b322095xx 2101001b322095xx  03DD 03E2

and there is our device listed with the appropriate wwn!  at this point i usually choose to refresh the database by running the following command:

[root@sanadmin]:[3.00.15]:[15:51:38]:[~]:[4624]#: symmask refresh Refresh Symmetrix FA/SE directors with contents of SymMask database 000190103951 (y/[n]) ? y Symmetrix FA/SE directors updated with contents of SymMask Database 000190103951

this will update/refresh the symmask database with the latest information.  okay, you should now be able to work with the host and pull in the new device, create your filesystem and off you go!  i will be covering how to undo all of the above in another post…

okay, i hope that this helps someone out there on the wire and best of luck from the dark side…

devnull