problem: you don’t have access to emc’s symmetrix management console (smc) or the ecc suite of tools and you need to create your metavolume from the command line to include masking, fa mapping placement, and a handy way to keep track of all the metavolumes you are creating and mapping. or, the guys you work with are terrified that you might learn what they know and won’t tell you anything!
to be honest, i have only used the smc a few times and ecc even fewer. i love the command line. i was born and raised on it (thank you again freddy mac!). i am sure it is some kind of control issue that could be linked to my childhood, but that is probably a topic covered under another blog somewhere so i will just leave that alone for now. i really need to thank rick naquin for sharing with me some steps i forgot from back in the day when i was at usi. i have taken those steps and have chuch’ed them up a bit with some new information that i have recently collected. so, here are the steps to create a metavolume from the command line and, of course, here are the caveats for this to work:
1. the host you are using (we use a rhel5 box) must have fiber channel connectivity/visibility to the symmetrix array in question. you will also need to have emc’s solutions enabler software loaded on the host (we are using v7.0 which i believe is the latest and greatest). you can check by running the following command on your rhel5 box:
2. if you have the software installed, you can confirm that you have visibility back to the array by running the following command (and you should see something similar to the following):
those tiny devices with the “gk” next to them are your gatekeeper devices. if you don’t see any of those, none of the commands that follow (well, the symconfigure commands anyway) will not work as they require communication with a gatekeeper device to function properly…or at all.
3. if you have visibility to more than one symmetrix disk array, you will need to know the serial number or at least the last two digits of the serial number. you can actually figure the last two digits of the serial number out by looking over the output of the syminq command above. taking a look at one of the lines from above we see:
if you look at the sixth column over (right after the microcode software version listing of 5773) you will see a long number that might appear to be random, but actually if you look at the first two digits they will correspond to the last two digits of your symmetrix array. for us, they are 51 and that is what i will use in the examples that follow.
4. all of your brocade/cisco zoning is done for the host you to which you want to allocate the storage. i say brocade/cisco because, let’s be honest, is anyone really using anything else with their symmetrix array?
5. you will notice that my command line prompt changes from time to time in the output below. i had to make some minor modifications to it so that the commands i am running can be seen…those are far more important than the time, version of bash, and current working directory!
those are all the caveats i can think of for now so let’s move on to the actual steps involved with creating our metavolume, selecting the devices we are going to use, masking them out to the host, and getting ourselves up and running with some emc storage!
1. picking the devices: this is really the first question i asked when i was first exposed to the command line back in the day – with no gui to see what is there, how will i know what devices i am going to pick to make my metavolume. or more importantly, how do i ensure that i don’t pick a device that is already in use! this first step revolves around the “symdev” command and its options. probably the most important task to attend to first is to refresh/re-read the current devices on the symmetrix with which you are working. yeah, you might think that nothing has changed, but trust me, things to change and especially if you are on a team where multiple team members have access to your array. to refresh everything just enter the following command:
you always want to be working with the latest information, so the symcfg discover command is important. to see a list of the devices that are available for use, but not assigned/mapped out to an fa you can run the following command:
(you can pipe it to more if you want to walk through the file and this output is extremely truncated at the end for the sake of brevity…and sanity! when you run the symdev list command without using the noport option, you will get a listing of every single device in the array…not what we want here)
so, as you can see, i have a number of RAID-6 devices above that i can use: 03e2, 03e3, 03e4, 03e5, and 03e6. we will go ahead and say that i am going to create a monster metavolume using four of these guys (the first four).
2. now that we know that we are using hypervolumes 03e2, 03e3, 03e4, and 03e5 we will use a method that i love and will credit to steven fletcher and wes “clinton” garner. they came up with a simple methodology that will allow you to keep track of all the changes/configurations/mappings by creating configuration files based on the client/hostname and whether or not the file is being used as a metavolume creation file or a metavolume mapping file. using the scenario i have here, we will say that i am creating this metavolume for the server “sanadmin”. so, for the filename i will use something very descriptive such as the following:
basically, the file you create with your favorite text editor (mine is vi) has the hostname of the server for which you are creating the metavolume, followed by the reason – in our case i am using the space for log files or something similar – the date the file was created (which should probably be the day you ran the symconfigure command using the file), the fact that it is a metavolume (which might be considered redundant, but hey, why not), and then finally, the fact that this is a “cmd” or command file for creating this metavolume. the “cmd” portion of the file name will make more sense once i talk about the mapping file.
again, what i love about this method is that you can then go back and look to see what you did, for which server, and on which date. i have been in some other shops where they just reuse the same file over and over again and i have also seen that type of methodology lead to pandemonium. the syntax for the file we would create using the scenario we have above will be the following:
be sure you use the meta head device number twice in the syntax…for us it will be device 03e2. now, if you are creating more than one metavolume (which you can do in the same file) you would simply follow the two lines above with the next two lines indicating which devices you are using and following the syntax above.
3. now that we have the devices selected and our metavolume command file created and ready we will use the symconfigure command to first test the syntax of the file and then to commit the changes for processing by the symmetrix array. now, you don’t have to run the “preview” option with the symconfigure command if you don’t want to, but why not use a little caution? the best analogy i can give you for not first running the “preview” option is the following: very rarely do i first close my eyes and then run across a busy freeway to get to the other side…so why not take a few minutes and “look both ways” before you commit the changes. the preview option will check the syntax of your file and i can honestly say that this option has saved me from disaster about a dozen times for just as many reasons. here is the output and the commands you would run to create the metavolume:
i forgot to mention the “prepare” option which i also run for completeness and to make sure there are no issues.
4. okay, so the session is complete but how can i tell if my device was actually created? we go back to the symdev command we ran back in step #1. you should expect to see something similar to the following:
as you can see, the meta head 03e2 is still listed and it now has the designation “(M)” next to its RW status. you can also see that the size has now changed as well and the devices that comprise the metavolume (with the exception of the meta head) are no longer listed. the output skips from 03e2 to 03e6. if you wanted to see the actual members listed out you would run the command “symdev -sid 51 list | more” and if we did that we would see the following:
the meta members are designated with a “(m)” and follow the meta head. so, now that we see that our metavolume has been created, we can confirm the meta head is device 03e2, and the size is now ~105gb total. we can now prepare the map file which will allow us to map the device to the front end host ports (i commonly call these fas and can’t recall hearing them called anything else) on which we want to provide access.
7. similar to step #1 (where we needed to figure out what devices we could safely use) we now need to figure out where to place the metavolume we have created. we need to figure out on which fas and at which address location. i will even show a preview example of what to expect when you attempt to place the device at a location that is already being used by another device (something you want to avoid doing!). to see the current address mappings on each fa you would run the following command:
from the output above we can see in the first column that the actual device id is listed and the last column over is the address mapping on that specifc fa and of course, in vintage emc parlance, in hex format looking down the list i have been placing the majority of these devices in order and it appears that 027 is the last address i have used. the next available address is 028 and so i will now run the command from above on each fa to ensure that the address 028 is available on each of the fas. another best practice i picked up along the way…when you are putting devices out there on the fas it makes your work so much easier come troubleshooting time if you place the device at the same address location on each fa. trust me, having had to troubleshoot scenarios where that is not the case is a nightmare from both a host and storage perspective.
i confirmed that 028 is available so we are now on to the next step which is creating the map file and mapping the new device to the fas we have selected to use (i will use 8 in my example).
6. again, i have to give props to steven and wes for the methodology here. just like the meta command file that we created in step #2 above, we will create a metavolume map file that will be used to map our device out to the fas. with the file we will also have an audit trail so we can always go back and reference the file to see what it was we did or what we thought we did on that day. the file name will be the following (i spiced it up by adding my initials for even more specificity should you have a team with multiple members):
keeping with the best practice of being as descriptive as possible we have our file name and now we will enter the following syntax into the file to map the device out to the fas (i will map this device to 8 of the ports we are using on both 8a and 9a and all on port #0):
the above is important because as you can see, you can place comments into your files as well. this is another great opportunity to maybe put in the change control number you are working under, the system poc information, or really anything else that will aid you in the future. now, that we have seen the syntax of the file, we can run the symconfigure commands below to check and commit our changes:
we can now confirm that our device is at the correct address space on the fa by running the previous symcfg command:
you can also run the symdev list command with the -noport option again to confirm that metavolume 03e2 no longer shows up as available…not a bad idea! so, where does that leave us? we have created a new device/metavolume, placed it out on the fas we want to access it on (and this is probably based on your zoning in caveat #4 above), confirmed that it is out there, have superb records of what we have done up to this point in our configuration files, and we are now ready to mask the device so that only the hbas in our host system can see the metavolume.
7. in order to make the device visible to the host (again, note caveat #4…zoning would be a completely different post ) we need to mask the device using the symmask and symmaskdb commands. first things first, let’s run a command to see if the wwn of the hba on our host is logged in (i have truncated this substantially for brevity…but you will still get the idea from the output below):
from this, we can see that the wwn of our host has logged in to fa 8a on the symmetrix (the host i am using for this post is the second one listed – ending in 95xx). if you don’t see your wwn there, a very good place to start troubleshooting is by looking at your zoning and making sure you don’t have any mistakes there. if the zoning is correct, you should be logged in to your fa and ready to roll.
8. once you see that your hba/wwn is logged in to the symmetrix, you can now mask out access to the metavolume you created for that specific host hba (and if you had more than one hba you would follow the same procedure for that hba as well):
you will do this for each fa (in our example i would run it for 8a/8b/8c/8d and 9a/9b/9c/9d). you can add “-noprompt” at the end to avoid seeing a message telling you that you have already masked out this device to an fa on subsequent symmask commands if you would like as well.
9. we now have our device masked out to our host wwn on the fas of our choice. in order to confirm that they are out there you will run the following command again:
and there is our device listed with the appropriate wwn! at this point i usually choose to refresh the database by running the following command:
this will update/refresh the symmask database with the latest information. okay, you should now be able to work with the host and pull in the new device, create your filesystem and off you go! i will be covering how to undo all of the above in another post…
okay, i hope that this helps someone out there on the wire and best of luck from the dark side…