Often times, there are complicated environments that have a large number of storage objects. Mapping them can be difficult, but often necessary to understand how data is being moved. Like many of my colleagues, documentation can be a chore and is prone to misinterpretations, as well as time consuming. Being a Linux engineer my first thought is "How do I automate this so I can do more interesting things?" After dragging my feet and going in kicking and screaming, I decided to dive into PowerShell. The following is an example of a dynamically created Visio 2010 diagram of the physical storage configuration of my NetApp Filer simulator.

Tools Used

• NetApp PowerShell Toolkit
• NetApp Visio Stencils (Extracted to "My Shapes" under Libraries->Documents)
• Visio 2010
• At minimum, RemoteSigned permissions (Set-ExecutionPolicy RemoteSigned)
• At least Read-Only Access to a NetApp Filer

How this diagram is created

1.  Script logs into the Filer
2. Data is collected
3. Script opens Visio and Stencils
4. Objects are placed on the workspace in logical order
5. Next version will have data from each point in a table (Latency, I/O, Growth Rate, Overwrite Rate, Days to Full) and exported to Sharepoint team portal
6. Data points will be active and retrievable to all whom has Sharepoint team portal access
7. Business Workflows will monitor the data points and take action when conditions are met (Create request for new storage, warn current use for remediation, etc...)

It won't turn any heads, but that is just a formatting process that is simple to design. Nothing with this process is particularly amazing (which is why I didn't include the code), but using the collection of tools like these can come in handy when trying to troubleshoot a complex system. Seeing the visual data paths and their associated  metrics helps put a scenario into perspective and helps move teams troubleshooting exercises to a solution based orientation rather than a problem based orientation (think of doing a maze from start-to-finish or from finish-to-start...which one is easier??). It also has the added benefit of creating nice diagrams if you are "artistically" challenged like me.

1) Layers 1-3 provide information about how data flows through the physical, data link, and network layers based on existing configurations

2) Layers 4-7 are accessed through a variety of network taps, agents on the source and destination, and a correlation engine to connect the data into information. Objects are analyzed for code performance, SQL query times are correlated back to hardware and OS performance metrics

3) Information collected is analyzed to create knowledge based on existing known workloads to establish a baseline

4) Conditions are set for potential business scenario planning with decision makers and executives based on known initiative/work loads

5) Scenarios are built and tested against proposed design

 
In the scenario here, the link between the first floor and second floor core switches is too small, causing network utilization to spike to 100%. This means that the Radiology application will have network queuing as a result of insufficient resources. Once the simulation completes, a new scenario can be cloned and created to remote the bottleneck and show how the application performs and what happens to queuing as a result.
 

The result of changing the interconnect can be seen to the left: point-to-point queuing delays are statistically zero, and point-to-point utilization on this same workload is less than 1%.

 
 
 
 
 
 
 
 
 
 
 
From the graph on the right, you can see that the application response time dropped from a previous peak of 60 seconds over the course of an hour, down on the 10Gb change scenario to a peak of 12.5 seconds.

You tell me....where is the guess work?

1) IPoIB. In this case IP stack is put above IB. You don’t need to rewrite your applications while you can utilize high throughput. On the other hand you will kill IB low latencies and won’t be able to utilize whole IB throughput capabilities.

2) Sockets Direct Protocol (SDP) which is designed to utilize IB RDMA capabilities and bypass TCP/IP stack. SDP can be used transparently w/o recompiling your application. It’s not that fast as native IB API but is better than IPoIB.

3) IB Verbs; the lowest API, User Direct Access Programming Library (uDAPL) which is based on IB Verbs, Message Passing Interface (MPI) or Unified Parallel C. Different versions of MPI and UPC can be based on either IB Verbs or uDAPL. I personally work with MPI and UPC so I will describe their installation over InfiniBand.

I have implemented all 3 types of technology to great success. The layer with the fastest speed and the lowest latencies is uDAPL, which I must say, takes some getting used to. It requires an abstraction layer between the IB interface, and the rest of the program that needs access to storage or process threads on the fabric will use that abstraction layer to send/receive data to targets available to the requesting server. One implementation was using iSCSI on IPoIB. The upside is that storage on an IP stack is very fast, but latency is the same as on Ethernet because of the protocol stack. I would equate its speed to that of 10g over FC; high speeds with a hanging nail of increased latency (If you are not proficient in OO programming, threading is almost impossible to do structurally in the main languages that are the most expansive - Java and Python).

Do reads and writes plague your applications? Do you wish you could get more "bang for the buck"? I know many companies that complain about I/O contention while balancing the tedious task of having redundancy built into their infrastructures in case of a "rainy day" failure (that is almost sure to occur). Many tier I storage vendors are moving to hybrid disk pools to add faster access, as well as mitigate the latency of RAID 1. However, many small and medium size businesses cannot afford the big 3, but anyone can take advantage of hybrid pools for a minimal price.

The first question any I.T. manager/architect/guy-who-is-responsible-for-storage should ask is "Am I using the tools that are available to me that support enterprise features?" Without understanding the intricate details of how many open source tools work, we will dive a little deep into the one feature EVERYONE should have in their arsenal: Logical Volume Manager

With budgets being squeezed from the down economy, everyone is scrambling to demonstrate their return on investment for any purchases needed to achieve the kind of performance that screams "Bonus in my review". However, LVM is readily available with the equipment you have lying around; you just need to learn to use it. What is great about LVM is its ability to abstract the disk layer into something far more human compatible than managing the disks locally. Now, you still have to understand the physical disk layer as LVM is built on top of disks, but the only steps required is partitioning. So, if you understand how fdisk works at a basic level, you are good to go! Yes, you can build your volumes directly on the disk, but partitioning allows you to divide your pools up for different work loads.

Now I'm no C programmer, but the above makes me want to learn C. I can envision the following workload written into LVM:

1) All disk timings are calculated on initialization
2) All disks are prioritized by the level at which they perform: lower the latency, higher the priority
3) In a hybrid RAID design like mine, I could feasibly write the above algorithm into LVM for such a server design.

It would be a fairly easy process/feature to add to LVM, and create a directly attached storage, or storage server for the mid-range, to enterprise,

(I so want to believe this is already a part of LVM, but don't know for sure yet as I am researching it.)

***This is a WIP****

1. Goto the options tab under your VM guests' settings
2. Select "Memory/CPU Hotplug" Setting
3. Enable memory hot add by selecting the enable memory hot add radial button
4. Enable CPU hot add by selecting the enable cpu hot add radial button
5. Select the OK button

The only time you will need to reboot to add either CPU or Memory will be this last time. Reboot the guest, add either memory or CPU, and then run the appropriate script for your desired function below:

#!/bin/bash
#This script will hot add vCPU's inside VMware guest systems
#Sonny Stormes - 2009
for CPU in $(ls /sys/devices/system/cpu/ | grep cpu | grep -v idle)

do
    CPU_DIR="/sys/devices/system/cpu/${CPU}"
    echo "Found cpu: \"${CPU_DIR}\" ..."
    CPU_STATE_FILE="${CPU_DIR}/online"
    if [ -f "${CPU_STATE_FILE}" ]; then
        STATE=$(cat "${CPU_STATE_FILE}" | grep 1)
        if [ "${STATE}" == "1" ]; then
          echo -e "\t${CPU} already online"
    else
      echo -e "\t${CPU} is new cpu, onlining cpu ..."
      echo 1 > "${CPU_STATE_FILE}"
  fi
    else
      echo -e "\t${CPU} already configured prior to hot-add"
  fi
done
-------------------------------------------------------------------------------------------------
#!/bin/bash
#This script will hot add memory into a VMware guest
#Sonny Stormes - 2009
if [ "$UID" -ne "0" ]
  then
  echo -e "You must be root to run this script.\nYou can 'sudo' to get root access"
  exit 1
fi

for MEMORY in $(ls /sys/devices/system/memory/ | grep memory)
do
    SPARSEMEM_DIR="/sys/devices/system/memory/${MEMORY}"
    echo "Found sparsemem: \"${SPARSEMEM_DIR}\" ..."
    SPARSEMEM_STATE_FILE="${SPARSEMEM_DIR}/state"
    STATE=$(cat "${SPARSEMEM_STATE_FILE}" | grep -i online)
    if [ "${STATE}" == "online" ]; then
      echo -e "\t${MEMORY} already online"
    else
      echo -e "\t${MEMORY} is new memory, onlining memory ..."
      echo online > "${SPARSEMEM_STATE_FILE}"
    fi
done

These functions are only good for adding either component. To remove is a more complicated process, but if enough interest is shown from the above, I'll complete my single script for both attaching and detaching CPU or memory.

Here is the basic premise:

1) Fibre Channel connections are your primary block transport. Set the initiator group on the storage unit as normal, export the blocks, and connect LUN's as normal.

2) Set up an iSCSI initiator to the storage unit and export the same LUN out the target of your storage system. Initiate the connection on both network ports out of the iSCSI TOE card.

3) Set up DM-Multipath. Out of the box, DM-Multipath will create a pseudo-device based on the SCSI_ID of each attached LUN and create a round-robin connection to each LUN (in count, you should have 6 total devices - 2 FC primary paths, 2 FC secondary paths, and 2 iSCSI IP network paths).

4) Set the 4 FC paths (primary and secondary) up accordingly with policies that let's the primary path be used during service time, the secondary path be a potential failover path in the event that either of the 2 primary paths are down, and a final policy that sets the IP network connections to the iSCSI target as dual tertiary paths that will be the final path of last resort.

Hang in there with me! I know that this sounds complicated, but over the next few posts, I'll be explaining each step in detail in a way that is easy for even a beginner to understand. I know that much of this sounds like a drunk man trying to explain why he is wearing a pink tutu in a park at 11am, but trust me when I say it is not that difficult.

At the end of this series, you can build a storage network that can withstand a charging rhino running from Godzilla (ok, maybe not a rhino, but certainly a koala bear wreaking havoc munching on FC cables or networks...if you have koala bears in your data center..after all..who doesn't?).

Until then, happy file and block exporting!

multipath {
    wwid 360a98000486e58526c34515944703277
    alias devicename
    mode 660
    uid 501
    gid 502
}

What's great about this is when DM-multipath instantiates the new mapper devices, they will all be set with the permissions, user, and group assignments that you want. No muss, no fuss!

Enjoy!

1) Set up your /etc/multipath.conf to use mpath_prio_alua rather then the default mpath_prio_ontap. I have not verified if mpath_prio_ontap reads the SCSI-3 ALUA commands, but I know the latter does, so it was easier to use it than the former.

2) Set the ALUA on the igroup with:
igroup set alua on

3) Reboot your hosts

4) Enjoy ALUA goodness!

With this configuration, when cf failover is initiated, your Red Hat server will know when the primary path is dead and use the secondary path without intervention. It really is a beautiful thing, and I recommend it for any storage heads out there.

From the CLI (where all Linux and Unix administration REALLY happens) execute the following:

echo 1 > /block/sys/<devicename>/device/delete

So if you had a disk /dev/sdb that was either a LUN you want to delete, or an attached device (SATA,SAS,IDE,yo mamma) by issuing the command

echo 1 > /block/sys/sdb/device/delete

You will have successfully removed it from the kernel and can unplug, deattach, throw out on its butt, the device in question, all without a reboot.

Ok, I'll take it to the next step and reverse the process:

To add a device on any bus (SATA,SAS,FC-HBA,iSCSI) the reverse is true...with some modifications:

echo "- - -" >/sys/class/scsi_host/host$NUMBER/scan

Where host$NUMBER is the number of the bus you want to scan. (The "- - -" means to look at every channel, every target, and every lun on that host). After you finish, check dmesg for your new device and BAM! You've successfully hotplugged a new LUN,SATA disk, SAS disk, USB disk

Peace!