SAN with ESOS and VMWare ESXi

Use ESOS to create a SCST device for VMWare ESXi

Alt text

General

ESOS is a Storage Solution for creating SCST devices (Group of Disks that gets assigned to a Target). Target refers to the PWWN of the HBA that is plugged into your Storage. ESOS is installed on a USB drive and loads its configuration into the systems RAM. An Initiator will be connected to the Target over a Fibre-Channel Switch to let VMWare access the Disks. The Initiator is also a PWWN of the HBA that is plugged in the VMWare ESXi Server.

Topology

I connected two Dell R620 Server over Fibre Channel to a Nexus Switch.

Alt text

Equipment

  • VMWare ESXi
  • ESOS Storage
  • Cisco Nexus 5548UP with FC license
  • 2x Dell R620 Server
  • 2x Dell 300GB Hard Disk
  • 2x QLE2560 8Gb FC PCIe HBA
  • 4x Cisco SFP 8G SW
  • 2x OM2 lc/lc multimode Fiber Cable

Physical Installation

First I put the HDDs in the Dell R620 Server, which will have ESOS Storage installed on it. You have to assign a vdisk for every physical Disk in the inbuilt Servers RAID Controller PERC H710 press CTRL + R at boot. The vdisks are needed if you want to create a software RAID with ESOS what we will do in our case.

Hint: You could also just use this inbuilt RAID Controller to create RAIDs and archieve redundancy without any additional software like ESOS.

Alt text

Then I connected the two QLE2560 HBAs to the PCI slot in both Servers.

Alt text

Here is a picture of the inside of the Dell R620.

Alt text

Then I inserted the four SFPs. Two in the HBAs and two in the N5K. I only had three Cisco SFPs but I used a SFP from another vendor. You just have to make sure you put this SFP in the HBA and not the N5K since the Switch only recognizes the Cisco SFPs. Also all SFPs have to be same speed (8Gbit).

Alt text

Lastly I plugged in the FC cables and done! Here is a picture of the topology. (Ignore the switch on the top and the ethernet cables.)

Alt text

OS Installation

I downloaded the ESOS zip file from

  • ESOS Storage and created a bootable USB stick with these commands:

    unzip esos-master_7842102_dgvszq.zip
lsblk #check which volume your usb device got
sudo umount /dev/sda1 #remove the logical disk from your system
cd esos-master_7842102_dgvszq/
sudo ./install.sh
/dev/sda

    Then in the Dell R620 BIOS I changed the Boot order so the Server first boots into the USB Stick. I choose the debug mode for ESOS since the production mode wasn't working for me.

    Nexus 5548UP

    Before we configure the N5K we have to make sure that the FC_FEATURES_PKG license is activated. You can check this with the command show license usage.

    Feature                      Ins  Lic   Status Expiry Date Comments
                                 Count
--------------------------------------------------------------------------------
FCOE_NPV_PKG                  No    -   Unused             -
FM_SERVER_PKG                 No    -   Unused             -
ENTERPRISE_PKG                Yes   -   Unused Never       -
FC_FEATURES_PKG               Yes   -   In use Never       -
VMFEX_FEATURE_PKG             No    -   Unused             -
ENHANCED_LAYER2_PKG           No    -   Unused             -
LAN_BASE_SERVICES_PKG         No    -   Unused             -
LAN_ENTERPRISE_SERVICES_PKG   No    -   Unused             -
--------------------------------------------------------------------------------

    The N5K config is pretty straightforward. I created VSAN 1000 and a Zone and added the two PWWNs to the Zone. Just be sure that you use the PWWN and not the NWWN.

    Here is the config with comments.

    version 7.3(7)N1(1)
hostname N5K
username admin password 5 *******  role network-admin
username CISCO password 5 *******  role network-admin
ip domain-name CISCO.net #for SSH
crypto key param rsa label SSH-KEY modulus 2048 #for SSH

slot 1 #I declared some interfaces as ethernet for another project.
  port 1-16 type ethernet
  port 17-32 type fc

vsan database
  vsan 1000 #is like VLAN in ethernet
  vsan 1000 interface fc1/17
  vsan 1000 interface fc1/19
fcdomain domain 10 static vsan 1000 #domain has to be unique on every switch

interface fc1/17
  no shutdown
interface fc1/19
  no shutdown
interface mgmt0
  vrf member management
  ip address 192.168.15.251/24 #for SSH

zone default-zone permit vsan 1000 #is like permit IP any any in ethernet. Is not really needed since we have the PWWNs.
zone name ESXi_zone vsan 1000 #zoning is like ACL in ethernet
    member pwwn 21:00:00:24:ff:0e:ac:7f #ESXi
    member pwwn 21:00:00:24:ff:0f:05:0a #ESOS
zoneset name ESXi vsan 1000
    member ESXi_zone
zoneset activate name ESXi vsan 1000

    Debug Commands

    sh flogi database #should show both PWWNs
sh fcdomain vsan 1000
sh vsan 1000 membership
show zone status vsan 1000 #should show defaul-zone: permit 

    N5K# sh flogi database 
--------------------------------------------------------------------------------
INTERFACE        VSAN    FCID           PORT NAME               NODE NAME       
--------------------------------------------------------------------------------
fc1/17           1000  0xd50020  21:00:00:24:ff:0f:05:0a 20:00:00:24:ff:0f:05:0a
fc1/19           1000  0xd50000  21:00:00:24:ff:0e:ac:7f 20:00:00:24:ff:0e:ac:7f

    ESOS Storage

    The Storage configuration is similar to the NAS configuration that I have done.

    Small tip: I like to connect to ESOS over SSH so I can switch between ESOS and N5K in the Bash pretty easily.

    Also you might need to use Debug Mode when starting ESOS since the primary mode does not work for me.


    Create the RAID. Press O -> Add Array. Alt text

    Create the File System and mount it. Press F -> Add File System. Alt text

    Create the virtual Disk. Press F -> Add VDisk File Alt text

    Now press T and enable the HBA as Target.

    The next three steps are the same steps as if we create a NAS so I used the three pictures of my NAS article.


    Create a Group. Press H -> Add Group Alt text

    Add the SCST Device to the Filesytem. Press D -> Add Device Alt text

    Add the SCST Device to the Host Group. Press D -> Map to Host Group Alt text


    Before we add the Initiator (PWWN of the ESXi HBA) we check that ESXi recognizes our HBA.

    Alt text

    Back in ESOS we press H -> add Initiator and the PWWN of ESXi should appear.

    Alt text

    And if we made everything right a session appears.

    Alt text

    VMWare ESXi

    Since I only have one ESXi host I connected to it directly instead of connecting to Vsphere so the GUI might look a bit different.


    Under Devices our SCST Device that we created in ESOS should now appear.

    Alt text

    Select it and choose new datastore.

    Alt text

    Go through the wizard and create the datastaore.

    Alt text

    Now if we create a VM we can select the new datastore as storage and voila we have successfully created a FC SAN.

    Alt text

    Thanks for reading my article. If you have any questions or recommendations you can message me via arvednetblog@gmail.com.