iSCSI
What is iSCSI?
iSCSI or internet SCSI, is really not that difficult to understand. Once you get your head around the basics, you can use this great technology just about on every system out there. Most vendors build it into their operating systems so you can immediately use it after installation.
In this section I will cover the basics on internet SCSI, the components that make it up and then how to install and use it on a Solaris 10 system.
The basics of internet SCSI. - How it works.
Components that make up internet SCSI. - What's used to make it work.
How to set up internet SCSI. - How can I use it in Solaris?
How to set up internet SCSI using ZFS. - How can I use internet SCSI with ZFS?
Remove targets and initiators - How can I remove initiator and target configurations?
What is iSCSI?
internet Small Computer System Interface.
Simply put, it's storage that is made available over the normal IP network to hosts or servers. That's really all there is to it.
Isn't that what NFS and CIFS does as well? Yep. that's exactly what NFS and CIFS does, but these storage models are what they call, file centric models. Stop frowning, I'll explain it in more detail.
You see, with NFS and CIFS, files and directory's are shared over the network. You share some files on the server side and then on the client side you mount these files and then immediately start to store mp3's or documents on it.
You don't have any control over what type of filesystem you just mounted, you just mount and use it.
With iSCSI, a disk is presented to the host instead of a mount point. The disk looks like a normal disk. You format it like a normal disk. You partition it like a normal disk and create filesystems on it, like a normal disk.
It's like connecting a USB drive to you system, only it's done over the IP network. You can either directly connect to the system that makes this storage available, or you can access the storage via standard Ethernet switches.
No need to invest in expensive fibre channel switches or infrastructure. It's all there, just use it.
The storage system that makes the storage avaiable is called the iSCSI target, and the system that uses this storage is called the iSCSI initiator. Clever hey. Below is a picture of this model.
Internet SCSI
Later on this page I'll show you how to setup a target and initiator on Solaris 10 and Windows 7. On windows 7, I'll just show you how to configure the initiator. The Solaris 10 system will be configured as the target.
Components of internet SCSI
Let's now look at the different components that internet SCSI is made up of. I don't want to spend too much time on this. It can become very complicated and that's not what I'm aiming for here. But you have to understand the theory behind this technology to use it correctly.
I will list the most common components here, with an explanation.
What is a iSCSI initiator?
The initiator is usually the device or port that will make use of whatever storage is presented to it. Initiators is not just found in internet SCSI, you will here this term with fibre channel and normal SCSI as well.
It could be a host bus adapter (HBA), hardware initiator, or software running on a server. Software initiators are very common and is, of course, much cheaper. They install as part of the internet SCSI software and no further software needs to be installed.
Hardware initiators are usually used when the system wants to boot from an internet SCSI target. But you can boot from a software initiator as well.
An initiator is like a boss in a company. The boss always initiates the work. He checks what needs to be done in order for the company or division to run smoothly. So, the initiator will make use of the storage or lun presented to it and use it to make the system run smoothly.
What is a iSCSI target?
A target makes some storage available to the initiator. This target could be a lun on a storage array or a lun made available from a normal server where some kind of storage is attached to it.
Most companies producing storage arrays have the internet SCSI target capability built in to their appliances. Some well known brands include, NetApp, Sun ZFS storage 7000, Sun Storage 2510 series and Fujitsu Eternus DX series. These are just a few, there are lot's of companies that make these internet SCSI storage devices.
What is an IQN?
IQN or iSCSI Qualified Name. This has to do with addressing the storage. You create a lun on the target and make this lun available to initiators. For the initiator to access the luns it has to communicate to a target.
It's very similar to fibre channel world wide names (WWN). Every port on fibre channel has a WWN, and the systems use these WWN's to map storage to it.
If you directly connect servers to storage then this is not much of a problem, cause the server can see only that storage connected to it. But, what if the storage is connected to several server? Why can we identify which luns should go where?
There is actually 2 ways that internet SCSI identifies targets and initiators. You can either use IQN's or Extended Unique Identifier (EUI). The IQN is most commonly used.
An example IQN would look like this:
iqn.1986-03.com.sun:01:ea7ec2f5ffff.4d9aa3d2
It looks more complicated than it really is.
iqn identifies the type of addressing. In this case it's iqn. You could also have eui.
1986-03 is the date string. This is the date that the domain was registered.
com.sun is the domain name of the company, reversed.
This first part, up to the : is assigned by the standards organization to a company that uses internet SCSI. This is fixed and should not be changed.
After the colon, :, you can use whatever you want to. The above example is from an Oracle Sun server, so they used their own type of naming scheme as shown above. You may want to use a more descriptive name after the colon. Maybe something like the server name and port, or something similar.
There is a RFC (Request For Comments) for iSCSI, RFC3720.
Below is an extract from the document where they show you how you how the naming scheme for IQN and EUI works:
The following are examples of internet SCSI qualified names that might be
generated by "EXAMPLE Storage Arrays, Inc."
Naming String defined by
Type Date Auth "example.com" naming authority
+--++-----+ +---------+ +--------------------------------+
| || | | | | |
iqn.2001-04.com.example:storage:diskarrays-sn-a8675309
iqn.2001-04.com.example
iqn.2001-04.com.example:storage.tape1.sys1.xyz
iqn.2001-04.com.example:storage.disk2.sys1.xyz
The format is "eui." followed by an EUI-64 identifier (16 ASCII-
encoded hexadecimal digits).
Example :
Type EUI-64 identifier (ASCII-encoded hexadecimal)
+--++--------------+
| || |
eui.02004567A425678D
Discovery methods
There are three discovery methods:
- Static discovery
- Send targets
- iSNS
Static discovery is a one on one mapping from the initiator to the target. You map to a specific iqn.
Send targets is where the initiator checks the iqn of a target to see what luns are available.
iSNS (Internet Storage Name service) is similar to a DNS (Domain Name Servive). The initiator will query the iSNS server to see what target luns are available.
The most common discovery is send targets and static. For larger environments you might want to have a look at iSNS.
What is CHAP?
Challenge Handshake Authentication Protocol (CHAP) has to do with security. If you want the initiator to authenticate itself to the target, you would setup CHAP.
You set a user and password on the target so that when the initiator connects to the target you specify the correct user and password in order to use the storage.
There is much more going on in the back ground but that has to do with the authentication protocol that CHAP uses. I'm not going into details with this. Read the RFC if you need more details.
General considerations
When setting up an internet SCSI network, use a separate network for internet SCSI. Lot's of data will travel through this network. You DO NOT want to use your normal public LAN for internet SCSI.
I'm going to say it again, use a separate network when using internet SCSI.
Don't use 10Mbit or 100Mbit networks. Take my word for it, it's too slow. You would need at least 1Gbit or higher.
For redundancy and speed I would also suggest you use some kind of multi pathing such as Internet Protocol Multi Pathing (IPMP), Link aggregation or Windows MPIO (Multi Path IO), if using a Windows client.
Use as many links as you can. Don't think that one 1Gbit channel is enough. Use two or more and the biggest you can afford. If you just want to play with internet SCSI then using one link is fine, but for serious server environments I would suggest that you do some research and planning.
The cool thing is that iSCSI uses normal IP networks. So, if you are familiar with networks, then you already have an advantage when setting up the network.
IF you need more information on internet SCSI then read the RFC 3720 (Opens in a new window) on it. It's a bit technical but if you are into reading RFC's then give it a go. It's very long and technical but it gives a lot of detail on this technology.
internet SCSI examples
My sample setup consists of two Solaris 10 systems. These are Oracle VM VirtualBox virtual machines with Solaris 10 installed on it. The one system will be my target and the other the initiator. I will use various types of backing store luns.
The backing store is where the actual storage is that you want to make available via internet SCSI. Here is a list of devices that you can make available as targets:
Files - You can create a file using mkfile, and then make this available as a target.
Block devices - Devices such as /dev/dsk/c0d0
SVM Solaris Volume Manager volumes - Devices like /dev/md/dsk/d100
ZFS volumes - This is the easiest way to make targets available. You create a pool, create a volume with the -V option and set the shareiscsi=on property. If you are going to use a server with a jbod connected to it, then I suggest you use ZFS to create the targets with. It's easy and fast.
We use the iscsitadm, in Solaris 10, to administrate targets, and iscsiadm to administrate initiators.
So, here goes.
Create target luns
First of all we need to create a base directory where the information for the targets will be kept. It's like an administration area that internet SCSI uses to keep configuration information. I will use a directory called, /iscsibase.
You could also create the base directory and then create targets in the base directory by just specifying the size and name of the target. This is a nice and simple setup cause you don't need to worry about where you created the targets. Later I will show you an example to do this.
We use the iscsitadm command and specify the base directory.
bash-3.00# mkdir /iscsibase
bash-3.00# iscsitadm modify admin -d /iscsibase
bash-3.00# iscsitadm show admin
iscsitadm:
Base Directory: /iscsibase
CHAP Name: Not set
RADIUS Access: Not set
RADIUS Server: Not set
iSNS Access: Not set
iSNS Server: Not set
iSNS Server Status: Not applicable
Fast Write ACK: Not set
We may use the iscsitadm show admin, command to display the administration area. As you can see, I have not setup any CHAP or iSNS information.
Great, all information about the targets will be kept in this base directory. This is also needed if you would like the targets to be persistent across reboots.
First example will just be a normal block device. I'll use the /dev/dsk/c0d1s6 device.
bash-3.00# iscsitadm create target -b /dev/dsk/c0d1s6 block1
bash-3.00# iscsitadm list target
Target: block1
i SCSI Name: iqn.1986-03.com.sun:02:9b7ff979-645e-42af-9cd4-c62061ccb041.block1
Connections: 0
bash-3.00# iscsitadm list target -v
Target: block1
i SCSI Name: iqn.1986-03.com.sun:02:9b7ff979-645e-42af-9cd4-c62061ccb041.block1
Connections: 0
ACL list:
TPGT list:
LUN information:
LUN: 0
GUID: 0
VID: SUN
PID: SOLARIS
Type: disk
Size: 1.1G
Backing store: /dev/dsk/c0d1s6
Status: online
bash-3.00#
Let's have a look at the command. You create targets with the iscsitadm create target command. The -b option is for the backing store. This is the actual piece of storage you want to make available as a target. In this case it's /dev/dsk/c0d1s6. I partitioned all the space in slice 6, so that's the slice I used. The last operand is the name of the target. In this example I called it block1
I then used the list target command to display the target name, iqn and whether there's any connections to it.
We then used the list target -v options to get a bit more information about the target. With the -v option, you also get information on the ACL, lun number and some other information about the target.
Using send targets on initiators to map targets
So now the target has been created and we need to discover it from the client system that will use this target.
We need to specify a discovery method to actually see the targets on the initiator.
On the client system, we'll use the iscsiadm command to configure the initiator.
I will use the send targets discovery method first just to show you how to easily discover luns or targets.
Below is a list of dicovery methods:
- static
- sendtargets
- iSNS
root@sol10u9 # iscsiadm modify discovery --sendtargets enable
root@sol10u9 # iscsiadm list discovery
Discovery:
Static: disabled
Send Targets: enabled
iSNS: disabled
root@sol10u9 #
Next, we need to specify a discovery IP address for the initiator to discover any targets on.
root@sol10u9 # iscsiadm add discovery-address 192.168.102.40
Cool stuff. Now, let's see if we can see the targets that are available on the discovered IP address.
root@sol10u9 # iscsiadm list target
Target: iqn.1986-03.com.sun:02:9b7ff979-645e-42af-9cd4-c62061ccb041.block1
Alias: block1
TPGT: 1
ISID: 4000002a0000
Connections: 1
root@sol10u9 #
Yep, we see it. So, how can we actually start to use it? If you cannot see the luns with format, then try the devfsadm -vC -i iscsi command.
root@sol10u9 # devfsadm -vC -i iscsi
root@sol10u9 # format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d0
/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
1. c0d1
/pci@0,0/pci-ide@1,1/ide@0/cmdk@1,0
2. c1d1
/pci@0,0/pci-ide@1,1/ide@1/cmdk@1,0
3. c2t600144F04E12032D000800276FC56E00d0
/scsi_vhci/disk@g600144f04e12032d000800276fc56e00
Specify disk (enter its number):
Cool stuff! I can see the lun. We can now format and create filesystems on the target just like any other disk. This is what makes internet SCSI such an attractive storage option.
Using static discovery on initiators to map targets
In the previous example, we used the send targets discovery method to discover targets. In the next example we'll use the static discovery method to discover and add targets.
We'll create another target on the target system and call the target, stat1
bash-3.00# iscsitadm create target -b /dev/dsk/c1d1s6 stat1
bash-3.00# iscsitadm list target
Target: block1
i SCSI Name: iqn.1986-03.com.sun:02:9b7ff979-645e-42af-9cd4-c62061ccb041.block1
Connections: 1
Target: stat1
i SCSI Name: iqn.1986-03.com.sun:02:bc72e80c-7542-6d4d-af47-930231fdf7c4.stat1
Connections: 0
Next, we need to tell the initiator that we want to use static discovery method. Let's do this on the initiator system.
root@sol10u9 # iscsiadm modify discovery --static enable
root@sol10u9 # iscsiadm list discovery
Discovery:
Static: enabled
Send Targets: enabled
iSNS: disabled
root@sol10u9 #
You will see that both the Static and Send Targets discovery methods are enabled. This is fine, you can use all of them if you want to.
We have to check whether we can see the target we created on the target system, before we can add the target.
root@sol10u9 # iscsiadm list discovery-address -v
Discovery Address: 192.168.102.40:3260
Target name: iqn.1986-03.com.sun:02:bc72e80c-7542-6d4d-af47-930231fdf7c4.stat1
Target address: 192.168.102.40:3260, 1
Target name: iqn.1986-03.com.sun:02:9b7ff979-645e-42af-9cd4-c62061ccb041.block1
Target address: 192.168.102.40:3260, 1
root@sol10u9 #
We can see the target. Now, we'll use the static method to add the target statically, so to speak.
root@sol10u9 # iscsiadm add static-config\
iqn.1986-03.com.sun:02:bc72e80c-7542-6d4daf47-930231fdf7c4.\
stat1,192.168.102.40
root@sol10u9 # iscsiadm list target
Target: iqn.1986-03.com.sun:02:bc72e80c-7542-6d4d-af47-930231fdf7c4.stat1
Alias: stat1
TPGT: 1
ISID: 4000002a0000
Connections: 1
Target: iqn.1986-03.com.sun:02:9b7ff979-645e-42af-9cd4-c62061ccb041.block1
Alias: block1
TPGT: 1
ISID: 4000002a0000
Connections: 1
root@sol10u9 #
Please Note: I had to split the command into three lines cause it doesn't fit into my page layout. The create command should be entered continuously on the command line. The backslash (\) above, just indicates that the command continues on the next line.
We can now use the format command to check if the lun is available.
root@sol10u9 # format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d0
/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
1. c0d1
/pci@0,0/pci-ide@1,1/ide@0/cmdk@1,0
2. c1d1
/pci@0,0/pci-ide@1,1/ide@1/cmdk@1,0
3. c2t600144F04E121BB2000800276FC56E00d0
/scsi_vhci/disk@g600144f04e121bb2000800276fc56e00
4. c2t600144F04E12032D000800276FC56E00d0
/scsi_vhci/disk@g600144f04e12032d000800276fc56e00
Specify disk (enter its number):
Okay, we see it. We can now go and partition and use the lun. With static discovery, you can specify which luns you want to add to the host. With send targets, all luns that are configured on the target will be seen by the initiator.
Using static discovery might be a good idea if you only need to discover one or two luns. If there are lots of luns from the the, send targets might be a good idea. It's all about planning. I cannot tell you which method to use. But the previous examples might help you decide.
Using internet SCSI with ZFS volumes. The easy way
If you find it difficult to use normal block devices or files to create targets, then I suggest using ZFS when creating targets. I would suggest it anyway if you are going to deploy internet SCSI using normal servers with storage connected to it. It's really easy creating internet SCSI targets using ZFS.
Let's get going.
I'll create a mirror pool with two disks and then create volumes out of this pool that will be used as targets on my initiator system.
bash-3.00# zpool create mypool mirror c0d1 c1d1
bash-3.00# zpool status
pool: mypool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c0d1 ONLINE 0 0 0
c1d1 ONLINE 0 0 0
errors: No known data errors
bash-3.00# df -h /mypool
Filesystem size used avail capacity Mounted on
mypool 1.0G 21K 1.0G 1% /mypool
bash-3.00# zfs create -V 200m mypool/vol1
bash-3.00# zfs create -V 200m mypool/vol2
bash-3.00# zfs list
NAME USED AVAIL REFER MOUNTPOINT
mypool 413M 643M 21K /mypool
mypool/vol1 207M 849M 16K -
mypool/vol2 207M 849M 16K -
I have created the pool and created 2 x 200Mbyte volumes called mypool/vol1 and mypool/vol2.
All that's needed is to set the shareiscsi=on property. This will enable the targets to be used by an initiator.
bash-3.00# zfs set shareiscsi=on mypool/vol1
bash-3.00# zfs set shareiscsi=on mypool/vol2
bash-3.00# iscsitadm list target
Target: mypool/vol1
i SCSI Name: iqn.1986-03.com.sun:02:fd1e1aca-c2df-cdbf-f314-ca226861152f
Connections: 0
Target: mypool/vol2
i SCSI Name: iqn.1986-03.com.sun:02:2751fb22-13a2-40ca-80ac-9217321effc6
Connections: 0
As you can see, it's very simple to create a internet SCSI share using ZFS.
We can now configure the initiator to use the targets.
root@sol10u9 # iscsiadm add discovery-address 192.168.102.40
root@sol10u9 # iscsiadm list target
Target: iqn.1986-03.com.sun:02:fd1e1aca-c2df-cdbf-f314-ca226861152f
Alias: mypool/vol1
TPGT: 1
ISID: 4000002a0000
Connections: 1
Target: iqn.1986-03.com.sun:02:2751fb22-13a2-40ca-80ac-9217321effc6
Alias: mypool/vol2
TPGT: 1
ISID: 4000002a0000
Connections: 1
Now we use format to partition and use the targets.
root@sol10u9 # format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0d0
/pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
1. c0d1
/pci@0,0/pci-ide@1,1/ide@0/cmdk@1,0
2. c1d1
/pci@0,0/pci-ide@1,1/ide@1/cmdk@1,0
3. c2t600144F04E134F0F000800276FC56E00d0
/scsi_vhci/disk@g600144f04e134f0f000800276fc56e00
4. c2t600144F04E134F10000800276FC56E00d0
/scsi_vhci/disk@g600144f04e134f10000800276fc56e00
Specify disk (enter its number):
The newly added luns are highlighted.
Remove internet SCSI target and initiator configuration
To remove a discovery address from the initiator you would do the following:
root@sol10u9 # iscsiadm modify discovery -s disable
root@sol10u9 # iscsiadm modify discovery -t disable
root@sol10u9 # iscsiadm list discovery
Discovery:
Static: disabled
Send Targets: disabled
iSNS: disabled
root@sol10u9 # iscsiadm remove discovery-address 192.168.102.40
root@sol10u9 # iscsiadm list discovery-address -v
That takes care of the initiator. Ther should be no more targets visible on the initiator system.
If we want to delete the target luns on the target system we would need to do the following:
bash-3.00# iscsitadm list target
Target: block1
i SCSI Name: iqn.1986-03.com.sun:02:9b7ff979-645e-42af-9cd4-c62061ccb041.block1
Connections: 0
Target: stat1
i SCSI Name: iqn.1986-03.com.sun:02:bc72e80c-7542-6d4d-af47-930231fdf7c4.stat1
Connections: 0
bash-3.00# iscsitadm delete target -u 0 block1
bash-3.00# iscsitadm delete target -u 0 stat1
bash-3.00# iscsitadm list target
bash-3.00#
That's it, the targets are deleted.
I have only showed you the basics on how to create target's and initiators. There's still a lot of stuff that you can do. I don't want to overwhelm you with options and operands. The information on this page should give you a good understanding of internet SCSI and how it works.
The best way to learn internet SCSI is to actually do it. Get yourself a system or two and start to create targets and initiators. Try to setup CHAP between the targets and initiators. Use some other features like multi pathing, ACL's and Target Portal Group Tag (TPGT).
Use the man pages and internet to get more information if you need to. I think that you would probably use some kind of hardware that will be used as the target and servers as initiators.
Return from iSCSI to Solaris 10
Back to What is My Computer
