Updated 27-May-2004 by RDR

*************************************************************************
Target Features:

1) Supports draft 20 (RFC 3720)
2) Supports multiple sessions, multiple connections per session
3) Supports multiple devices
4) Supports multiple listening sockets
5) Supports header digest, data digest
6) Supports ErrorRecoveryLevel 1
7) Supports IPv4 and IPv6 addresses.
8) Supports 4 modes for emulating target storage


*************************************************************************
Naming on the target:

In the following discussion, the following terms are used:

target platform - the computer hardware on which the target module code is
		loaded and executed.  This platform must run the Linux
		operating system.

target modules - the software package that is loaded as modules to implement
		the functionality of an iscsi target.  There are 2 modules,
		both created in the src/target directory:
		unh_iscsi_target.o - is the iscsi part of the target, which
			implements the iscsi protocol and deals with TCP/IP.
		unh_scsi_target.o - is the scsi part of the target, which
			implements either the emulation of a scsi target or
			the interface to a real scsi target, depending on
			the mode of operation
		These modules provide access to a set of target controllers.

target controller - one of several objects accessible to an iscsi initiator via
		the target modules.  These are identified by numbers 0, 1, 2,
		..., MAX_TARGETS-1.  Each controller in turn provides access to
		a set of target logical units, which are given numbers (called
		LUNs) 0, 1, 2, ..., MAX_LUNS-1.

target logical unit - the storage object itself.  This is identified in the
		target modules by a controller number and a Logical Unit Number
		(LUN).  All SCSI commands are delivered by iSCSI to a single
		target logical unit.


The target iscsi module defines target controllers that have the names:
	iqn.2002-10.edu.unh.iol.iscsi.draft20-target:0
	iqn.2002-10.edu.unh.iol.iscsi.draft20-target:1
	iqn.2002-10.edu.unh.iol.iscsi.draft20-target:2
	. . .
where the number following the colon (:) at the end of each name is the
number of the controller referenced by that name.  These are the ONLY names
that can be requested by an initiator during the login phase with the key
TargetName=.  For example, an initiator could send the following key in the
first login request pdu it sends on a new connection:

	TargetName=iqn.2002-10.edu.unh.iol.iscsi.draft20-target:2

The association between this name and the actual controller on the target
platform depends on the mode in which the target scsi module is running.
There are 4 modes, as explained below:
	MEMORYIO
	DISKIO
	FILEIO
	GENERICIO

Each target controller itself can further define Logical Units (LUs) which
are the actual storage objects and which are referred to by Logical Unit
Numbers (LUNs) 0, 1, 2, ...  The association of an LU to physical storage
depends on which mode the target scsi module is running in.  To an initiator,
each [controller number, LUN] pair represents a separate mountable storage
device.


*************************************************************************
Modes of Target scsi Module Operation:

Before compiling the UNH scsi target module, you need to set options in the
file scsi_target.h.  The most important option is the mode of operation that
the target scsi module will run in.  There are 4 choices: MEMORYIO mode,
DISKIO mode, FILEIO mode and GENERICIO mode.  To set one of these options,
you need to uncomment the one you want and comment out all other options in the
source file scsi_target.h.  For example, the following selects the DISKIO mode:

//# define MEMORYIO	/* for performance studies */
# define DISKIO		/* to a real disk */
//# define FILEIO	/* to a file on the system */
//# define GENERICIO	/* when the scsi_do_req is not working */


Note -- you MUST select exactly one of these options.  Chaos will result
if you try to select more than one, or if you do not select any.
The distributed version selects FILEIO.

MEMORYIO
--------
When MEMORYIO is selected, there are no real storage objects on the target
platform.  When the target scsi module is asked to send data to an initiator,
the target scsi module will just send garbage data.  When the target scsi
module is asked to receive data from an initiator, it will just throw
the data away.  Thus the initiator is unable to read back what it just wrote,
because the target scsi module does not actually store anything sent to it.
This mode is used only to test the performance of the iSCSI/TCP/IP/network
stack.

In MEMORYIO mode, the target scsi module initializes itself to recognize
MAX_TARGETS target controllers, where MAX_TARGETS is defined in scsi_target.h
(the distributed version sets this to 8).  Furthermore, each controller is set
up to recognize MAX_LUNS logical units, where MAX_LUNS is defined in
scsi_target.h (the distributed version sets this to 16).


DISKIO
------
When installing a target scsi module compiled with DISKIO selected, the target
scsi module will search all available scsi disk HBAs (or tape HBAs if
TAPE_DEVICE is defined) on the target platform and will assign a controller
number to each of these HBAs, starting with 0 for the first HBA it finds.
The number of LUNs per HBA depends on the scsi HBA itself, since that HBA may
be any type of scsi controller with any number of actual storage devices
behind it.

Note: Before installing a target scsi module compiled in DISKIO mode, be sure
that none of the scsi disks (or tapes) are mounted on the target platform
(i.e., be sure they exist as just devices, not mounted devices).  This is
because the initiator will eventually mount these on the initiator platform
via the target modules, and the initiator will then believe that the device is
"local" to the initiator platform.  If it is also mounted on the target
platform, chaos will result.  


FILEIO
------
FILEIO mode is the most flexible mode of target scsi module operation.
It allows you to configure as many target controllers and luns as you want,
each of which is a file or a device accessible via the file system on the
target platform.  Normally, these objects are ordinary files in the host file
system of the target platform, but to the initiator they look just like
local disks attached to the initiator platform.  The contents of these files
is determined entirely by the actions of the initiator, and when used like
local disks each file will contain a super block, and a complete,
self-contained file system.

By default, when the target scsi module is installed in FILEIO mode, it
creates (if not already created) MAX_FILE_TARGETS times MAX_FILE_LUNS files.
These two defines are in the file scsi_target.c, and are set to 2 for
MAX_FILE_TARGETS and 4 for MAX_FILE_LUNS in the distributed version,
which means that 8 files are created (this emulates 2 target controllers,
numbers 0 and 1, with each controller having 4 LUNs, numbers 0, 1, 2 and 3).
These 8 default files in the distributed version have the names:
	scsi_disk_file_0_0
	scsi_disk_file_0_1
	scsi_disk_file_0_2
	scsi_disk_file_0_3
	scsi_disk_file_1_0
	scsi_disk_file_1_1
	scsi_disk_file_1_2
	scsi_disk_file_1_3
where the first digit is the controller number and the second digit is the LUN.
By redefining MAX_FILE_TARGETS and MAX_FILE_LUNS, you can create as many
default files as you want, up to a maximum of MAX_TARGETS and MAX_LUNS.
The size of each default file is FILESIZE blocks, and the number of bytes
in each block is BLOCKSIZE.  These two symbols are defined in the file
scsi_target.h, and in the distributed version are set to (4096 * 1024) for
FILESIZE, and 512 for BLOCKSIZE (which means that the maximum size of each
default file is 4096 * 1024 * 512 = 2,147,483,648 bytes = 2 Gigabytes).
However, storage for these files is allocated only as it is used.

In addition to the default files, which are created (if they do not
already exist) when a target scsi module compiled for FILEIO mode is loaded,
it is possible to dynamically add or remove other files while the target
scsi module is running.  This is done via the /proc file system by writing
appropriate commands to the file /proc/scsi_target/scsi_target.
To add a new file, use the addfile command.  To remove an existing file,
use the delfile command. For example,

	echo > /proc/scsi_target/scsi_target \
  "addfile targetid=1 lun=4 filename=xyz blocksinfile=262144 blocksize=512"

would create (if it does not already exist) a file called "xyz" and would
cause the target scsi module to regard this as LUN 4 of target controller 1.
In order to access this controller, the name given by the initiator during
a login negotiation would be:

	TargetName=iqn.2002-10.edu.unh.iol.iscsi.draft20-target:1

where the number after the colon (:) is the target controller number 1.
The initiator would see this mountable storage device as having a block size of
512 bytes per block, and a total of 262144 blocks (for a total of
134,217,728 bytes = 128 Megabytes), and would have to refer to it as LUN 4
in all SCSI commands sent to it.

When adding a new file, the targetid parameter must have a value less than
MAX_TARGETS, and the lun parameter must have a value less than MAX_LUNS.
In addition, there must not already be a file with that targetid and lun
combination in use.

Note that the initiator uses the number after the colon (:) in the
TargetName value to select which of the many target controllers on the target
module it wishes to connect to.  The initiator does NOT use the name of
the file (xyz in the above example), because, in theory, the initiator does
not even know that the storage object is implemented as a file -- the initiator
simply sees it as a storage device of some sort.  There is no way to have
the UNH iscsi target module recognize any other TargetName values.

Note also that within a given target controller, the use of the LUN is
handled by the SCSI subsystem on that initiator platform.  The LUN number is
generated by that SCSI subsystem and is given to the iscsi initiator by
that SCSI subsystem.  The iscsi initiator simply passes it on to the target
iscsi module, which passes it to the target scsi module where it is used in
DISKIO mode (along with the target controller number) to select the appropriate
file.

Finally, note that the name following the filename= parameter can be any
existing file or device on the target platform that can be accessed via
the Linux file system interface (i.e., open, close, read, write, seek).
In particular, it can be a block special device in the /dev directory,
which means that FILEIO mode can be used to enable the target to access
non-scsi disks.

To remove an existing file, whether one added by a previous addfile
command or one created by default, use the delfile command.  For example,

	echo > /proc/scsi_target/scsi_target "delfile targetid=1 lun=4"

would delete the file identified by target controller 1 and lun 4 from the
target scsi module's internal table.  This does NOT delete the corresponding
file -- it just prevents the target scsi module from accessing it as
controller 1 lun 4.  The file itself, and any data stored in it, persist
in the file system of the target platform.

Note that in both the addfile and delfile commands, all parameters
shown in the above examples are required and must appear in the order shown.

*************************************************************************
Coordinating naming on initiator and target platforms

Because of the flexibility provided, there is a lot of confusion about
how names on the iscsi initiator platform are associated with names on the
iscsi target platform, and how devices on the initiator platform are associated
with devices (or files) on the target platform.  Here is an attempt to clarify
the situation.

The initiator can communicate simultaneously with many different remote
target platforms, and with many different target controllers on each target
platform.  To do this, the initiator requires one session per remote target
controller (regardless of which target platform that controller is on).
Therefore, the UNH initiator allows you to manage, configure, and run multiple
independent sessions simultaneously, up to a maximum of MAX_TARGET_IDS
sessions, where MAX_TARGET_IDS is defined on the initiator in the file
src/initiator/iscsi_initiator.h (in the distributed version this is defined
as 10).

To refer to these different sessions, the iscsi_manage and iscsi_config
tools (in the src/cmd directory) used to configure the initiator accept a
parameter target= followed by a number, which must be a non-negative number
less than MAX_TARGET_IDS.  This number is a way of indicating which session
slot in the initiator you wish to use for this session.  When the target=
parameter is omitted, it defaults to a value of 0.  For simple systems,
you only need to communicate with one target controller and hence you don't
need to use the target= parameter (which is why that parameter is omitted
in most sample scripts).

Please note that this target=number is completely independent of the
number used after the colon (:) in the TargetName= key sent by the
initiator to the UNH target modules.  The number in the TargetName= key
selects which of several target controllers in the UNH target modules the
initiator wants to connect to.  This is independent of the target=number in the
iscsi_manage and iscsi_config tools that selects the session on the initiator
that is doing the connecting.  A better choice of the parameter for the
iscsi_manage and iscsi_config tools would have been session=number,
not target=number, but historically target=number was used because of the
SCSI convention that devices are always addressed by 4 numbers,
[host, channel, target, lun], and we stayed with those labels (channel is
always 0 for the unh software, and therefore cannot be specified anywhere).

Within a single initiator session, there can be many different TCP/IP
connections to the same target controller.  You can control this by specifying
the parameter cid= followed by a number in the iscsi_config tool (of course,
you also have to provide the ip= and port= parameters to identify the
IP address and port number for the connection).  Since normally there is only
one connection per session, the cid= parameter is usually omitted and a cid
value of 0 is used.  The number following a cid= parameter must be in the
range [0..255], and is otherwise arbitrary (obviously different connections
within the same session require different numbers).  As far as iscsi is
concerned, each connection within a session is equivalent in terms of its
ability to transmit commands and data between an initiator module and a target
module.  SCSI does not know that multiple connections exist, and a scheduling
algorithm in the iscsi initiator module decides which connection to use
when sending a command.

Independently of the number of TCP/IP connections in a session,
a single remote target controller can support multiple Logical Units (LUs),
each of which is an independent device which can be mounted independently
by the initiator.  The iscsi initiator itself does not really worry
much about LUNS, since most SCSI subsystems use the SCSI REPORT LUNS
command to discover what luns are available on targets.  However,
you can explicitly bring up a new lun (or take down an existing lun)
by using the iscsi_config tool with the appropriate lun number.
There is no lun parameter to the iscsi_manage tool, because all session and
connection parameters apply to all luns in the session, so no distinction is
necessary.  If the lun parameter is omitted when using the iscsi_config
tool, the default value of 0 is assumed, and many simple target devices
will only support lun 0.

Note that unlike the number of connections in a session and the cid number,
the number of LUNs in a session and the LUN is visible to the SCSI system
on the initiator.


*************************************************************************
Setting up target portal groups:

Before compiling the UNH target iscsi module, you need to define the
IP addresses, port numbers and target portal group tags the target iscsi
module will use to listen for connections from initiators.  This is
done by filling in the table called iscsi_portal_groups[] in the file
src/target/iscsi_portal_group.c.  This table can have a maximum of
MAX_PORTAL entries, where MAX_PORTAL is defined in the file
src/target/iscsi_portal_group.h.  In the distributed version MAX_PORTAL
is defined as 16.

When the target iscsi module is loaded, it will start up one thread
for each item in the iscsi_portal_groups[] table, and the sole job of
that thread is to listen on the indicated IP address and port for
connections from initiators.  When such a connection is received,
the thread will start the login process on that connection.

Instructions and examples of table entries are given in comments in the
file iscsi_portal_group.c.  As distributed, the target will listen on the
iscsi WKP (3260) and on the iscsi system port (860) for all IP addresses
defined by the host OS, and these addresses will be considered
to belong to portal group 1.

Note that both IPV4 and IPV6 addresses are acceptable.

It is possible to dynamically add and remove items from the table
iscsi_portal_groups[] by writing appropriate commands to the appropriate
file in the /proc/scsi_target/iscsi_target/ directory using the program
iscsi_config, found in the src/cmd directory.

For example, the command:

	./iscsi_config up server ip=192.168.10.45 port=5000 tag=3 host=0

will make a new entry into the iscsi-portal_groups[] table and start a
new kernel thread in the target iscsi module that will listen for IPv4
connections at IP address 192.168.10.45 port 5000.  This address-port
combination is considered to be part of portal group 3.  The total number
of entries in the iscsi_portal_groups[] table, including those defined at
compile-time and those defined dynamically, is MAX_PORTAL, which in the
distributed version is 16.

The command:

	./iscsi_config down server ip=192.168.10.45 port=5000 tag=3 host=0

will search the iscsi_portal_groups[] table for an entry with IPv4
address 192.168.10.45 and port number 5000, and if found, will remove
that entry from the table and will terminate the kernel thread listening
for new connections from initiators on that address-port combination.

Note that for all commands to iscsi_config, the first parameter must be
up or down, but all other parameters can appear in any order.  If a
parameter is omitted, its value is assumed to be 0 by default.  The
parameter "server" is required to indicate that it is the target iscsi
module (not the iscsi initiator module) that is being configured.


*************************************************************************
How to install the UNH iSCSI reference implementation Target:

1) load the target scsi module,
   using /sbin/insmod src/target/unh_scsi_target.o
2) load the target iscsi module,
   using /sbin/insmod src/target/unh_iscsi_target.o
3) use the tool ./iscsi_manage to set keys for use during login negotiations
4) use the tool ./iscsi_config to add listening addresses
5) if the scsi_target was compiled for FILEIO mode, write appropriate commands
   to /proc/scsi_target/scsi_target to add new files

At any point after step 2 the target module is ready to accept connections
from initiators.  The file target-install in the src/cmd directory
gives an example of a sequence of commands for steps 1 2 and 3 above.


*************************************************************************
How to use a disk on the target platform from the initiator platform:

After the target modules have been loaded and set up (as described above)
on the target platform, do the following on the initiator platform
(which is expected to be a completely different computer):
1) Load the iscsi initiator module, using
   /sbin/insmod src/initiator/unh_iscsi_initiator.o
   (see the script ini-install in the src/cmd/ directory)
2) Use the iscsi_manage tool to set up the parameters to be negotiated
   during the login session.  This includes the TargetName parameter.
3) Use the iscsi_config tool to define the ip address and port number
   of the target and to make the connection and perform the login.
   (see the script ini-manage in the src/cmd/ directory, which demonstrates
   use of both the iscsi_manage and iscsi_config tools)
4) Once the connection is made and iSCSI has logged in successfully
   (if there is an error, check the /var/log/messages file on both the
   initiator and target platforms for details), the initiator sees another
   disk on its system.  If it is the only scsi device on the system, it will
   be given the name /dev/sda, otherwise, it will be /dev/sdb or /dev/sdc etc.
5) You can run the normal Linux utility programs (/sbin/fdisk, /sbin/mke2fs, 
   /sbin/e2fsck, etc) ON THE INITIATOR PLATFORM to create new partition tables,
   file systems, etc. on the disk. The OS on the initiator platform just thinks
   it has a local scsi disk, and everything that you can do to a real scsi disk
   on the initiator platform you can do to the one you are connected to via
   iscsi. Once you are connected, you NEVER do anything on the target platform
   -- you always access everything from the initiator platform. The target
   platform is used only for storage. In fact, all access to the disk MUST come
   only from the initiator platform, because the OS on the initiator platform
   knows nothing about the OS on the target platform (and vice versa), so it
   thinks it has complete control over the disk, and it uses this control to
   allocate blocks, maintain inodes, etc.  If both the initiator platform and
   the target platform operating systems try to do this at the same time,
   they will surely mess up the disk (and probably crash the systems), since
   they are not coordinating their updates with each other.
6) Once the file system is made on the disk, you can mount it just like you 
   would mount any other (hard) scsi disk on the initiator platform.  The
   partitions will be called /dev/sda1, /dev/sda2, etc.  Once it is mounted on
   the initiator platform, you access it on the initiator platform just like
   you would access any other file system on the initiator platform -- you can
   cd to directories on it, you can make new directories on it using mkdir,
   you can copy files to it using cp, etc., etc., etc.


*************************************************************************
How to remove the target module:

1) ensure that there are no connections from initiators to the target modules
2) remove the iscsi_target module first, using /sbin/rmmod unh_iscsi_target
3) then remove the scsi_target module using /sbin/rmmod unh_scsi_target

Steps 2 and 3 are done by the script target-uninstall in the src/cmd/ directory.

*************************************************************************
