System Design

Table of Contents

• Introduction
• Current Practice
• Functions of the Media Library Manager
• MLM and Device I/O
• MLM Interfaces
• Internal Architecture

This document provides an introduction to the goals, architecture, and implementation strategy for MLM, SGI's Media Library Manager. Individual components of the Media Library Manager are described in detail in their respective design documents:


List of Documents

• Administrative Command Line Interface
• MLM Core Design Description
• Client and Language Version Negotiation (HELLO)
• Client and Administrative Model and Programming Interface (CAPI and AAPI)
• Object Definitions for AAPI and CAPI
• Abstract Library Interface (ALI)
• Library Control Program (LCP)
• Abstract Drive Interface (ADI) and Drive Control Program (DCP)
• System Start-Up and Restart
• Pre-Defined Tokens and Descriptions
• Interprocess Communication and Authentication
• Glossary of Terms

Introduction

The Media Library Manager, MLM, is a software "middleware" product which allows multiple applications to manage, store and retrieve removable media. Throughout this document, we refer to a unit of removable media as a cartridge-- this might be a tape cartridge, a tape reel, an optical disk, or a removable magnetic disk. Retrieval may be performed either using a wide range of robotic library devices or by means of a human operator. MLM will support a wide range of robotic library devices, and includes facilities to make it easy for SGI or partner companies to add support for additional robotic devices.

MLM is a library management program -- it does not provide end-user application functionality, nor does it become involved in data I/O operations to cartridges. It is expected that MLM will be used in the full range of traditional tertiary storage management applications:

Direct Device Access
The user reads and writes cartridges directly, using a utility program such as tar, cpio, or dd.

Backup
This is characterized by frequent, automatic writes of user files and infrequent, user-requested reads of data from the tertiary medium. Typically, automatic aging and recycling of cartridges is performed. Examples of applications which perform backups include NetWorker and xfsdump; a wide range of additional third-party applications exist in this realm.

Archive
Archiving typically is invoked manually by the user, both for writing information to the tertiary storage medium (after which it is typically deleted from disk), and for retrieving data from the tertiary store. Cartridges are typically retained long-term, until the user wishes to recycle the cartridge. NetWorker currently provides archiving facilities; a user writing a tar format file of the user's own files and then deleting them from disk is a simple form of archiving.

Hierarchical Storage Management (HSM)
HSM is traditionally characterized by automatic, "transparent" movement of data between tertiary storage and and disk-based filesystems. User files which have not been accessed for some time are moved to the tertiary medium without the user's knowledge; when the user accesses a file which has been moved from disk, the user experiences a delay while the file is retrieved from tertiary storage and restored to disk "transparently."

MLM is designed to be completely independent of user applications, so it could also be used to manage robotic library units for which no computer access of data is expected. An example of such a device would be a broadcast video library unit equipped only with videotape units, such as an Odetics Broadcast Library or a Sony Betacart Library. MLM could be used to keep track of all of the videotapes at the site, whether inside a library or not, and it would control one or more robotic library units, but the SGI computer(s) which control the libraries would neither read nor write the videotapes themselves. Using custom application software and signalling protocols specified by the videotape deck manufacturers, the SGI systems could perform standard control operations on the videotape machines.


Current Practice

The "traditional" approach to managing removable media robotic devices is to have an application which understands how to use the particular robotic device. This approach has several obvious difficulties:

• a robotic device driver must be written for the application for each type of device which is to be managed
• introduction of a new robot type implies writing new code for the specific application and re-releasing that application
• each robot is managed by exactly one application and cannot be shared between multiple applications

The traditional model is shown in Figure 1.

Conventional Robotic Library Management

Figure 1.


Functions of the Media Library Manager

The Media Library Manager MLM solves these problems by implementing two standard interfaces-- one for applications which use cartridges and one for robotic (or non-robotic) library units which store and retrieve cartridges. MLM implements a set of core functions which allow client applications to allocate, deallocate, mount, and unmount cartridges in a manner which is independent of the actual library storage device involved.

Client applications may choose their own essentially arbitrary naming scheme for cartridges. Each client application operates within a separate name space, so different applications may re-use the same name for distinct cartridges. Full protection is also afforded between multiple client applications, as they do not have visibility to cartridges of any other application which might be utilizing MLM. (Administrative functions do permit specific cartridges to be moved from one application to another.)

Library storage systems, including both robotic and non-robotic (manual) systems, are supported by MLM using a distinct interface. This interface is designed to make it easy to implement new libraries, while providing sufficient richness to handle large, complex library units.

Both the Client Application Programming Interface (CAPI) and the Abstract Library Interface (ALI) are designed so that applications and libraries may be added to the system without software updates to the main body of MLM. Applications or interface programs are simply installed at appropriate points in the directory hierarchy and then administrative commands provided by MLM are used to inform MLM of the existence of new applications or library units. These administrative commands may be performed on a running system.

Media Library Manager
Figure 2.

MLM accepts high-level requests from applications and turns them into a set of low-level robotic controller operations to accomplish the task. It also schedules competing requests from different applications, establishes and enforces cartridge pools on a per-application basis, and provides a mapping from logical cartridge names (an application's view) to physical cartridge numbers (as used by robots).


MLM and Device I/O

MLM manages the cartridge library and mounts and dismount cartridges; once a cartridge is mounted for a client application, the client application performs I/O to the cartridge using standard UNIX I/O interfaces. MLM is not involved in the I/O operations performed by the client application to and from standard UNIX devices, as shown in Figure 3.


I/O and Control Paths
Figure 3.


MLM Interfaces

To provide maximum portability across different vendors and operating systems and to ensure a simple and extensible interface, MLM utilizes "typeable" ASCII commands across all major interfaces. The language used by clients is CAPI, the Client Application Programming Interface; ALI, the Abstract Library Interface, is used by MLM to interface to library control programs. Although these languages are somewhat richer than many of the standard interfaces in use today on the Internet such as TCP and nntp, the languages are designed to be simple to generate and replies are designed to be simple to parse.


Media Library Manager Primary Interfaces

Note: update picture to include DCPs
Figure 4.

As shown in Figure 4, there are several MLM interfaces in addition to the Client Applications Programming Interface (CAPI), and the Abstract Library Interface (ALI). The Abstract Drive Interface (ADI) allows MLM to perform operations on I/O devices before and after a client program utilizes the device. MLM uses the ADI to verify that a cartridge has been properly inserted into the drive, optionally to look at an initial sequence of data on the cartridge to determine if the proper cartridge has been loaded, and, when the client program is done with the drive, to obtain any statistical information which the drive might have on its operation and/or the condition of the cartridge, and finally to unload the cartridge from the drive.

The CAPI, ALI, and ADI will be published by SGI as freely available language specifications. Other organizations are welcome to implement programs which utilize these interfaces with no royalty payments to SGI. SGI welcomes input on these interfaces, and will evaluate suggestions for extensions. SGI expects to retain editorial control over these interfaces until such time as they stabilize and can be submitted to appropriate standards groups.

SGI expects to develop a repertoire of Library Control Programs (LCPs) for many of the common robotic libraries which are on the market, as well as a Library Control Program for manual mount operations in which human operators are used in lieu of or in addition to robotic storage libraries. SGI will also encourage vendors of robotic devices to implement their own LCPs; SGI expects to not implement an LCP for a robotic device if the manufacturer of the device chooses to implement their own LCP.

SGI will also develop a repertoire of Drive Control Programs (DCPs) for many of the common tape drives on the market, including both drives which SGI sells and those which are available through third parties on SGI's systems. In early versions of MLM, SGI does not plan to implement DCPs for non-tape devices, but in later versions, it is expected that SGI will provide support for devices such as magneto-optical disks, CDROM readers and writers, removable disk cartridges, and other removable media devices. As with LCP's, SGI will encourage vendors of drives to develop their own DCPs. Vendors may also need to develop device drivers for their drives.

Our experience with third-party peripherals has been that the most successful path to having successful, functional devices on SGI systems is through SGI's Gold Seal program. Under the Gold Seal program, SGI agrees with the device manufacturer to implement appropriate control software and to distribute this software as part of SGI's standard system software releases. An expanded Gold Seal program will provide implementation services for device drivers, DCPs, and LCPs.

A number of Administrative Interfaces are used to communicate between a number of command-line and graphical-interface administration systems which are included as a part of MLM.

All external MLM interfaces, including the Client Applications Programming Interface (CAPI), the Abstract Library Interface (ALI), the Abstract Drive Interface (ADI), and Administrative Interfaces are designed to be operable either within a single host computer system or in a networked environment with multiple computers. To ensure that cartridges are protected against unauthorized access, including attacks across a network, MLM implements a set of Interprocess Communication and Authentication functions. These functions are provided in library form and are linked in with applications so that the ASCII strings may be validated by both MLM and the client or interface program.


Internal Architecture

MLM is organized as a set of cooperating processes. The central core of the system is the MLM Server, which is a single, multithreaded process on the server. The server accepts connections from multiple clients and acts on requests by connecting to library and drive control programs. It maintains a Persistent Storage (database) which contains information on all cartridges in the system, as well as descriptions of authorized applications and their capabilities, known libraries and devices, library and drive control programs, etc.

Figure 5 illustrates the functional units present in MLM's internal architecture. The dotted lines in the diagram indicate example physical machine boundaries.


Media Library Manager Internal Architecture

Figure 5.

Click to view larger image