IC R3 supports two messaging servers: a multi-protocol message transfer agent (MTA), and a multi-protocol message store (MS).
The MTA implements both the X.400/MHS 1988 and Internet standards for messaging and includes support for the following:
- Protocol channels for message exchange using X.400 (1984, 1988, and 1992 versions), SMTP, DECnet Mail, UUCP, and fax.
- Class 2 Fax Access Unit providing delivery of fax messages.
- Submission and delivery of messages in a wide range of formats including X.400 (P3), RFC822 submission/delivery, P3-file, shell and structured delivery.
- Body-part and content-type conversion facilities including MIME-MHS.
- Distribution-list processing.
- Comprehensive authorization support.
- An API for the development of additional channels is provided.
The above functionality includes numerous major enhancements since the last release.
The Message Store provides a message store database with a defined API, a P7 server [X.413], an LMAP server, and a message delivery channel for the MTA. Indirect message submission is provided via the P7 and LMAP servers.
The ISODE Consortium MTA is a high-performance, high-capability multi-protocol message transfer agent. It implements both the X.400/MHS 1988 and SMTP standards for messaging including their related content-types and addressing. It supports sophisticated and extensible content and body-part-type conversion capabilities. This provides the capability to gateway between different types of mail system, in particular between X.400 and Internet mail.
The capabilities of the MTA are based on the service elements and protocols defined by the MHS (MOTIS) series of standards [X.400, ISO/IEC 10021] for the Message Transfer Service. All the Basic Services are supported. All the Essential Optional User Facilities are supported. All the Additional Optional User Facilities are supported with the exception of deferred delivery cancellation and explicit conversion.
The following functional groups are supported:
- Conversion (CV)
- Distribution List (DL)
- Redirection (RED)
- Latest Delivery (LD)
- Return of Contents (RoC)
- Use of Directory (DIR) (excluding use of Directory Name when
O/R Address does not identify a user and use of Directory to determine a recipient's capabilities),
- 1984 Interworking (84IW) (excluding internal trace information).
The Physical Delivery (PD) and Security (SEC) functional groups are not supported.
The operation of an MTA is based around a queue. The queue is managed by a Queue Manager process (qmgr) which invokes various so-called channels to process messages. Processing can include conversion when necessary but mostly involves relaying messages to the next MTA along their route or performing final delivery for local users.
In IC R3, the message security element type definitions have been aligned with the most recent X.400 specifications.
Protocol channels are used to exchange messages with adjacent MTAs. Different channels are used to support different standards and network environments. Most channels support both inbound and outbound message transfer. Inbound transfer is generally invoked by the remote system and the appropriate channel is started by a network daemon upon reception of the connection. Outbound channels are run under the control of the queue manager.
The X.400 channels support both the 1984 and 1988 recommendations including the mts-transfer (P1-1988, RTSE normal-mode), mts-transfer-protocol (P1-1988, RTSE X.410(84)-mode) and mts-transfer-protocol-1984 (P1-1984, RTSE X.410(84)-mode) application contexts.
Full RTSE recovery is supported for both inbound and outbound transfers.
All three application contexts can be supported by a receiving channel on a single Session Address.
The sending channel will perform downgrading of P1-1988 to P1-1984 when sending using mts-transfer-protocol-1984. A message will be non-delivered if downgrading fails (according to the rules of X.411, Annex B).
RTSE recovery in the mts-transfer and mts-transfer-protocol application contexts, P1-1988 to P1-1984 downgrading, and single-address multi-context responder channel capability are significant enhanced functionality in IC R3.0.
MTA-MTA authentication is password-based with an access control table, and message authentication and authorization will be the same in IC R3 as IC R2.
MTA-provided X.400 strong authentication services may be added in a future release.
The Simple Mail Transfer Protocol (SMTP) [RFC821] conforming to the host requirements for messaging [RFC 1123] are supported. The inbound channel can operate in stand-alone mode or from the standard TCP network daemon (usually inetd).
Interworking with DECnet Mail is available for Sun systems with Sunlink DNI.
The Unix-to-Unix-Copy subsystem can used with the UUCP channels.
The fax access unit supports outgoing faxes via fax modems with Class 2 capability.
Drivers for two other (almost obsolete) modems are also available. These are the Panasonic Systemfax 250 and the Fujitsu dexNet200.
Fax headers may be customized with a logo, and facsimile (graphics) and text body parts may be mixed.
Message submission and delivery are independent mechanisms. Local submission and delivery may be achieved by various supplied channels and programs and an API is available.
- P3 Submission and Delivery
A channel pair implements P3 protocol access to the ISODE Consortium MTA.
Furthermore, the p3-file channel provides local submission and delivery of X.400 P2 messages. Messages are exchanged via files in the format of P3 protocol submission and delivery PDUs. This mechanism is compatible with that of certain other MTAs.
Programs that look like the standard mail and sendmail programs are provided and these can be used as substitutes for the normal UNIX utilities subject to a few restrictions.
The SMTP channel can also be used for RFC822 submission and this is a common mechanism employed by user agents, particularly MH and PC agents.
The 822-local channel provides for local delivery of messages in RFC822 format. Sophisticated local (per-system and per-user) tailoring is provided and both standard mailbox formats are supported. Tools are provided to give users asynchronous notification of the delivery of new messages.
The shell channel can be used to deliver messages to programs.
- Application Programming Interface
A proprietary API provides full control over the submission mechanism and can be used for both the MTS (UA to MTA message submission) and MTA (MTA to MTA message transfer) services.
The MTA provides both content-conversion and body-part conversion facilities. It also provides for normalization of message header information.
Body parts are more correctly called encoded information types (EITs). Individual body-parts can be converted from one to another using conversion filters. Typically this is used for converting a text body part from one character set to another: for example, from T.61 (teletext) to IA5 (international alphabet 5, similar to ASCII) as specified in [X.408]. Such conversions are often necessary as part of content-type conversion.
The necessary conversions are calculated when a message is first submitted and they may be re-evaluated as processing progresses.
Content-type conversion is used to convert between the formats of different messaging standards: for example, between X.400 P2 and RFC822. It is done by exploding a message content into its individual elements (usually a header and one or more body parts). These elements are then individually converted as if they were normal EITs. Finally, the resultant parts are flattened to produce the output content type.
- X.400-Internet Interoperability
Channels to convert between RFC822 and X.400 P2 according to the rules of [RFC1327] are provided. Conversion of message headers is performed according to [RFC1494], [RFC1495] and [RFC1496].
- MIME &λτ;--&γτ; MHS Content Conversion
Channels are provided for the conversion of the message body-part types of X.400 and those of MIME [RFC1521, RFC1522].
It is often desirable to rewrite header information in particular to normalize addresses by rewriting the address in some canonical form. Header normalization is provided and uses the generic EIT conversion capabilities. Channels for the normalization of RFC822 and P2 headers are provided.
- File-transfer Body Part (FTBP) Submission, Conversion & Spooling
The X.420(1992) Recommendation adds the specification of a file-transfer body part (FTBP), based on FTAM document types. IC R3.0 includes functionality to support the submission, conversion and delivery of such body parts.
A conversion filter is provided that will encode an FTBP as text (using uuencode) so that it may be relayed to destinations not capable of receiving FTBPs. This will probably be replaced with a MIME-compatible encoding in a future release.
A delivery mechanism is provided for FTBPs which cannot be handled directly by a user's user agent. This is in the form of a filter which stores the transferred FTBP in a spool area together with catalogue information about it, and then replaces the FTBP in the message itself with a text body part describing the replaced FTBP.
The user may retrieve the stored file and catalogue information with the fex command. This filter and the fex program, specifically the location of each user's spool area, are configured using a table with the same format as that used by the 822-local channel.
A simple user-agent program is provided to enable a user to send a file as an FTBP.
Expansion of distribution lists is provided by two channels, one which uses plain files for the specification of list membership, and another which uses the Directory.
The list expansion facilities are content-type independent. They can work with any type of message content which can be handled by the MTA.
Tools are provided for list maintenance.
The heart of the MTA configuration is a straightforward tailor file. This is used to specify:
- the available channels (protocol, submission & delivery, conversion, and various housekeeping ones);
- the tables used for routing, address-format conversion (according to RFC1327), and those used by individual channels;
- the EITs (body parts and header types) understood by the various channels;
- many other individual items of configuration.
Outside the tailor file, tables are used for many aspects of configuration. Some of these can be replaced or augmented by use of the X.500 Directory and the Domain Name System (DNS).
A large part of configuration, in particular addressing and routing, may be performed using data stored in the X.500 Directory according to the Internet MHS-DS specification. Local MHS users, typically Message Store users, may be entirely configured using the Directory. A GUI is provided for management.
The main elements of configuration are to do with addressing and consequent message routing. The system is extensible so that simple use of the MTA only requires simple configuration. The use of each of the more sophisticated capabilities requires use of incremental configuration elements.
The following are the main elements related to addressing and routing:
- local users with no interconnection to other MTAs;
- configuration as a pure X.400 MTA with a small number (possibly one) of bi-lateral connections;
- configuration as a pure RFC822 Internet-connected MTA with DNS-based configuration of communications links;
- a gateway between X.400 and RFC822 mail systems;
- mail hub operation supporting multiple addressing domains;
- Support for separate internal and external addressing styles;
- additional facilities such as fax access unit and distribution lists.
- Authorization
Comprehensive authorization allows for control on security and accounting grounds and enables policy-based routing. Overall control is based upon a 3-level hierarchy of pairwise (sender and receiver) elements: channels, adjacent MTAs, users. This can be refined on the basis of message content and size.
The IC R3 MTA supports a rich set of management facilities.
Graphics and terminal-based console programs allow the state of an MTA to be monitored. This can provide an overview or examine the system in progressively greater detail down to the level of an individual message.
In privileged mode, connections can be enabled or disabled at the levels of system, channel, peer MTA, message and recipient.
An SNMP agent permits integration of MTA monitoring into a general network-management system.
Comprehensive logging facilities allow the tracking and accounting of all messages and provide extensive trouble-shooting and debugging capabilities.
The Multi-Protocol Message Store was introduced in IC R2. It serves as an intermediary between User Agents and the Message Transfer Agent, accepting delivery of messages on the user's behalf and storing them for subsequent retrieval. The Message Store also provides an indirect submission service and facilities for searching, listing and deleting messages. The additional functionality of the Multi-Protocol Message Store, as compared with a standard X.413 Message Store, allows partial and complete messages to be stored and manipulated within the Message Store prior to submission.
The key to the Multi-Protocol Message Store is its internal API, which makes the Message Store multi-protocol in two senses:
- It stores multiple message formats in a single abstract format. For IC R3, P2 and MIME message formats are supported.
- It allows access by multiple protocols. For IC R3.0, P7(88), P3(88), and LMAP are provided.
The Lightweight Mail Access Protocol (LMAP), an added-value ISODE Consortium proprietary protocol, defines both the service specification of the API and the access protocol used to provide remote access for native client applications. LMAP takes many of its design cues from the successful Lightweight Directory Access Protocol (LDAP). The protocol's simplicity is intended to facilitate the wide-scale deployment of X.400 User Agents on workstations and PCs.
The Message Store itself consists of several components:
- a database that implements the LMAP protocol semantics;
- a P7 process which uses the database to provide an X.413 Message Store service;
- a Message Store delivery channel which allows the PP Message Transfer Agent to deliver messages into the Message Store.
- Security
LMAP and P7 are defined to have simple and strong authentication. Simple authentication is available from the server, P7 and LMAP libraries in IC R3. In IC R4, strong authentication will be provided by the P7 and LMAP servers and protocol libraries.
The appropriate X.400(88) P1 security protocol elements will be accessible through both protocols and in the MTA and MS library routines.
The Lightweight Mail Access Protocol provides facilities for accessing a message store, including message submission, and access to and management of folders, messages and components of messages.
Some of the design goals for the Lightweight Mail Access Protocol were:
- Provision of a lightweight X.400 access protocol, in the style of LDAP, to facilitate the implementation and deployment of X.400 UAs on small machines.
- Support for RFC 822 messaging, to facilitate mixed RFC 822/X.400 working and to provide an RFC 822 message access protocol superior to others available.
- Provision of a means to give protocol access to various LAN mail systems. A particular goal was to design a protocol which could be implemented as a MAPI DLL (providing integration with various PC network mail systems).
In addition, the protocol design aimed to provide:
- A core protocol that is simple but with good extensibility.
- An abstraction that is a useful basis for a User Agent.
- A protocol that can easily and efficiently map on to a range of underlying message database technologies.
- Integration with LDAP, so that combined mail and directory operations can be handled over the same association.
- Facilities to allow a message to be represented as either an atomic object or in a structured manner, as required by client applications.
- Why Not Use an Existing Protocol?
The primary reason for designing a new messaging protocol, instead of using the existing P7 or IMAP (Interactive Mail Access Protocol) protocols, as the basis for the ISODE Consortium's Message Store is that the very flexible information and access model offered by P7 makes building a database which is cleanly aligned to the protocol very difficult. Using such a database to support other protocols (such as RFC 822) is still more problematical.
Other reasons for the design of a new protocol included:
- Both P7 and IMAP have dissimilar client models for submission and access. This increases the complexity of both the Message Store and User Agents.
- The ASN.1 describing P7 is complex, necessitating extensive libraries for the manipulation of the resulting data structures. This could be a particular problem on small machine implementations.
- P7 does not provide any general facilities for folder and message manipulation.
- P7's generalized treatment of body parts as attributes is awkward to implement efficiently for large body parts.
- The LMAP Information Model
LMAP represents information (messages) in two ways, as objects and as attributes:
Objects are typed entities which are named within an LMAP Message Store; including folders, messages, message headers, message envelopes and atomic Body Parts.
Objects are (potentially large) collections of attributes. The protocol transfer of objects is designed to allow fragmented reception or transmission, removing the need for applications to hold the whole of a large object in memory.
Attributes are typed entities which are found within objects. They are identified by their type, rather than by a name.
An attribute consists of a type/value pair; the value of an attribute is either a plain text encoding (i.e. an rfc822-mailbox attribute might be stored as "ic.info@isode.com") or a text wrapping around an ASN.1 encoding (a P7 ContentIdentifier of "12345", represented in ASN.1 as "[APPLICATION 10] 12345" would be stored as "6A053132333435").
Types are identified either by object identifiers (in a printable string encoding), or well-known strings ("lm-rfc822-mailbox" for example).
The abstract operations provided by the LMAP protocol are as follows:
Bind Associate a client with the LMAP Message Store.
Unbind Terminate an association.
Create Create a new instance of a specified object type (folder, message or bodypart).
Remove Delete a particular instance of an object.
Modify Change an existing object (e.g., add attributes, change bodypart).
Move Move a message or a range of messages from one folder to another.
Copy Copy an atomic body part or message to make a new instance.
ModifyAnnotation Alter the annotations associated with an object. Annotations are attributes associated with an object which may be modified by the user but are not accessed when a message is submitted.
Retrieve Transfer objects from the Message Store to the user. The mode of transfer of objects can be controlled.
Sort Sort messages in a folder keyed by a selected attribute.
Scan Provide ordered listings of folders, messages and bodyparts.
Find Search collections of messages for particular attribute values.
Abort Terminate a partially completed Retrieve, Search or Scan operation.
VerifyAddress Verify an address to the maximum extent possible by the Message Store prior to submission.
Submit Submit a particular message or probe to the MTA.
Delivery Identify to the Message Store that a message under construction is complete. This operation is typically used by a delivery channel to indicate that the message concerned can be "seen" by a UA accessing the same mailbox.
The database is the heart of the Message Store. It represents messages as collections of attributes, stored in a simple text-based encoding (rather than as complex ASN.1 structures). This abstraction makes the storage of messages from diverse protocols (such as X.400 and RFC 822) straightforward.
Several criteria were set out when considering the design and implementation of the Message Store Database:
- It should be robust, so that the reconstruction of all indices from the underlying structures should be possible.
- It should be optimized for small (one or two kilobyte), simple messages, as it is expected that these will form the bulk of the messages in the envisaged target environment.
- There should be a minimal overhead per folder. Folder selection should be fast.
- Folder scanning should be fast.
- The mapping from IP Message ID to Message Store object-ID should be straightforward and quick.
- It should be possible to extract a message in pieces and to add and incrementally manipulate a non-delivered message.
- A search on common information should be reasonably quick.
- It should include mechanisms for efficient and effective management of the database.
- Implementation
The initial implementation of the Message Store database maps onto a storage system based on a mixture of table and standard UNIX files. The design provides for the use of alternative mappings (for example, on to a relational database), through careful separation in the API of protocol and storage operations within the database implementation.
The database models each message as a directory, containing a set of files which hold the attributes making up the message. Higher level directories perform the function of "folders", segregating messages into manageable groups and optimizing the process of searching for individual messages. Included messages are held as subdirectories below message directories.
The database allows the retrieval of objects in structured or unstructured form, as convenient for client applications. Thus, for example, a header object could either be retrieved as a list of attributes (structured form), or a stream of bytes containing the ASN.1 encoding of the header object (unstructured P2-header form), or a stream of bytes containing an RFC822 header (unstructured RFC822-header form). Conversion from structured to unstructured form is performed "on the fly" by the database.
The Message Store supports submission and delivery using the X.411 service (P3 protocol). This provides both MTA location and product independence, and maximal functionality. This is the preferred mechanism.
Co-located submission and delivery closely tied to the IC MTA and with restricted functionality is also supported.
The P7 Server process offers an X.413 Message Store service to P7 User Agents and X.411 (P3) submission and delivery services (MTS) for communication with an MTA.
For the Message Store service, the process accepts remote operation requests framed in the P7 protocol and performs the conversions required to map these operations onto the LMAP protocol. Where facilities are not directly provided by LMAP (for example, the storage of FetchDefaults), the P7 Server uses LMAP to implement them. Persistent per-user configuration information is stored in annotations attached to the user's root folder.
For the MTS, submission and delivery are supported using P3 and co-located submission is also supported.
The Message Store Delivery Channel provides the co-located link from the MTA to the Message Store. The process is a standard MTA delivery channel which uses the LMAP Client API to deliver messages from the MTA into the Message Store database. It handles both RFC 822 and X.400 messages. The only modification required to the MTA to enable the use of the Message Store is to the tailor file and tables.
The process of delivering a message into the Message Store is essentially one of mapping the elements of structures passed to the Delivery Channel to the corresponding LMAP attributes, performing necessary syntax conversions and creating appropriate LMAP objects containing these attributes. Once the objects which make up a message have all been created, a "Deliver" operation makes the message available to users of that mailbox.
Graphical and terminal-based console tools to allow the state of the Message Store and associated Server processes to be monitored are planned for later IC releases. Control of many areas of Message Store functionality will be possible when activated in privileged mode. A set of SNMP agents to allow the integration of Message Store control and monitoring into a general network-management system are available. IC R3 provides hooks within the areas to be monitored.
The protocol provides the following facilities for users to the management of their own messages:
- Folder and message management is provided by the LMAP Create, Delete, Move and Copy operations.
- Modification of security credentials is provided by LDAP operations acting on X.500 Directory entries with particular names of user and MS. As the LDAP protocol operations use the same association as the LMAP operations and are handled by the same server process, the choice of whether "real" X.500 access takes place or is "faked" by the server process is a runtime tailoring option. This allows the use of the LMAP Message Store without the need for an X.500 directory to be present.
- Modification of Scan defaults (that is, the attributes to return by default in a Scan result) is performed in the same manner as modification of security credentials.
- Network Administrator Management
The main emphasis of this type of management is envisaged to be monitoring, rather than control. This would be appropriate for an environment where there are many Message Stores, with centralized operational control. This release includes an SNMP MIB (based on MADMAN) and associated agent processes, to allow the use of third-party management tools.
The X/Open API to Electronic Mail (X.400) Message Transfer (XMT) divides an MTA into two software components, a mail system gateway and an X.400 gateway services. An implementation of XMT is provided in this releases. This will facilitate the development and support of mail gateways that can be integrated with the IC MTA.
For IC R3, XMT has been verified to support Boston Software Works (BSW) mail gateways.
The XMT API is described in the X/Open CAE Specification, API to Electronic Mail (X.400), Issue 2 available from the X/Open Company Limited.
A 'C' language binding to the LMAP protocol as previously described is provided. This provides a client API for use in User Agent interactions with the Message Store.