William E. Johnston

Lawrence Berkeley Laboratory1

University of California

Berkeley, CA, 94720

Computer science and telecommunications groups from fifteen Bay Area organizations have formed a testbed to develop and deploy the infrastructure needed to support a diverse set of distributed applications in a large-scale, metropolitan IP over ATM network environment. These organizations are:

+Apple Computer +DEC - Palo Alto Systems Research Center

+Hewlett-Packard Laboratories +International Computer Science Institute

+Lawrence Berkeley Laboratory +Lawrence Livermore National Laboratory

+NASA Ames Research Center +Pacific Bell-Broadband Development Group

+Sandia National Laboratories +Silicon Graphics, Inc.

+SRI International +Stanford University

+Sun Microsystems, Inc. +University of California, Berkeley

+Xerox Palo Alto Research Center

This paper describes the architecture of the testbed, and some of the issues of running IP over ATM.

1.0 Internet Architecture and Asynchronous Transport Mode Networks

While there are potentially several models for data transport using ATM networks, the principle focus of the BAGNet testbed is the use of ATM networks to provide a link-level service for IP (Internet Protocol) datagrams. Once this is accomplished then all Internet services will operate over the ATM network, and interoperate with the many other link-level networks that comprise the global Internet.

The architecture of the Internet is that of collections of “hot potato” packet routers that communicate with each other regarding the probable best way to get packets to their destination. These routers communicate with each other over a very heterogeneous collection of interconnects, and over LANs between routers and end hosts.

To first order, there are arguably two basic Internet services: unicast and multicast. In the former, packets are labeled with source and destination addresses of communicating end systems (hosts). TCP (transmission control protocol) provides logical circuits between processes, and UPD (user datagram protocol) and RTP (realtime transport protocol) provide messaging. IP multicast is an addressing mechanism whereby many hosts can receive the same datagram, and the real trick is figuring out the minimum number of routers that have to replicate datagrams in order that all interested parties receive them. IP multicast is not supported by many of the commercial routers used in the Internet so a IP multicast backbone (“mbone”) overlays the larger IP network by having multicast-capable routers that are separated by incapable routers communicate with each other via TCP. See [1].

Asynchronous Transfer Mode (as opposed to the synchronous operation of SONET, which is the typical physical-layer protocol for ATM) is a circuit-oriented, micro-packet (“cell”) protocol. ATM is circuit oriented in the sense that ATM cells follow the same path for the life-time of the connection. So, like a telephone call, an elaborate signaling process precedes data transmission in order to identify the entire path that data carrying cells will follow. This circuit-establishing signaling process comes in two flavors, permanent virtual circuits (PVCs) and switched virtual circuits (SVCs). As the names imply, PVCs are “virtual wires” and changed infrequently, and SVCs are dynamic (they may occur at the frequency of TCP connection establishment). ATM defines a number of transport capabilities called adaptation layers (AALs), one of which (AAL-5) was tailored for IP.

Given that it seems fairly clear that the telecommunications industry will be deploying SONET and ATM infrastructure for the bulk of future high speed transport, the question is how to map Internet protocols to ATM. The reason for attempting this mapping, rather than scrapping Internet protocols in favor of data transport directly over ATM, is that in the Internet there is a great deal of experience in designing and building general purpose distributed systems. These systems, and the semantics of the IP-based transport, have grown up together, and it would not be a trivial (or even ultimately useful) task to adapt distributed applications to the semantics of an ATM-based transport. Additionally, the global Internet will be a very heterogeneous environment for a long time to come, and it is the function of IP to hide this heterogeneity from the transport and applications layer.

Further strengthening the case for successfully mapping IP onto ATM is that it is also seems clear that ATM is going to provide the fast, inexpensive, and scalable local area network technology in the future. It is already the case that ATM host interface cards are less expensive than FDDI interfaces, and the cost of ATM is dropping rapidly.

For these (and other) reasons, five of the six national gigabit testbeds that are intended to develop the network technology for the next generation high speed Internet, are ATM based. BAGNet is another example of such a testbed, thought it is focused on a wide variety of issues for the IP over ATM mapping rather than strictly on very high speed.

This paper describes some of the issues being addressed in BAGNet.

2.0 BAGNet Architecture

Pacific Bell’s CalREN program has awarded to each participant a grant covering the cost of two years of OC-3 (155 Mbits/sec), ATM service. The service is defined as best-effort virtual circuits to the other BAGNet sites, and up to a 15 db loss from each BAGNet site to the Pacific Bell equipment.

2.1 The Physical-level

The network backbone consists of several Pacific Bell operated, Central Office ATM switches connected via a SONET network. SONET is a topic of its own, but for the purposes of this discussion a few brief comments are in order.

SONET is the optical fiber implementation of the “synchronous digital hierarchy.” SDH specifies a synchronous communication mechanism and a set of interrelated communication speeds. It provides a set of framing conventions that allow many different “flows” of different bandwidths to be carried on a single physical circuit. These flows can migrate in and out of streams through the use of multiplexers and other SONET-specific equipment. SONET “networks” are relatively static, with the flows typically corresponding to physical end points of equipment. Reconfiguring a SONET network is something that typically takes minutes to hours. SONET is designed for high speed, long haul communication, with OC-48 (ª 2.5 gigabits/sec) being the typical bandwidth between SONET equipment operated by the telecommunication industry in the wide area. SONET streams are typically subdivided into several OC-12 (622 mbits/sec) and OC-3 (155 mbits/sec) flows operating between ATM (or other communications) equipment. SONET equipment is relatively expensive, and in BAGNet SONET is used primary to interconnect the Pacific Bell central office ATM switches, and to get the user switch ports (as opposed to the inter-switch trunking ports) to within 15 db fiber loss of the customer site. That is, SONET from the Pacific Bell ATM switch will be used to reach the central office that is within about 10 km (15 db fiber loss) of the customer site.

The overall Pacific Bell, Bay Area ATM network is used by organizations other than those identified as part of BAGNet (some of these are partnered with BAGNet organizations, and some are working independently). All together there are about 30 sites connecting to the metropolitan ATM network at the 155 mbits/sec rate. (The two currently defined access rates for the Pacific Bell ATM network are OC-3 (155 mbits/sec) and DS-3 (45 mbits/sec).) Figure 1 illustrates the ATM-level architecture of the Bay Area portion of the network. There are also connections to the Monterey Bay area, with inter-LATA ATM service being provided by Sprint, and there is a similar Pacific Bell network in the Los Angles area.

At the BAGNet sites there are several different strategies for connecting to the Pacific Bell network. The important issue is that Pacific Bell OC-3 ATM service is provided on single mode fiber. Single mode fiber is used almost exclusively in wide area and metropolitan area fiber-based networks, as opposed to the multimode fiber common in LAN (site) environments. Multimode fiber is used, or example, to support campus-wide FDDI and Ethernet. Several different types of ATM LAN equipment (switches and workstations) make up the “customer premise” equipment at the BAGNet sites. This equipment, even if SONET, OC-3 based, usually uses multimode fiber. So there is typically a “special” piece of equipment needed to attach to the Pacific Bell network interface (the carrier network - user network interface, or “UNI”). Figure 2 illustrates the two most common strategies. A third strategy, not illustrated, is to use a single-mode to multi-mode SONET converter. While this should be a simple device compared to other SONET and ATM equipment, there are not currently any volume commercial sources of this device, so it tends to be an expensive solution to the UNI problem.

2.2 The Link-level

The Pacific Bell, Bay Area ATM network is currently PVC based and fully meshed.This means that there is a fixed set of virtual circuits between every communicating pair of end-nodes in the network. The reason for using PVCs instead of SVC is that the much more complex mechanisms needed to support SVCs have not yet been implemented by all of the ATM switch vendors, and for SVCs to work all of the switches have to talk to each other during the circuit setup phase.

A virtual circuit is a path through a collection of ATM switches. The mechanism of providing this path is that there is a table in every switch with entries corresponding to every path through the switch. Note that neither PVC or SVC directly addresses routing. (That is, how do you determine by what physical path - sequence of switches - you get from source to destination?) Routing is currently done by “hand.”

2.3 The Network-level

ATM “Adaptation Layer - 5” (AAL-5) is used to carry IP packets. AAL-5 defines structure that organizes collections of cells to carry larger unites of data. AAL-5 has no multiplexing and no cell sequence numbers. The basic idea is to provide an efficient and reliable way to transport a protocol data unit (PDU). A PDU of 1 to 65535 bytes is structured in a cell sequence marked by a trailer. Cells are not sequence numbered within the PDU under the assumption that cells may be dropped, but not reordered by the ATM network. The entire PDU is protected by a 32 bit CRC.

IP packets with IEEE 802.2 Logical Link Control (LLC) headers are placed in the AAL-5 PDU. (See [2].) The use of LLC allows identification of the protocol (i.e. IP is encapsulated via 802.2). Other protocols could be transported using this mechanism.)

2.4 The Internet-level

There are a number of ways that the semantics of IP and its associated protocols might map to the ATM environment, and this is an area of active research in the computer science community. The BAGNet testbed group elected to use the model proposed in RFC - 1577: “Classical IP and ARP over ATM” [3]. This RFC specifies one way that the model of a “classical IP” network can be implemented in an ATM environment. Adhering to this model will provide us with insights into how ATM networks will work in the Internet environment. The essential elements of the model are:

The IP protocol stack treats the ATM host adapter as a network interface operating in a LAN-based paradigm;
A logical IP subnet (LIS) is a collection of ATM end-nodes and interconnects configured as one IP subnet (the IP end-nodes have the same IP network/subnet number and address mask);
All IP end-nodes in the LIS must be connected to the ATM network and must be able to communicate with each other (the LIS must have fully meshed PVCs, or SVCs established on demand);
Each LIS member communicates independently of members of other LISs that may be defined for the “same” ATM network (IP end-nodes configured in different logical IP subnets must communicate via an IP router - they may not set up a direct VC, even though this is possible);
All end-nodes must have a mechanism for mapping IP addresses to ATM addresses within the LIS (i.e. hosts must support the ATM Address Resolution Protocol).

2.5 Conclusion

There is a great deal of work to be done in order to make ATM a well integrated transport mechanism for the Internet protocols. In the process of doing this work both IP and ATM will be improved technically (and already have been), and more knowledge will be built up about the higher level question of universal transport models. (An important contribution to this higher level model has been made in the book “Realizing the Information Future” [4].)

More information about BAGNet, including an expanded version of this paper, may be found at http://www-itg.lbl.gov/BAGNet.html.

2.6 Acknowledgments

The work in BAGnet is done by many people from the fifteen constituent organizations. This note is a brief report on some of that work. The BAGNet executive is co-chaired by Bill Johnston, LBL (wejohnston@lbl.gov), Marjory Johnson, NASA Ames (mjj@riacs.edu), and Dan Swinehart, Xerox PARC (swinehart@parc.xerox.com. The IP task group is chaired by Berry Kercheval, Xerox PARC (kerch@parc.xerox.com).

3.0 References

M. Macedonia and D. Brutzman, “MBone Provides Audio and Video Across the Internet”, IEEE COMPUTER magazine, pp. 30-36, April 1994. (Available from ftp://taurus.cs.nps.navy.mil/pub/mbmg/mbone.html .)

J. Heinanen, “Multiprotocol Encapsulation over ATM Adaptation Layer 5”, Internet Engineering Task Force, Request for Comments (RFC) 1483, July, 1993. (Available from http://ds.internic.net/ds/dspg1intdoc.html .)

M. Laubach, “Classical IP and ARP over ATM”, Internet Engineering Task Force, Request for Comments (RFC) 1577, Jan., 1994. (Available from http://ds.internic.net/ds/dspg1intdoc.html .)

“Realizing the Information Future”, National Academy Press, 2101 Constitution Avenue, NW, Box 285, Washington, DC 20055, telephone: 800-624-6242. Also available from http://www.nas.edu:70/1/nap/online/rtif/, or from ftp://ftp.nas.edu/pub/reports/realizing_the_information_future.