INS mid-level documentation

Purpose:

This documentation is intended to give an overview of the use of the INS system, with an emphasis on more implementation details than the papers but at a higher level than the Java docs.

1.0 Introduction

In this document, we are describing the INS 2.0 system, in the WIND.ins CVS package. This version uses the Client Library to interact with the INR nodes, incorporates the spanning tree overlay network code, and includes full vspace support. Version 1.1 of the system, which at the time of writing is still used for most demos, is described at the end of this document in Section 7.0.

1.1 Components of INS

The INS is composed of three main software components:

These components are located in the wind.ins package (WIND/ins directory), which currently has the following subdirectories:

All the system code is effectively in subpackages of the ins package; the INR is in ins.inr, the DSR in ins.dsr, the Client Library in ins.api. This simplifies the CLASSPATH.The apps are however not in an ins package, so they can be moved around and expanded upon easily. It is important not to use the files in the INS 1.1 directories (i.e. WIND/apps, WIND/dsr, WIND/sys) -- they do not interoperate with INS 2.0 and can be confusing, particularly if they are in the CLASSPATH and the wrong apps get called.

1.2 Sample Command Lines

We start off with a description of how to run the boiler-plate Floorplan demo in a very self-contained fashion, with a DSR and INR set up on the local machine, and everything also set up locally.

Some services need to be run from any directory so that configuration information (such as which printers to monitor) may be read from the .conf file in their directory. This can be eliminated by placing all the .conf files in a single directory where they were all run from, etc.

2.0 Using INS

The command lines in Section 1.2 represent a first pass in using INS. Here, we look at the options for running the different components of INS in more detail.

2.1 DSR

The DSR has no command line arguments. Its job is to track all the INRs in a given domain and all their vspaces. The main request it receives is to get the INRs which host a given vspace. It gets this information from INRs, which periodically (now ~30 minutes) announce which vspaces they host, with expiration times.

It assumes that it binds on UDP port 5678 of its host, thus there is a maximum of one per host (ideally only a few per domain, which all mirror each other). Applications/INRs expect, unless told otherwise, that the DSR will be at dsr.domainname.

In practice, it is helpful to have separate DSRs, particularly for debugging. That way, if bugs come up, INRs get taken down and restarted, or things get recompiled, everything can easily be restarted with a clean state. In system-level debugging, it's typical to have a DSR, INR, and test applications all running self-contained on one or a few local machines, and then to stop and restart most/all of the components between runs.

2.2 INR

The INR provides the backbone of functionality for INS. Any number of them, hosting any number of vspaces, may be deployed, since they can all find each other and form a spanning tree network using the DSR (which they in turn all advertise their existence to).

The INR's command-line syntax is as follows:

java ins.inr.Resolver hostedvspaces [-d DSRname] [-udp udpport] [-tcp tcpport] [-n neighbor names]

For the most part, the only the hostedvspaces (mandatory) and [-d DSR] parts are used. There is typically no use for specifying the server's port numbers or neighbors in application development. There are no commandline options to change the topology/overlay network (that's more something that should be done for now by modifying the related resolver code.

The [-d DSRname] option is used to override using dsr.domainname as the DSR to connect to. For the examples in 1.2, this was directed to a DSR on the localhost.

The hostedvspaces field is a list of vspaces which this INR should host locally. This may be a single vspace like wind or a list like floor1,floor2,floor3 (no spaces). A colon is used to specify than the INR hosts a vspace owned by another domain/DSR (see section in thesis on domain-qualified wide area vspaces): printers:mit.edu,lcsprinters,printers:cs.berkeley.edu.

(this paragraph may be skipped at first) This hostedvspaces field is further extended for aggregate vspaces/unions of vspaces. To specify that a vspace is a parent of some children, either () parentheses or {} braces are used. For example, if the lcs vspace is defined as the parent of floor1,floor2,floor3, the hostedvspaces field can be written as lcs(floor1,floor2,floor3). The parentheses also define the child vspaces as being hosted locally, whereas the braces don't necessarily (the equivalent to the previous with braces would be lcs{floor1,floor2,floor3},floor1,floor2,floor3) -- this allows hierarchical vspaces with remotely-hosted children.

To use Andrew Lau's indexed name structure, -DuseIndex=true should be added between java and ins.inr.Resolver on the command line.

There are a few interactive console debugging features -- some commands, such as getNameTree, can be typed while it is running to see the current status of the name-tree (see main thread in the Resolver). In addition, some messages are printed out depending on the value of Resolver.DEBUG and more can be included.

2.3 Applications

Here, we first look at the Application tester, which is a test driver that works well for verifying the correctness of system-level changes, and then we examine the different services associated with Floorplan.

Application tester

The application tester is a test driver that can be used to test all the main requests that an application might ask of INS. It can also be used to test running applications, by sending specific packets and otherwise interacting with them.

The program, ins.api.AppTests, is designed to be hacked and to be very hackable. The available commands can be browsed and their parameters determined by looking through the code. Briefly, it allows:

LocationServer service

The location server provides two main functions: returning a map surrounding a location and returning the child's coordinates on a parent's map (e.g. where 504 is on the floor5 map).

The data for the LocationServer is stored in files in a directory structure. The LocationServer.conf configuration file, which is usually read from the directory where the LocationServer is invoked, contains information on where to find this location directory. The configuration file also specifies the names of the files which are assumed to contain the floor images and which file has the coordinate data.

The subdirectories in the location are patterned aftr how the location name-specifier looks. In the current release, the location is indicated by the [location=...] name-specifier branch, plus the vspace. Location is properly defined by the the [location=...]/vspace pair. Once it reads to the end of the [location=...] branch of the name-specifier, it uses the coordinates or image information in the directory.

In the current release, coordinates are retrieved by late-binding and maps (due to their size) by early-binding.

With the introduction of vspaces, it is assumed that a location server, if it serves a vspace (which is also in the configuration file), that it serves every knowable location in the vspace. This is consistent with the size assumptions of vspaces.

Floorplan service

Floorplan is a tool to graphically display discovered services, but it is not necessary to use any of them -- camera and printer can be used on their own, but that requires knowing the full name-specifier of the service (e.g. [service=camera][location=...]...).

Floorplan uses the [service=...] field to determine which viewer application to run. This is why the configuration file is necessary. While in some ways this is incomplete (it would be nice to be able to download the correct viewer!) and the need for a configuration file inelegant, it is analogous to a web browser storing the plug-ins that it knows about -- netscape might have the Flash and Acrobat viewers installed, so it has records the mime-type to viewer mappings.

In addition, applications that want to be displayed in the floorplan are now required to have [floorplanshow=true] in their name-specifier. This originated as a hack to prevent receiver clients (which are also [service=printer], just [entity=client]) from being erroneously displayed. But, this type of field could be used as a pointer for where a viewer could be automatically downloaded.

Each floorplan map panel is techically a separate INS application (in the same JVM) with its own threads, which at this point does not seem to be problem at all performance-wise. But, it might be wise to consolidate some of the functionality or multiplex them to a single INS "application" (this is way low-priority for now).These floorplan maps each send a discovery request to the INR periodically (now about every 3-5 seconds, depending on processing time). The scalability of this could be improved upon by implementing an expiring "push," rather than "pull" version of discoverNames (requires modifying ins/Application and inr/AppManager, plus keying off events in the RouteManager). Services that it discovers which are on that page are represented with the icons in the configuration file, while further away services are represented by hyper-linked dots.

When a service or dot is pressed, the event handler calls a routine in floorplan which attempts to invoke the proper application by Java reflection. The application is loaded, and it tries to invoke the application using the same type of constructor by which the original floorplan was invoked. That is, if the user manually specified the DSR, that constructor is used, or the constructor for peering to a single INR, or the constructor for searching for a DSR. It is assumed that invoking the constructor will be enough to start the threads that run the application.

For good clean-up behavior, since many of these viewer applications may be running in the same JVM, Application implements a cleanUp method which kills the announcer and receiver threads and frees the sockets. When a viewer service is closed by the user (i.e. hits the "X" in the upper right hand corner of a GUI), the application should clean up after itself, run cleanUp, and not call System.exit.

There could be some improvements to work well across vspaces, such as creating proxy services that advertise another vspace (effectively allowing hyper-linking to another disjoint vspace). This would help scalablility (particularly reducing the magnitude of discovery requests) and be straightforward to implement.

Camera service

The camera service has two components: the transmitter and receiver.

The Transmitter requires some configuration information, particularly how often to send images, how to retrieve the images, and what location to advertise the camera as being at (the last could foreseeably be retrieved from the LocationManager, but that may not always be the appropriate way in practice).

In Windows, the practice is to run the quickcam capture program in separately such that it dumps an image in a directory frequently. The file is then read every few seconds by the camera transmitter service and transmitted. In Linux, the capture program is executed every time, and the result is read from the capture program's standard out stream. The Transmitter.conf configuration options are complete enough that it allows all this to be specified. The configuration file also lists which vspaces to advertise the camera in.

The Receiver takes as an argument the name-specifier of which discovered camera to listen to. This is normally supplied by the Floorplan application, which originally discovered the Transmitter. But, it is possible to run Receiver by hand with this information.

Printer service

Printer is similar to camera -- there is a gateway and client. The gateway proxies for a set of printers, whereas the client connects to any/all the gateways to use the appropriate printers.

The LPRGateway, like the camera transmitter, requires configuration. The LPRGateway.conf file consists of a list of printers to proxy for, along with their locations. A "root" location is allowed that is prepended to all the locations to shorten the descriptions. And, relevant vspaces to advertise in is also in the file.

The printer gateway is a good service to browse to see how early-binding works and how application metrics may be changed on the fly. Periodically, this gateway "lpq"s all the printers to count the number of outstanding jobs, and it uses this information to modify the advertised application metric.

The LPRClient, like the camera receiver, needs a name-specifier argument to point which printer the client is to converse with. This is typically supplied by the Floorplan application, but it needs to be furnished if the LPRClient is to be run by hand apart from Floorplan.

3.0 Application Development

The Client Library provides the basic functionality that services and applications need to function in INS. This includes functions for the "big three" functions that an INR supports:discovery (discoverNames), late-binding (sendMessage, overriding receivePacket), and early-binding (getHostByiName). But, it also includes a number of helper functions to do things such as talking with the location server and sending name announcements to advertise the service.

3.1 "Hello World"

First, we look at a minimal boiler-plate INS application. It should:

In addition, we would like the application to do something, which typically involves sending and receiving late-binding packets. Here is a "hello world"-type application that sends hello world messages to itself whenever the user presses enter:


import ins.api.*;
import ins.namespace.*;
import ins.inr.Packet;
import java.io.*;

public class hello extends Application {

    NameSpecifier ourName = new NameSpecifier("[service=hello][vspace=test]");

    // constructors
    public hello() throws Exception { super(); init(); }
    public hello(String dsrname) throws Exception { super(dsrname); init(); }
    public hello(String inrname, int inrport) throws Exception { 
	super(inrname, inrport); init(); }

    // common init routine to start the name announcer
    void init() {
	// start announcing ourselves to the system
	startAnnouncer(new NameSpecifier[]{ ourName }); 
    }


    // sends a "hello" message through INS late-binding to ourselves
    void sendHello() {       
	sendMessage(ourName, ourName, "hello world".getBytes());
    }

    // Called whenever a packet is received
    public void receivePacket(Packet p) {
	System.out.println(new String(p.data));
    }

    public static void main(String [] args) throws Exception {
	hello h = new hello("localhost");

	// send a hello message to ourselves whenever enter is pressed
	BufferedReader br =
	    new BufferedReader(new InputStreamReader(System.in));
	while (br.readLine()!=null)
	    h.sendHello();
    }
}

Note that a DSR and INR need to be running for this program to work. The INR should make sure to host the "test" vspce.

We now discuss some of the relevant features of this hello program. This includes a look at constructors, name announcements, and the sending/receiving code.

First off, there are three possible constructors for the INS Application class:

It is a good idea to provide matching constructors for all three, and then to have a common init() routine for them to call when done. In the code above, we simply chose to use the DSR on "localhost" (see hello h = new hello("localhost"); line).

As a reference, the floorplan application and its associated services are good examples for how command line parameters can be parsed to to the desired constructor -- no directives yield the first of the three, [-d DSRname] yields the second, and [-p INRname INRport] yields the last.

Beyond that, the application/service should announce its name so that others may know of its existence. This occurs with the startAnnouncer method, which starts a thread that sends periodic advertisements. There are multiple versions of the startAnnouncer method, the simplest of which just takes in an array of NameSpecifiers to announce. This usually is sufficient. Other versions of the method allow the application-defined metrics or hostrecords for early-binding to be changed (see the printer application code). The list of names being announced may be modified, added to, deleted from, and the whole process may be stopped.

Sending a packet is done with sendMessage. There are several versions of this method -- the simplest just takes a source namespecifier, destination namespecifier, and a byte array to be sent. Other versions allow the user to specify whether the packet is sent multicast, and (less relevant) whether its source INR should be embedded in the packet header and be announced.

An application can receive late-binding packets by overriding the receivePacket(Packet p) method. Whenever the underlying thread receives a packet for the application, it calls this method. The method here just prints the data in the packet.

3.2 Early-binding

The early binding mechanism provides a way for a service to announce information about itself so that an application may contact it directly. While any protocol may be used, it typically is TCP-- the service starts a TCP server thread and then registers the port number so that it may be properly announced. The application can then look up this information using getHostByiName to contact the server.

For services, once the early-binding TCP (or other protocol) server is running, it must register the protocol name and port number. There are two ways to do this:

The application gets this information by using getHostByiName. A query to filter the services, such as [service=printer][location=*][vspace=wind] is passed to this routine, as well as a flag signifying whether all matching names, or just the one with the highest metric, should be returned.

The result of the getHostByiName query is an array of EarlyBindingRecord, which contain a metric an a HostRecord (ins.inr.HostRecord). The getInetAddress method can be called on the hostrecord to determine the machine name to connect to, and the getPortForTransport method can be called (e.g. hostRecord.getPortForTransport("TCP")) to get the port number on that machine. Or, the application can look through the list of available transport protocols. In any case, once the application knows which service it wants to choose, it can connect to it using the early-binding information in the HostRecord.

3.3 Discovery

Discovery requests are similar to early-binding requests -- a query is sent for a set of names and a response is returned. However, instead of EarlyBindingRecords, an array of matching NameSpecifiers is returned. This can be sorted through and successively refined until the correct service is found.

The method to use for this is discoverNames. The simplest version of the method takes only the NameSpecifier query (such as [service=printer][vspace=floor5]), and it returns a NameSpecifier array. Another version of the method takes a flag, which is relevant for use in making an aggregate vspace browser (this flag can enable or disable searching child vspaces for unions of vspaces), but for the most part the former is sufficient.

3.4 Location

Location can be retrieved from the cricket devices using the LocationManager. This LocationManager (not to be confused with the LocationServer) is not an INS service. Rather, it is a server that reads data from the cricket by the serial port and makes this data available to the clients via a well-known UDP socket that can be queried. The clients have helper functions, known as getLocation and getLocationNameSpecifier which do the querying.

The main reason for separate getLocation and getLocationNameSpecifier functions in the API is that the amount of data which comes from the cricket beacons is very small (4 bytes -- room 504 gets announced as "0504"). The getLocation routine gets that "0504" string from the LocationManager.

The getLocationNameSpecifier method attempts to get a NameSpecifier which more accurately describes the total contents. The request is passed to the LocationManager, which returns a namespecifier in the form [location=...504]]][vspace=...]. How does the LocationManager get this extra information (floor, building...)? Right now, it's hard-coded into the LocationManager by the -context and -vspace flags. This hard-coding needs to change eventually, perhaps by the beacon advertising a server IP:port where it could get the full information, but it appears that the LocationManager is the correct module where this type of information should be known.

As a security precaution, the LocationManager only accepts requests to its server from entities on the localhost. Thus, a client only knows where it is (unless some other client advertises its existence).

4.0 INR Architecture

The Intentional Name Resolver internally has a prototypical server architecture: it receives packets and often sends a packet in response. We look at the different components of the architecture, including the receiving subsystem, the forwarding subsytem, the internal applications, and the neighboring system.

4.1 Receiving packets

INRs receive packets either by UDP or TCP before they are handed off to the rest of the INR. Most of the packets come in by UDP, but some of the routing/overlay network code uses TCP. The UDP packets come into the system via the UDPForwardThread, which waits for new packets to receive and passes them on to the forwarder. The TCP packets come in by various TCPForwardThreads (due to the nature of TCP, there is a thread per neighbor connection), and are then passed on to the forwarder. The Communicator is used as an abstraction for the sockets.

As an important note, packets are not kept around (except in a few special cases with aggregate vspaces, when they are copied internally anyway) -- they are either forwarded or dropped rather than buffered. The operation is largely single-threaded as well (there is a single UDP receive/send thread), which eliminates the need to new large buffers for every received UDP packet.

These packets are in the INS packet format, which is encapsulated by the Packet class. Briefly, these packets consist of a source namespecifier, a destination namespecifier, data, a multi/anycast flag, and options data.

4.2 Forwarding packets

All received packets are sent to the Forwarder, which is responsible for forwarding the packets to an outside component (e.g. a user application or service) or to an internal application (e.g. the overlay network manager or the discovery request responder). The packet may also be dropped. By convention, packets with destination namespecifier containing a [control-msg=...] field are usually intended for internal applications.

Packets come in from the receiver threads to either the Forward or ForwardByTCP methods in Forwader.java and then are routed to either the anycast or multicast methods. Both any/multi-cast go through a set of functions: they check whether the source name is in the system to infer a route, they check whether the destination vspace is hosted locally (and forward the message to another INR otherwise), they do a lookup of the destination namespecifier on the appropriate name-tree, and they use the results from the name-tree lookup to forward the packet.

An INR assumes that it has full knowledge of every vspace which it hosts, so it assumes that if it has the vspace's name-tree (i.e. it hosts the vspace), the results from a lookup are authoritative, at least in a soft-state sense. If it does not host the vspace, it can send a request to the DSR for an INR which does and simply forward the message.

The result from the name-tree lookup, which is a name-record, may be of one of two types: an remote destination or an internal application. It will have a last hop INR or IP address to forward the message to in the former case. The any/multi-cast routines just have to figure out the which (or all) destinations to forward the message to, and this should be set. For an internal application, there will be a pointer to a the handling class, which implements the IHandler Java interface. Here, the INR just calls the receive method of the internal application.

At this point, the message either gets to a remote destination or internal application.

4.3 Internal applications

Here is a list of important internal applications:

4.4 Data structures

The main data structures in the INR keep track of the relevant vspaces, neighbors, and name-specifiers.

All (and only) the name-specifiers for a given vspace are stored on the vspace's NameTree. All the nametrees in the system are stored in the VSNameTree structure, which consists of vspaces mapped their respective nametrees. The VSNameTree is the authoritative list of which vspaces are "hosted" by the system and are authoritatively known.

For the network/neighboring information, other INRs, as determined by unique address and port combinations, are represented by the Node data structure. This structure contains the address and port information, as well as the list of vspaces which it hosts. There is also expiration time and statistical data. These nodes can be stored on two lists:

All the Nodes in the system that host given vspaces are stored in the VSResolvers structure. This maps a vspace name to a set of Nodes. This includes vspaces that are not locally hosted (for instance, if an INR only hosts vspace X but occasionally needs to send packets to vspace Y, it caches the list of INRs that host Y, as well as keeping all its potential neighbors in vspace X). The INR does not necessarily have a TCP connection or relationship with these nodes. Rather, the elements in VSNeighbors are there either as candidates to neighbor with or as places hosting non-local vspaces where packets may be forwarded to.

A subset of the Nodes in VSResolvers are also in VSNeighbors. These represent the actual current neighbors on the overlay network to whom connections are made and periodic communication is expected. Like VSResolvers, this structure maps vspace names to a set of Nodes, but it is restricted to the set of vspaces that are locally hosted (i.e. present in VSNameTree) and it typically is only a subset of the INRs in those vspaces. Neighbors represents the name and route announcement network as well.

To maintain data consistency, there is only one total copy of the Node data structure in memory for an INR. This is between the VSNeighbors and VSResolvers structures, and between the different vspace entries in each of these structures.

5.0 Vspaces

Vspaces are an application-defined partitioning mechanism that may be used to enhance the scalability of INS. The design and higher-level picture, as well as many other details, are found in Jeremy's thesis, but we will look briefly at the practical aspects of vspaces.

For a small application, there is no need to have multiple vspaces. Using a single "wind" vspace is fine for that (which we do in the current demo). And, a vspace is a little too heavy-weight to be used on a per-room basis (that is why I defined location to be the combination of a vspace and a [location=...] name-specifier branch).

However, as the system grows, there may be many "natural" divisions by which the names may be split, such as by building, lab, or floor. Vspaces try to reflect the structure by which some groups are relatively self-contained or related. Applications may advertise themselves to multiple vspace. Or, aggregate vspaces (see command lines in Section 2.2) may be used to "glue" these split vspaces together at a higher usage but lower advertisement cost.

With the introduction of "harder" state, rather than pure soft-state (i.e., split the state maintenance for names into soft state and hard state -- see William's thesis), the advertisement bandwidths in the paper and in the Jeremy's thesis are not as bad anymore, so the system is much more scalable within a given vspace than we previously said (Jeremy has registered his views on the difficulty of doing this correctly. William thinks that reducing bandwidths used by unnecessary advertisements at the cost of careful programming and clean code for this technique is worthwhile to do).

INRs are said to "host" a number of vspaces -- each vspace they host, they know, within a soft-state precision, all the vspace's names. They are started with this information on the command line, though there is no reason that a load-balancing protocol could not adjust this on the fly (in fact, there are routines in Resolver.java to produce and remove vspaces, which were designed for this purpose). This list of vspaces is advertised periodically to the DSR, which can in turn allow other INRs to look up this information.

Practically, portions like the floors in LCS and the LCS building can/ should be encapsulated with vspaces, rather than using a long [location=...] hierarchy. In other words, the printer could be in [location=517] in the [vspace=lcs5] vspace, rather than in [location=mit[building=ne43[floor=5[room=504]]]] in the [vspace=mit] vspace. For the purposes of a floorplan-like application, proxy services could advertise the [vspace=lcs5] vspace in the [vspace=lcs] vspace, perhaps as [service=vspace][vspacename=lcs5][vspace=lcs], which would allow hyperlinking. It would be very straightforward to produce a proxy service that advertised all these vspaces, and it would make discovery more efficient by drastically reducing the number of irrelevant services that are searched and returned.

6.0 Limitations

This INS release has its share of areas that are not be resolved and could be improved upon, with different priority levels.

7.0 The "Old System"

Since the 1.1 release is still used in demos, we present some documentation for using it. The two versions are incompatible (e.g. packet formats, APIs are different). But they may coexist on the same machine, since their DSRs use different port numbers, and INRs are essentially server applications that bind to any port. The 2.0 system is much much easier to program with the client library, so it is recommended that all development be done for the new system. Version 2.0 also supports the overlay network, does not require an effective INR per application, and it supports vspaces better.

Here is how to run the old floorplan demo, self-contained on a single machine. See some of the longer descriptions about using the services in the 2.0 system:

Draft by jjlilley@mit.edu (5/18/00) and wadjie@lcs.mit.edu (5/27/00).