In a MANET, nodes are often mobile and there may even exist mobile subnetworks which are interconnected through a backbone [7] [26]. In flat MANETs, routing is challenging in the presence of mobility because links become unpredictable as a result of the dynamic nature of these networks. This becomes even more challenging in cases where there are mobile sub-MANETs. In such setups, the routing scheme deployed in the backbone network must be able to maximize the backbone bandwidth, enhancing throughput and reducing end-to-end delays with respect to schemes without a backbone. It must do so without compromising (in fact, possibly enhancing) scalability and fault tolerance.

Distributed Hash Tables (DHTs) are a form of a distributed database that can store and retrieve information associated with a key, in a network of peers/nodes that can join and leave the network at any time. In this paper, we leverage this distributed data sharing functionality to provide extra information with which each gateway can carry out routing through the backbone efficiently. By leveraging this functionality, different forms of mobility, including single nodes switching cluster membership and whole clusters merging/splitting, can be managed. The DHT strategy is used as a lookup strategy in structured P2P systems to specify the location of objects in peers, and it provides all the needed services to look up objects in a decentralized P2P system. The DHT retains mapped information about nodes and peers in the form of key/value pairs (k, v) so that data can be easily located in the overlay network. In DHTs, each peer maintains a storage space to keep a hash table. Many structured P2P systems like Chord [27] or OneHopOverlay4MANET [12] are based on a DHT.

We divide a flat MANET of N nodes comprising h nodes and g gateways into c clusters. Each cluster is therefore a MANET on its own and has (at least) one gateway node. The gateway nodes are connected in a backbone MANET and connect their local clusters to the rest of the network. Each cluster is basically a separate and independent routing domain, running its own intra-domain routing protocol. Similarly, the backbone, which is its own a MANET, may require a routing protocol to coordinate routing among the gateway nodes.

Within a cluster, we selected OLSR [28] as our routing protocol. While this addresses intra-cluster connectivity, routing between nodes in different clusters is more complex. For example, a node A₁ in cluster A needs to route a packet to a node B₁ in cluster B, as shown in Figure 1. The source node A₁ creates an IP data packet using the address of the destination node B₁. A₁ does not know how to forward the packet to B₁ because its routing table does not have an entry for B₁. The OLSR protocol supports an optional auxiliary functionality, “Host and Network Association (HNA) Message”, whereby a gateway announces reachability of other nodes or subnets to its local cluster members. The gateways in each cluster (here GW_A for cluster A and GW_B for cluster B) will periodically broadcast HNA messages to all nodes in their cluster to advertise reachability of nodes outside the cluster through them (Step 1 in Figure 1). We advertise a global default route (0.0.0.0/0.0.0.0). Each node will create an entry in its routing table that IP forwarding defaults to when looking up a route. Packets destined for nodes outside the local cluster will consequently be forwarded to the appropriate cluster gateway. In case a cluster has multiple gateways, each advertising reachability of the outside world, the closest gateway (in terms of hop count) will be used.

Once a cluster gateway received a data packet destined to a node outside its cluster, it will forward it over the backbone. This requires a routing solution/protocol, simply running another instance of OLSR on the nodes forming the backbone will not suffice. Consider the example in Figure 1: once GW_A receives a packet destined for B₁, it needs a way to forward it to the appropriate gateway, here GW_B. We developed two solutions for this problem: Flooding and DHT-based Routing.

Flooding: Flooding is a simple approach for the MANET backbone. A source node in the backbone will broadcast a packet in the backbone once, and all other nodes, except the destination gateway, will rebroadcast these packets. The gateway that knows (from its local routing table) that the destination node is a member of its local cluster will forward the data packet into its local cluster, using the intra-cluster routing protocol. In this case, the backbone has no need of a specific (unicast) routing protocol. We implemented a flooding protocol to use as a base case to evaluate the performance of our solution.

DHT-Based Unicast Routing: Using a unicast protocol like OLSR in the backbone will carry out routing functions among the members of the backbone network (the gateways nodes). Each gateway GW_X participates in a P2P overlay to advertise reachability to all nodes X in its local cluster by storing a key-value pair (X, GW_X) as follows. Gateways learn about all nodes in their cluster through OLSR, which will populate a local routing table. Using this information, each gateway node GW advertises reachability to all nodes N in its cluster by storing the following key-value pair in the DHT: (k, GW) (Step 2 in Figure 1). Key k is generated by hashing each node’s IP address.

Figure 2 shows the data packet forwarding, assuming that the network is static. Once A₁ has a data packet to transmit, it will consult its routing table. Using the default route created by the HNA messages, it routes the packet to its gateway GW_A (Step 3 in Figure 2). Once the gateway receives such packets, it executes SHA-1 on the IP address of B₁ to generate key k. It then queries the DHT using this key. This retrieves the associated gateway, GW_B, for B₁’s cluster (Step 4 in Figure 2). GW_A forwards the packet to GW_B through the backbone (Step 5 in Figure 2) using tunneling (using IP-in-IP encapsulation). Once GW_B receives the packet, it decapsulates the datagram and forwards the packet within its cluster based on the local instance of the intra-domain routing protocol (Step 6 in Figure 2).

This approach is similar to on-demand routing protocols such as AODV or DSR: routes to new destinations are only discovered when the first data packet is destined to such a node. However, our DHT lookups are done via unicast routing in the P2P overlay, rather than broadcasting/flooding the backbone. Furthermore, unlike such protocols, any topology changes in the backbone will be hidden from the gateways (i.e., no route maintenance), reducing the need to rediscover routes. Connectivity among gateways is maintained by the OLSR instance executing in the backbone.

We expanded this basic scheme to include routing support for various mobility scenarios which are likely to result in MANET environments: mobility within the local cluster, whole clusters joining or leaving, clusters merging or splitting, and nodes switching cluster membership or completely disappearing (can no longer be reached). Nodes moving within their local cluster are simply a variation of the static scenario. The challenging mobility scenarios are when clusters merge/split or a node changes cluster membership. All nodes use their (single) radio to communicate on a fixed frequency. Once a node joins a different cluster, it will exchange OLSR control messages with other nodes, effectively joining this cluster. Upon leaving a cluster, the node will stop this control message exchange, allowing other nodes to detect its absence. Similarly, when groups of nodes come into radio contact with each other, they will form in essence one MANET, exchanging routing information with other nodes, forming a single (merged) cluster with multiple gateways. Once these groups separate physically, they will also stop forming a single cluster, splitting into two or more disjoint clusters with their own gateway(s).

In the basic routing case we described earlier, there was no need to update the DHT with any information because the node-gateway mapping stored in the DHT stayed the same for the entire network lifetime. With mobility, it becomes necessary for the gateways to update the DHT with recent routing information regarding the nodes they can now reach or no longer reach. For example, when two clusters (A and B) merge, the ensuing cluster will have two gateways GW_A and GW_B who will now both advertise reachability of all cluster nodes in the DHT. Similarly, when the clusters split, the gateways will have to delete entries they initially advertised in the DHT, for nodes they can no longer reach. In the case of nodes switching cluster membership, a gateway GW_X to the cluster X which the node n arrived in will learn about the presence of n in its cluster via the intra-domain routing protocol, OLSR, and will advertise to the DHT that it can now reach node n. Similarly, the gateway GW_Y to cluster Y which node n left will have to update the DHT that it can no longer reach node n. Temporarily, depending on the order in which these updates occur, a node may be advertised as being reachable through no gateway, one gateway, or both gateways.

To support these scenarios, we store more than one gateway IP address under the key representing a destination node. We serialize a list of potential gateways and store this as the value v in the PUT message. This way, a source gateway that retrieves this information can use the hop count information in its routing table to select the closest gateway. While selecting the closest gateway through the backbone does not necessarily translate to the shortest path to a destination node, it serves to minimize the traffic through the backbone. Figures 3-5 show the way the DHT is updated to support the different mobility scenarios. In each of these figures, hosts A₁, A₂ and gateway GW_A are members of cluster A while hosts B₀, B₁, B₂ and gateway GW_B are members of cluster B. B₂ is a host previously in cluster B that recently moved to cluster A. The arrows represent DHT queries and responses while the dotted lines indicate wireless connections (potentially multi-hop). In order to provide a solution that supports all our mobility scenarios, every gateway queries the DHT before adding an entry to it.

Queries are distinguished from another via different code names: ADD_ROUTE, DELETE_ROUTE, LOCAL_UPDATES. Gateways will handle these queries differently. The three mobility cases together with the steps involved in updating the DHT in each case are described below.

1) ADD_ROUTE

Initially, every gateway advertises reachability to all hosts in its cluster in the DHT as explained earlier. Subsequently, in events like clusters merging or hosts joining, the gateways will learn about these new hosts after their routing tables have been updated by the intra-domain routing protocol, OLSR. The gateway then hashes the individual IP addresses of the new hosts and queries the DHT. This query will either be successful and return a valid value, a NULL value, or completely fail. According to the result of a query for a host B₂, a gateway GW_A will do the following:

a) Retrieving a valid value from the DHT implies that an entry for B₂ already exists in the DHT. If the associated list of gateways already contains GW_A, this entry need not be updated (Step [5a] in Figure 5). Otherwise, GW_A will add its IP address to the existing entry and update the DHT (Step [1] in Figure 3).

b) Returning a NULL value implies that no entry exists for B₂ in the DHT. GW_A will then hash the IP address of B₂ to generate key k and initiate a PUT message to advertise its reachability to N in the DHT with the message: PUT(k, GW_A) (Step [4] in Figure 4).

c) When a query fails (neither returns a valid value nor a NULL value), GW_A chooses a random back-off interval and re-issues the query to the DHT.

2) DELETE_ROUTE

When a host B₂ leaves a cluster, or clusters split, gateways learn that one or multiple hosts left after their routing tables have been updated by the intra-cluster routing protocol. The gateways will hash the IP address of a host B₂ to generate key k and query the DHT, which will return the value currently stored against key k (Step [2] in Figure 3, [3] in Figure 4, or [6] in Figure 5). If the value retrieved contains GW_A’s IP address, it removes its IP address from the list and updates the DHT (Step [2] in Figure 3). Otherwise, for a single entry corresponding to its IP address, it removes the entry, and updates the DHT with a NULL value (Step [3] in Figure 4).

It is difficult to synchronize the DHT updates between GW_A and GW_B for host B₂. Consequently, race conditions may occur. For example, consider Figure 4 and assume that both gateways queried the DHT concurrently. GW_A updates the entry by adding itself to the list of gateways reaching B₂. This information will be lost when GW_B overrides the entry with NULL (believing itself to have been the only gateway registered for the node. To handle this, each gateway will periodically check with the DHT to make sure that its update was successful (Steps [5] and [6] in Figure 5).

3) LOCAL_UPDATES

Each gateway maintains a separate data structure, TEMP_ROUTING_TABLE, which caches all the routing information for the different destinations X, Y, …, Z retrieved from the DHT. With mobility, this routing information becomes stale. To check that the information is still valid, the gateways query the DHT periodically to refresh this information. If this information is not used, the routing cache expires (soft state).

In the static scenario, all nodes are uniformly distributed in their different cluster areas, and remain members of the same cluster for the entire network lifetime. Table 1 summarizes our key simulation parameters. Initially, all 40 nodes participate in a flat network where all nodes are hosts who participate to support routing in the network and 30 of the 40 nodes exchange ping messages with each other. The 40 hosts are then divided into 2, 4, 5, 6, 8 and 10 clusters having 2, 4, 5, 6, 8 and 10 gateways. The clusters are MANETs and the gateways are part of a backbone MANET through which they inter-connect all the clusters. Gateways are equipped with two radios with which they can communicate over two separate interfaces: the cluster they belong to, and the backbone network. The hosts, on the other hand, only have a single radio for communication within the cluster they belong to. The resulting network is a two-tier hierarchical network structure, with the gateways serving to provide connectivity between members of their cluster and external networks. In the 2, 4, 5, 6, and 8 cluster configurations, 8, 6, 5, 4, and 2 gateways respectively, do not participate in the backbone network but exist to support routing in the clusters they belong to. Thus, they behave like hosts but do not send nor receive any ping messages. In the flat network configuration (one cluster), all hosts are uniformly distributed in the network area and they assume this position throughout the entire simulation duration. In the other cluster configurations, each cluster is assigned a unique area in the network area and the member nodes are also uniformly distributed within their cluster space. Similarly, nodes in their individual clusters assume the same initial position for the entire simulation duration. Simulations were run for 900 simulated seconds. The first 210 seconds was allowed for the routing tables of each node to be populated and for the gateway nodes to build the overlay network and reach stability. The results reported here are the averages over 10 repetitions for each scenario, using different random number seeds each time. Graphs for each metric also show the margin of error by plotting the 95% confidence intervals. At times, these CIs are quite narrow and therefore not easily visible.

We explored 4 different ping packet sizes, but the results and trends are very similar. So for brevity sake, we report on only one set of results, using a ping packet size of 256 bytes. In all figures, the (F) suffix in the legend refers to the metric value when using Flooding as routing solution in the backbone, similarly (D) refers to the metric value when using the proposed DHT-based approach.

With flooding, as the number of clusters increases, PSR deteriorates sharply (Figure 6(a)). We also observed that the number of packets transmitted in the backbone, as well as packet collisions and packet drops, increase drastically with the number of clusters. The backbone handles more traffic because of the redundant rebroadcasts of each data packet. With multiple gateways re-transmitting a just received packet, there is a high probability that these successive transmissions result in collisions, even when adding random jitter. Overall, as the number of clusters increases, for each ping packet size and across the different ping packet sizes, the frequent rebroadcasts utilize most of the backbone channel resources. Queues start to build up and packets are being dropped, resulting in fewer ping packet exchanges being successful.

For the proposed DHT-based solution, as shown in Figure 6(a), PSR is consistently high for pings that cross the backbone (i.e., sender and receiver are in different clusters), as shown by the blue line. In fact, the success ratio is almost as high as for pings among nodes in the same cluster (the green line). The backbone traffic does grow with the number of clusters, but at a slower rate than was the case for flooding. We also experience almost no collisions and dropped packets. With the DHT-based approach, ping packets are transmitted through a unicast tunnel, so the MAC protocol will retry packet transmissions a certain number of times, ensuring the high PSR across the backbone. Overall, the protocol delivered over 95% of the packets to their destinations, which remains constant for all numbers of clusters and ping packet sizes.

Figure 6(b) shows PRTT as we increase the number of clusters. Both approaches have similar values for the one- and two-cluster scenarios but begin to vary significantly beginning with the four-cluster scenario. The one-cluster scenario has no backbone through which data needs to be routed and is expected to produce similar results. The two-cluster scenario involves only two gateways.

The flooding solution will behave like a unicast solution since every broadcast packet in the backbone will be directed to just one gateway node, who happens to be the gateway to the destination host. In this case, there will be no redundant broadcasts. However, as we increase the number of clusters further, the round-trip time in the flooding solution (red line) begins to increase. We also note that it gets worse as we increase the ping packet size. Again, the underlying reason is the increased number of (redundant) packet transmissions in the backbone, forcing new ping packets to be queued for longer periods before gaining access to the channel.

The DHT-based solution achieves lower PRTT (blue line, mostly overlapping with the green line), and remains almost constant across all scenarios. For every packet a gateway GW_X is routing for the first time to an external destination, GW_X queries the DHT to obtain the routing information required to route that packet. During this period, the first few packets for which routing information is not available are queued and are forwarded once the gateway successfully retrieves routing information from the DHT. Although there is some delay incurred by this route discovery process, it is in the order of milliseconds. Since the nodes remain in the same cluster for the entire simulation duration, the routing information retrieved is cached and reused for successive packet forwarding in the same direction. In the backbone, which is of more concern as it can become a bottleneck with heavy network loads, packets are forwarded over a unicast shortest-hop route. As discussed earlier, the DHT solution introduces new routing overheads, which are the DHT maintenance messages and the OLSR control messages, but these control messages have fixed intervals of 5 seconds and 2 seconds respectively. This control overhead increases as the number of gateway nodes increases but does not impose as high a load on the backbone as the flooding solution. PRTT in the DHT-based solution remained at about 2.5 ms for all cluster scenarios (and also across all the ping packet sizes). Again the performance of inter-cluster pings, using the DHT-based solution, is very comparable to the performance of local (intra-cluster) pings, which clearly is not the case for flooding.

The routing overhead counts the average number of maintenance messages the protocol incurs in carrying out topology maintenance and routing. As hierarchical network architectures are widely deployed to reduce routing overheads and increase scalability in ultimately large-scale MANETs, this metric gives an insight into the extent to which clustering reduces the overall routing overheads when compared to a flat MANET architecture. In Figure 7, the blue line shows the total number of HELLO messages, the green line the total number of TC messages, the orange line the total number of HNA messages and the red line shows the sum of all control messages. In the flat network (single cluster), there are no active gateways. All nodes exchange HELLO messages every 2 seconds and MPRs propagate TC messages every 5 seconds. In the flat network, there are many MPRs because nodes are not within reach of each other. This accounts for the high number of TC messages flooded in the MANET as seen in Figure 7.

As the number of clusters increases, the nodes are limited to smaller areas and more nodes are within reach of each other. The number of MPRs, therefore, begins to reduce and completely vanishes in the ten-cluster scenario. The number of TC message transmissions is about 130,000 messages in the one cluster scenario but drops to about 20,000 messages in the 2 cluster scenario. It continues to reduce as the number of clusters increased. HNA messages introduce additional overhead in the hierarchical design, to enable gateways to advertise reachability to external clusters to nodes within their local cluster. HNA messages are propagated every 5 seconds and forwarded only via the MPRs. This slightly increases the number of control messages. With fewer nodes per cluster as the number of clusters increases, the total number of HNA messages reduces, similar to the number of TC messages. The number of HELLO message transmissions remains constant across all cluster scenarios: every node sends these messages every 2 seconds and these messages are never forwarded. Since the number of nodes remains constant for all scenarios, the number of HELLO messages remains unchanged. Overall, the total OLSR control messages drop significantly going from a flat network (one cluster) to two clusters: from over 150,000 to about 40,000. They drop even further as we increase the number of clusters: from 38,000, 35,000, and 30,000 to 25,000 respectively.

Figure 8 shows the total traffic in the backbone for our two routing solutions. This total traffic includes the ping packets, the OLSR control messages, as well as all traffic attributed to the DHT overlay in case of the DHT routing solution. For the case of a single cluster, there is no backbone, hence both solutions show 0 packets transmitted in this case. The total traffic increases almost linearly when using flooding as the backbone routing solution, even though there is no DHT overlay to build and maintain, and no GET or PUT messages required to perform

the backbone routing task. The total backbone traffic for the DHT-based backbone shows much less variability with the number of clusters, and is almost constant as we vary the number of clusters. Again, this confirms that the DHT-based solution scales better as we increase the number of clusters.

These reductions in protocol overheads also improve the routing protocol performance within a cluster, with PSR rising from 70% in the case of a single cluster to almost 100% in networks with more clusters (Figure 6(a)). Similarly, PRTT drops from about 14 ms in the case of a flat network/single cluster to 1.5 ms as the number of clusters increases (Figure 6(b)). This clearly demonstrates that large-scale MANETs benefit from hierarchical architectures: overall routing overhead is reduced and routing performance increases.

We assume that clusters are formed by the application requirements, and nodes belonging to a cluster mostly move as a group. We therefore use a group mobility model called Reference Point Group Mobility (RPGM) model [33] to model this behaviour. In RPGM, each group has a “logical center” whose movement defines the behavior of the whole cluster/MANET, including location, speed, direction, acceleration, etc. The nodes are evenly distributed in the geographic scope of a group. Each node in the group follows the movement of the group center, with some degree of freedom to allow for topology changes within a group over time. We also model scenarios where a single node leaves a cluster and joins (temporarily) an alternative cluster. More specifically, we defined and explored the following three mobile scenarios:

• Intra-Cluster Mobility: All nodes are mobile within their clusters only. This mobility scenario is a variation of what happens in the static scenarios, with the nodes now moving within their cluster area.

• Nodes Switching Clusters: Some selected hosts switch their cluster membership. The remaining nodes stay within their cluster.

• Whole Clusters Merging and Splitting: Clusters as a whole will be mobile, with the group of nodes comprising the cluster moving according to the RPGM model (including the gateways). Since nodes within all clusters communicate locally over the same communication channel, when two cluster come into proximity with each other, they overlap physically and logically to form a single cluster.

The traffic and evaluation metrics for the mobility scenarios are the same ones we used in the static scenario. We fix the number of clusters at 4, allowing for enough distinct non-trivial clusters to study merging and splitting. The 6 non-participating gateways serve to support routing in the various cluster where they belong, resulting in clusters of 10 nodes each. BonnMotion [33] generated mobility traces based on RPGM which were used directly in OMNeT++. We varied the average mobility speed from 0 to 20 ms⁻¹ in steps of 5. The mobility traces for hosts switching cluster membership were handcrafted. We allowed 210 seconds for network stabilization as was the case in the static scenarios, and the statistics were collected over the next 3390 seconds.

As with the static scenarios, we are particularly interested in the performance of the network protocol deployed in the backbone network. For the intra-cluster routing case, we measure the impact of mobility on the metrics measured, in comparison with the static scenarios discussed above, and we use this as a base case for the other mobility scenarios. Graphs for each metric and all mobility scenarios are then plotted and the margin of error displayed on all graphs represents 95% confidence intervals.

Figure 9 shows the Ping Success Rates when nodes were mobile within their clusters, hosts switched cluster membership and clusters merged/split. The circles represent the intra-cluster ping successes and the squares in Figure 9(a) and Figure 9(c) represent the inter-cluster ping successes. In Figure 9(b), the squares represent PSR when the mobile host was the ping destination and the triangles represent PSR when the mobile host was the ping source. In Figure 9(b) the number of clusters which each host switched to over the entire network lifetime ranges from 0 to 4 in steps of 1. Five (5) nodes switched their cluster membership but at different times from each other with no particular order or interval. However, each node spent about 600 seconds in the first four clusters and 990 seconds in the fifth cluster.

For both approaches, the presence of mobility brought about frequent link breakages which resulted in packet losses and as a result, declining PSR. In the DHT-based approach, PSR across clusters remains almost as high as PSR within a local cluster, with some deterioration when clusters merge and split more often under higher mobility rates (Figure 9(c)). When hosts switch between multiple

clusters, routing information becomes stale more frequently, contributing to the declining ping success rates. Figure 9(b) also shows that PSR is, in general, independent of whether the fixed node or the mobile node initiates the ping exchange. This is somewhat expected, as we need, in both cases, a forward and a reverse route to complete the ping exchange.

With flooding, in addition to the link breaks, which become more frequent as mobility increases, the performance deteriorates for the intra-clusters mobility model and when clusters merged/split because of the high traffic load and frequent collisions in the backbone as discussed earlier. We therefore conclude that our design supports routing efficiently under different forms of mobility and outperforms the flooding approach. Mobility has a negative impact on PSR, and this is true for any routing protocol: When nodes move, they have to be rediscovered in order to be reached. Thus, messages exchanged during this rediscovery phase will be lost (either dropped, as no route is yet available, or sent to the old cluster due to stale routing information).

Figure 10 shows the Ping Round Trip Time for all mobile scenarios. Pings delivered within the clusters experienced low PRTT (green line) because the nodes are within reach of each other most of the time. For flooding, PRTT is again higher than for the DHT-based approach, with the same explanation we gave earlier. In the DHT-based solution, packets are only queued for the initial route discovery phase of our protocol and packets are delivered with a unicast protocol in single hops through the backbone for most cases. When hosts are mobile within their clusters and when clusters merged/split as seen in Figure 10(a) and Figure 10(c), the round trip times are similar for all mobility speeds and the confidence intervals show that the results overlap. The gap is a bit wider for the case of single nodes switching cluster, but overall, PRTT with the DHT-based approach is comparable to the intra-cluster PRTT and is consistently and significantly lower than that of the flooding approach.