A Routing Architecture for Satellite Networks

Introduction Satellite networks present some interesting challenges for packet networking. The entire topology is continually in motion, with links that are far less reliable than what is common in terrestrial networks. Some changes to link connectivity can be anticipated due to orbital mechanics. This document proposes a scalable routing architecture for satellite networks based on existing routing protocols and mechanisms, enhanced with scheduled link connectivity change information. This document proposes no protocol changes.

Related Work A survey of related work can be found in . Link state routing for satellite networks has been considered in

Terms and Abbreviations Downlink: The half of a ground link leading from a satellite to a ground station. Gateway: A ground station that participates as part of the network and acts as the interconnect between satellite constellations and the planetary network. Gateways have a much higher bandwidth than user stations, have ample computing capabilities, and perform traffic engineering duties. Multiple gateways are assumed to exist, with each serving a portion of the network. GEO: Geostationary Earth Orbit. A satellite in GEO has an orbit that is synchronized to planetary rotation, so it effectively sits over one spot on the planet. Ground link: A link between a satellite and a ground station. IGP: Interior Gateway Protocol. A routing protocol that is used within a single administrative domain. Note that 'gateway' in this context is semantically equivalent to 'router' and has no relationship to the 'gateway' used in the rest of this document. IS-IS: Intermediate System to Intermediate System routing protocol. An IGP that is commonly used by service providers. ISL: Inter-satellite link. Frequently a free space laser. L1: IS-IS Level 1 L1L2: IS-IS Level 1 and Level 2 L2: IS-IS Level 2 LEO: Low Earth Orbit. Local gateway: Each user station is associated with a single gateway in its region. LSP: IS-IS Link State Protocol Data Unit. An LSP is a set of packets that describe a node's connectivity to other nodes. MEO: Medium Earth Orbit. SID: Segment Identifier Stripe: A set of satellites in a few adjacent orbits. These form an IS-IS L1 area. SR: Segment Routing Uplink: The half of a ground link leading from a ground station to a satellite. User station: A ground station interconnected with a small end user network.

Topological Considerations Satellites travel in specific orbits around their parent planet. Some of them have their orbital periods synchronized to the rotation of the planet, so they are effectively stationary over a single point. Other satellites have orbits that cause them to travel across regions of the planet gradually or quite rapidly. Respectively, these are typically known as Geostationary Earth Orbits (GEO), Medium Earth Orbit (MEO), or Low Earth Orbit (LEO), depending on altitude. This discussion is not Earth-specific; as we get to other planets, we can test this approach's generality. Satellites may have data interconnections with one another through Inter-Satellite Links (ISLs). Due to differences in orbits, ISLs may be connected temporarily, with periods of potential connectivity computed through orbital mechanics. Multiple satellites may be in the same orbit but separated in time and space, with a roughly constant separation. Satellites in the same orbit may have ISLs that have a higher duty cycle than cross-orbit ISLs but are still not guaranteed to always be connected.

Ground stations can communicate with one or more satellites that are in their region. Some ground stations have a limited capacity and communicate with only a single satellite at a time. These are known as user stations. Other ground stations that may have richer connectivity and higher bandwidth are commonly called gateways and provide connectivity between the satellite network and conventional wired networks. Gateways serve ground stations that are in their geographic region and are replicated as necessary to meet the needs of the service. Traffic from one ground station to another may need to traverse a path across multiple satellites via ISLs.

Link Changes Like conventional network links, ISLs and ground links can fail at any time. However, unlike conventional links, there are predictable times when ISLs and ground links can potentially connect and disconnect. These predictions can be computed and cataloged in a schedule that can be distributed to relevant network elements. Predictions of a link connecting are not a guarantee: a link may not connect for a variety of reasons. Predictions of a link disconnecting due to orbital mechanics are effectively guaranteed, as the underlying physics is extremely unlikely to improve unexpectedly.

Scalability Some proposed satellite networks are fairly large, with tens of thousands of proposed satellites. A key concern is the ability to reach this scale and larger. As we know, the key to scalability is the ability to create hierarchical abstractions, so a key question of any routing architecture will be about the abstractions that can be created to contain topological information. Normal routing protocols are architected to operate with a static but somewhat unreliable topology. Satellite networks lack the static organization of terrestrial networks, so normal architectural practices may not apply and alternative approaches may need consideration.

Assumptions In this section, we discuss some of the assumptions that are the basis for this architectural proposal. The data payload is IP packets. There may be a terrestrial network behind each gateway that may interconnect to the broader Internet. The architecture of the terrestrial network is assumed to be a typical IS-IS and BGP deployment and is not discussed further.

Traffic Patterns We assume that the primary use of the satellite network is to provide access from a wide range of geographic locations. We assume that providing high bandwidth bulk transit between peer networks is not a goal. It has been noted that satellite networks can provide lower latencies than terrestrial fiber networks . This proposal does not preclude such applications but also does not articulate the mechanisms necessary for user stations to perform the necessary traffic engineering computations. Low-latency applications are not discussed further. As with most access networks, we assume that there will be bidirectional traffic between the user station and the gateway, but that the bulk of the traffic will be from the gateway to the user station. We expect that the uplink from the gateway to the satellite network to be the bandwidth bottleneck, and that gateways will need to be replicated to scale the uplink bandwidth. We assume that it is not essential to provide optimal routing for traffic from user station to user station. If this traffic is sent first to a gateway and then back into the satellite network, this would be acceptable as the traffic volume is very low. This type of route is commonly called a 'hairpin' and is not discussed further. We assume that traffic for a user station should enter the network through a gateway that is in some close topological proximity to the user station. This is to maximize the practical capacity of the satellite network. Similarly, we assume that user station traffic should exit the network through the gateway that is in the closest topological proximity to the user station. This architecture does not preclude gateway-to-gateway traffic across the satellite constellations, but it does not seek to optimize it. We assume that a user station registers with one satellite at a time, forming a temporary association that is relayed to the local gateway. The mechanism for this registration is outside of the scope of this document.

Stochastic Connectivity We assume that links in general will be available when scheduled. As with any network, there will be failures, and the schedule is not a guarantee, but we also expect that the schedule is mostly accurate. We assume that at any given instant, there are enough working links and aggregate bandwidth to run the network and support the traffic demand. If this assumption does not hold, then no routing architecture can magically make the network more capable. Satellites that are in the same orbit may be connected by ISLs. These are called intra-orbit ISLs. Satellites that are in different orbits may also be connected by ISLs. These are called inter-orbit ISLs. We assume that, in general, intra-orbit ISLs have higher reliability and persistence than inter-orbit ISLs. We assume that the network is connected (in the graph theory sense) at all times, even if some links are down. This implies that there is always some path to the destination. In the extreme case where there is no such path, we assume that it is acceptable to drop the payload packets. We do not require buffering of traffic when a link is down. Instead, traffic should be rerouted.

Problem Statement The goal of the routing architecture is to provide an organizational structure to protocols running on the satellite network such that topology information is conveyed through relevant portions of the network, that paths are computed across the network, and that data can be delivered along those paths, and the structure can scale to a very large network without undue changes to the organizational structure.

Forwarding Plane The end goal of a network is to deliver traffic. In a satellite network where the topology is in a continual state of flux and the user stations are frequently changing their association with the satellites, having a highly flexible and adaptive forwarding plane is essential. Toward this end, we propose to use MPLS as the fundamental forwarding plane architecture . Specifically, we propose to use a Segment Routing (SR) based approach with an MPLS data plane , where each satellite is assigned a node Segment Identifier (SID). A path through the network can be then expressed as a label stack of node SIDs. IP forwarding is not used within the internals of the satellite network, although each satellite may be assigned an IP address for management purposes. Existing SR label stack compression algorithms may be used, so that the label stack need only contain the significant waypoints along the path. This implies that the label stack operates as a form of loose source routing through the network. If there is an unexpected topology change in the network, then the IGP will compute a new path to the next waypoint, allowing packet delivery despite ISL failures. We assume that there is a link-layer mechanism for a user station to associate with a satellite. User stations will have an IP address that is assigned from a prefix managed by its local gateway. The mechanisms for this assignment and its communication to the end station are not discussed herein but might be similar to DHCP . User station IP addresses change infrequently and do not reflect their association with their first-hop satellite. Gateways and their supporting terrestrial networks advertise a prefix into the global Internet for all of its local user stations. User stations may be assigned a node SID, in which case MPLS forwarding can be used for all hops to the user station. Alternatively, if the user station does not have a node SID, then the last hop from the satellite to the end station can be performed based on the destination IP address of the packet. This does not require a full longest prefix match lookup as the IP address is merely a unique identifier at this point. Similarly, gateways may be assigned a node SID. A possible optimization is that a single SID value be assigned as a global constant to always direct traffic to the topologically closest gateway. If traffic engineering is required for traffic that is flowing to a gateway, a specific path may be encoded in a label stack that is attached to the packet by the user station or by the first-hop satellite. Gateways can also perform traffic engineering by using different paths and label stacks for different traffic flows. Routing a single traffic flow across multiple paths has proven to cause performance issues with transport protocols, so that approach is not recommended.

IGP Selection The IETF currently actively supports two Interior Gateway Protocols (IGPs): OSPF and IS-IS . OSPF requires that the network operate around a backbone area, with subsidiary areas hanging off of the backbone. While this works well for traditional terrestrial networks, this does not seem appropriate for satellite networks, where there is no centralized portion of the topology. IS-IS has a different hierarchical structure, where Level 1 (L1) areas are connected sets of nodes, and then Level 2 (L2) is a connected subset of the topology that intersects all of the L1 areas. Individual nodes can be L1, L2, or both (L1L2). In particular, we propose that all nodes in the network be L1L2 so that local routing is done based on L1 information and then global routing is done based on L2 information. IS-IS also has the interesting property that it does not require interface addresses. This feature is commonly known as 'unnumbered interfaces'. This is particularly helpful in satellite topologies because it implies that ISLs may be used flexibly. Sometimes an interface might be used as an L1 link to another satellite and a few orbits later it might be used as an L1L2 link to a completely different satellite without any reconfiguration or renumbering. Scalability for IS-IS can be achieved through the use of a proposal known as Area Proxy . With this proposal, all of the nodes in an L1 area combine their information into a single L2 Link State Protocol Data Unit (LSP). This implies that the size of the L1 Link State Database (LSDB) scales as the number of nodes in the L1 area and the size of the L2 LSDB scales with the number of L1 areas. The Area Proxy proposal also includes the concept of an Area SID. This is useful because it allows traffic engineering to construct a path that traverses areas with a minimal number of label stack entries. Suppose, for example, that a network has 1,000 L1 areas, each with 1,000 satellites. This would then mean that the network supports 1,000,000 satellites, but only requires 1,000 entries in its L1 LSDB and 1,000 entries in its L2 LSDB; numbers that are easily achievable today. The resulting MPLS label table would contain 1,000 node SIDs from the L1 (and L2) LSDB and 1,000 area SIDs from the L2 LSDB. If each satellite advertises an IP address for management purposes, then the IP routing table would have 1,000 entries for the L1 management addresses and 1,000 area proxy addresses from L2. In this proposal, IS-IS does not carry any IP routes other than those that are in the satellite topology. In particular, there are no IP routes for user stations or the remainder of the Internet.

Stripes and Areas A significant problem with any link state routing protocol is that of area partition. While there have been many proposals for automatic partition repair, none has seen significant production deployment. It seems best to simply avoid this issue altogether and ensure that areas have an extremely low probability of partitioning. As discussed above, intra-orbit ISLs are assumed to have higher reliability and persistence than inter-orbit ISLs. However, even intra-orbit ISLs are not sufficiently reliable to avoid partition issues. Therefore, we propose to group a small number of adjacent orbits as an IS-IS L1 area, called a stripe. We assume that for any given reliability requirement, there is a small number of orbits that can be used to form a stripe that satisfies the reliability requirement. Stripes are connected to other adjacent stripes using the same ISL mechanism, forming the L2 topology of the network. Each stripe should have multiple L2 connections and should never become partitioned from the remainder of the network. By using a stripe as an L1 area, in conjunction with Area Proxy, the overhead of the architecture is greatly reduced. Each stripe contributes a single LSP to the L2 LSDB, completely hiding all of the details about the satellites within the stripe. The resulting architecture scales proportionately to the number of stripes required, not the number of satellites. MEO and GEO constellations that have intra-constellation ISLs can also form an IS-IS L1L2 area. Satellites that lack intra-constellation ISLs are better as independent L2 nodes.

Traffic Engineering Forwarding in this architecture is straightforward. A path from a gateway to a user station on the same stripe only requires a single node SID for the satellite that provides the downlink to the user station.

X \ \ X \ \ X ---> x User Station \ Figure 2: On-stripe forwarding ]]> Similarly, a user station returning a packet to a gateway need only provide a gateway node SID. For off-stripe forwarding, the situation is a bit more complex. A gateway would need to provide the area SID of the destination stripe/area plus the node SID of the downlink satellite. For return traffic, user stations or first-hop satellites would want to provide the area SID for the gateway as well as the gateway SID.

X \ \ \ X --- X \ \ \ \ X --- X \ \ \ U ---> D User Station \ Figure 3: Off-stripe forwarding ]]> As an example, consider a packet from an Internet destination S to a user station D. A local gateway L has injected a prefix covering D into BGP and advertised it globally. The packet is forwarded to L using legacy IP forwarding. When L receives the packet, it performs a lookup in a pre-computed forwarding table. This contains a SID list for the user station that has already been converted into a label stack. Suppose that the user station is currently associated with a different stripe so that the label stack will contain an area label A and a label for the satellite associated with the user station U, resulting in a label stack (A, U). The local gateway forwards this into the satellite network. The first hop satellite now forwards based on the area label A at the top of the stack. All area labels are propagated as part of the L2 topology. This forwarding continues until the packet reaches a satellite that is adjacent to the destination area. That satellite performs a penultimate hop pop on label A, removing that label and forwarding the packet into the destination area. The packet is now forwarded based on the remaining label U, which was propagated as part of the L1 topology. The last satellite forwards the packet based on the destination address D and forwards the packet to the user station. The return case is similar. The label stack, in this case, consists of a label for the local gateway's area, A', and the label for the local gateway, L, resulting in the stack (A', L). The forwarding mechanisms are similar to the previous case. Very frequently, access networks congest due to oversubscription and the economics of access. Network operators can use traffic engineering to ensure that they are getting higher efficiency out of their networks by utilizing all available paths and capacity near any congestion points. In this particular case, the gateway will have information about all of the traffic that it is generating and can use all of the possible paths through the network in its topological neighborhood. Since we're already using SR, this is easily done just by adding more explicit SIDs to the label stack. These can be additional area SIDs, node SIDs, or adjacency SIDs. Path computation can be performed by traditional Path Computation Elements (PCE). Each gateway or its PCE will need topological information from all of the areas that it will route through. It can do this by being a participant in the IGP, either directly, via a tunnel, or another delivery mechanism such as BGP-LS . User stations do not participate in the IGP. Traffic engineering for traffic into a gateway can also be provided by an explicit SR path on the traffic. This can help ensure that ISLs near the gateway do not congest with traffic for the gateway. These paths can be computed by the gateway or PCE and then distributed to the first-hop satellite or user station, which would then apply them to traffic. The delivery mechanism is outside of the scope of this document.

Scheduling The most significant difference between terrestrial and satellite networks from a routing perspective is that some of the topological changes that will happen to the network can be anticipated and computed. Both link and node changes will affect the topology and the network should react smoothly and predictably. The management plane is responsible for providing information about scheduled topological changes. The exact details of how the information is disseminated are outside of the scope of this document but could be done through a YANG model . Scheduling information needs to be accessible to all of the nodes that will make routing decisions based on the topological changes in the schedule, so information about an L1 topological change will need to be circulated to all nodes in the L1 area and information about L2 changes will need to propagate to all L2 nodes, plus the gateways and PCEs that carry the related topological information. There is very little reaction that the network should do in response to a topological addition. A link coming up or a node joining the topology should not have any functional change until the change is proven to be fully operational based on the usual liveness mechanisms found within IS-IS. Nodes may pre-compute their routing table changes but should not install them before all of the relevant adjacencies are flooded. The benefits of this pre-computation appear to be very small. Gateways and PCEs may also choose to pre-compute paths based on these changes, but should be careful to not install paths using the new parts of the topology until they are confirmed to be operational. If some path pre-installation is performed, gateways and PCEs must be prepared for the situation where the topology does not become operational and may need to take alternate steps instead, such as reverting any related pre-installed paths. The network may choose to not do any pre-installation or pre-computation in reaction to topological additions, at a small cost of some operational efficiency. Topological deletions are an entirely different matter. If a link or node is to be removed from the topology, then the network should act before the anticipated change to route traffic around the expected topological loss. Specifically, at some point before the topology change, the affected links should be set to a high metric to direct traffic to alternate paths. This is a common operational procedure in existing networks when links are taken out of service, such as when proactive maintenance needs to be performed. This type of change does require some time to propagate through the network, so the metric change should be initiated far enough in advance that the network converges before the actual topological change. Gateways and PCEs should also update paths around the topology change and install these changes before the topology change takes place. The time necessary for both IGP and path changes will vary depending on the exact network and configuration.

Future Work This architecture needs to be validated by satellite operators, both via simulation and operational deployment. Meaningful simulation hinges on the exact statistics of ISL connectivity, and that information is not publicly available at present.

Deployment Considerations The network behind a gateway is expected to be a normal terrestrial network. Conventional routing architectural principles apply. An obvious approach would be to extend IS-IS to the terrestrial network, applying L1 areas as necessary for scalability. The terrestrial network may have one or more BGP connections to the broader Internet. Prefixes for user stations should be advertised into the Internet near the associated gateway. If gateways are not interconnected by the terrestrial network, then it may be advisable to use one autonomous system per gateway. Otherwise, one autonomous system may be used for the entire terrestrial network.

Security Considerations This document discusses one possible routing architecture for satellite networks. It proposes no new protocols or mechanisms and thus has no new security impact. Security for IS-IS is provided by and .

Acknowledgements We would like to thank Dino Farinacci, Olivier De jonckere, Eliot Lear, and Dean Bogdanovic for their comments.

IANA Considerations This document makes no requests for IANA.