Test Protocol for One-way IP Capacity Measurement

The IETF's efforts to define Network and Bulk Transport Capacity have been chartered and finally progressed after over twenty years. Over that time, the performance community has seen development of Informative definitions in for Framework for Bulk Transport Capacity (BTC), RFC 5136 for Network Capacity and Maximum IP-layer Capacity, and the Experimental metric definitions and methods in , Model-Based Metrics for BTC. This memo looks at the problem of measuring Network Capacity metrics defined in where the method deploys a feedback channel from the receiver to control the sender's transmission rate in near-real-time. Although there are several test protocol already available for support and manage active measurements, this protocol is a major departure from their operation: UDP transport is used for all setup, test activation, and control messages, and for results feedback (not TCP), simplifying operations. TWAMP and STAMP use the philosophy that one host is a Session-Reflector, sending test packets every time they receive a test packet. This protocol supports a one-way test with periodic status messages returned to the sender. These messages are also a basis for on-path Round-trip delay measurements, which are a key input to the load adjustment search algorithm. OWAMP supports one-way testing with results Fetch at the end of the test session. This protocol supports a one-way test and requires periodic status messages returned to the sender to support the load adjustment search algorithm. The security features of OWAMP and TWAMP have been described as "unusual", to the point that IESG approved their use while also asking that these methods not be used again. Further, the common OWAMP and TWAMP approach to security is over 15 years old at this time. Note: the -00 update of this draft will be the last that describes version 8 of the protocol in the running code. Updates -01 and -02 of the draft correspond to version 9 of the protocol, which strives to allow interoperability with version 8.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

The scope of this memo is to define a protocol to measure the Maximum IP-Layer Capacity metric and according to the standardized method. The continued goal is to harmonize the specified metric and method across the industry, and this protocol supports the specifications of IETF and other Standards Development Organizations. All active testing protocols currently defined by the IPPM WG are UDP-based, but this protocol specifies both control and test protocols using UDP transport. Also, the control protocol continues operating during testing to convey results and dynamic configurations. The primary application of the protocol described here is the same as in Section 2 of where: The access portion of the network is the focus of this problem statement. The user typically subscribes to a service with bidirectional access partly described by rates in bits per second.

This section gives an informative overview of the communication protocol between two test end-points (without expressing requirements: later sections provide details and requirements). One end-point takes the role of server, listening for connection requests on a well-known destination port from the other end-point, the client. The client requires configuration of a test direction parameter (upstream or downstream test, where the client performs the role of sender or receiver, respectively) as well as the hostname or IP address of the server in order to begin the setup and configuration exchanges with the server. The protocol uses UDP transport and has four phases: Setup Request and Response Exchange: The client requests to begin a test by communicating its protocol version, intended security mode, and jumbo datagram support. The server either confirms matching configuration or rejects the connection. The server also communicates the ephemeral port for further communication when accepting the client's request. Test Activation Request and Response: the client composes a request conveying parameters such as the testing direction, the duration of the test interval and test sub-intervals, and various thresholds. The server then chooses to accept, ignore or modify any of the test parameters, and communicates the set that will be used unless the client rejects the modifications. Note that the client assumes that the Test Activation exchange has opened any co-located firewalls and network address/port translators for the test connection (in response to the Request packet on the ephemeral port) and the traffic that follows. If the Test Activation Request is rejected or fails, the client assumes that the firewall will close the address/port combination after the firewall's configured idle traffic time-out. Test Stream Transmission and Measurement Feedback Messages: Testing proceeds with one end-point sending load PDUs and the other end-point receiving the load PDUs and sending frequent status messages to communicate status and transmission conditions there. The feedback messsages are input to a load-control algorithm at the server, which controls future sending rates at either end-point as needed. The choice to locate the load-control algorithm at the server, regardlesss of transmiision direction, means that the algorithm can be updated more easily at a host within the network, and at a fewer number of hosts than the number of clients. Stopping the Test: When the specified test duration has been reached, the server initiates the phase to stop the test by setting the STOP1 indication in load PDUs or status feedback messages. The client acknowledges by setting the STOP2 in further load PDUs or messages, and a graceful connection termination at each end-point follows. (Since the load PDUs and feedback messages are used, this phase is kind of a sub-phase of 3.) If the Test traffic stops or the communication path fails, the client assumes that the firewall will close the address/port combination after the firewall's configured idle traffic time-out.

For Parameters related to the Maximum IP-Layer Capacity Metric and Method, please see Section 4 of .

All messages defined in this section SHALL use UDP transport. The hosts SHALL calculate and include the UDP checksum, or check the UDP checksum as neccessary.

The client SHALL begin the Control protocol connection by sending a Setup Request message to the server's control port. The client SHALL simultaneously start a test initiation timer so that if the control protocol fails to complete all exchanges in the allocated time, the client software SHALL exit (close the UDP socket and indicate an error message to the user). (Note: in version 8, the watchdog time-out is configured, in udpst.h, as #define WARNING_NOTRAFFIC 1 // Receive traffic stopped warning threshold (sec) #define TIMEOUT_NOTRAFFIC (WARNING_NOTRAFFIC + 4) or 5 seconds) The Setup Request message PDU SHALL be organized as follows:

The UDP PDU format layout SHALL be as follows (big-endian AB):

When the client generates the authDigest, the calculation SHALL cover the entire header (9 fields). The current Unix time SHALL be read and inserted immediately prior to the calculation (as immediately as possible, as are all preeding fields.

When the server receives the Setup Request it SHALL validate the request by checking the protocol version, the maxBandwidth requested for the test, the modifierBitmap for use of options such as Jumbo datagram status and traditional MTU (1500 bytes), and the authentication data if utilized. The value in the authUnixTime field is a 32-bit time stamp and a 5 minute tolerance window (+/- 2.5 minutes) is used to prevent the replay of a Setup Request. All other fields would remain valid if the authUnixTime field was omitted from the PDU. The authUnixTime is covered by the authDigest hash. If the client has selected options for: Jumbo datagram support status (modifierBitmap), Traditional MTU (modifierBitmap), Authentication mode (optional) that do not match the server configuration, the server MUST reject the Setup Request. Note that a server implemenation of protocol version 9 allows backward compatibility with version 8 when in use by the client. (Note: in version 8, the watchdog time is configured, in udpst.h, as #define WARNING_NOTRAFFIC 1 // Receive traffic stopped warning threshold (sec) #define TIMEOUT_NOTRAFFIC (WARNING_NOTRAFFIC + 4) or 5 seconds) If the Setup Request must be rejected (due to any of the reasons in the Command response codes listed below), a Setup Response SHALL be sent back to the client with a corresponding command response value indicating the reason for the rejection, unless the server requires Authentication, in which case the Setup Request SHOULD fail silently. The exception is for operations support: server administrators using Authentication are permitted to send a Setup Response to support operations and troubleshooting.

uint16_t maxBandwidth;// Required bandwidth (added in v9) uint16_t testPort; // Test port on server (available port in Response) uint8_t modifierBitmap;// Modifier bitmap (replaced jumboStatus, table below) uint8_t authMode; // Authentication mode uint32_t authUnixTime;// Authentication time stamp unsigned char authDigest[AUTH_DIGEST_LENGTH] // 32 octets, MBZ cmdResponse Code Field: Command Server Response Codes (CSRP) CHSR_CRSP_NONE 0 = None CHSR_CRSP_ACKOK 1 = Acknowledgement CHSR_CRSP_BADVER 2 = Bad Protocol Version CHSR_CRSP_BADJS 3 = Invalid Jumbo datagram option CHSR_CRSP_AUTHNC 4 = Unexpected Authentication in Setup Request CHSR_CRSP_AUTHREQ 5 = Authentication missing in Setup Request CHSR_CRSP_AUTHINV 6 = Invalid authentication method CHSR_CRSP_AUTHFAIL 7 = Authentication failure CHSR_CRSP_AUTHTIME 8 = Authentication time is invalid in Setup Request CHSR_CRSP_NOMAXBW 9 = No Maximum test Bit rate specified CHSR_CRSP_CAPEXC 10 = Server Maximum Bit rate exceeded CHSR_CRSP_BADTMTU 11 = MTU option does not match Server maxBandwidth Field MSB Code Bit: CHSR_USDIR_BIT 0x8000 Bandwidth upstream direction bit, Set for Upstream modifierBitmap Code Field: Setup CHSR_JUMBO_STATUS 0x01 = set to use Jumbo datagram sizes above 1Gbps CHSR_TRADITIONAL_MTU 0x02 = set to use datagrams for 1500 byte packets ]]> There is a set of Command Response codes, beginning with: "2 = Bad Protocol Version", one of which SHOULD be communicated to indicate the cause when an error condition detected and testing cannot proceed:

The exceptional circumstances when a server would not communicate the appropriate Command Response Code for an error condition are when the Setup Request PDU size is not correct (for supported versions of the protocol), the control ID is invalid, or a directed attack has been detected, in which case the server will allow setup attempts to terminate silently. Attack detection is beyond the scope of this specification. When indicating a Bad Protocol Version error, the server SHALL update the protocolVer field in the Setup Response to indicate the current version supported. If the server finds that the Setup Request matches its configuration and is otherwise acceptable, the server SHALL initiate a new connection for the client, using a new UDP socket allocated from the UDP ephemeral port range. Then, the server SHALL start a watchdog timer (to terminate the connection in case the client goes silent), and sends the Setup Response back to the client (see below for composition). When the Setup Request is accepted by the server, a Setup Response SHALL be sent back to the client with a corresponding command response value indicating 1 = Acknowledgement.

(Note: in version 8, the watchdog time-out is configured at 5 seconds) The Setup Response SHALL include the port number at the server for the new socket, and this UDP port-pair SHALL be used for all subsequent communication. The server SHALL confirm or populate the values of: Protocol version Jumbo datagram support status (modifierBitmap), Traditional MTU (modifierBitmap), Authentication mode (authDigest will be all zeroes if used) Required Bandwidth Test Port (ephemeral port) Modifier Bitmap authUnixTime retuned as-received for the client's use on the new connection in its Setup Response, and the authentication digest MUST Be Zero (MBZ). Finally, the new UDP connection associated with the new socket and port number is opened, and the server awaits communication there. If a Test Activation Request is not subsequently received from the client on this new port number before the watchdog timer expires, the server SHALL close the socket and deallocate the port.

When the client receives the Setup Response from the server, the client SHALL check: the message PDU for correct length and formatting (fields have values in-range, beginning with the fields listed below) the controlID, to validate the type of message (Setup) the cmdResponse value IF the cmdResponse value indicates an error the client SHALL display/report a relevant message to the user or management process and exit. If the client receives a Command Server Response code (CRSP) that is not equal to one of the codes defined above, then the client MUST terminate the connection and terminate operation of the current Setup Request. If the Command Server Response code (CRSP) value indicates success the client SHALL compose a Test Activation Request with all the test parameters it desires, such as the test direction, the test duration, etc.

This section is divided according to the sending and processing of the client, server, and again at the client. All messages defined in this section SHALL use UDP transport. The hosts SHALL calculate and include the UDP checksum, or check the UDP checksum as neccessary.

Upon a successful setup, the client SHALL then send the Test Activation Request to the UDP port number the server communicated in the Setup Response. The client SHALL compose Test Activation Request as follows:

Note: uint16_t srIndexConf is the table index of the configured fixed or starting send rate (depending on whether CHTA_SRIDX_ISSTART is cleared or set respectively). The server MAY allow the client to specify any fixed or starting send rate. Otherwise, the server MAY enforce a maximum of the fixed or starting send rate which the client can successfully request. If the client's Test Activation Request exceeds the server's configured maximum, the server MUST either reject the request, or coerce the value to the configured maximum, and communicate that maximum to the client in the Test Activation Response. The client can of course choose to end the test, as appropriate. The UDP PDU format of the Test Activation Request is as follows (big-endian AB):

The client SHALL use the configuration for cmdRequest (upstream or downstream) cmdResponse is cleared (all zeroes) and the remaining fields are populated based on default values or command-line options requested in the Setup Request and confirmed by the server in the Setup Response. @@@@@ We could add the authDigest to the Test Activation request/response. THEN, we would explain that + Use of optional Authenticated mode requires checking the validity of authDigest in this phase + The time stamp in the PDU MUST be within 5 minutes (+/- 2.5 minutes) of the current time at the recipient. @@@@@

After the server receives the Test Activation Request on the new connection, it MUST choose to accept, ignore or modify any of the test parameters. When the server sends the Test Activation Response, it SHALL set the cmd Response field to: uint8_t cmdResponse;// Command response (set to 1, ACK, or 2 error) The server SHALL repeat all test parameters to indicate changes to the client. If the client has requested an upstream test, the server SHALL include the transmission parameters from the first row of the sending rate table in the Sending Rate Structure (defined below), OR use the parameters from the configured send rate index (srIndexConf) of the sending rate table, or starting rate index (indicated in the Test Activation modifierBitmap) when these options are present. The remaining 28 octets of the Test Activation Response (normally read from the first row of the sending rate table) are called the Sending Rate Structure, and SHALL be organized as follows:

with

Note that the server additionally has the option of completely rejecting the request and sending back an appropriate command response value: uint8_t cmdResponse; // Command response (set to 2, error) If activation continues, the new connection is prepared for an upstream OR downstream test. In the case of a downstream test, the server SHALL prepare to send with either a single timer to send status PDUs at the specified interval OR dual timers to send load PDUs based on the transmission parameters from the first row of the sending rate table in the Sending Rate Structure, OR the transmission parameters of the configured send rate index (srIndexConf) of the sending rate table, or starting rate index (indicated in the Test Activation modifierBitmap) when these options are present. The server SHALL then send a Test Activation Response back to the client, update the watchdog timer with a new time-out value, and set a test duration timer to eventually stop the test. The new connection is now ready for testing.

When the client receives the Test Activation Response, it first checks the command response value. If the client receives a Test Activation Command Response value that indicates an error, the client SHALL display/report a relevant message to the user or management process and exit. If the client receives a Test Activation Command Response value that is not equal to one of the codes defined above, then the client MUST terminate the connection and terminate operation of the current Setup Request. If the client receives a Test Activation Command Response value that indicates success (CHTA_CRSP_ACKOK) the client SHALL update its configuration to use any test parameters modified by the server. Next, the client SHALL prepare its connection for either an upstream test with dual timers set to send load PDUs (based on the starting transmission parameters sent by the server), OR a downstream test with a single timer to send status PDUs at the specified interval. Then, the client SHALL stop the test initiation timer, set a new time-out value for the watchdog timer, and start the timer (in case the server goes quiet). The connection is now ready for testing.

This section describes the testing phase of the protocol. The roles of sender and receiver vary depending whether the direction of testing is from server to client, or the reverse. All messages defined in this section SHALL use UDP transport. The hosts SHALL calculate and include the UDP checksum, or check the received UDP checksum before further processing, as neccessary.

Testing proceeds with one end point sending load PDUs, based on transmission parameters from the sending rate table, and the other end point receiving the load PDUs and sending status messages to communicate the traffic conditions at the receiver. The watchdog timer at the receiver SHALL be reset each time a test PDU is received. See non-graceful test stop in Section 8 for handling the watchdog/NOTRAFFIC time-out expiration at each end-point. When the server is sending Load PDUs in the role of sender, it SHALL use the transmission parameters directly from the sending rate table via the index that is currently selected (which was based on the feedback in its received status messages). However, when the client is sending load PDUs in the role of sender, it SHALL use the discreet transmission parameters that were communicated by the server in its periodic status messages (and not referencing a sending rate table). This approach allows the server to control the individual sending rates as well as the algorithm used to decide when and how to adjust the rate. The server uses a load adjustment algorithm which evaluates measurements, either it's own or the contents of received feedback messages. This algorithm is unique to udpst; it provides the ability to search for the Maximum IP Capacity that is absent from other testing tools. Although the algorithm depends on the protocol, it is not part of the protocol per se. The current algorithm (B) has three paths to its decision on the next sending rate: When there are no impairments present (no sequence errors, low delay variation), resulting in sending rate increase. When there are low impairments present (no sequence errors but higher levels of delay variation), so the same sending rate is retained. When the impairment levels are above the thresholds set for this purpose and "congestion" is inferred, resulting in sending rate decrease. The algorithm also has two modes for increasing/decreasing the sending rate: A high-speed mode to achieve high sending rates quickly, but also back-off quickly when "congestion" is inferred from the measurements. Any two consecutive feedback intervals that have a sequence number anomaly and/or contain an upper delay variation threshold exception in both of the two consecutive intervals, count as the two consecutive feedback measurements required to declare "congestion" within a test. A single-step mode where all rate adjustments use the minimum increase or decrease of one step in the sending rate table. The single step mode continues after the first inference of "congestion" from measured impairments. On the other hand, the test configuration MAY use a fixed sending rate requested by the client, using the field below: uint16_t srIndexConf; // Configured sending rate index The client MAY communicate the desired fixed rate in its activation request. The reasons to conduct a fixed-rate test include stable measurement at the maximum determined by the load adjustment (search) algorithm, or the desire to test at a known subscribed rate without searching. The Load PDU SHALL have the following format and field definitions:

The Test Load UDP PDU format is as follows (big-endian AB):

The receiver SHALL send a Status PDU to the sender during a test at the configured feedback interval. The watchdog timer at the test PDU sender SHALL be reset each time a Status PDU is received. See non-graceful test stop in Section 8 for handling the watchdog/NOTRAFFIC time-out expiration at each end-point. @@@@ To Do: What protections from bit errors (checksum) or on-path attacks (something stronger) are warrented for the Status PDUs? These PDUs are a key part of the server-client control loop. Added a requirement to calculate and include/check the UDP checksum. The Status Header PDU SHALL have the following format and field definitions:

The Status feedback UDP payload PDUs format is as follows (big-endian AB):

Note that the Sending Rate Structure (28 octets) is defined in the Test Activation section. Also note that the Sub-interval Saved Stats Structure (52 octets) SHALL be included (and populated as required when the server is in the receiver role) as defined below. The Sub-interval saved statistics structure for received traffic measurements SHALL be organized and formatted as follows:

Note that the 52 octet saved statistics structure above has slight differences from the 40 octets that follow in the status feedback PDU, particularly the time-related fields. Upon receiving the Status Feedback PDU or expiration of the feedback interval, the server SHALL perform calculations required by the Load adjustment algorithm and adjust its sending rate, or signal that the client do so in its role as as sender. @@@@ To Do: Additional measurements, like interface byte counters from a client at a residential gateway, would change the Status Feedback PDU (and the protocol version number as a result). Interface byte counters seem useful for specific circumstances, such as when the client application has acces to an interface that sees all traffic to/from a service subscriber's location.

When the test duration timer on the server expires, it SHALL set the connection test action to STOP and mark all outgoing load or status PDUs with a test action of STOP1. uint8_t testAction; // Test action (server sets STOP1) This is simply a non-reversible state for all future messages sent from the server. When the client receives a load or status PDU with the STOP1 indication, it SHALL finalize testing, display the test results, and also mark its connection with a test action of STOP (so that any PDUs received subsequent to the STOP1 are ignored). With the test action of the client's connection set to STOP, the very next expiry of a send timer for either a load or status PDU SHALL cause the client to schedule an immediate end time to exit. The client SHALL then send all subsequent load or status PDUs with a test action of STOP2 uint8_t testAction; // Test action (client sets STOP2) as confirmation to the server, and a graceful termination of the test can begin. When the server receives the STOP2 confirmation in the load or status PDU, the server SHALL schedule an immediate end time for the connection which closes the socket and deallocates it. In a non-graceful test stop, the watchdog/NOTRAFFIC time-outs at each end-point will expire (sometimes at one end-point first), notifications in logs, STDOUT, and/or formateed output SHALL be made, and the test action of each end-point's connection SHALL be set to STOP.

The architecture of the method REQUIRES two cooperating hosts operating in the roles of Src (test packet sender) and Dst (receiver), with a measured path and return path between them. The duration of a test duration, parameter I, MUST be constrained in a production network, since this is an active test method and it will likely cause congestion on the Src to Dst host path during a test.

This section is for the benefit of the Document Shepherd's form, and will be deleted prior to final review. Much of the development of the method and comparisons with existing methods conducted at IETF Hackathons and elsewhere have been based on the example udpst Linux measurement tool (which is a working reference for further development) . The current project: is a utility that can function as a client or server daemon requires a successful client-initiated setup handshake between cooperating hosts and allows firewalls to control inbound unsolicited UDP which either go to a control port [expected and w/authentication] or to ephemeral ports that are only created as needed. Firewalls protecting each host can both continue to do their job normally. This aspect is similar to many other test utilities available. The firewall at the server will need to open a limited range of ephemeral ports to complete the second exchange: Test Activtion (where the client communicates to the server on an ephemeral destination port *assigned by the server*). is written in C, and built with gcc (release 9.3) and its standard run-time libraries allows configuration of most of the parameters described in Sections 4 and 7. supports IPv4 and IPv6 address families. supports IP-layer packet marking.

Active metrics and measurements have a long history of security considerations. The security considerations that apply to any active measurement of live paths are relevant here. See and . When considering privacy of those involved in measurement or those whose traffic is measured, the sensitive information available to potential observers is greatly reduced when using active techniques which are within this scope of work. Passive observations of user traffic for measurement purposes raise many privacy issues. We refer the reader to the privacy considerations described in the Large Scale Measurement of Broadband Performance (LMAP) Framework , which covers active and passive techniques. There are some new considerations for Capacity measurement as described in this memo. Cooperating client and server hosts and agreements to test the path between the hosts are REQUIRED. Hosts perform in either the server or client roles. One way to assure a cooperative agreement employs the optional Authorization mode through the use of the authDigest field and the known identity associated with the key used to create the authDigest field. Other means are also possible, such as access control lists at the server. It is REQUIRED to have a user client-initiated setup handshake between cooperating hosts that allows firewalls to control inbound unsolicited UDP traffic which either goes to a control port [expected and w/authentication] or to ephemeral ports that are only created as needed. Firewalls protecting each host can both continue to do their job normally. Client-server authentication and integrity protection for feedback messages conveying measurements is RECOMMENDED. To accomodate different host limitations and testing circumstances, different modes of operation are recommended:

integrity trade-off (lightweight encryption may see more deployment) + synergy: we are already using the OpenSSL library in the running code (for Authentication) New: we think this mode D might not be used very often, the demands on hosts to generate and measure at Gbps rates usually require all the cycles they can allocate to the measurement process. Pre-WG 00 proposal below: A. Unauthenticated mode (for all phases) AND B. OPTIONAL Authenticated set-up only SHA-256 HMAC time-window verification (5 min time stamp verification) (could add silent failure option) -=-=-=-=-=-=-=-=-=-Above options exist in Running Code -=-=-=-=-=- C. Encrypted setup and test-activation (currently using OpenSSL Library, so KISS, but may be too slow for test packets) -=-=-=-=--=- Old/lowpower host performance impacts -=-=-=-=-=-=- D. Encrypted feedback messages (maybe split into Integrity and encrypt?) E. Integrity protection for test packets SHA-256 HMAC F. Encrypted test packets (maybe also valuable to defeat compression on links) ]]> Hosts MUST limit the number of simultaneous tests to avoid resource exhaustion and inaccurate results. Senders MUST be rate-limited. This can be accomplished using a pre-built table defining all the offered load rates that will be supported (Section 8.1). The recommended load-control search algorithm results in "ramp up" from the lowest rate in the table. Service subscribers with limited data volumes who conduct extensive capacity testing might experience the effects of Service Provider controls on their service. Testing with the Service Provider's measurement hosts SHOULD be limited in frequency and/or overall volume of test traffic (for example, the range of I duration values SHOULD be limited). The exact specification of these features was hopefully accomplished during this protocol development.

This memo requests IANA to assign a "well-known" UDP port for the Test Setup phase of protocol operation.

Thanks to Ruediger Geib, Lincoln Lavoie, Can Desem, and Greg Mirsky for reviewing this draft and providing helpful suggestions and areas for further development. Ken Kerpez and Chen Li have provided helpful reviews. Brian Weis provided an early SEC-DIR review; version 02 captures clarifications and further versions will take on the protocol changes suggested.