Sandflow Consulting LLC

San Mateo CA United States of America pal@sandflow.com

University of New South Wales

Sydney Australia d.taubman@unsw.edu.au

WIT avtcore JPEG 2000 J2K HTJ2K low latency scalable streaming This document defines the RTP payload format for the streaming of a video signal encoded as a sequence of JPEG 2000 codestreams. The format allows sub-codestream latency, such that the first RTP packet for a given image can be emitted before the entire image is available to or encoded by the sender.

Status of This Memo This is an Internet Standards Track document. This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at .

Copyright Notice Copyright (c) 2025 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents () in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.

Table of Contents

• .  Introduction
• .  Requirements Language
• .  Media Format Description
• .  Video Signal Description
• .  Payload Format
  • .  General
  • .  RTP Fixed Header Usage
  • .  Main Packet Payload Header
  • .  Body Packet Payload Header
• .  JPEG 2000 Codestream
• .  Sender
  • .  Main Packet
  • .  RTP Packet Filtering
  • .  Resync Point
  • .  PTSTAMP Field
  • .  RES Field
  • .  Extra Information
  • .  Unassigned Values
  • .  Extension Values
  • .  Code-Block Caching
• .  Receiver
  • .  PTSTAMP
  • .  QUAL
  • .  RES
  • .  Extra Information
  • .  Unassigned Values
  • .  Extension Values
  • .  Code-Block Caching
• .  Media Type
  • .  General
  • .  Definition
• . Mapping to the Session Description Protocol (SDP)
• . Congestion Control
• . IANA Considerations
• . Security Considerations
• . References
  • .  Normative References
  • .  Informative References
• .  Pixel Formats
• .  Signal Formats
• .  Sample Formats
• Authors' Addresses

Introduction The Real-time Transport Protocol (RTP), which is specified in , provides end-to-end network transport functions for transmitting real-time data but does not define the characteristics of the data itself (the payload), which varies across applications and is defined in companion RTP payload format documents. This RTP payload format specifies the streaming of a video signal encoded as a sequence of JPEG 2000 codestreams (see for a primer on the structure of JPEG 2000 codestreams). JPEG 2000 is a flexible image codec that supports resolution and quality scalability, lossy-to-lossless coding, non-iterative optimal rate control, and high dynamic range, multi-channel, and subsampled images. These features have made it a mainstay in high-performance applications, including medical, geospatial, archival, cinema, studio post-production, and TV production. In addition to supporting a variety of frame-scanning techniques (progressive, interlaced, and progressive segmented frame) and image characteristics, the payload format supports real-time image transmission (live streaming), where image content is encoded, transmitted, and decoded continuously as it is being produced, with minimal latency. Target applications include real-time TV production over IP , remote presence, surveillance, etc. Specifically:

• The payload format allows sub-codestream latency such that the first RTP packet of a given codestream can be emitted before the entire codestream is available. Specifically, the payload format does not rely on the JPEG 2000 PLM (Packet length, main header) and PLT (Packet length, tile-part header) marker segments for recovery after RTP packet loss since these markers can only be written after the codestream is complete and are thus incompatible with sub-codestream latency. Instead, the payload format includes payload header fields (ORDH, ORDB, POS, and PID) that indicate whether the RTP packet contains a resynchronization (resync) point and how a recipient can restart codestream processing from that resync point. This contrasts with , which also specifies an RTP payload format for JPEG 2000 but relies on codestream structures that cannot be emitted until the entire codestream is available.
• As in , the payload format defines an extended sequence number, which extends the standard (16-bit) sequence number of the RTP Fixed Header by storing additional (high-order) bits in the payload header (ESEQ field). This enables the payload format to accommodate high data rates without ambiguity, since the standard sequence number will roll over very quickly for high data rates likely to be encountered in this application. For example, the standard sequence number will roll over in 0.5 seconds with a 1 Gbps video stream with RTP packet sizes of at least 1000 octets, which can be a problem for detecting loss and out-of-order RTP packets, particularly in instances where the round-trip time is greater than the rollover period (0.5 seconds in this example).
• The payload header optionally contains a temporal offset (PTSTAMP) relative to the first RTP packet with the same value of RTP timestamp field (). The higher resolution of PTSTAMP compared to the timestamp allows receivers to recover the sender's clock more rapidly.

In addition to support for sub-codestream latency and high-precision sender clock recovery, the payload format improves on by supporting:

• code-block caching for screen content (see );
• progressive segmented frame (PsF) video support (see ); and
• explicit colorspace signaling (see ).

Finally, the payload format also makes use of the unique scalability features of JPEG 2000 to allow an intermediate system or recipient to discard resolution levels and/or quality layers merely by inspecting RTP packet headers (QUAL and RES fields), without having to parse the underlying codestream (see ).

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. In case of conflict between the contents of a figure and the prose, the prose takes precedence.

Media Format Description The following summarizes the structure of the JPEG 2000 codestream, which is specified in detail in . NOTE: As described in , a JPEG 2000 codestream allows capabilities defined in any part of the JPEG 2000 family of standards, including those specified in and . JPEG 2000 represents an image as one or more components, each uniformly sampled on a common rectangular reference grid. For example, an image can consist of the customary Y (luma), C_b (blue-difference chroma), and C_r (red-difference chroma) components, with the C_b and C_r being subsampled by a factor of two compared to the Y component. An image can be further divided into contiguous rectangular tiles that are each independently coded and decoded. JPEG 2000 codes each image as a standalone codestream. A codestream consists of (i) marker segments, which contain coding parameters and metadata, and (ii) coded data. For convenience to the reader, the following lists both the abbreviated and full names of marker segments that are mentioned in this specification (several other marker segments are defined by JPEG 2000 and can be present in a codestream):

CAP:

Extended Capabilities

COC:

Coding style component

COD:

Coding style default

COM:

Comment

EOC:

End of codestream

PLM:

Packet length, main header

PLT:

Packet length, tile-part header

SOC:

Start of codestream

SOD:

Start of data

SOT:

Start of tile-part

The codestream starts with an SOC marker segment and ends with an EOC marker segment. The main header of the codestream consists of marker segments between the SOC and first SOT marker segment and contains information that applies to the codestream in its entirety. It is generally impossible to decode a codestream without its main header. The rest of the codestream consists of additional marker segments (tile-part headers) interleaved with coded image data. At the heart of JPEG 2000 coding is the wavelet transform, which decomposes the image into successive resolution levels, with each level related to the next one by a spatial factor of two, i.e., each successive resolution level has half the horizontal and half the vertical resolution of the previous one. The coded image data ultimately consists of code-blocks, each containing coded samples belonging to a rectangular (spatial) region within one resolution level of one component. Code-blocks are further collected into precincts, which, accordingly, represent code-blocks belonging to a spatial region within one resolution level of one component. The coded image data can be arranged into several progression orders, which dictates which aspect of the image appears first in the codestream (in terms of byte offset). The progression orders are parameterized according to:

Position (P)

The first lines of the image come before the last lines of the image.

Component (C)

The first component of the image comes before the last component of the image.

Resolution Level (R)

The information needed to reconstruct the lower spatial resolutions of the image come before the information needed to reconstruct the higher spatial resolutions of the image.

Quality Layer (L)

The information needed to reconstruct the most significant bits of each sample come before the information needed to reconstruct the least significant bit of each sample.

For example, in the PRCL progression order, the information needed to reconstruct the first lines of the image come before that needed to reconstruct the last lines of the image, and within a collection of lines, the information needed to reconstruct the lower spatial resolutions of the image come before the information needed to reconstruct the higher spatial resolutions. This progression order is particularly useful for subframe latency operations.

Video Signal Description This RTP payload format supports three distinct techniques for scanning video frames:

• Progressive frame
• Interlaced frame, where each frame consists of two fields. Field 1 occurs earlier in time than Field 2. The height in lines of each field is half the height of the image.
• Progressive segmented frame (PsF), where each frame consists of two segments. Segment 1 contains the odd lines (1, 3, 5, 7, ...) of a frame, and Segment 2 contains the even lines (2, 4, 6, 8, ...) of the same frame, where lines from the top of the frame to the bottom of the frame are numbered sequentially starting at 1.

All frames are scanned left to right, top to bottom.

Payload Format

General

Packetization of a Sequence of JPEG 2000 Codestreams (Not to Scale) <--------------- Codestream (image) --------------> | | <----- Extended Header -----> | | | | +-----+-//-+-----+-//-+-----+--------//-----+-----+-----+--------- | SOC | .. | SOT | .. | SOD | ............. | EOC | P | SOC ... +-----+-//-+-----+-//-+-----+--------//-----+-----+-----+--------- | | | <----------> Main header | | | +------------------------------+------+--//-+-----------+--------- | Main | Body | ... | Body | Main ... +------------------------------+------+--//-+-----------+--------- | | <--------- RTP Packet --------->

In , P denotes (optional) padding bytes. See for expansions of SOC, SOD, SOT, and EOC. Each RTP packet, as specified in , is either a Main Packet or a Body Packet. A Main Packet consists of the following ordered sequence of structures concatenated without gaps:

• the RTP Fixed Header;
• a Main Packet Payload Header, as specified in ; and
• the payload, which consists of a JPEG 2000 codestream fragment.

A Body Packet consists of the following ordered sequence of structures concatenated without gaps:

• the RTP Fixed Header;
• a Body Packet Payload Header, as specified in ; and
• the payload, which consists of a JPEG 2000 codestream fragment.

When concatenated, the sequence of JPEG 2000 codestream fragments emitted by the sender MUST be a sequence of JPEG 2000 codestreams where two successive JPEG 2000 codestreams MAY be separated by one or more padding bytes (see ). The sender MUST set the value of each padding byte to zero. The receiver MUST ignore the values of the padding bytes. The JPEG 2000 codestreams MUST conform to . NOTE 1: Padding bytes can be used to achieve constant bit rate transmission. A JPEG 2000 codestream fragment, and thus an RTP packet, does not necessarily contain complete JPEG 2000 packets, as defined in . A JPEG 2000 codestream Extended Header consists of the bytes between, and including, the SOC marker and the first SOD marker. NOTE 2: The concept of the JPEG 2000 codestream Extended Header is specific to this document and is distinct from the JPEG 2000 codestream main header, which is defined in . The codestream main header consists of the bytes between, and including, the SOC marker and the first SOT marker. The codestream main header is a subset of the codestream Extended Header (see ). The payload of a Body Packet MUST NOT contain any bytes of the JPEG 2000 codestream Extended Header. The payload of a Main Packet MUST contain at least one byte of the JPEG 2000 codestream Extended Header and MAY contain bytes other than those of the JPEG 2000 codestream Extended Header. A payload MUST NOT contain bytes from more than one JPEG 2000 codestream.

RTP Fixed Header Usage The following RTP header fields have a specific meaning in the context of this payload format:

marker

1

The payload contains an EOC marker.

0

Otherwise

timestamp

The timestamp is the presentation time of the image to which the payload belongs. The timestamp clock rate is 90 kHz. The timestamp of successive progressive frames MUST advance at regular increments based on the instantaneous video frame rate. The timestamp of Field 1 of successive interlaced frames MUST advance at regular increments based on the instantaneous video frame rate, and the Timestamp of Field 2 MUST be offset from the timestamp of Field 1 by one half of the instantaneous frame period. The timestamp of both segments of a progressive segmented frame MUST be equal. The timestamp of all RTP packets of a given image MUST be equal.

sequence number

The low-order bits of the extended sequence number. The high-order bits of the extended sequence number are contained in the ESEQ field, which is specified in . The extended sequence number is calculated as follows: <extended sequence number> = <ESEQ field> * 65536 + <sequence number field of the RTP Fixed Header>

Main Packet Payload Header specifies the structure of the payload header. Fields are interpreted as unsigned binary integers in network order.

Structure of the Main Packet Payload Header 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |MH | TP |ORDH |P|XTRAC| PTSTAMP | ESEQ | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |R|S|C| RSVD |*| PRIMS | TRANS | MAT | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | XTRAB | | ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * RANGE

MH (Codestream Main Header Presence):

2 bits

0

The RTP packet is a Body Packet.

1

The RTP packet is a Main Packet, and the codestream has more than one Main Packet. The next RTP packet is a Main Packet.

2

The RTP packet is a Main Packet, and the codestream has more than one Main Packet. The next RTP packet is a Body Packet.

3

The RTP packet is a Main Packet, and the codestream has exactly one Main Packet.

TP (Image Type):

3 bits Indicates the scanning structure of the image to which the payload belongs.

0

Progressive frame.

1

Field 1 of an interlaced frame, where the first line of the field is the first line of the frame.

2

Field 2 of an interlaced frame, where the first line of the field is the second line of the frame.

3

Field 1 of an interlaced frame, where the first line of the field is the second line of the frame.

4

Field 2 of an interlaced frame, where the first line of the field is the first line of the frame.

5

Segment 1 of a progressive segmented frame, where the first line of the image is the first line of the frame.

6

Segment 2 of a progressive segmented frame, where the first line of the image is the second line of the frame.

7

Extension value. See Sections and .

ORDH (Progression Order Flag, Main Packet):

3 bits Specifies the progression order used by the codestream and whether resync points are signaled. See for details about progression orders.

0

Resync points are not necessarily signaled. The progression order can vary over the codestream.

1

The progression order is LRCP for the entire codestream. The first resync point is specified in every Body Packet that contains one or more resync points.

2

The progression order is RLCP for the entire codestream. The first resync point is specified in every Body Packet that contains one or more resync points.

3

The progression order is RPCL for the entire codestream. The first resync point is specified in every Body Packet that contains one or more resync points.

4

The progression order is PCRL for the entire codestream. The first resync point is specified in every Body Packet that contains one or more resync points.

5

The progression order is CPRL for the entire codestream. The first resync point is specified in every Body Packet that contains one or more resync points.

6

The progression order is PRCL for the entire codestream. The first resync point is specified in every Body Packet that contains one or more resync points.

7

The progression order can vary over the codestream. The first resync point is specified in every Body Packet that contains one or more resync points.

ORDH MUST be 0 if the codestream consists of more than one tile. NOTE 1: Only ORDH = 4 and ORDH = 6 allow sub-codestream latency streaming. NOTE 2: Progression order PRCL is defined in . The other progression orders are specified in .

P (Precision Timestamp Presence):

1 bit

0

PTSTAMP is not used.

1

PTSTAMP is used.

XTRAC (Extension Payload Length):

3 bits Length, in multiples of 4 bytes, of the XTRAB field.

PTSTAMP (Precision Timestamp):

12 bits PTSTAMP = (timestamp + TOFF) mod 4096, if P = 1 in the Main Packet of this codestream. TOFF is the transmission time of this RTP packet, in the timebase of the timestamp clock and relative to the first RTP packet with the same timestamp value. TOFF = 0 in the first RTP packet with the same timestamp value. PTSTAMP = 0, if P = 0 in the Main Packet of this codestream. NOTE 3: As described in Sections and , PTSTAMP is intended to improve clock recovery at the receiver and only applies when the transmission time of two consecutive RTP packets with identical timestamp fields differ by no more than 45 ms (which is 4095/90,000). addresses the general case when an RTP packet is transmitted at a time other than its nominal transmission time.

ESEQ (Extended Sequence Number High-Order Bits):

8 bits The high-order bits of the extended sequence number. NOTE 4: The low-order bits of the extended sequence number are stored in the sequence number field of the RTP Fixed Header. specifies the formula to compute the extended sequence number.

R (Codestream Main Header Reuse):

1 bit Determines whether Main Packet and codestream main header information can be reused across codestreams.

1

All Main Packets in this stream, as identified by the value of the SSRC field in the RTP Fixed Header:

• MUST have identical Main Packet Payload Headers, with the exception of their TP, MH, ESEQ, and PTSTAMP fields;
• MUST contain the same codestream main header information (with the exception of the SOT and COM marker segments and any pointer marker segments); and
• MUST NOT contain bytes other than Extended Header bytes.

0

Otherwise

S (Parameterized Colorspace Presence):

1 bit

0

Component colorimetry is not specified; it is left to the session or the application. PRIMS, TRANS, MAT, and RANGE MUST be zero.

1

Component colorimetry is specified by the PRIMS, TRANS, MAT, and RANGE fields. The codestream components MUST conform to one of the combinations in . Mapping of Codestream Components to Color Channels

Combination Name	Component Index
	0	1	2	3
Y	Y
YA	Y	A
RGB	R	G	B
RGBA	R	G	B	A
YCbCr	Y	CB	CR
YCbCrA	Y	CB	CR	A

Channel A is an opacity channel. The minimum sample value (0) indicates a completely transparent sample, and the maximum sample value (as determined by the bit depth of the codestream component) indicates a completely opaque sample. The opacity channel MUST map to a component with unsigned samples.

C (Code-Block Caching Usage):

1 bit

0

Code-block caching is not in use.

1

Code-block caching is in use. R MUST be equal to 1.

RSVD (Reserved):

4 bits Unassigned value. See Sections and .

RANGE (Video Full Range Usage):

1 bit Value of the VideoFullRangeFlag specified in .

PRIMS (Color Primaries):

8 bits One of the ColourPrimaries values specified in .

TRANS (Transfer Characteristics):

8 bits One of the TransferCharacteristics values specified in .

MAT (Color Matrix Coefficients):

8 bits One of the MatrixCoefficients values specified in .

XTRAB (Extension Payload):

variable Allows the contents of the Main Packet Payload Header to be extended in the future. The size of the field is determined by . See also Sections and .

Body Packet Payload Header specifies the structure of the Body Packet Payload Header. Fields are interpreted as unsigned binary integers in network order.

Structure of the Body Packet Payload Header 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |MH | TP |RES |*|QUAL | PTSTAMP | ESEQ | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | POS | PID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ * ORDB

MH:

See .

TP:

See .

RES (Resolution Levels):

3 bits

0

The payload can contribute to all resolution levels.

Otherwise

The payload contains at least one byte of one JPEG 2000 packet belonging to resolution level (N_L + RES - 7) but does not contain any byte of any JPEG 2000 packet belonging to lower resolution levels. N_L is the number of decomposition levels of the codestream.

ORDB (Progression Order Flag, Body Packet):

1 bit

0

No resync point is specified for the payload.

1

The payload contains a resync point.

ORDB MUST be 0 if the codestream consists of more than one tile.

QUAL (Quality Layers):

3 bits

0

The payload can contribute to all quality layers.

Otherwise

The payload contributes only to quality layer index QUAL or above.

PTSTAMP:

See .

ESEQ:

See .

POS (Resync Point Offset):

12 bits Byte offset from the start of the payload to the first byte of the resync point belonging to the precinct identified by PID. POS MUST be 0 if ORDB = 0.

PID (Precinct Identifier):

20 bits Unique identifier of the precinct of the resync point. PID = c + s * num_components where:

c:

Index (starting from 0) of the image component to which the precinct belongs.

s:

Sequence number that identifies the precinct within its tile-component.

num_components:

Number of components of the codestream.

If PID is present, the payload MUST NOT contain codestream bytes from more than one precinct. PID MUST be 0 if ORDB = 0. NOTE: PID is identical to precinct identifier I specified in .

JPEG 2000 Codestream A JPEG 2000 codestream consists of the bytes between, and including, the SOC and EOC markers, as defined in . The JPEG 2000 codestream MAY include capabilities beyond those specified in , including those specified in and . NOTE 1: The Rsiz parameter and CAP marker segments of each JPEG 2000 codestream contain detailed information on the capabilities necessary to decode the codestream. NOTE 2: The caps media type parameter defined in allows applications to signal required device capabilities. NOTE 3: The block coder specified in improves throughput and reduces latency compared to the original arithmetic block coder defined in . For interlaced or progressive segmented frames, the height specified in the JPEG 2000 main header MUST be the height in lines of the field or the segment, respectively. If any decomposition level involves only horizontal decomposition, then no decomposition level MUST involve only vertical decomposition; conversely, if any decomposition level involves only vertical decomposition, then no decomposition level MUST involve only horizontal decomposition.

Sender

Main Packet A Main Packet MAY contain zero or more bytes of the JPEG 2000 codestream Extended Header. The sender MUST emit either a single Main Packet with MH = 3, or one or more Main Packets with MH = 1 followed by a single Main Packet with MH = 2. The Main Packet Payload Headers fields MUST be identical in all Main Packets of a given codestream, with the exception of:

• MH;
• ESEQ; and
• PTSTAMP.

RTP Packet Filtering An intermediate system MAY strip out RTP packets from a codestream that are of no interest to a particular client, e.g., based on a resolution or a spatial region of interest.

Resync Point A resync point is the first byte of JPEG 2000 packet header data for a precinct and for which PID < 2²⁴. NOTE 1: Resync points cannot be specified if the codestream consists of more than one tile (ORDB and ORDH are both equal to zero). NOTE 2: A resync point can be used by a receiver to process a codestream even if earlier RTP packets in the codestream have been corrupted, lost, or deliberately discarded by an intermediate system. As a corollary, resync points can be used by an intermediate system to discard RTP packets that are not relevant to a given rendering resolution or region of interest. Resync points play a role similar to pointer marker segments, albeit tailored for high-bandwidth, low-latency streaming applications.

PTSTAMP Field A sender SHOULD set P = 1, but only if it can generate PTSTAMP accurately. PTSTAMP can be derived from the same clock that is used to produce the 32-bit timestamp field in the RTP Fixed Header. Specifically, a sender maintains, at least conceptually, a 32-bit counter that is incremented by a 90 kHz clock. The counter is sampled at the point in time when each RTP packet is transmitted and the 12 least significant bits of the sample are stored in the PTSTAMP field. If P = 1, then the transmission time TOFF (as defined in ) for two consecutive RTP packets with identical timestamp fields MUST NOT differ by more than 4095.

RES Field A sender SHOULD set RES > 0 whenever possible. NOTE: While a sender can always safely set RES = 0, this makes it more difficult to discard RTP packets based on resolution, as described in .

Extra Information The sender MUST set the value of XTRAC to 0. Future updates to this RTP payload format can permit other values.

Unassigned Values The sender MUST set unassigned values to 0. Unassigned values are available for assignment by future updates to this RTP payload format. As specified in , these future assigned values are ignored by receivers that conform to this specification. In contrast to extension values (see ), this mechanism allows updates to this RTP payload format that remain compatible with receivers that conform to this specification.

Extension Values A sender MUST NOT use an extension value.

Code-Block Caching This section only applies if C = 1. A sender can improve bandwidth efficiency by only occasionally transmitting code-blocks corresponding to static portions of the video and otherwise transmitting empty code-blocks. When C = 1, a receiver maintains a simple cache of previously received code-blocks, which it uses to replace empty code-blocks (see ). A sender alone determines which code-blocks are replaced with empty code-blocks and when the replacement occurs. The sender cannot, however, determine with certainty the state of the receiver's cache; for example, some code-blocks might have been lost in transit, the sender doesn't know exactly when the receiver started processing the stream, etc. A code-block is empty if:

• it does not contribute code-bytes as specified in the parent JPEG 2000 packet header; or
• it contains an HT cleanup segment and the first two bytes of the Magsgn byte-stream are between 0xFF80 and 0xFF8F (if the code-block conforms to ).

NOTE: The last condition allows the encoder to insert padding bytes to achieve a constant bit rate even when a code-block does not contribute code-bytes, as suggested in Annex F.4 of .

Receiver

PTSTAMP Receivers can use PTSTAMP values to accelerate sender clock recovery since PTSTAMP typically updates more regularly than timestamp.

QUAL A receiver can discard RTP packets where QUAL > N if it is interested in reconstructing an image that only incorporates quality layers N and below.

RES The JPEG 2000 coding process decomposes an image using a sequence of discrete wavelet transform (DWT) stages. Optional discarding of Body Packets based on the value of the RES field when decoding a reduced resolution image, in the case where N_L = 5 and all DWT stages consist of both horizontal and vertical transforms. The image has nominal width and height of W x H.

Decomposition Level	Resolution Level	Sub-bands	Keep all Body Packets with RES equal to or less than this value...	...to decode an image with at most these dimensions
1	5	HL1,LH1,HH1	7	W x H
2	4	HL2,LH2,HH2	6	(W/2) x (H/2)
3	3	HL3,LH3,HH3	5	(W/4) x (H/4)
4	2	HL4,LH4,HH4	4	(W/8) x (H/8)
5	1	HL5,LH5,HH5	3	(W/16) x (H/16)
5	0	LL5	2	(W/32) x (H/32)

illustrates the case where each DWT stage consists of both horizontal and vertical transforms, which is the only mode supported in . The first stage transforms the image into (i) the image at half-resolution (LL1 sub-bands) and (ii) residual high-frequency data (HH1, LH1, HL1 sub-bands). The second stage transforms the image at half-resolution (LL1 sub-bands) into the image at quarter resolution (LL2 sub-bands) and residual high-frequency data (HH2, LH2, HL2 sub-bands). This process is repeated N_L times, where N_L is the number of decomposition levels as defined in the COD and COC marker segments of the codestream. The decoding process reconstructs the image by reversing the coding process, starting with the lowest resolution image stored in the codestream (LL_NL). As a result, it is possible to reconstruct a lower resolution of the image by stopping the decoding process at a selected stage. For example, in order to reconstruct the image at quarter resolution (LL2), only sub-bands with index greater than 2 (e.g., HL3, LH3, HH3, HL4, LH4, HH4, etc.) are necessary. In other words, a receiver that wishes to reconstruct an image at quarter resolution could discard all RTP packets where RES >= 6 since those RTP packets can only contribute to HL1, LH1, HH1, HL2, LH2, and HH2 sub-bands. In the case where all DWT stages consist of both horizontal and vertical transforms, the maximum decodable resolution is reduced by a factor of 2^{7 - N} if all Body Packets where RES > N are discarded. Optional discarding of Body Packets based on the value of the RES field when decoding a reduced resolution image, in the case where N_L = 5 and some DWT stages consist of only horizontal transforms. The image has nominal width and height of W x H.

Decomposition Level	Resolution Level	Sub-bands	Keep all Body Packets with RES equal to or less than this value...	...to decode an image with at most these dimensions
1	5	HL1,LH1,HH1	7	W x H
2	4	HL2,LH2,HH2	6	(W/2) x (H/2)
3	3	HX3	5	(W/4) x (H/2)
4	2	HX4	4	(W/8) x (H/2)
5	1	HX5	3	(W/16) x (H/2)
5	0	LX5	2	(W/32) x (H/2)

illustrates the case where some DWT stages consist of only horizontal transforms, as specified in Annex F of . A receiver can therefore discard all Body Packets where RES is greater than some threshold value if it is interested in decoding an image with its resolution reduced by a factor determined by the threshold value, as illustrated in Tables and .

Extra Information The receiver MUST accept values of XTRAC other than 0 and MUST ignore the value of XTRAB, whose length is given by XTRAC. Future updates to this RTP payload format can specify XTRAB contents such that this content can be ignored by receivers that conform to this specification.

Unassigned Values The receiver MUST ignore unassigned values (see additional discussion in ).

Extension Values The receiver MUST discard an RTP packet that contains any extension value.

Code-Block Caching This section only applies if C = 1. When C = 1, the sender can improve bandwidth efficiency by only occasionally transmitting code-blocks corresponding to static portions of the video and otherwise transmitting empty code-blocks (see ). When decoding a codestream, the following procedures apply for each code-block in the codestream:

• If the code-block in the codestream is empty, the receiver MUST replace it with a matching code-block from the cache, if one exists.
• If the code-block in the codestream is not empty, the receiver MUST replace any matching code-block from the cache with the code-block in the codestream.

Two code-blocks are matching if the following characteristics are identical for both: spatial coordinates, resolution level, component, sub-band, and value of the TP field of the parent RTP packet.

Media Type

General This RTP payload format is identified using the media type defined in . This media type has been registered in accordance with using the template in .

Definition

Type name:

video

Subtype name:

jpeg2000-scl

Required parameters:

N/A

Optional parameters:

pixel:

Specifies the pixel format used by the video sequence. The parameter MUST be a URI-reference as specified in . If the parameter is a relative-ref as specified in , then it MUST be equal to one of the pixel formats specified in , and the RTP header and payload MUST conform to the characteristics of that pixel format. If the parameter is not a relative-ref, the specification of the pixel format is left to the application that defined the URI. If the parameter is not specified, the pixel format is unspecified.

sample:

Specifies the format of the samples in each component of the codestream. The parameter MUST be a URI-reference as specified in . If the parameter is a relative-ref as specified in , then it MUST be equal to one of the formats specified in , and the stream MUST conform to the characteristics of that format. If the parameter is not a relative-ref, the specification of the sample format is left to the application that defined the URI. If the parameter is not specified, the sample format is unspecified.

width:

Maximum width in pixels of each image. Integer between 0 and 4,294,967,295. The parameter MUST be a sequence of 1 or more digits. If the parameter is not specified, the maximum width is unspecified.

height:

Maximum height in pixels of each image. Integer between 0 and 4,294,967,295. The parameter MUST be a sequence of 1 or more digits. If the parameter is not specified, the maximum height is unspecified.

signal:

Specifies the sequence of image types. The parameter MUST be a URI-reference as specified in . If the parameter is a relative-ref as specified in , then it MUST be equal to one of the signal formats specified in , and the image sequence MUST conform to that signal format. If the parameter is not a relative-ref, the specification of the pixel format is left to the application that defined the URI. If the parameter is not specified, the stream consists of an arbitrary sequence of image types.

caps:

The parameter contains a list of sets of constraints to which the stream conforms, with each set of constraints identified using an absolute-URI defined by an application. The parameter MUST conform to the uri-list syntax expressed as follows using ABNF : uri-list = absolute-URI *(";" absolute-URI) The application that defines the absolute-URI MUST ensure that it does not contain any ";" character and MUST associate it with a set of constraints to which the stream conforms. Such constraints can, for example, include the maximum height and width of images. If the parameter is not specified, constraints beyond those specified in this document are unspecified.

cache:

The value of the parameter MUST be either false or true. If the parameter is true, the field C MAY be 0 or 1; otherwise, the field C MUST be 0. If the parameter is not specified, then the parameter is equal to false.

Encoding considerations:

This media type is framed and binary. See .

Security considerations:

JPEG 2000 is a flexible image format. As a result, the size of the memory structures required to process JPEG 2000 images can vary greatly depending on the characteristics of the image and the encoding parameters. For example, the JPEG 2000 syntax allows image height and width up to 2³² - 1 pixels, which is also captured in the syntax of the height and width parameters of this media type. Therefore, implementations SHOULD take care when processing input that influences the size of memory structures and SHOULD fail gracefully when resource constraints are exceeded. See also .

Interoperability considerations:

The RTP stream is a sequence of JPEG 2000 images. An implementation that conforms to the family of JPEG 2000 standards can decode and attempt to display each image.

Published specification:

RFC 9828

Applications that use this media type:

video streaming and communication

Fragment identifier considerations:

N/A

Additional information:

N/A

Person & email address to contact for further information:

Pierre-Anthony Lemieux <pal@sandflow.com>

Intended usage:

COMMON

Restrictions on Usage:

This media type depends on RTP framing and hence is only defined for use with RTP as specified in . Transport within other framing protocols is not defined at the time.

Author:

Pierre-Anthony Lemieux <pal@sandflow.com>

Change controller:

IETF

Mapping to the Session Description Protocol (SDP) The mapping of the payload format media type and its parameters to SDP, as specified in , MUST be done according to .

Congestion Control Since this RTP payload format can be used in a variety of applications across many different contexts, there is no single congestion control mechanism that will work for all. Below is a non-exhaustive list of example mechanisms that can be used:

• a sender can offer several alternative streams for a given video signal, each with a different data rate corresponding to a different compression ratio;
• a sender can modulate the data rate within a stream by modulating the resolution, frame rate, or compression ratio of the underlying video signal; or
• an intermediate system can lower the data rate of a stream by discarding resolution levels and/or quality layers, as specified in .

As suggested in , RTCP feedback can be used in the data rate adaptation process.

IANA Considerations IANA has registered the media type specified in .

Security Considerations RTP packets using the payload format specified in this document are subject to the security considerations discussed in and in any applicable RTP profile such as , , , and . However, as discusses, it is not an RTP payload format's responsibility to discuss or mandate what solutions are used to meet the basic security goals like confidentiality, integrity, and source authenticity for RTP in general. This responsibility lies with anyone using RTP in an application. They can find guidance on available security mechanisms and important considerations in . Applications SHOULD use one or more appropriate strong security mechanisms. The rest of this section discusses the security impacting properties of the payload format itself. This RTP payload format and its media decoder do not exhibit any significant non-uniformity in the receiver-side computational complexity for RTP packet processing and thus are unlikely to pose a denial-of-service threat due to the receipt of pathological data. In addition, the RTP payload format does not contain any active content. Security considerations related to the JPEG 2000 codestream contained in the payload are discussed in .

References Normative References ITU-T ITU-T ITU-T ITU-T ITU-T In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. This memorandum describes RTP, the real-time transport protocol. RTP provides end-to-end network transport functions suitable for applications transmitting real-time data, such as audio, video or simulation data, over multicast or unicast network services. RTP does not address resource reservation and does not guarantee quality-of- service for real-time services. The data transport is augmented by a control protocol (RTCP) to allow monitoring of the data delivery in a manner scalable to large multicast networks, and to provide minimal control and identification functionality. RTP and RTCP are designed to be independent of the underlying transport and network layers. The protocol supports the use of RTP-level translators and mixers. Most of the text in this memorandum is identical to RFC 1889 which it obsoletes. There are no changes in the packet formats on the wire, only changes to the rules and algorithms governing how the protocol is used. The biggest change is an enhancement to the scalable timer algorithm for calculating when to send RTCP packets in order to minimize transmission in excess of the intended rate when many participants join a session simultaneously. [STANDARDS-TRACK] A Uniform Resource Identifier (URI) is a compact sequence of characters that identifies an abstract or physical resource. This specification defines the generic URI syntax and a process for resolving URI references that might be in relative form, along with guidelines and security considerations for the use of URIs on the Internet. The URI syntax defines a grammar that is a superset of all valid URIs, allowing an implementation to parse the common components of a URI reference without knowing the scheme-specific requirements of every possible identifier. This specification does not define a generative grammar for URIs; that task is performed by the individual specifications of each URI scheme. [STANDARDS-TRACK] This document specifies the procedure to register RTP payload formats as audio, video, or other media subtype names. This is useful in a text-based format description or control protocol to identify the type of an RTP transmission. [STANDARDS-TRACK] Internet technical specifications often need to define a formal syntax. Over the years, a modified version of Backus-Naur Form (BNF), called Augmented BNF (ABNF), has been popular among many Internet specifications. The current specification documents ABNF. It balances compactness and simplicity with reasonable representational power. The differences between standard BNF and ABNF involve naming rules, repetition, alternatives, order-independence, and value ranges. This specification also supplies additional rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications. [STANDARDS-TRACK] RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This memo defines the Session Description Protocol (SDP). SDP is intended for describing multimedia sessions for the purposes of session announcement, session invitation, and other forms of multimedia session initiation. This document obsoletes RFC 4566. Informative References SMPTE This document describes a profile called "RTP/AVP" for the use of the real-time transport protocol (RTP), version 2, and the associated control protocol, RTCP, within audio and video multiparticipant conferences with minimal control. It provides interpretations of generic fields within the RTP specification suitable for audio and video conferences. In particular, this document defines a set of default mappings from payload type numbers to encodings. This document also describes how audio and video data may be carried within RTP. It defines a set of standard encodings and their names when used within RTP. The descriptions provide pointers to reference implementations and the detailed standards. This document is meant as an aid for implementors of audio, video and other real-time multimedia applications. This memorandum obsoletes RFC 1890. It is mostly backwards-compatible except for functions removed because two interoperable implementations were not found. The additions to RFC 1890 codify existing practice in the use of payload formats under this profile and include new payload formats defined since RFC 1890 was published. [STANDARDS-TRACK] This document describes the Secure Real-time Transport Protocol (SRTP), a profile of the Real-time Transport Protocol (RTP), which can provide confidentiality, message authentication, and replay protection to the RTP traffic and to the control traffic for RTP, the Real-time Transport Control Protocol (RTCP). [STANDARDS-TRACK] This document serves to register and document the standard MIME types associated with the ISO/IEC 15444 standards, commonly known as JPEG 2000 (Joint Photographic Experts Group). [STANDARDS-TRACK] This memo specifies a packetization scheme for encapsulating uncompressed video into a payload format for the Real-time Transport Protocol, RTP. It supports a range of standard- and high-definition video formats, including common television formats such as ITU BT.601, and standards from the Society of Motion Picture and Television Engineers (SMPTE), such as SMPTE 274M and SMPTE 296M. The format is designed to be applicable and extensible to new video formats as they are developed. [STANDARDS-TRACK] Real-time media streams that use RTP are, to some degree, resilient against packet losses. Receivers may use the base mechanisms of the Real-time Transport Control Protocol (RTCP) to report packet reception statistics and thus allow a sender to adapt its transmission behavior in the mid-term. This is the sole means for feedback and feedback-based error repair (besides a few codec-specific mechanisms). This document defines an extension to the Audio-visual Profile (AVP) that enables receivers to provide, statistically, more immediate feedback to the senders and thus allows for short-term adaptation and efficient feedback-based repair mechanisms to be implemented. This early feedback profile (AVPF) maintains the AVP bandwidth constraints for RTCP and preserves scalability to large groups. [STANDARDS-TRACK] An RTP profile (SAVP) for secure real-time communications and another profile (AVPF) to provide timely feedback from the receivers to a sender are defined in RFC 3711 and RFC 4585, respectively. This memo specifies the combination of both profiles to enable secure RTP communications with feedback. [STANDARDS-TRACK] This memo describes an RTP payload format for the ISO/IEC International Standard 15444-1 | ITU-T Rec. T.800, better known as JPEG 2000. JPEG 2000 features are considered in the design of this payload format. JPEG 2000 is a truly scalable compression technology allowing applications to encode once and decode many different ways. The JPEG 2000 video stream is formed by extending from a single image to a series of JPEG 2000 images. [STANDARDS-TRACK] This document describes a method to inform Real-time Transport Protocol (RTP) clients when RTP packets are transmitted at a time other than their 'nominal' transmission time. It also provides a mechanism to provide improved inter-arrival jitter reports from the clients, that take into account the reported transmission times. [STANDARDS-TRACK] This document defines procedures for the specification and registration of media types for use in HTTP, MIME, and other Internet protocols. This memo documents an Internet Best Current Practice. The Real-time Transport Protocol (RTP) is used in a large number of different application domains and environments. This heterogeneity implies that different security mechanisms are needed to provide services such as confidentiality, integrity, and source authentication of RTP and RTP Control Protocol (RTCP) packets suitable for the various environments. The range of solutions makes it difficult for RTP-based application developers to pick the most suitable mechanism. This document provides an overview of a number of security solutions for RTP and gives guidance for developers on how to choose the appropriate security mechanism. This memo discusses the problem of securing real-time multimedia sessions. It also explains why the Real-time Transport Protocol (RTP) and the associated RTP Control Protocol (RTCP) do not mandate a single media security mechanism. This is relevant for designers and reviewers of future RTP extensions to ensure that appropriate security mechanisms are mandated and that any such mechanisms are specified in a manner that conforms with the RTP architecture.

Pixel Formats defines pixel formats. Defined Pixel Formats

NAME	SAMP	COMPS	TRANS	PRIMS	MAT	VFR	Mapping in
rgb444sdr	4:4:4	RGB	1	1	0	0, 1	RGB
rgb444wcg	4:4:4	RGB	1	9	0	0, 1	RGB
rgb444pq	4:4:4	RGB	16	9	0	0, 1	RGB
rgb444hlg	4:4:4	RGB	18	9	0	0, 1	RGB
ycbcr420sdr	4:2:0	YCbCr	1	1	1	0	YCbCr
ycbcr422sdr	4:2:2	YCbCr	1	1	1	0	YCbCr
ycbcr422wcg	4:2:2	YCbCr	1	9	9	0	YCbCr
ycbcr422pq	4:2:2	YCbCr	16	9	9	0	YCbCr
ycbcr422hlg	4:2:2	YCbCr	18	9	9	0	YCbCr

Each pixel format is characterized by the following:

NAME:

Identifies the pixel format

SAMP:

4:2:0

The C_b and C_r color channels are subsampled horizontally and vertically by 1/2.

4:2:2

The C_b and C_r color channels are subsampled horizontally by 1/2.

4:4:4

No color channels are subsampled.

COMPS:

RGB:

Each codestream contains exactly three components, associated with the R, G, and B color channels, in order.

YCbCr:

Each codestream contains exactly three components, associated with the Y, C_b, and C_r color channels, in order.

TRANS:

Identifies the transfer characteristics allowed by the pixel format, as defined in .

PRIMS:

Identifies the color primaries allowed by the pixel format, as defined in .

MAT:

Identifies the matrix coefficients allowed by the pixel format, as defined in .

VFR:

Allows values of the VideoFullRangeFlag defined in .

Signal Formats

prog:

The stream MUST only consist of a sequence of progressive frames.

psf:

Progressive segmented frame (PsF) stream. The stream MUST only consist of an alternating sequence of first segment and second segment.

tff:

Interlaced stream. The stream MUST only consist of an alternating sequence of first field and second field, where the first line of the first field is the first line of the frame.

bff:

Interlaced stream. The stream MUST only consist of an alternating sequence of first field and second field, where the first line of the first field is the second line of the frame.

Sample Formats

8

All components consist of unsigned 8-bit integer samples.

10

All components consist of unsigned 10-bit integer samples.

12

All components consist of unsigned 12-bit integer samples.

16

All components consist of unsigned 16-bit integer samples.

Authors' Addresses Sandflow Consulting LLC

San Mateo CA United States of America pal@sandflow.com

University of New South Wales

Sydney Australia d.taubman@unsw.edu.au