Internet-Draft RTCP Messages for Point Cloud Prioritiza September 2024
Engelbart, et al. Expires 29 March 2025 [Page]
Workgroup:
Audio/Video Transport Core Maintenance
Internet-Draft:
draft-engelbart-avtcore-rtcp-point-cloud-roi-00
Published:
Intended Status:
Standards Track
Expires:
Authors:
M. Engelbart
Technical University Munich
J. Ott
Technical University Munich
L. Kondrad
Nokia Technologies

RTCP Messages for Point Cloud Prioritization

Abstract

This document specifies RTCP messages and RTP header extensions for exchanging parameters of real-time streamed point clouds. A sender can notify receivers of the currently applied parameters, such as selected regions, and their parameters, such as the respective resolutions and included point attributes. A receiver can request updates to the same parameters using RTCP feedback messages.

About This Document

This note is to be removed before publishing as an RFC.

The latest revision of this draft can be found at https://mengelbart.github.io/draft-engelbart-avtcore-rtcp-point-cloud-roi/draft-engelbart-avtcore-rtcp-point-cloud-roi.html. Status information for this document may be found at https://datatracker.ietf.org/doc/draft-engelbart-avtcore-rtcp-point-cloud-roi/.

Discussion of this document takes place on the Audio/Video Transport Core Maintenance Working Group mailing list (mailto:avt@ietf.org), which is archived at https://mailarchive.ietf.org/arch/browse/avt/. Subscribe at https://www.ietf.org/mailman/listinfo/avt/.

Source for this draft and an issue tracker can be found at https://github.com/mengelbart/draft-engelbart-avtcore-rtcp-point-cloud-roi.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 29 March 2025.

Table of Contents

1. Introduction

A point cloud is a set of data points in a three-dimensional coordinate system where each point is defined by its three coordinates. Point clouds can represent three-dimensional environments, such as a vehicle's surroundings. Each point in a point cloud may optionally be associated with additional attributes. Attributes can, for example, be colors or reflectance of objects in the scene. Sequences of point clouds can be generated by sensors such as Lidar ("light detection and ranging"), Radar ("radio detection and ranging"), or a multi-camera setup. Due to the high number of points in a scene, the bandwidth requirements of transmitting point clouds over a network are often higher than those of streaming video. In video streaming, efficient codecs are used to reduce the bandwidth requirement. Similar codecs are being developed for point clouds. However, when streaming point cloud data, consumers of the data usually aren't equally interested in each point. For further processing, focusing on some regions of a point cloud stream is often sufficient, allowing the producer of a point cloud sequence to filter out points of lower interest to further reduce the bandwidth requirements and prioritize bandwidth usage for points of higher importance. When selecting the regions to prioritize and deciding which points can be excluded from transmission, producers and consumers need a mechanism to signal a) which regions of a scene are currently being prioritized and b) request updates to prioritize different regions in the future. This document describes such a mechanism for real-time transmission of point clouds building on the RTP Control protocol RTCP, which is part of the Real-time Transport Protocol RTP. Additionally, this document provides RTP header extensions for senders to inform receivers about the currently applied parameters. The information might be useful for receivers in determining if an RTCP message requesting an update was received and acted upon by the sender.

In Section 4, this document first defines a set of shared encoding schemes to represent point cloud parameters. Section 5 and Section 6 define RTCP messages and RTP header extensions, respectively. The RTCP messages and RTP header extensions reference the encoding scheme in Section 4. Section 7 defines the SDP parameters required to negotiate the usage of the presented RTCP messages and header extensions.

2. Conventions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

3. Definitions and Abbreviations

4. Region and Parameter Encodings

This section introduces encodings for point cloud regions and their parameters, such as the included attributes and the level of detail. It also provides an encoding for the definition of a viewport, including a dynamically adaptable precision.

The encodings described in the following subsections are reused in RTCP feedback messages and RTP header extension as described in Section 5 and Section 6.

4.1. Octree Encoding of Point Cloud Regions

This section defines an encoding that is reused in RTCP message types to index regions in a point cloud. The encoding uses a recursive octree encoding to signal the presence of certain regions of a point cloud. The encoding can be used to set parameters for the present regions, e.g., receivers can signal individual priority of every region or sets of attributes to include in every region.

The root node in every tree is a single byte where every bit indicates whether a child node of an octant is present. Child nodes are appended in pre-order traversal order. A zero-byte encodes a leaf node. The order of octants by the signs of the points of the X-, Y-, and Z-coordinates in each octant is given in Table 1.

Regions in the octree are considered present when a leaf node encoding that region is present. For example, a single zero-byte encodes a single leaf node, and the root region covering the complete space is present. An encoded value of the two bytes 0x4000 would indicate that only the octant with X<0, Y>=0, and Z>=0 is present.

The octree encoding can be used in absolute and relative forms. In the absolute form, the bounding box of the octree is implicitly defined by the space covered by the point cloud-generating source. In the relative form, the bounding box can be explicitly defined by setting the min and max values of the X-, Y-, and Z-coordinates. The coordinates are prefixed before the octree encoding as four-byte integers each, as shown in Figure 1.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Min X                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Min Y                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Min Z                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Max X                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Max Y                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Max Z                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | octree encoding                                             ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 1: Relative octree encoding using an explicitly defined bounding box

4.2. Region-Dependent Attributes Encoding

Attributes are additional values optionally associated with every point in a point cloud. The available attributes depend on the context of an application and thus need to be configured out-of-band. The attribute encoding described in this section requires mapping attributes to a bitmask value. Section 7 describes one option to negotiate the mapping when using SDP.

To signal the presence of attributes per region, senders and receivers can use the octree encoding presented in Section 4.1. For every region presented in the octree encoding, N bytes will be used to indicate the presence of attributes by setting the bits of the bitmask of the respective attributes to true. The size of N depends on the number of attributes and is implicitly given by the smallest number of bytes that can represent the largest bitmask negotiated during signaling.

Bitmask values can be shared among attributes to allow signaling of attribute sets that always occur in combination.

5. Format of RTCP Feedback Messages

This document describes RTCP message types that can be used to signal interest in the prioritization of regions and their parameters. Receivers can signal an interest in receiving a region with a higher priority. Different regions can be requested in different resolutions or with different sets of attributes included. Additionally, receivers can request a viewport precision update.

A sender can acknowledge a parameter update using the RTP header extensions described in Section 6. Receivers should not retransmit the same request multiple times to avoid unnecessary overhead, but if the receiver can assume that a request was lost, it may retransmit the request. The request should not be retransmitted earlier than at least one RTT after the first request was transmitted.

The following sections define the available message types in detail. All RTCP feedback messages use the common header format shown in Figure 2. The first eight bytes of the header follow the format of RTCP message headers defined in [RFC3550]. The common header is followed by an additional byte consisting of flags describing the payload.

The payload of the RTCP messages following the header contains one or more of the parameter encodings in the following order:

  1. Absolute/Relative Octree Encoding

  2. (optional) Priority

  3. (optional) Attribute Encoding

  4. (optional) Level-of-Detail Encoding

The semantics of the different payload formats are explained in the following subsections.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |V=2|P|   FMT   |       PT      |          length               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  SSRC of packet sender                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Res. |R|P|A|L| Payload                                      ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2: Common RTCP feedback message header

The fields V, P, SSRC and length are used as defined in the RTP specification [RFC3550]. The respective meaning is summarized below:

version (V): 2 bits

This field identifies the RTP version. The current version is 2.

padding (P): 1 bit

If set, the padding bit indicates that the packet contains additional padding octets at the end that are not part of the control information but are included in the length field.

Feedback message type (FMT): 5 bits

The feedback message type is XX (TODO: Use correct value).

Payload type (PT): 8 bits

The RTCP packet type is PSFB (206).

Relative (R): 1 bit

This field distinguishes between absolute and relative region requests (see Section 4.1. If the bit is set, the message contains a relative octree encoding and the minimum and maximum fields described in Figure 1 are present.

Priority: (P): 1 bit

If set, the RTCP message contains a priority request and the octree encoding is followed by one byte indicating the requested priority for each region encoded as leaf node in the octree encoding.

Attributes (A): 1 bit

If this bit is set, the RTCP message contains an attribute request and the octree encoding is followed by an attribute encoding for every region encoded as leaf node in the octree encdoing.

Level-of-Detail (L): 1 bit

If this bit is set, the RTCP message contains a level-of-detail request and the octree encoding (or the attribute encoding, if it is present) is followed by a level-of-detail encoding for every region encoded as leaf node in the octree encoding.

5.1. Region Requests

A receiver can use region of Interest signaling to indicate interest in some areas of a point cloud, and a sender can react by prioritizing the regions of interest when allocating bandwidth.

The region interest request uses the standard header defined in Figure 2.

5.2. Priority Requests

Receivers can append a priority encoding to the octree to signal fine-grained priorities per region. A priority is a value encoded as a single byte where a higher value indicates a higher priority. A priority request contains a priority byte for every region encoded as a leaf node in the preceding octree encoding.

5.3. Attribute Requests

By setting the header Flag A to 1, the receiver can include an attribute encoding as described in Section 4.2 to the feedback message to request attribute sets for every region encoded as a leaf node of the octree. The attribute encoding length depends on the number of negotiated attributes and their associated bitmask values. A single byte can indicate up to eight attributes or attribute sets. For example, if the two attributes, color, and reflectance, are negotiated with bitmask values of 0x01 and 0x02, one byte will be appended to the octree for every leaf node where the bits 0x01 and 0x02 are set to one for every region, which should include these attributes.

5.4. TODO: Region-Dependen Resolution or Level-of-Detail Request

  • TODO: Using the same mechanism as for attrbiutes, we can signal resolution per region, but we need to define how to express resolution.

5.5. Examples

An example encoding of a relative octree encoding followed by attribute and level-of-detail encodings:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |V=2|P|   FMT   |       PT      |          length               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                  SSRC of packet sender                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Res. |1|0|1|1| Payload                                      ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Min X                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Min Y                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Min Z                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Max X                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Max Y                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                            Max Z                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | octree encoding                                             ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | attribute encoding                                          ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | level-of-detail encdoing                                    ...
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 3: A RTCP feedback message containing a relative octree encoding, attribute encoding and level-of-detail encoding.

6. Format of RTP Header Extensions

RTP senders use RTP header extensions to acknowledge the applied parameters to the receiver. The parameters chosen by the sender may differ from the ones requested by the receiver. The header extension uses the two-byte header form defined in [RFC8285]. The payload of the header extension element uses the same format as the RTCP messages defined in Section 5. Each extension element begins with the same one-byte header (Figure 4) followed by an octree encoding and optional priority, attribute, or level-of-detail encodings.

    0
    0 1 2 3 4 5 6 7
   +-+-+-+-+-+-+-+-+
   |  Res. |R|P|A|L|
   +-+-+-+-+-+-+-+-+
Figure 4: Common RTP header extension payload header

7. SDP Parameters

This section defines SDP parameters for negotiating usage of the RTCP messages and RTP header extension described in this document.

7.1. RTCP Feedback Messages

The rtcp-fb attribute is extended to indicate the capability to send or receive the RTCP feedback defined in this document. This document adds a new parameter "oerr" to the "ccm" feedback value defined in [RFC5104]. The "oerr" (Octree Encoded Region Request) parameter indicates support for the RTCP feedback message defined in Section 5.

rtcp-fb-ccm-param =/ SP "oerr" ; Octree Encoded Region Request

7.2. RTP Header Extensions

The URI for indicating support for the RTP Header extension described in Section 6 is "urn:ietf:params:rtp-hdrext:octree-region".

8. IANA Considerations

8.1. RTCP Feedback Message

The following value are requested to be registered as FMT value in the "FMT Values for PSFB Payload Types" Registry:

Name:

OERR

Long Name:

Octree Encoded Region Request

Value:

TODO

Reference:

TODO: This document

8.2. RTP Header Extension

The following URI is requested to be added to the RTP Compact Header Extensions registry:

Extension URI:

urn:ietf:params:rtp-hdrext:octree-region

Description:

Octree Encoded Region Information

Contact:

mathis.engelbart@gmail.com

Reference:

TODO: This document

9. Security Considerations

TODO

10. RFC Editor Considerations

Note to RFC Editor: This section may be removed after carrying out all the instructions of this section.

TODO: Consider

11. Normative References

[RFC2119]
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, , <https://www.rfc-editor.org/rfc/rfc2119>.
[RFC3550]
Schulzrinne, H., Casner, S., Frederick, R., and V. Jacobson, "RTP: A Transport Protocol for Real-Time Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, , <https://www.rfc-editor.org/rfc/rfc3550>.
[RFC5104]
Wenger, S., Chandra, U., Westerlund, M., and B. Burman, "Codec Control Messages in the RTP Audio-Visual Profile with Feedback (AVPF)", RFC 5104, DOI 10.17487/RFC5104, , <https://www.rfc-editor.org/rfc/rfc5104>.
[RFC8174]
Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, , <https://www.rfc-editor.org/rfc/rfc8174>.
[RFC8285]
Singer, D., Desineni, H., and R. Even, Ed., "A General Mechanism for RTP Header Extensions", RFC 8285, DOI 10.17487/RFC8285, , <https://www.rfc-editor.org/rfc/rfc8285>.

Acknowledgments

TODO acknowledge.

Authors' Addresses

Mathis Engelbart
Technical University Munich
Jörg Ott
Technical University Munich
Lukasz Kondrad
Nokia Technologies