CDNI Logging Extensions

1.1. Log Generation

As described in section 2.2.1 of the CDNI Logging Interface [RFC7937], logging is generated by the downstream CDN (dCDN), to record all significant task completions, events, and failures. It is expected that all devices acting as caching nodes SHALL generate logging information, including devices performing request routing and other control functions. It is to be noted that this does not preclude caching nodes from generating and reporting transactions that are in progress and not fully completed, such as progressive downloads that may run for a longer duration and are therefore considered good candidates for partial reporting prior to fully completing a transaction. The individual implementations are, however, expected to make a determination as to when it is appropriate to generate reports for partial transactions.¶

1.2. Log Retention

The caching node SHALL retain logs for a specified duration. The specific duration is up to individual implementations to determine. Presumably, in the absence of logs being uploaded to or acquired by the CDN, it is possible the size of logging information would grow exponentially, therefore, it is also RECOMMENDED to define a shorter duration for upload/acquisition, such as a period of a few hours.¶

1.3. Accounting

Logging information is essential for accounting. The logging information provided by the caching node enables the CDNs to compute the total amount of traffic delivered by every cache for a given content service provider (CSP), as well allow the CDNs to determine associated bandwidth usage (peak or percentile) and key performance indicators, such as maximum (or minimum) number of simultaneous sessions over a given period of time.¶

1.4. Analytics And Reporting

The goals of analytics include gathering all relevant information in order to be able to develop statistics on content download, analyze user behavior, and monitor the performance and quality of content delivery. For instance, logging information enables CDN providers to report on content consumption (e.g., delivered sessions per content) in a specific geographic area.¶

The goal of reporting is to gather all relevant information to monitor the performance and quality of content delivery, and allow detection of delivery issues. For instance, reporting could track the average delivery throughput experienced by end users in a given region for a specific CSP or content set over a period of time.¶

1.5. Content Protection

The goal of content protection is to monitor and prevent unauthorized access, misuse, modification, and denial of access to content. A set of information is logged in a CDN for security purposes. In particular, a record of access to content is usually collected to permit the CSP to detect infringements of content delivery policies and other abnormal end-user behaviors.¶

1.6. Private Data Protection

The goal of logging transforms is to modify the outputs of fields that may include private information to comply with the confidentiality requirements of both the uCDCN and dCDN. The uCDN configurations are bundled within the transport configuration to support configurations that may be specific to the jurisdictions of individual reporting endpoints and service footprints. Additionally, the dCDN can advertise field transformations that have been applied in the footprint and capabilities interface.¶

3.1. JSON

The JSON container/file format consists of a single root object with file metadata, fields common to all records, and an array of log records.¶

Where the JSON format MUST be referenced from other configurations (e.g., in the container-format property of MI.LoggingTransport), it MAY be referred to by the string: "json_v1"¶

Property: shortname¶

• Description: A unique identifier for the dCDN.¶

• Type: String¶

• Mandatory-to-Specify: No¶

Property: timestamp-start-ns¶

• Description: The starting timestamp for the time range for which the log file was generated. This can be earlier than the timestamp of the file's first log record.¶

• Type: Integer time in nanoseconds since Unix epoch¶

• Mandatory-to-Specify: No¶

Property: timestamp-start-iso8601¶

• Description: The starting timestamp for the time range for which the log file was generated. This can be earlier than the timestamp of the file's first log record.¶

• Type: ISO-8601-1:2019¶

• Mandatory-to-Specify: No¶

Property: timestamp-end-ns¶

• Description: The ending timestamp for the time range for which the log file was generated. This can be later than the timestamp of the file's last log record.¶

• Type: Integer time in nanoseconds since Unix epoch¶

• Mandatory-to-Specify: No¶

Property: timestamp-end-iso8601¶

• Description: The ending timestamp for the time range for which the log file was generated. This can be later than the timestamp of the file's last log record.¶

• Type: ISO-8601-1:2019¶

• Mandatory-to-Specify: No¶

Property: account-id¶

• Description: Account ID as defined by the account-id property of MI.LoggingMetadata.¶

• Type: String¶

• Mandatory-to-Specify: No¶

Property: metadata¶

• Description: Metadata associated with configuration of the logging records, e.g., associated record-type.¶

• Type: Object. Valid properties:¶

  • record-type: String. The record type string defining the format of included logging records. See section 6. (Section 5)¶

  • transforms: Object. The MI.LoggingTransforms object applied to the record set. See section 7.4. (Section 6.4)¶

  • secret-stores: Array. Any MI.SecretStore objects referenced from MI.LoggingTransforms.¶

  • include-headers: Array. Any optionally included request or response header fields. See section 7.1. (Section 6.1)¶

• Mandatory-to-Specify: Yes¶

Property: records¶

• Description: The log records consisting of an array wherein all properties are valid log record fields as described in Section 5. (Section 4)¶

• Type: Array of objects¶

• Mandatory-to-Specify: Yes¶

The following is an example of the JSON container format:¶

{
  "shortname": "exampleCDN",
  "timestamp-start-iso8601": "2023-01-15T14:00:00Z",
  "timestamp-end-iso8601": "2023-01-15T14:59:59Z",
  "account-id": "foobarMedia",
  "metadata": {
    "record-type": "opencaching_standard_json_v1",
    "transforms": [
      {
        "record-fields": [
                 "c-ip"
             ],
        "operations": [
                 {
            "type": "MI.LoggingTransformMaskIp",
            "value": {
              "mask-lsb-v4": 4,
              "mask-lsb-v6": 16
            }
          }
             ]
      }
    ]
  },
  "records": [
    {
      "timestamp-ns": 1672938865000000000,
      "cs-uri": "https://cdn.example.com/assets/b874bc5b/index.m3u8",
      "sc-status": "200",
      "sc-total-bytes": 1784,
      "s-upstream-header-size-bytes": 208,
      "s-upstream-total-bytes": 1784
    }
  ]
}

3.2. Newline Delimited Records

A newline delimited container consists of individual log records separated by a linefeed (UNIX) or linefeed plus carriage return (Windows). The format of the log records is specified independently of the record delimiter.¶

The following is an example of the Newline Delimited container format:¶

1672579231000000,US/TN/MEM/38138,https://cdn.example.com/content/index.m3u8,200,14678,EXAMPLECDN,14678
1672579289000000,US/TN/MEM/38138,https://cdn.example.com/content/1.m4s,200,3887658,EXAMPLECDN,3887658

3.3. Archive Containers

File-based transport modes MAY bundle a set of log files together in a gzipped tar archive. When utilizing tar_v1 as the container-type, the properties of the MI.LoggingContainerMetadata object refer to the generation of the archive file, and the archive-metadata property of that object contains another MI.LoggingContainerMetadata object that specifies the configuration for the individual files within that archive.¶

For example, if the archive container is specified with an interval of 86400, and the inner archive-metadata has an interval of 3600, a tar.gz archive will be produced once a day containing 24 individual log files, each containing one hour of log records.¶

3.4. Protobuf Containers

Each log MAY be serialized using protobuf protocol buffers. This document provides the protobuf definition languages (.proto files) for different record types in Appendix section 10.1 (Section 8.1)¶

4.1. Fields

5.1. Log Record Formats

{
  "timestamp-ns": 1672938865000000000,
  "cs-uri": "https://cdn.example.com/assets/b874bc5b/index.m3u8",
  "sc-status": "200",
  "sc-total-bytes": 1784,
  "s-upstream-header-size-bytes": 208,
  "s-upstream-total-bytes": 1784,
}

5.2. Predefined Log Record Types

• A predefined log record type is a set of fields in a defined textual or binary format (JSON, CSV, whitespace delimited, or protobuf). This specification includes three predefined record types:¶

  • A ‘minimal' type optimized for data size, including only the fields necessary to perform billing functions.¶

  • A ‘standard' type with a set of fields likely to be useful for operational monitoring.¶

  • An ‘extended' type with an exhaustive field set likely to cover most operational and troubleshooting use cases.¶

{
    "generic-metadata-type": "MI.LoggingMetadata",
    "generic-metadata-value": {
            "transports": [{
                    "type": "MI.LoggingTransportKafka",
                    "record-type": "opencaching_standard_json_v1",
                    "include-headers": ["cs-hdr-Range", "sc-hdr-Content-Length"],
                    "account-id": "dCDN1",
                    "endpoint": {
                            "bootstrap-servers": ["kafka.ucdn.example.com:9092"],
                            "topic": "dcdn-logs"
                    }
            }]
    }
}

5.3. Custom Record Types

It is permitted with the out-of-band agreement of the participants to use a record type not defined in this specification. In this case, the format of the record type is entirely dependent on cooperation between the uCDN and dCDN outside the scope of CDNI. A unique identifier SHOULD be assigned for every custom record type, and it MAY be utilized in place of record types defined in this specification wherever a record-type value is required.¶

6.1. MI.LoggingMetadata

Property: transports¶

• Description: An array of MI.LoggingTransport objects.¶

• Type: Array¶

• Mandatory-to-Specify: No¶

Property: pull-transport-metadata¶

• Description: An object that can be used to configure certain properties of dCDN pull transports.¶

• Type: MI.LoggingPullTransportMetadata¶

• Mandatory-to-Specify: No¶

6.2. MI.LoggingPullTransportMetadata

Property: sftp-public-keys¶

• Description: An array of SSH public keys that the dCDN SHOULD authorize on SFTP logging endpoints.¶

• Type: Array of strings¶

• Mandatory-to-Specify: No¶

6.3. MI.LoggingTransport

MI.LoggingTransport objects are enclosed in the MI.LoggingMetadata object and instruct the dCDN how to transmit log files or streams to the uCDN.¶

Property: transport-type¶

• Description: The type discriminator for the endpoint property. The following values are supported:¶

  • MI.LoggingTransportKafka¶

  • MI.LoggingTransportSFTP¶

  • MI.LoggingTransportS3¶

• Type: String¶

• Mandatory-to-Specify: Yes¶

Property: record-type¶

• Description: The format of the individual log records. This MUST be a valid value as defined in Section 6 (Section 5)¶

• Type: String¶

• Mandatory-to-Specify: Yes¶

Property: container-metadata¶

• Description: The configuration for the generation of log files when operating in file-based modes.¶

• Type: MI.LoggingContainerMetadata¶

• Mandatory-to-Specify:¶

  • File or object-based transports: Yes¶

  • Streaming transports: No¶

Property: include-headers¶

• Description: Additional header fields to include, dependent on dCDN support via the allow-include-headers flag of MI.LoggingMetadata.¶

• Type: Ordered array of header fields in sc-hdr-* and cs-hdr-* naming format¶

• Mandatory-to-Specify: No¶

Property: account-id¶

• Description: An account identifier value that the dCDN must include in log records.¶

• Type: String¶

• Mandatory-to-Specify: No¶

Property: endpoint¶

• Description: An object with any configuration necessary for the specific logging transport, discriminated on the type property. Objects for each transport type are described in the sections below.¶

• Type: Object¶

• Mandatory-to-Specify: Yes¶

Property: transforms¶

• Description: Transformations on sets of fields as defined in an MI.LoggingTransform. The sets of record-fields field names MUST NOT overlap for the transport.¶

• Type: Array of MI.LoggingTransforms objects¶

• Mandatory-to-Specify: No¶

{
    "generic-metadata-type": "MI.LoggingMetadata",
    "generic-metadata-value": {
            "transports": [{
                    "transport-type": "MI.LoggingTransportKafka",
                    "record-type": "opencaching_standard_json_v1",
                    "account-id": "dCDN1",
                    "endpoint": {
                            "bootstrap-servers": ["kafka.ucdn.example.com:9092"],
                            "topic": "dcdn-logs"
                    }
            }]
    }
}

{
    "generic-metadata-type": "MI.LoggingMetadata",
    "generic-metadata-value": {
            "transports": [{
                    "type": "MI.LoggingTransportSFTP",
                    "record-type": "opencaching_standard_json_v1",
                    "container-metadata": {
                            "container-type": "json_v1",
                            "filename-template": "uCDN/logs/%Y-%m/%d-access.log.%n",
                            "interval": 86400,
                            "maximum-size": "100M"
                    },
                    "account-id": "dCDN1",
                    "endpoint": {
                            "host": "logs.ucdn.example.com:22",
                            "username": "dcdnlogs"
                    }
            }]
    }
}

{
    "generic-metadata-type": "MI.LoggingMetadata",
    "generic-metadata-value": {
            "transports": [{
                    "type": "MI.LoggingTransportS3API",
                    "record-type": "opencaching_standard_json_v1",
                    "container-metadata": {
                            "container-type": "json_v1",
                            "name-template": "%Y-%m-%d-access.log",
                            "interval": 86400
                    },
                    "account-id": "dCDN1",
                    "endpoint": {
                            "host": "s3.amazonaws.com",
                            "access-key-id": "xxxxxxxxx",
                            "access-key-secret": {
                                    "secret-store-id": "store-1",
                                    "secret-store-path": "/logging/dCDN1/dcdnlogs/s3"
                            },
                            "bucket-name": "dcdnlogs"
                    }
            }]
    }
}

6.4. MI.LoggingTransforms

Logging fields can contain personal user information that cannot be shared upstream or stored in the operating jurisdictions. These operations allow for obfuscation and anonymization of logging record fields.¶

An array of MI.LoggingTransforms objects can be enclosed in the MI.LoggingTransport object transform field for uCDN configured obfuscation.¶

The dCDN specifies privacy obfuscations in the FCI.Logging advertisement applied-transforms field.¶

Property: record-fields¶

• Description: The list of privacy-sensitive log record fields for which obfuscation is applied.¶

• Type: Array of strings¶

• Mandatory-to-Specify: Yes¶

Property: transforms¶

• Description: The transform operations to apply on the record fields, in order.¶

• Type: Array of MI.LoggingTransform objects¶

• Mandatory-to-Specify: Yes¶

{
    "generic-metadata-type": "MI.LoggingMetadata",
    "generic-metadata-value": {
            "transforms": [{
                    "record-fields": [
                            "c-uri",
                            "c-hdr-Referer"
                    ],
                    "operations": [{
                            "type": "MI.LoggingTransformUrlParam",
                            "value": {
                                    "replace-param": "^.?[iI][dD]$|.*_[iI][dD]$",
                                    "strip-params": False
                            }
                    }]
            }]
    }
}

{
    "generic-metadata-type": "MI.LoggingMetadata",
    "generic-metadata-value": {
            "transforms": [{
                    "record-fields": [
                            "c-uri",
                            "c-hdr-Referer"
                    ],
                    "operations": [{
                            "type": "MI.LoggingTransformUrlPath",
                            "value": {
                                    "path-trim": -1,
                                    "replace-path": "s/sid=[^\/]+/sid-XXXX/g"
                            }
                    }]
            }]
    }
}

{
    "generic-metadata-type": "MI.LoggingMetadata",
    "generic-metadata-value": {
            "transforms": [{
                    "record-fields": [
                            "c-ip"
                    ],
                    "transforms": [{
                            "type": "MI.LoggingTransformMaskIp",
                            "value": {
                                    "mask-lsb-v4": 4,
                                    "mask-lsb-v6": 16
                            }
                    }]
            }]
    }
}

{
    "generic-metadata-type": "MI.LoggingMetadata",
    "generic-metadata-value": {
            "transforms": [{
                    "record-fields": [
                            "c-ip"
                    ],
                    "transforms": [{
                            "type": "MI.LoggingTransformHash",
                            "value": {
                                    "function": "MD5",
                                    "key": {
                                            "secret-store-id": "store-1",
                                            "secret-path": "/logging/hmac/c-ip"
                                    }
                            }
                    }]
            }]
    }
}

{
    "generic-metadata-type": "MI.LoggingMetadata",
    "generic-metadata-value": {
            "transforms": [{
                    "record-fields": [
                            "c-ip"
                    ],
                    "operations": [{
                            "type": "MI.LoggingTransformEncrypt",
                            "value": {
                                    "algorithm": "AEAD-XCHACHA20-POLY1305-IETF",
                                    "key": {
                                            "secret-store-id": "store-1",
                                            "secret-path": "/logging/encrypt/c-ip"
                                    }
                            }
                    }]
            }]
    }
}

{
    "generic-metadata-type": "MI.LoggingMetadata",
    "generic-metadata-value": {
            "transports": [{
                    "type": "MI.LoggingTransportS3API",
                    "record-type": "opencaching_standard_json_v1",
                    "container-metadata": {
                            "container-type": "json_v1",
                            "name-template": "%Y-%m-%d-access.log",
                            "interval": 86400
                    },
                    "endpoint": {
                            "host": "s3.amazonaws.com",
                            "bucket-name": "dcdnlogs"
                    },
                    "transforms": [
                            {
                                "generic-metadata-type": "MI.LoggingTransforms",
                                "generic-metadata-value": {
                                        "transforms": [{
                                                "record-fields": [
                                                        "c-ip"
                                                ],
                                                "operations": [{
                                                                "type": "MI.LoggingTransformEncrypt",
                                                                "value": {
                                                                        "algorithm": "RC4",
                                                                        "key": {
                                                                                "secret-store-id": "store-1",
                                                                                "secret-path": "/logging/encrypt/c-ip"
                                                                        },
                                                                        "id": 622,
                                                                        "wrap-id": true
                                                                }
                                                        },
                                                        {
                                                                "type": "MI.LoggingTransformEncode",
                                                                "value": {
                                                                        "encoding": "BASE64"
                                                                }
                                                        }
                                                ]
                                        }]
                                }
                            }
                    ]
            }]
    }
}

6.5. MI.LoggingFilenameTemplate

The MI.LoggingFilenameTemplate type is used to specify the path and filename for files created by batch mode MI.LoggingTransport and FCI.LoggingTransport services.¶

If the resulting filename is longer than 255 characters, it will be truncated to 255 characters in length, including the filename extension that will be retained. Fields that MAY be truncated are noted in the table below.¶

It is the responsibility of the dCDN to ensure that file names are unique. It is strongly RECOMMENDED that the %id field is used to ensure the uniqueness requirement is satisfied.¶

The overall path length is limited to 32 KiB, and each path component is limited to 255 characters. Any element exceeding the length limitation will be truncated to the maximum length.¶

An instance of MI.LoggingFilenameTemplate is a string with the following supported substitution variables:¶

6.6. MI.LoggingContainerMetadata

This object defines configuration parameters used in file-based logging transports to define how files are created and formatted.¶

Property: container-type¶

• Description: The container type as defined in section 7 (Section 3)¶

• Type: String. The following values are supported:¶

  • cdni_v1¶

  • json_v1¶

  • linefeed_v1¶

  • carriage_return_v1¶

  • tar_v1¶

• Mandatory-to-Specify: Yes¶

Property: filename-template¶

• Description: The template to be used for files pushed by a batch mode transport.¶

• Type: MI.LoggingFilenameTemplate¶

• Mandatory-to-Specify: No¶

Property: include-metadata-sidecar¶

• Description: Used to specify if a metadata sidecar file is generated for each log file. See section 7.1.1 (Section 3.1.1)¶

• Type: Boolean¶

• Mandatory-to-Specify: No¶

Property: interval¶

• Description: For file-based logging transports, the time period for which a single log file SHOULD cover before being pushed to the endpoint, with the duration in seconds.¶

• Type: Integer¶

• Mandatory-to-Specify: No¶

Property: maximum-size¶

• Description: For batch-type logging transports, the maximum file or object size before the log file SHOULD be pushed to the endpoint and a new file created for further entries. The value of the format is "<Integer><Specifier>" where Specifier is one of the following:¶

  • M: megabyte¶

  • G: gigabyte¶

• Type: String¶

• Mandatory-to-Specify: No¶

Property: archive-metadata¶

• Description: Used with the tar_v1 container type, it specifies the container metadata for files inside of the archive.¶

• Type: MI.LoggingContainerMetadata¶

• Mandatory-to-Specify: No¶

7.1. FCI.LoggingPush

The FCI.Logging object described in [RFC8008] is restricted to announcing available record types and OPTIONAL fields supported by the dCDN from those listed in [RFC7937].¶

This specification includes more record types that can be used by the uCDN in the log configuration as well as other possible metadata that can be used as container formats and possible pull transport methods.¶

To permit the dCDN to announce multiple record types, container formats, and transport methods for the same footprint, this capability extends that described in [RFC8008] with a new capability dedicated to the push mechanisms.¶

The following is an FCI.LoggingPush example:¶

{
    "capability-type": "FCI.LoggingPush",
    "capability-value": {
            "record-types": [
                    "opencaching_minimal_json_v1",
                    "opencaching_extended_whitespace_v1"
            ],
            "allow-include-headers": true,
            "container-formats": ["json_v1", "carriage_return_v1"],
            "applied-transforms": [{
                    "record-fields": [
                            "c-ip"
                    ],
                    "operations": [{
                            "type": "MI.LoggingTransformMaskIp",
                            "value": {
                                    "mask-lsb-v4": 4,
                                    "mask-lsb-v6": 16
                            }
                    }]
            }]
    }
}

Property: record-types¶

• Description: The supported list of CDNI logging record-types.¶

• Type: Array of strings corresponding to an entry from the "CDNI logging record-types" registry [RFC7937], or this specification¶

• Mandatory-to-Specify: Yes¶

Property: container-formats¶

• Description: The supported list of container formats according to this specification.¶

• Type: Array of strings corresponding to available container formats¶

• Mandatory-to-Specify: No. If only streaming transports are used, container formats are not used. If not included, no non-streaming containers would be available.¶

Property: allow-include-headers¶

• Description: A flag indicating support for custom headers configured by the uCDN for inclusion in the log records.¶

• Type: Boolean¶

• Mandatory-to-Specify: No¶

Property: applied-transforms¶

• Description: Transforms applied by the dCDN on all logs to the upstream. The sets of record-fields field names MUST NOT overlap for the transport.¶

• Type: Array of MI.LoggingTransforms objects¶

• Mandatory-to-Specify: No¶

Property: sftp-public-keys¶

• Description: An array of SSH public keys that the dCDN SHOULD authorize on SFTP logging endpoints.¶

• Type: Array of strings¶

• Mandatory-to-Specify: No¶

7.2. FCI.LoggingSource

A dCDN can provide information on available logging endpoints to the uCDN that do not require metadata configuration. The FCI.LoggingSource capability object can be announced with the relevant information for the uCDN to get the log from the dCDN.¶

The dCDN could provide different endpoints, depending on footprint and/or delegated hosts. It could be possible that the sources in FCI.LoggingSource in the dCDN are not available for all delegated hosts or all available footprints, permitting flexibility and control of its own infrastructure (e.g., a dCDN could limit logging endpoints to suitable hosts included in an agreement between a dCDN and uCDN).¶

Property: hosts¶

• Description: One or more uCDN hosts for which the logs are available for pull logging transports. A redirecting host SHOULD be a host that was published in an MI.HostMatch object by the uCDN as defined in section 4.1.2 of [RFC8006].¶

• Type: A list of endpoint objects (see section 4.3.3 of [RFC8006])¶

• Mandatory-to-Specify:No. If absent or empty, the logging endpoint is available for all hosts of the redirecting uCDN.¶

Property: transports¶

• Description: List of available sources for an uCDN to obtain the logs for its delegated services¶

• Type: Array of FCI.LoggingPullTransport objects¶

• Mandatory-to-Specify: No¶

The following is an FCI.LoggingSource example that shows use of an S3 bucket for retrieving JSON formatted logs in a JSON container:¶

{
    "capability-type": "FCI.LoggingSource",
    "capability-value": {
            "hosts": [
                    "a.service123.ucdn.example.com",
                    "b.service123.ucdn.example.com"
            ],
            "transports": [{
                    "type": "FCI.LoggingPullTransportS3",
                    "record-type": "opencaching_standard_json_v1",
                    "container-metadata": {
                            "container-type": "json_v1",
                            "filename-template": "%Y-%m-%d-access.%i.log",
                            "interval": 86400
                    },
                    "account-id": "dCDN1",
                    "endpoint": {
                            "host": "s3.amazonaws.com",
                            "access-key-id": {
                        "secret-store-id": "store-1-cms",
                        "secret-value": "MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBAD
AFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEApJeXzsUS1jbAyNtQiJ9um9IMIHW5B2g+gHn
XdNSTyd33OEfTR6yLSZihBlFbHpY3qSzK1CX7RF5Oz3SqLDW+r3i1D/aHbVXwQbviWHEv
Hterql8l9VDm2FCNaDx5vihdbtvng3+/vdJNNMMhmovwZL5uhPsK81DkKwZCvznMMWt8Y
dNSFGT62f73ash7Eg/mS54IUyYOJHYrXEkRLSjvl0j+JqcIR8hCOCA78+5bS4MgfdsS9x
xSwQTrPru6EdTivMDKE/jlKg7li8lWdirWqtv0za5gLmH5T+zslXIoklwERAE50Jj8FxZ
D98EikKH8DAa+JeFsBm6Z1+yVFsWucTBMBgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBw
s1riXA6m336zRbsiKtrVgCA267133v2zD/wjFQHXrKSJfd/2YJaxPskgdmQaVlgWCw=="
                      },
                            "bucket-name": "dcdnlogs"
                    }
            }]
    }
}

7.3. FCI.LoggingPullTransport

Property: transport-type¶

• Description: The type discriminator for the endpoint property. The following values are supported:¶

  • FCI.LoggingPullTransportAtomFeed¶

  • FCI.LoggingPullTransportSFTP¶

  • FCI.LoggingPullTransportS3¶

• Type: String¶

• Mandatory-to-Specify: Yes¶

Property: record-type¶

• Description: The format of the individual log records. This must be a valid value as defined in section 5. (Section 4)¶

• Type: String¶

• Mandatory-to-Specify: Yes¶

Property: container-metadata¶

• Description: The configuration for the generation of log files when operating in file-based modes.¶

• Type: MI.LoggingContainerMetadata¶

• Mandatory-to-Specify:¶

  • File or object-based transports: Yes¶

  • Streaming transports: No¶

Property: transforms¶

• Description: A list of transformations to be applied to sets of record fields before transport. A record field name SHOULD NOT appear more than once.¶

• Type: Array of MI.LoggingTransforms objects¶

• Mandatory-to-Specify: No¶

Property: endpoint¶

• Description: An object with any configuration necessary for the specific logging transport, discriminated on the type property. Objects for each transport type are described below.¶

• Type: Object¶

• Mandatory-to-Specify: Yes¶

7.4. FCI.LoggingPullTransportAtomFeed

Property: feed-href¶

• Description: The URL of the CDNI logging feed in Atom format per [RFC7937].¶

• Type: String¶

• Mandatory-to-Specify: Yes¶

The following is an FCI.LoggingSourceAtomFeed example:¶

{
    "capability-type": "FCI.LoggingSource",
    "capability-value": {
            "hosts": [
                    "a.service123.ucdn.example.com",
                    "b.service123.ucdn.example.com"
            ],
            "transports": [{
                    "transport-type": "FCI.LoggingPullTransportAtomFeed",
                    "record-type": "opencaching_standard_json_v1",
                    "container-metadata": {
                            "container-type": "cdni_v1",
                            "filename-template": "%Y-%m-%d-access.%i.log",
                            "interval": 86400
                    },
                    "endpoint": {
                            "feed-href": "https://occ.example.com/logs/index.atom"
                    }
            }]
    }
}

7.5. FCI.LoggingPullTransportSFTP

Property: host¶

• Description: The hostname of the SFTP server, which MUST be specified as either hostname or hostname:port.¶

• Type: String¶

• Mandatory-to-Specify: Yes¶

Property: username¶

• Description: The remote username to use for connecting to the SFTP server.¶

• Type: String¶

• Mandatory-to-Specify: Yes¶

Property: password¶

• Description: The remote password to use for connecting to the SFTP server when using password-based authentication. This field is not used when SSH keys are provided by the uCDN via MI.LoggingPullTransportMetadata.¶

• Type: MI.SecretValue¶

• Mandatory-to-Specify: No¶

The following is an FCI.LoggingPullTransportSFTP example:¶

{
    "capability-type": "FCI.LoggingSource",
    "capability-value": {
            "hosts": [
                    "a.service123.ucdn.example.com",
                    "b.service123.ucdn.example.com"
            ],
            "transports": [{
                    "type": "FCI.LoggingPullTransportSFTP",
                    "record-type": "opencaching_standard_json_v1",
                    "container-metadata": {
                            "container-type": "json_v1",
                            "filename-template": "%Y-%m-%d-access.%i.log",
                            "interval": 86400
                    },
                    "endpoint": {
                            "host": "logs.dcdn.example.com",
                            "username": "logsucdn"
                    }
            }]
    }

}

7.6. FCI.LoggingPullTransportS3API

Property: host¶

• Description: The hostname to connect to the S3-compatible API.¶

• Type: String¶

• Mandatory-to-Specify: Yes¶

Property: access-key-id¶

• Description: The access key ID to use when connecting to the API.¶

• Type: String¶

• Mandatory-to-Specify: Yes¶

Property: access-key-secret¶

• Description: The access key secret to use when connecting to the API.¶

• Type: MI.SecretValue¶

• Mandatory-to-Specify: Yes¶

Property: bucket-name¶

• Description: The bucket name to use to pull log file objects.¶

• Type: String¶

• Mandatory-to-Specify: Yes¶

The following is an FCI.LoggingPullTransportS3API example:¶

{
    "capability-type": "FCI.LoggingSource",
    "capability-value": {
            "hosts": [
                    "a.service123.ucdn.example.com",
                    "b.service123.ucdn.example.com"
            ],
            "transports": [{
                    "type": "FCI.LoggingPullTransportS3API",
                    "record-type": "opencaching_standard_json_v1",
                    "container-metadata": {
                            "container-type": "json_v1",
                            "name-template": "%Y-%m-%d-access.log",
                            "interval": 86400
                    },
                    "endpoint": {
                            "host": "xxxxxxxxxxxxx",
                            "access-key-id": "xxxxxxxxx",
                            "access-key-secret": {
                                    "secret-store-id": "store-1",
                                    "secret-store-path": "/logging/dCDN1/dcdnlogs/s3"
                            },
                            "bucket-name": "ucdnlogs"
                    }
            }]
    }
}

8.1. Protobuf Record Type Definitions

This subsection provides the protobuf definitions in the proto file format for the protobuf record types defined in Section 4.4 (Section 3.4)¶

syntax = "proto3";
message OCLogRequestV1 {
  google.protobuf.Timestamp timestamp = 1;
  uint64 s_time_total_ms = 3;
  string c_groupid = 4;
  Method cs_method = 5;
  float cs_version = 6;
  string cs_uri = 7;
  string cs_uri_transformed = 8;
  uint32 sc_status = 9;
  uint64 sc_total_bytes = 10;
  string ccid = 11;
  string s_sid = 12;
  CacheStatus s_cached = 13;
  uint64 sc_header_size_bytes = 14;
  string c_ip = 15;
  string s_shortname = 16;
  string s_id = 17;
  uint64 s_time_first_ms = 18;
  uint64 s_sdur_ms = 19;
  uint64 s_time_upstream_ms = 20;
  uint64 s_upstream_header_size_bytes = 21;
  uint64 s_upstream_content_size_bytes = 22;
  repeated HttpHeader cs_hdr = 23;
  repeated HttpHeader sc_hdr = 24;
}
// Available at github.com/streaming-video-technology-alliance/logging

9.1. Field Privacy

Logging records may contain a user's personally identifiable information (PII), which may be inappropriate for the dCDN to share, and/or for the uCDN to store. This is particularly important when the dCDN is supporting custom headers. This specification provides MI.LoggingTransforms defined in section 7.4. (Section 6.4) as a mechanism for truncating, replacing, encrypting, and/or hashing record field values.¶

The uCDN can request transformations be applied to record fields with the transforms property of the MI.LoggingTransport object defined in section 7.3. (Section 6.3) The dCDN advertises transformations applied to record fields with the applied-transforms property of FCI.LoggingPush defined in section 8.1 (Section 7.1)¶

9.2. Field Encryption

In using the MI.LoggingTransformEncrypt defined in section 7.4.9 (Section 6.4.9), the nonce SHOULD NOT be reused across caching nodes using a counter permutation. When the uniqueness of nonces cannot be ensured, you can use the XChaCha20 algorithm with a nonce from a random source with sufficient entropy, or you can take the hash of a weaker random source, concatenate it with the message and hash it with a cryptographic hash function that is safe against length-extension attacks, such as BLAKE2 or the HMAC construction.¶

Rivest Cipher 4 (RC4) is listed to support annotating values originating from a packet cores' header enrichment network function for sharing subscriber or network identifiers with web applications. It SHOULD NOT be used as a security construct for concealing data in logs. An implementation would effectively be a no-op passing the value with bound credentials. It is RECOMMENDED to chain another MI.LoggingTransformEncrypt operation with a modern algorithm for actual confidentiality.¶

9.3. Field Length

It is important to consider using an MI.LoggingTransformTruncate in implementations that MAY have constraints on individual field record lengths. The dCDN can advertise these field length constraints in the applied-transforms property of FCI.LoggingPush and the transforms property of FCI.LoggingPullTransport*.¶

It is strongly RECOMMENDED to include an MI.LoggingTransformTruncate when including any cs-hdr-* OR sc-hdr-* fields in the logging format to protect against any denial of service (DoS) risks that arise from inadvertently malformed or actively malicious values in the headers.¶

9.4. Atom Endpoints

Authentication and authorization mechanisms for the Atom feed are outside the scope of this document.¶