]> Google

garyillyes@google.com

Mozilla

mt@lowentropy.net

Web and Internet Transport AI Preferences skynet training wheel Content creators and other stakeholders might wish to signal their preferences about how their content might be consumed by automated systems. This document defines how preferences can be signaled as part of the acquisition of content in HTTP. This document updates RFC 9309 to allow for the inclusion of usage preferences. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the AI Preferences Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction The automated consumption of content by crawlers and other machines has increased significantly in recent years. This is partly due to the training of machine-learning models. Content creators and other stakeholders, such as distributors, might wish to express a preference regarding the types of usage they consider acceptable. Entities that might use that content need those preferences to be expressed in a way that is easily consumed by an automated system. This document describes two mechanisms for associating preferences with content:

• A Content-Usage header field for HTTP ; see .
• A Content-Usage directive for the Robots Exclusion Protocol (colloquially known as "robots.txt") ; see .

For automated systems that use HTTP to gather content, these allow for the automated gathering of preferences in the same way that content is obtained.

Preference Expressions The format of preference expressions is defined in the preference vocabulary . The preference vocabulary defines:

• what preferences can be expressed,
• how multiple expressions of preference are combined, and
• how those preferences are turned into strings or byte sequences for use in a protocol.

This document only defines how the strings or byte sequences are conveyed so that the preferences can be associated with content.

Examples A server that provides content using HTTP could signal preferences about how that content is used with the Content-Usage header field as follows: Alternatively, or additionally, a server might include the same directive in its "robots.txt" file:

Embedded Preferences Embedding preferences is expected to be an effective means of associating preferences with content, because it ensures that metadata is always associated with content. This document, however, does not define any specific means of embedding preferences in content. The main challenge with embedding preferences is that a different method might be needed for each content type. That is, a different storage or serialization model of conveying the preferences might need to be defined for each format whether it represent audio, documents, images, video, or other types of content. Furthermore, some content types, such as plain text (text/plain), offer no standardized means of embedding preferences. The mechanisms in this document can be applied to any content type, provided that the content is obtained using HTTP (and maybe FTP). Future work might define how preferences might be indicated for alternative content distribution or acquisition methods, such as email.

Registry-Based Preferences This document does not define a means of using unique identifiers and a registry for associating preferences. Registry-based approaches might be applicable in certain contexts, particularly where embedding is impractical or unavailable. Additionally, a registry might enable persistent association of preferences across distribution channels.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

HTTP Content-Usage Header Field The Content-Usage field is a structured field dictionary, as defined in . This field follows the vocabulary and processing rules in . This field indicates usage preferences regarding the content of the HTTP message. That is, the representation data, as defined in , not the resource. Servers MUST retain any preferences associated with a request if the content of that request is used to answer later requests. For example, the content of a PUT request that is used to answer subsequent GET requests. Note that servers that have not been updated to understand this field will not comply with this requirement. The Content-Usage field does not have any special effect on caching.

Robots Exclusion Protocol Content-Usage Rule The core function of Robots Exclusion Protocol format (or the "robots.txt" file) is to describe the expectations of the server operator about which paths can be crawled. This document adds a new rule that associates usage preferences with different paths. This new rule applies to any paths that can be crawled; paths that cannot be crawled have no associated usage preferences. A Content-Usage rule is added to the set of potential rules that can be included in a Group for "robots.txt". The rule ABNF pattern from is extended as shown in .

Extended robots.txt ABNF ]]>

Each group contains zero or more Content-Usage rules. Each Content-Usage rule consists of a path and a usage preference. The path might be absent or empty; if a path present, a SP or HTAB separates it from the usage preference. Note that the usage preference expression encoding does not use an ABNF definition, relying instead on the definitions in .

Content-Usage Rule Semantics Each group in the file applies to a set of crawlers, identified by product token as defined in . The Allow and Disallow rules determine what resources can be crawled, using the rule that has the longest matching path prefix, as defined in . This creates a two-stage arrangement that distinguishes acquisition and usage. Acquisition relies on Allow/Disallow rules; usage preference relies on Content-Usage rules. Any Content-Usage rules determine the usage preferences for resources using the same path prefix matching rules as defined for Allow and Disallow. That is, the path prefix length is determined by counting the number of bytes in the encoded path. Usage preferences apply only to those resources that can be crawled according to Allow/Disallow rules; no preferences are implied for resources that are disallowed. Paths specified for Content-Usage rules use the same percent-encoding rules as used for Allow/Disallow rules, as defined in . In particular, SP (U+20) and HTAB (U+09) characters need to be replaced with "%20" and "%09" respectively. The ordering of rules in a group carries no semantics. Thus, Content-Usage rules can be interleaved with Allow and Disallow rules. If there are Content-Usage rules that have identical paths and conflicting usage preferences, these preferences apply separately according to the process defined in . Note that this differs from the Allow/Disallow rules, where a conflict leads to the more permissive option, allowing crawling. A crawlers can cache a "robots.txt" file for up to 24 hours, following HTTP Cache-Control semantics defined in ; see for details. Updates to preferences within the period that a file is cached might not be visible.

Processing Content-Usage Rules To process a Content-Usage rule, a parser identifies lines with the "Content-Usage" label. This requires that SP and HTAB characters are ignored, before and after the label, in addition to before and after the COLON (":", U+3A) separator. The remainder of the line - up to either the first CR (U+0D), LF (U+0A), or octothorpe ("#", U+23) - is the rule value. The first character of the rule value will be "/" (U+2F) if a non-empty path is specified. Paths always start with a "/" character, so a rule value that starts with any other character indicates that the path is absent. If a path is specified, the path ends immediately before the first SP (U+20) or HTAB ("U+09") character. The remainder of the rule value is the usage preference expression. If a path is absent, the entire rule value is the usage preference expression. The usage preference is encoded using the exemplary format defined in . The parsing and processing rules from Sections and of apply. Note that a usage preference expression is processed as a sequence of bytes, rather than Unicode text; see .

When Preferences Apply A crawler that fetches resources uses the copy of "robots.txt" that is current at the time of the fetch to determine which usage preferences apply to those resources. defines how a "robots.txt" file can be cached. This means that updates to "robots.txt" do not retroactively apply to resources. Changes to "robots.txt" that affect usage preferences therefore only apply after a crawler has retrieved the updated "robots.txt" and subsequently retrieved the affected resource again.

Example shows a simple "robots.txt" document.

Example robots.txt file

A crawler that identifies as "ExampleBot" uses the second group. That crawler would be able to obtain all content and apply usage preferences of "ai=y" as defined in . All other crawlers use the first group. This allows crawling of all content other than resources under "/never/". Of those resources, those under "/ai-ok/" have an associated usage preference of "train-ai=y" and all other resources have a usage preference of "train-ai=n". Sample of usage preferences for different paths

Path	Crawl	Usage Preference
/test	yes	train-ai=n
/never/test	no	N/A
/ai-ok/test	yes	train-ai=y

Security Considerations Processing usage preferences involves the parsing of text that is produced by potential adversaries. Different guidelines for robust parsing can be found in and . describes security considerations for "robots.txt". A "robots.txt" file can be up to 500KiB of text. This specification does not increase this limit.

IANA Considerations The Content-Usage HTTP header field defined in is added to the "HTTP Field Name" registry established in :

Field Name:

Content-Usage

Status:

permanent

Structured Type:

Dictionary

Reference:

Comments:

None

References Normative References This document describes a set of data types and associated algorithms that are intended to make it easier and safer to define and handle HTTP header and trailer fields, known as "Structured Fields", "Structured Headers", or "Structured Trailers". It is intended for use by specifications of new HTTP fields. This document obsoletes RFC 8941. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching. A Uniform Resource Identifier (URI) is a compact sequence of characters that identifies an abstract or physical resource. This specification defines the generic URI syntax and a process for resolving URI references that might be in relative form, along with guidelines and security considerations for the use of URIs on the Internet. The URI syntax defines a grammar that is a superset of all valid URIs, allowing an implementation to parse the common components of a URI reference without knowing the scheme-specific requirements of every possible identifier. This specification does not define a generative grammar for URIs; that task is performed by the individual specifications of each URI scheme. [STANDARDS-TRACK] Open Future Mozilla In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Informative References The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document defines HTTP caches and the associated header fields that control cache behavior or indicate cacheable response messages. This document obsoletes RFC 7234.

Acknowledgments TODO acknowledge.