From 24cc6b0db8ef9c718925d14b329f21938e5d2b1b Mon Sep 17 00:00:00 2001 From: Rasmus Dahlberg Date: Tue, 20 Apr 2021 12:28:28 +0200 Subject: started on our in-progress (re)design documents --- doc/api.md | 247 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 247 insertions(+) create mode 100644 doc/api.md (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md new file mode 100644 index 0000000..760663b --- /dev/null +++ b/doc/api.md @@ -0,0 +1,247 @@ +# System Transparency Logging: API v0 +This document describes details of the System Transparency logging API, +version 0. The broader picture is not explained here. We assume that you have +read the System Transparency design document. It can be found [here](https://github.com/system-transparency/stfe/blob/design/doc/design.md). + +**Warning.** +This is a work-in-progress document that may be moved or modified. + +## Overview +The log implements an HTTP(S) API: +- Requests that add data to the log use the HTTP POST method. The HTTP content +type is `application/x-www-form-urlencoded`. The posted data are key-value +pairs. Binary data must be base64-encoded. +- Requests that retrieve data from the log use the HTTP GET method. The HTTP +content type is `application/x-www-form-urlencoded`. Input parameters are +key-value pairs. +- Responses are JSON objects. The HTTP content type is `application/json`. +- Error messages are human-readable strings. The HTTP content type is +`text/plain`. + +We decided to use these web formats for requests and responses because the log +is running as an HTTP(S) service. In other words, anyone that interacts with +the log is most likely using these formats already. The other benefit is that +all requests and responses are human-readable. This makes it easier to +understand the protocol, troubleshoot issues, and copy-paste. We favored +compatibility and understandability over a wire-efficient format. + +Note that we are not using JSON for signed and/or logged data. In other words, +a submitter that wishes to distribute log responses to their user base in a +different format may do so. The forced (de)serialization parser on _end-users_ +is a small subset of Trunnel. Trunnel is an "idiot-proof" wire-format that the +Tor project uses. + +## Primitives +### Cryptography +The log uses the same Merkle tree hash strategy as [RFC 6962, §2](https://tools.ietf.org/html/rfc6962#section-2). +The hash functions must be [SHA256](https://csrc.nist.gov/csrc/media/publications/fips/180/4/final/documents/fips180-4-draft-aug2014.pdf). +The log must sign tree heads using [Ed25519](https://tools.ietf.org/html/rfc8032). +The log's witnesses must also sign tree heads using Ed25519. + +All other parts that are not Merkle tree related also use SHA256 as the hash +function. Using more than one hash function would increases the overall attack +surface: two hash functions must be collision resistant instead of one. + +We recommend that submitters sign using Ed25519. We also support RSA with +[deterministic](https://tools.ietf.org/html/rfc8017#section-8.2) +or [probabilistic](https://tools.ietf.org/html/rfc8017#section-8.1) +padding. Supporting RSA is suboptimal, but excluding it would make the log +useless for many possible adopters. + +### Serialization +We use the [Trunnel](https://gitweb.torproject.org/trunnel.git) [description language](https://www.seul.org/~nickm/trunnel-manual.html) +to define (de)serialization of data structures that need to be signed or +inserted into the Merkle tree. Trunnel is more expressive than the +[SSH wire format](https://tools.ietf.org/html/rfc4251#section-5). +It is about as expressive as the [TLS presentation language](https://tools.ietf.org/html/rfc8446#section-3). +A notable difference is that Trunnel supports integer constraints. The Trunnel +language is also readable by humans _and_ machines. "Obviously correct code" +can be generated in C and Go. + +A fair summary of our Trunnel usage is as follows. + +All integers are 64-bit, unsigned, and in network byte order. A fixed-size byte +array is put into the serialization buffer in-order, starting from the first +byte. These basic types are concatenated to form a collection. You should not +need a general-purpose Trunnel (de)serialization parser to work with this +format. If you have one, you may use it though. The main point of using +Trunnel is that it makes a simple format explicit and unambiguous. + +TODO: URL-encode _or_ JSON? I think we should only need one. Always doing HTTP +POST would also ensure that input parameters don't show up in web server logs. + +#### Merkle tree head +Tree heads are signed by the log and its witnesses. It contains a timestamp, a +tree size, and a root hash. The timestamp is included so that monitors can +ensure _liveliness_. It is the time since the UNIX epoch (January 1, 1970 +00:00:00 UTC) in milliseconds. The tree size specifies the current number of +leaves. The root hash fixes the structure and content of the Merkle tree. + +``` +struct tree_head { + u64 timestamp; + u64 tree_size; + u8 root_hash[32]; +}; +``` + +The serialized tree head must be signed using Ed25519. A witness must only sign +the log's tree head if it is consistent with prior history and the timestamp is +roughly correct. A timestamp is roughly correct if it is not backdated or +future-dated more than 12 hours. + +#### Merkle tree leaf +The log supports a single leaf type. It contains a checksum, a signature +scheme, a signature that the submitter computed over that checksum, and the hash +of the public verification key that can be used to verify the signature. + +``` +const ALG_ED25519 = 1; // RFC 8032 +const ALG_RSASSA_PKCS1_V1_5 = 2; // RFC 8017 +const ALG_RSASSA_PSS = 3; // RFC 8017 + +struct tree_leaf { + u8 checksum[32]; + u64 signature_scheme IN [ + ALG_ED25519, + ALG_RSASSA_PKCS1_V1_5, + ALG_RSASSA_PSS, + ]; + union signature[signature_scheme] { + ALG_ED25519: u8 ed25519[32]; + default: u8 rsa[512]; + } + u8 key_hash[32]; +} +``` + +A key-hash is included in the leaf so that it can be attributed to the signing +entity. A hash, rather than the full public verification key, is used to force +the verifier to locate the appropriate key and make an explicit trust decision. + +## Public endpoints +Every log has a base URL that identifies it uniquely. The only constraint is +that it must be a valid HTTP(S) URL that can have the `/st/v0/` suffix +appended. For example, a complete endpoint URL could be +`https://log.example.com/2021/st/v0/get-signed-tree-head`. + +### get-signed-tree-head +``` +GET /st/v0/get-signed-tree-head +``` + +Input key-value pairs: +- `type`: either the string "latest", "stable", or "cosigned". + - "latest": ask for the most recent signed tree head. + - "stable": ask for a recent signed tree head that is fixed for some period + of time. + - "cosigned": ask for a recent cosigned tree head. + +Output: +- On success: status 200 OK and a signed tree head. The response body is +defined by the following [schema](https://github.com/system-transparency/stfe/blob/design/doc/schema/sth.schema.json). +- On failure: a different status code and a human-readable error message. + +### get-proof-by-hash +``` +POST /st/v0/get-proof-by-hash +``` + +Input key-value pairs: +- `leaf_hash`: a base64-encoded leaf hash that identifies which `tree_leaf` the +log should prove inclusion for. The leaf hash is computed using the RFC 6962 +hashing strategy. In other words, `H(0x00 | tree_leaf)`. +- `tree_size`: the tree size of a tree head that the proof should be based on. + +Output: +- On success: status 200 OK and an inclusion proof. The response body is +defined by the following [schema](https://github.com/system-transparency/stfe/blob/design/doc/schema/inclusion_proof.schema.json). +- On failure: a different status code and a human-readable error message. + +### get-consistency-proof +``` +POST /st/v0/get-consistency-proof +``` + +Input key-value pairs: +- `new_size`: the tree size of a newer tree head. +- `old_size`: the tree size of an older tree head that the log should prove is +consistent with the newer tree head. + +Output: +- On success: status 200 OK and a consistency proof. The response body is +defined by the following [schema](https://github.com/system-transparency/stfe/blob/design/doc/schema/consistency_proof.schema.json). +- On failure: a different status code and a human-readable error message. + +### get-leaves +``` +POST /st/v0/get-leaves +``` + +Input key-value pairs: +- `start_size`: zero-based index of the first leaf to retrieve. +- `end_size`: index of the last leaf to retrieve. + +Output: +- On success: status 200 OK and a list of leaves. The response body is +defined by the following [schema](https://github.com/system-transparency/stfe/blob/design/doc/schema/leaves.schema.json). +- On failure: a different status code and a human-readable error message. + +The log may truncate the list of returned leaves. However, it must not be an +empty list on success. + +### add-leaf +``` +POST /st/v0/add-leaf +``` + +Input key-value pairs: +- `leaf_checksum`: the checksum that the submitter wants to log in base64. +- `signature_scheme`: the signature scheme that the submitter wants to use. +- `tree_leaf_signature`: the submitter's `tree_leaf` signature in base64. +- `verification_key`: the submitter's public verification key. It is serialized +as described in the corresponding RFC, then base64-encoded. +- `domain_hint`: a domain name that indicates where the public verification-key +hash can be downloaded in base64. Supported methods: DNS and HTTPS +(TODO: docdoc). + +Output: +- On success: HTTP 200. The log will _try_ to incorporate the submitted leaf +into its Merkle tree. +- On failure: a different status code and a human-readable error message. + +The submitted entry will not be accepted if the signature is invalid or if the +downloaded verification-key hash does not match. The submitted entry may also +not be accepted if the second-level domain name exceeded its rate limit. By +coupling every add-leaf request with a second-level domain, it becomes more +difficult to spam the log. You would need an excessive number of domain names. +This becomes costly if free domain names are rejected. + +The log does not publish domain-name to key bindings because key management is +more complex than that. + +Public logging should not be assumed until an inclusion proof is available. An +inclusion proof should not be relied upon unless it leads up to a trustworthy +signed tree head. Witness cosigning can make a tree head trustworthy. + +TODO: the log may allow no `domain_hint`? Especially useful for v0 testing. + +### add-cosignature +``` +POST /st/v0/add-cosignature +``` + +Input key-value pairs: +- `signature`: a base64-encoded signature over a `tree_head` that is fixed for +some period of time. The cosigning witness retrieves the tree head using the +`get-signed-tree-head` endpoint with the "stable" type. +- `key_hash`: a base64-encoded hash of the public verification key that can be +used to verify the signature. + +Output: +- HTTP status 200 OK on success. Otherwise a different status code and a +human-readable error message. + +The key-hash can be used to identify which witness signed the log's tree head. +A key-hash, rather than the full verification key, is used to force the verifier +to locate the appropriate key and make an explicit trust decision. -- cgit v1.2.3 From aa8f64c0ed18f384a6af1ade6268b35ec60dac85 Mon Sep 17 00:00:00 2001 From: Rasmus Dahlberg Date: Tue, 20 Apr 2021 21:45:03 +0200 Subject: added shard_hint --- doc/api.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index 760663b..0f873e4 100644 --- a/doc/api.md +++ b/doc/api.md @@ -119,6 +119,39 @@ A key-hash is included in the leaf so that it can be attributed to the signing entity. A hash, rather than the full public verification key, is used to force the verifier to locate the appropriate key and make an explicit trust decision. +#### Shard hint +The log is only accepting new leaves during a predefined time interval. We +refer to this time interval as the log's _shard_. Sharding can simplify log +operations because it becomes explicit when the log can be shutdown. + +Unlike X.509 certificates that already have a validity range, a checksum does +not have any such information. Therefore, we require the submitter to sign a +_shard hint_. A shard hint is composed of a prefix and a tree leaf. + +``` +struct shard_hint { + u64 prefix; + struct tree_leaf leaf; +} +``` + +The log will check that the signed `shard_hint` can be verified using the +submitter's public verification key. The prefix could be anything and may +repeat. This API documentation assumes that the prefix is set to zero. + +As long as the `shard_hint` signature is not revealed, no one but the submitter +can submit a leaf that the log will accept. Therefore, the good Samaritan +cannot submit all leaves from an earlier shard into a newer one. The +`shard_hint` does not prevent the _legitimate submitter_ from reusing an earlier +submission in a future shard. + +Note the importance of letting the submitter decide if an entry is logged again +or not. If the log has a rate limiting function, replayed submissions could +deny service in a new shard. In practise we expect submitters to not log a +leaf again. Once an inclusion proof and a cosigned tree head is available, you +have all the necessary proofs. These proofs continue to be valid after the log +shuts down because the verification process is non-interactive. + ## Public endpoints Every log has a base URL that identifies it uniquely. The only constraint is that it must be a valid HTTP(S) URL that can have the `/st/v0/` suffix @@ -199,6 +232,7 @@ Input key-value pairs: - `leaf_checksum`: the checksum that the submitter wants to log in base64. - `signature_scheme`: the signature scheme that the submitter wants to use. - `tree_leaf_signature`: the submitter's `tree_leaf` signature in base64. +- `shard_hint_signature`: the submitter's `shard_hint` signature in base64. - `verification_key`: the submitter's public verification key. It is serialized as described in the corresponding RFC, then base64-encoded. - `domain_hint`: a domain name that indicates where the public verification-key -- cgit v1.2.3 From 66b3d09b526c3c1e8d5f2d9a92deba497ca8124c Mon Sep 17 00:00:00 2001 From: Rasmus Dahlberg Date: Mon, 26 Apr 2021 12:48:22 +0200 Subject: moved shard_hint into tree_leaf --- doc/api.md | 114 ++++++++++++++++++++++++++++++++++--------------------------- 1 file changed, 63 insertions(+), 51 deletions(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index 0f873e4..b5d54e6 100644 --- a/doc/api.md +++ b/doc/api.md @@ -85,72 +85,74 @@ struct tree_head { }; ``` -The serialized tree head must be signed using Ed25519. A witness must only sign -the log's tree head if it is consistent with prior history and the timestamp is -roughly correct. A timestamp is roughly correct if it is not backdated or -future-dated more than 12 hours. +The serialized tree head must be signed using Ed25519. A witness must not +cosign a tree head if it is inconsistent with prior history or if the timestamp +is backdated or future-dated more than 12 hours. #### Merkle tree leaf -The log supports a single leaf type. It contains a checksum, a signature -scheme, a signature that the submitter computed over that checksum, and the hash -of the public verification key that can be used to verify the signature. +The log supports a single leaf type. It contains a message, a signature scheme, +a signature that the submitter computed over the message, and a hash of the +public verification key that can be used to verify the signature. ``` -const ALG_ED25519 = 1; // RFC 8032 -const ALG_RSASSA_PKCS1_V1_5 = 2; // RFC 8017 -const ALG_RSASSA_PSS = 3; // RFC 8017 +const SIGNATURE_SCHEME_ED25519 = 1; // RFC 8032 +const SIGNATURE_SCHEME_RSASSA_PKCS1_V1_5 = 2; // RFC 8017 +const SIGNATURE_SCHEME_RSASSA_PSS = 3; // RFC 8017 -struct tree_leaf { +struct signature_ed25519 { + u8 signature[32]; +}; + +struct signature_rsassa { + u64 num_bytes IN [ 256, 384, 512 ]; + u8 signature[num_bytes]; +}; + +struct message { + u64 shard_hint; u8 checksum[32]; +}; + +struct tree_leaf { + struct message message; u64 signature_scheme IN [ - ALG_ED25519, - ALG_RSASSA_PKCS1_V1_5, - ALG_RSASSA_PSS, + SIGNATURE_SCHEME_ED25519, + SIGNATURE_SCHEME_RSASSA_PKCS1_V1_5, + SIGNATURE_SCHEME_RSASSA_PSS, ]; union signature[signature_scheme] { - ALG_ED25519: u8 ed25519[32]; - default: u8 rsa[512]; + SIGNATURE_SCHEME_ED25519: struct signature_ed25519 ed25519; + default: struct signature_rsassa rsassa; } u8 key_hash[32]; } ``` -A key-hash is included in the leaf so that it can be attributed to the signing -entity. A hash, rather than the full public verification key, is used to force -the verifier to locate the appropriate key and make an explicit trust decision. - -#### Shard hint -The log is only accepting new leaves during a predefined time interval. We -refer to this time interval as the log's _shard_. Sharding can simplify log -operations because it becomes explicit when the log can be shutdown. - -Unlike X.509 certificates that already have a validity range, a checksum does -not have any such information. Therefore, we require the submitter to sign a -_shard hint_. A shard hint is composed of a prefix and a tree leaf. +Unlike X.509 certificates that already have validity ranges, a checksum does not +have any such information. Therefore, we require that the submitter selects a +_shard hint_. The selected shard hint must be in the log's _shard interval_. A +shard interval is defined by a start time and an end time. Both ends of the +shard interval are inclusive and expressed as the number of milliseconds since +the UNIX epoch (January 1, 1970 00:00:00 UTC). -``` -struct shard_hint { - u64 prefix; - struct tree_leaf leaf; -} -``` +Sharding simplifies log operations because it becomes explicit when a log can be +shutdown. A log must only accept logging requests that have valid shard hints. +A log should only accept logging requests during the predefined shard interval. +Note that _the submitter's shard hint is not a verified timestamp_. The +submitter should set the shard hint as large as possible. If a roughly verified +timestamp is needed, a cosigned tree head can be used. -The log will check that the signed `shard_hint` can be verified using the -submitter's public verification key. The prefix could be anything and may -repeat. This API documentation assumes that the prefix is set to zero. +Without a shard hint, the good Samaritan could log all leaves from an earlier +shard into a newer one. Not only would that defeat the purpose of sharding, but +it would also become a potential denial-of-service vector. -As long as the `shard_hint` signature is not revealed, no one but the submitter -can submit a leaf that the log will accept. Therefore, the good Samaritan -cannot submit all leaves from an earlier shard into a newer one. The -`shard_hint` does not prevent the _legitimate submitter_ from reusing an earlier -submission in a future shard. +The signed message is composed of the selected shard hint and the submitter's +checksum. It must be possible to verify the signature using the specified +signature scheme and the submitter's public verification key. -Note the importance of letting the submitter decide if an entry is logged again -or not. If the log has a rate limiting function, replayed submissions could -deny service in a new shard. In practise we expect submitters to not log a -leaf again. Once an inclusion proof and a cosigned tree head is available, you -have all the necessary proofs. These proofs continue to be valid after the log -shuts down because the verification process is non-interactive. +A key-hash is included in the leaf so that it can be attributed to the signing +entity. A hash, rather than the full public verification key, is used to force +the verifier to locate the appropriate key and make an explicit trust decision. ## Public endpoints Every log has a base URL that identifies it uniquely. The only constraint is @@ -229,10 +231,10 @@ POST /st/v0/add-leaf ``` Input key-value pairs: -- `leaf_checksum`: the checksum that the submitter wants to log in base64. +- `shard_hint`: the shard hint that the submitter selected. +- `checksum`: the checksum that the submitter wants to log in base64. - `signature_scheme`: the signature scheme that the submitter wants to use. -- `tree_leaf_signature`: the submitter's `tree_leaf` signature in base64. -- `shard_hint_signature`: the submitter's `shard_hint` signature in base64. +- `signature`: the submitter's signature over `tree_leaf.message` in base64. - `verification_key`: the submitter's public verification key. It is serialized as described in the corresponding RFC, then base64-encoded. - `domain_hint`: a domain name that indicates where the public verification-key @@ -279,3 +281,13 @@ human-readable error message. The key-hash can be used to identify which witness signed the log's tree head. A key-hash, rather than the full verification key, is used to force the verifier to locate the appropriate key and make an explicit trust decision. + +## Summary of log parameters +- **Public key**: an Ed25519 verification key that can be used to verify the +log's tree head signatures. +- **Log identifier**: the hashed public verification key using SHA256. +- **Shard interval**: the time during which the log accepts logging requests. +The shard interval's start and end are inclusive and expressed as the number of +milliseconds since the UNIX epoch. +- **Base URL**: where the log can be reached over HTTP(S). It is the prefix +before a version-0 specific endpoint. -- cgit v1.2.3 From 83d38bfc5c3b9304953d04a4679658e3c2645367 Mon Sep 17 00:00:00 2001 From: Rasmus Dahlberg Date: Mon, 26 Apr 2021 15:12:57 +0200 Subject: drafty experiment where we would only use percent encoding --- doc/api.md | 206 ++++++++++++++++++++++++++++++++++--------------------------- 1 file changed, 116 insertions(+), 90 deletions(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index b5d54e6..174d2c9 100644 --- a/doc/api.md +++ b/doc/api.md @@ -8,28 +8,18 @@ This is a work-in-progress document that may be moved or modified. ## Overview The log implements an HTTP(S) API: -- Requests that add data to the log use the HTTP POST method. The HTTP content -type is `application/x-www-form-urlencoded`. The posted data are key-value -pairs. Binary data must be base64-encoded. -- Requests that retrieve data from the log use the HTTP GET method. The HTTP -content type is `application/x-www-form-urlencoded`. Input parameters are -key-value pairs. -- Responses are JSON objects. The HTTP content type is `application/json`. -- Error messages are human-readable strings. The HTTP content type is -`text/plain`. - -We decided to use these web formats for requests and responses because the log -is running as an HTTP(S) service. In other words, anyone that interacts with -the log is most likely using these formats already. The other benefit is that -all requests and responses are human-readable. This makes it easier to -understand the protocol, troubleshoot issues, and copy-paste. We favored -compatibility and understandability over a wire-efficient format. - -Note that we are not using JSON for signed and/or logged data. In other words, -a submitter that wishes to distribute log responses to their user base in a -different format may do so. The forced (de)serialization parser on _end-users_ -is a small subset of Trunnel. Trunnel is an "idiot-proof" wire-format that the -Tor project uses. +- Requests that add data to the log use the HTTP POST method. +- Request that retrieve data from the log use the HTTP GET method. +- The HTTP content type is `application/x-www-form-urlencoded` for requests and +responses. This means that all input and output are expressed as key-value +pairs. Binary data must be hex-encoded. + +We decided to use percent encoding for requests and responses because it is a +_simple format_ that is commonly used on the web. We are not using percent +encoding for signed and/or logged data. In other words, a submitter may +distribute log responses to their end-users in a different format that suit +them. The forced (de)serialization parser on _end-users_ is a small subset of +Trunnel. Trunnel is an "idiot-proof" wire-format that the Tor project uses. ## Primitives ### Cryptography @@ -49,6 +39,13 @@ padding. Supporting RSA is suboptimal, but excluding it would make the log useless for many possible adopters. ### Serialization +Log requests and responses are percent encoded. Percent encoding is a smaller +dependency than an alternative parser like JSON. It is comparable to rolling +your own minimalistic line-terminated format. Some input and output data is +binary: cryptographic hashes and signatures. Binary data must be expressed as +hex before percent-encoding it. We decided to use hex as opposed to base64 +because it is simpler, favoring simplicity over efficiency on the wire. + We use the [Trunnel](https://gitweb.torproject.org/trunnel.git) [description language](https://www.seul.org/~nickm/trunnel-manual.html) to define (de)serialization of data structures that need to be signed or inserted into the Merkle tree. Trunnel is more expressive than the @@ -62,13 +59,12 @@ A fair summary of our Trunnel usage is as follows. All integers are 64-bit, unsigned, and in network byte order. A fixed-size byte array is put into the serialization buffer in-order, starting from the first -byte. These basic types are concatenated to form a collection. You should not -need a general-purpose Trunnel (de)serialization parser to work with this -format. If you have one, you may use it though. The main point of using -Trunnel is that it makes a simple format explicit and unambiguous. - -TODO: URL-encode _or_ JSON? I think we should only need one. Always doing HTTP -POST would also ensure that input parameters don't show up in web server logs. +byte. A variable length byte array first declares its length as an integer, +which is then followed by that number of bytes. These basic types are +concatenated to form a collection. You should not need a general-purpose +Trunnel (de)serialization parser to work with this format. If you have one, you +may use it though. The main point of using Trunnel is that it makes a simple +format explicit and unambiguous. #### Merkle tree head Tree heads are signed by the log and its witnesses. It contains a timestamp, a @@ -160,91 +156,124 @@ that it must be a valid HTTP(S) URL that can have the `/st/v0/` suffix appended. For example, a complete endpoint URL could be `https://log.example.com/2021/st/v0/get-signed-tree-head`. +The HTTP status code is 200 OK to indicate success. A different HTTP status +code is used to indicate failure. The log should set the "error" key to a +human-readable value that describes what went wrong. For example, +`error=invalid+signature`, `error=rate+limit+exceeded`, or +`error=unknown+leaf+hash`. + ### get-signed-tree-head ``` GET /st/v0/get-signed-tree-head ``` -Input key-value pairs: -- `type`: either the string "latest", "stable", or "cosigned". - - "latest": ask for the most recent signed tree head. - - "stable": ask for a recent signed tree head that is fixed for some period +Input: +- "type": either the string "latest", "stable", or "cosigned". + - latest: ask for the most recent signed tree head. + - stable: ask for a recent signed tree head that is fixed for some period of time. - - "cosigned": ask for a recent cosigned tree head. - -Output: -- On success: status 200 OK and a signed tree head. The response body is -defined by the following [schema](https://github.com/system-transparency/stfe/blob/design/doc/schema/sth.schema.json). -- On failure: a different status code and a human-readable error message. + - cosigned: ask for a recent cosigned tree head. + +Output on success: +- "timestamp": `tree_head.timestamp` as a human-readable number. +- "tree_size": `tree_head.tree_size` as a human-readable number. +- "root_hash": `tree_head.root_hash` in hex. +- "signature": an Ed25519 signature over `tree_head`. The result is +hex-encoded. +- "key_hash": a hash of the public verification key that can be used to verify +the signature. The public verification key is serialized as in RFC 8032, then +hashed using SHA256. The result is hex-encoded. + +The "signature" and "key_hash" fields may repeat. The first signature +corresponds to the first key hash, the second signature corresponds to the +second key hash, etc. The number of signatures and key hashes must match. ### get-proof-by-hash ``` POST /st/v0/get-proof-by-hash ``` -Input key-value pairs: -- `leaf_hash`: a base64-encoded leaf hash that identifies which `tree_leaf` the +Input: +- "leaf_hash": a hex-encoded leaf hash that identifies which `tree_leaf` the log should prove inclusion for. The leaf hash is computed using the RFC 6962 -hashing strategy. In other words, `H(0x00 | tree_leaf)`. -- `tree_size`: the tree size of a tree head that the proof should be based on. +hashing strategy. In other words, `SHA256(0x00 | tree_leaf)`. +- "tree_size": a human-readable tree size of the tree head that the proof should +be based on. -Output: -- On success: status 200 OK and an inclusion proof. The response body is -defined by the following [schema](https://github.com/system-transparency/stfe/blob/design/doc/schema/inclusion_proof.schema.json). -- On failure: a different status code and a human-readable error message. +Output on success: +- "tree_size": human-readable tree size that the proof is based on. +- "leaf_index": human-readable zero-based index of the leaf that the proof is +based on. +- "inclusion_path": a node hash in hex. + +The "inclusion_path" may be omitted or repeated to represent an inclusion proof +of zero or more node hashes. The order of node hashes follow from our hash +strategy, see RFC 6962. ### get-consistency-proof ``` POST /st/v0/get-consistency-proof ``` -Input key-value pairs: -- `new_size`: the tree size of a newer tree head. -- `old_size`: the tree size of an older tree head that the log should prove is -consistent with the newer tree head. +Input: +- "new_size": human-readable tree size of a newer tree head. +- "old_size": human-readable tree size of an older tree head that the log should +prove is consistent with the newer tree head. + +Output on success: +- "new_size": human-readable tree size of a newer tree head that the proof +is based on. +- "old_size": human-readable tree size of an older tree head that the proof is +based on. +- "consistency_path": a node hash in hex. -Output: -- On success: status 200 OK and a consistency proof. The response body is -defined by the following [schema](https://github.com/system-transparency/stfe/blob/design/doc/schema/consistency_proof.schema.json). -- On failure: a different status code and a human-readable error message. +The "consistency_path" may be omitted or repeated to represent a consistency +proof of zero or more node hashes. The order of node hashes follow from our +hash strategy, see RFC 6962. ### get-leaves ``` POST /st/v0/get-leaves ``` -Input key-value pairs: -- `start_size`: zero-based index of the first leaf to retrieve. -- `end_size`: index of the last leaf to retrieve. +Input: +- "start_size": human-readable index of the first leaf to retrieve. +- "end_size": human-readable index of the last leaf to retrieve. -Output: -- On success: status 200 OK and a list of leaves. The response body is -defined by the following [schema](https://github.com/system-transparency/stfe/blob/design/doc/schema/leaves.schema.json). -- On failure: a different status code and a human-readable error message. +Output on success: +- "shard_hint": `tree_leaf.message.shard_hint` as a human-readable number. +- "checksum": `tree_leaf.message.checksum` in hex. +- "signature_scheme": human-readable number that identifies a signature scheme. +- "signature": `tree_leaf.signature` in hex. +- "key_hash": `tree_leaf.key_hash` in hex. -The log may truncate the list of returned leaves. However, it must not be an -empty list on success. +All fields may be repeated to return more than one leaf. The first value in +each list refers to the first leaf, the second value in each list refers to the +second leaf, etc. The size of each list must match. + +The log may return fewer leaves than requested. At least one leaf must be +returned on HTTP status code 200 OK. ### add-leaf ``` POST /st/v0/add-leaf ``` -Input key-value pairs: -- `shard_hint`: the shard hint that the submitter selected. -- `checksum`: the checksum that the submitter wants to log in base64. -- `signature_scheme`: the signature scheme that the submitter wants to use. -- `signature`: the submitter's signature over `tree_leaf.message` in base64. -- `verification_key`: the submitter's public verification key. It is serialized -as described in the corresponding RFC, then base64-encoded. -- `domain_hint`: a domain name that indicates where the public verification-key -hash can be downloaded in base64. Supported methods: DNS and HTTPS -(TODO: docdoc). - -Output: -- On success: HTTP 200. The log will _try_ to incorporate the submitted leaf -into its Merkle tree. -- On failure: a different status code and a human-readable error message. +Input: +- "shard_hint": human-readable number in the log's shard interval that the +submitter selected. +- "checksum": the cryptographic checksum that the submitter wants to log in hex. +- "signature_scheme": human-readable number that identifies the submitter's +signature scheme. +- "signature": the submitter's signature over `tree_leaf.message`. The result +is hex-encoded. +- "verification_key": the submitter's public verification key. It is serialized +as described in the corresponding RFC. The result is hex-encoded. +- "domain_hint": a domain name that indicates where `tree_leaf.key_hash` can be +retrieved as a DNS TXT resource record in hex. + +Output on success: +- None The submitted entry will not be accepted if the signature is invalid or if the downloaded verification-key hash does not match. The submitted entry may also @@ -260,23 +289,20 @@ Public logging should not be assumed until an inclusion proof is available. An inclusion proof should not be relied upon unless it leads up to a trustworthy signed tree head. Witness cosigning can make a tree head trustworthy. -TODO: the log may allow no `domain_hint`? Especially useful for v0 testing. - ### add-cosignature ``` POST /st/v0/add-cosignature ``` -Input key-value pairs: -- `signature`: a base64-encoded signature over a `tree_head` that is fixed for -some period of time. The cosigning witness retrieves the tree head using the -`get-signed-tree-head` endpoint with the "stable" type. -- `key_hash`: a base64-encoded hash of the public verification key that can be -used to verify the signature. +Input: +- "signature": an Ed25519 signature over `tree_head`. The result is +hex-encoded. +- "key_hash": a hash of the public verification key that can be used to verify +the signature. The public verification key is serialized as in RFC 8032, then +hashed using SHA256. The result is hex-encoded. -Output: -- HTTP status 200 OK on success. Otherwise a different status code and a -human-readable error message. +Output on success: +- None The key-hash can be used to identify which witness signed the log's tree head. A key-hash, rather than the full verification key, is used to force the verifier -- cgit v1.2.3 From 57a600662e98f86fc103f2671a5ec9602e1b7dd0 Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Mon, 3 May 2021 15:38:51 +0200 Subject: Incorporate changes from recent discussions. Remove all RSA support. Motivation: Simpler format for tree_leaf. Replace percent-encoding with headers for indata and key/value in body for outdata. Motivation: ':' is exactly what we want and it works for output data (responses) and not only for input data (requests). Don't POST. Motivation: We don't need the complexity of POST since we don't ever send a lot of data to the log. Split up the get-signed-tree-head endpoint into three separate without input data. Motivation: More explicit API plus easier debugging. Change timestamps and shard hints to use seconds rather than milliseconds. Motivation: time(1) and time(2). --- doc/api.md | 190 ++++++++++++++++++++++++++++++++++--------------------------- 1 file changed, 105 insertions(+), 85 deletions(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index 174d2c9..2d54001 100644 --- a/doc/api.md +++ b/doc/api.md @@ -1,25 +1,29 @@ # System Transparency Logging: API v0 This document describes details of the System Transparency logging API, version 0. The broader picture is not explained here. We assume that you have -read the System Transparency design document. It can be found [here](https://github.com/system-transparency/stfe/blob/design/doc/design.md). +read the System Transparency Logging design document. It can be found [here](https://github.com/system-transparency/stfe/blob/design/doc/design.md). **Warning.** This is a work-in-progress document that may be moved or modified. ## Overview The log implements an HTTP(S) API: -- Requests that add data to the log use the HTTP POST method. -- Request that retrieve data from the log use the HTTP GET method. -- The HTTP content type is `application/x-www-form-urlencoded` for requests and -responses. This means that all input and output are expressed as key-value -pairs. Binary data must be hex-encoded. - -We decided to use percent encoding for requests and responses because it is a -_simple format_ that is commonly used on the web. We are not using percent -encoding for signed and/or logged data. In other words, a submitter may -distribute log responses to their end-users in a different format that suit -them. The forced (de)serialization parser on _end-users_ is a small subset of -Trunnel. Trunnel is an "idiot-proof" wire-format that the Tor project uses. + +- Requests to the log use the HTTP GET method. +- Input data (in requests) and output data (in responses) are + expressed as ASCII-encoded key/value pairs. +- Requests use HTTP request headers for input data while responses use + the HTTP message body for output data. +- Binary data is hex-encoded before being transmitted. + +The motivation for using a text based key/value format for request and +response data is that it's simple to parse. Note that this format is not being +used for the serialization of signed or logged data, where a more +well defined and storage efficient format is desirable. +A submitter may distribute log responses to their end-users in any +format that suits them. The (de)serialization required for +_end-users_ is a small subset of Trunnel. Trunnel is an "idiot-proof" +wire-format in use by the Tor project. ## Primitives ### Cryptography @@ -32,19 +36,14 @@ All other parts that are not Merkle tree related also use SHA256 as the hash function. Using more than one hash function would increases the overall attack surface: two hash functions must be collision resistant instead of one. -We recommend that submitters sign using Ed25519. We also support RSA with -[deterministic](https://tools.ietf.org/html/rfc8017#section-8.2) -or [probabilistic](https://tools.ietf.org/html/rfc8017#section-8.1) -padding. Supporting RSA is suboptimal, but excluding it would make the log -useless for many possible adopters. - ### Serialization -Log requests and responses are percent encoded. Percent encoding is a smaller -dependency than an alternative parser like JSON. It is comparable to rolling -your own minimalistic line-terminated format. Some input and output data is -binary: cryptographic hashes and signatures. Binary data must be expressed as -hex before percent-encoding it. We decided to use hex as opposed to base64 -because it is simpler, favoring simplicity over efficiency on the wire. +Log requests and responses are transmitted as ASCII-encoded key/value +pairs, for a smaller dependency than an alternative parser like JSON. +Some input and output data is binary: cryptographic hashes and +signatures. Binary data must be Base16-encoded, also known as hex +encoding. Using hex as opposed to base64 is motivated by it being +simpler, favoring ease of decoding and encoding over efficiency on the +wire. We use the [Trunnel](https://gitweb.torproject.org/trunnel.git) [description language](https://www.seul.org/~nickm/trunnel-manual.html) to define (de)serialization of data structures that need to be signed or @@ -57,9 +56,9 @@ can be generated in C and Go. A fair summary of our Trunnel usage is as follows. -All integers are 64-bit, unsigned, and in network byte order. A fixed-size byte -array is put into the serialization buffer in-order, starting from the first -byte. A variable length byte array first declares its length as an integer, +All integers are 64-bit, unsigned, and in network byte order. Fixed-size byte +arrays are put into the serialization buffer in-order, starting from the first +byte. Variable length byte arrays first declare their length as an integer, which is then followed by that number of bytes. These basic types are concatenated to form a collection. You should not need a general-purpose Trunnel (de)serialization parser to work with this format. If you have one, you @@ -70,7 +69,7 @@ format explicit and unambiguous. Tree heads are signed by the log and its witnesses. It contains a timestamp, a tree size, and a root hash. The timestamp is included so that monitors can ensure _liveliness_. It is the time since the UNIX epoch (January 1, 1970 -00:00:00 UTC) in milliseconds. The tree size specifies the current number of +00:00:00 UTC) in seconds. The tree size specifies the current number of leaves. The root hash fixes the structure and content of the Merkle tree. ``` @@ -86,50 +85,25 @@ cosign a tree head if it is inconsistent with prior history or if the timestamp is backdated or future-dated more than 12 hours. #### Merkle tree leaf -The log supports a single leaf type. It contains a message, a signature scheme, -a signature that the submitter computed over the message, and a hash of the +The log supports a single leaf type. It contains a shard hint, a checksum over whatever the submitter wants to log a checksum for, +a signature that the submitter computed over the shard hint and the checksum, and a hash of the public verification key that can be used to verify the signature. ``` -const SIGNATURE_SCHEME_ED25519 = 1; // RFC 8032 -const SIGNATURE_SCHEME_RSASSA_PKCS1_V1_5 = 2; // RFC 8017 -const SIGNATURE_SCHEME_RSASSA_PSS = 3; // RFC 8017 - -struct signature_ed25519 { - u8 signature[32]; -}; - -struct signature_rsassa { - u64 num_bytes IN [ 256, 384, 512 ]; - u8 signature[num_bytes]; -}; - -struct message { +struct tree_leaf { u64 shard_hint; u8 checksum[32]; -}; - -struct tree_leaf { - struct message message; - u64 signature_scheme IN [ - SIGNATURE_SCHEME_ED25519, - SIGNATURE_SCHEME_RSASSA_PKCS1_V1_5, - SIGNATURE_SCHEME_RSASSA_PSS, - ]; - union signature[signature_scheme] { - SIGNATURE_SCHEME_ED25519: struct signature_ed25519 ed25519; - default: struct signature_rsassa rsassa; - } + u8 signature[32]; u8 key_hash[32]; } ``` -Unlike X.509 certificates that already have validity ranges, a checksum does not -have any such information. Therefore, we require that the submitter selects a +Unlike X.509 certificates which already have validity ranges, a checksum does not +carry any such information. Therefore, we require that the submitter selects a _shard hint_. The selected shard hint must be in the log's _shard interval_. A shard interval is defined by a start time and an end time. Both ends of the -shard interval are inclusive and expressed as the number of milliseconds since -the UNIX epoch (January 1, 1970 00:00:00 UTC). +shard interval are inclusive and expressed as the number of seconds since +the UNIX epoch (January 1, 1970 00:00 UTC). Sharding simplifies log operations because it becomes explicit when a log can be shutdown. A log must only accept logging requests that have valid shard hints. @@ -143,11 +117,15 @@ shard into a newer one. Not only would that defeat the purpose of sharding, but it would also become a potential denial-of-service vector. The signed message is composed of the selected shard hint and the submitter's -checksum. It must be possible to verify the signature using the specified -signature scheme and the submitter's public verification key. +checksum. It must be possible to verify the signature using the +submitter's public verification key. -A key-hash is included in the leaf so that it can be attributed to the signing -entity. A hash, rather than the full public verification key, is used to force +Note that the way `shard_hint` and `chekcsum` are serialized with +regards to signing differs from how they're being transmitted to the +log. + +A key hash is included in the leaf so that the leaf can be attributed to the +submitter. A hash, rather than the full public verification key, is used to motivate the verifier to locate the appropriate key and make an explicit trust decision. ## Public endpoints @@ -162,32 +140,76 @@ human-readable value that describes what went wrong. For example, `error=invalid+signature`, `error=rate+limit+exceeded`, or `error=unknown+leaf+hash`. -### get-signed-tree-head +### get-tree-head-cosigned +Returns the latest cosigned tree head. Used by ordinary users of the log. + ``` -GET /st/v0/get-signed-tree-head +GET /st/v0/get-tree-head-cosigned ``` Input: -- "type": either the string "latest", "stable", or "cosigned". - - latest: ask for the most recent signed tree head. - - stable: ask for a recent signed tree head that is fixed for some period - of time. - - cosigned: ask for a recent cosigned tree head. +- None Output on success: -- "timestamp": `tree_head.timestamp` as a human-readable number. -- "tree_size": `tree_head.tree_size` as a human-readable number. -- "root_hash": `tree_head.root_hash` in hex. -- "signature": an Ed25519 signature over `tree_head`. The result is -hex-encoded. +- "timestamp": `tree_head.timestamp` ASCII-encoded decimal number, seconds since the UNIX epoch. +- "tree_size": `tree_head.tree_size` ASCII-encoded decimal number. +- "root_hash": `tree_head.root_hash` hex-encoded. +- "signature": hex-encoded Ed25519 signature over `tree_head` serialzed as described in section `Merkle tree head`. - "key_hash": a hash of the public verification key that can be used to verify -the signature. The public verification key is serialized as in RFC 8032, then -hashed using SHA256. The result is hex-encoded. +the signature. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then +hashed using SHA256. The hash value is hex-encoded. The "signature" and "key_hash" fields may repeat. The first signature corresponds to the first key hash, the second signature corresponds to the second key hash, etc. The number of signatures and key hashes must match. +### get-tree-head-to-sign +Returns the latest tree head to be signed by log witnesses. Used by +witnesses. + +``` +GET /st/v0/get-tree-head-to-sign +``` + +Input: +- None + +Output on success: +- "timestamp": `tree_head.timestamp` ASCII-encoded decimal number, seconds since the UNIX epoch. +- "tree_size": `tree_head.tree_size` ASCII-encoded decimal number. +- "root_hash": `tree_head.root_hash` hex-encoded. +- "signature": hex-encoded Ed25519 signature over `tree_head` serialzed as described in section `Merkle tree head`. +- "key_hash": a hash of the public verification key that can be used to verify +the signature. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then +hashed using SHA256. The hash value is hex-encoded. + +There is exactly one `signature` and one `key_hash` field. The +`key_hash` refers to the log's signing key. + + +### get-tree-head-latest +Returns the latest tree head, signed only by the log. Used for debug. + +``` +GET /st/v0/get-tree-head-latest +``` + +Input: +- None + +Output on success: +- "timestamp": `tree_head.timestamp` ASCII-encoded decimal number, seconds since the UNIX epoch. +- "tree_size": `tree_head.tree_size` ASCII-encoded decimal number. +- "root_hash": `tree_head.root_hash` hex-encoded. +- "signature": hex-encoded Ed25519 signature over `tree_head` serialzed as described in section `Merkle tree head`. +- "key_hash": a hash of the public verification key that can be used to verify +the signature. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then +hashed using SHA256. The hash value is hex-encoded. + +There is exactly one `signature` and one `key_hash` field. The +`key_hash` refers to the log's signing key. + + ### get-proof-by-hash ``` POST /st/v0/get-proof-by-hash @@ -260,11 +282,9 @@ POST /st/v0/add-leaf ``` Input: -- "shard_hint": human-readable number in the log's shard interval that the +- "shard_hint": human-readable decimal number in the log's shard interval that the submitter selected. -- "checksum": the cryptographic checksum that the submitter wants to log in hex. -- "signature_scheme": human-readable number that identifies the submitter's -signature scheme. +- "checksum": the cryptographic checksum that the submitter wants to log in hex. note: fixed length 64 bytes, validated by the server somehow - "signature": the submitter's signature over `tree_leaf.message`. The result is hex-encoded. - "verification_key": the submitter's public verification key. It is serialized @@ -314,6 +334,6 @@ log's tree head signatures. - **Log identifier**: the hashed public verification key using SHA256. - **Shard interval**: the time during which the log accepts logging requests. The shard interval's start and end are inclusive and expressed as the number of -milliseconds since the UNIX epoch. +seconds since the UNIX epoch. - **Base URL**: where the log can be reached over HTTP(S). It is the prefix before a version-0 specific endpoint. -- cgit v1.2.3 From c82c4e1266c5e8fbe08cb0f6140caea2723ef205 Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 4 May 2021 14:51:36 +0200 Subject: be explicit with key type; define struct message, for tree_leaf Specify who's verification key -- log, witness or submitter. Move shard_hint and checksum in tree_leaf into its own struct, for a more explicit definition of what to be signed. --- doc/api.md | 59 ++++++++++++++++++++++++++++++++++------------------------- 1 file changed, 34 insertions(+), 25 deletions(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index 2d54001..0fa445a 100644 --- a/doc/api.md +++ b/doc/api.md @@ -87,14 +87,18 @@ is backdated or future-dated more than 12 hours. #### Merkle tree leaf The log supports a single leaf type. It contains a shard hint, a checksum over whatever the submitter wants to log a checksum for, a signature that the submitter computed over the shard hint and the checksum, and a hash of the -public verification key that can be used to verify the signature. +submitter's public verification key, that can be used to verify the signature. ``` +struct message { + u64 shard_hint; + u8 checksum[32]; +}; + struct tree_leaf { - u64 shard_hint; - u8 checksum[32]; - u8 signature[32]; - u8 key_hash[32]; + struct message; + u8 signature_over_message[32]; + u8 key_hash[32]; } ``` @@ -116,17 +120,20 @@ Without a shard hint, the good Samaritan could log all leaves from an earlier shard into a newer one. Not only would that defeat the purpose of sharding, but it would also become a potential denial-of-service vector. -The signed message is composed of the selected shard hint and the submitter's -checksum. It must be possible to verify the signature using the -submitter's public verification key. +The signed message is composed of the chosen `shard_hint` and the +submitter's `checksum`. It must be possible to verify +`signature_over_message` using the submitter's public verification +key. -Note that the way `shard_hint` and `chekcsum` are serialized with +Note that the way `shard_hint` and `checksum` are serialized with regards to signing differs from how they're being transmitted to the log. -A key hash is included in the leaf so that the leaf can be attributed to the -submitter. A hash, rather than the full public verification key, is used to motivate -the verifier to locate the appropriate key and make an explicit trust decision. +A `key_hash` of the key used for signing `message` is included in +`tree_leaf` so that the leaf can be attributed to the submitter. A +hash, rather than the full public key, is used to motivate the +verifier to locate the appropriate key and make an explicit trust +decision. ## Public endpoints Every log has a base URL that identifies it uniquely. The only constraint is @@ -155,8 +162,8 @@ Output on success: - "tree_size": `tree_head.tree_size` ASCII-encoded decimal number. - "root_hash": `tree_head.root_hash` hex-encoded. - "signature": hex-encoded Ed25519 signature over `tree_head` serialzed as described in section `Merkle tree head`. -- "key_hash": a hash of the public verification key that can be used to verify -the signature. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then +- "key_hash": a hash of the public verification key (belonging to either the log or to one of its witnesses), which can be used to verify +the most recent `signature`. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then hashed using SHA256. The hash value is hex-encoded. The "signature" and "key_hash" fields may repeat. The first signature @@ -179,16 +186,16 @@ Output on success: - "tree_size": `tree_head.tree_size` ASCII-encoded decimal number. - "root_hash": `tree_head.root_hash` hex-encoded. - "signature": hex-encoded Ed25519 signature over `tree_head` serialzed as described in section `Merkle tree head`. -- "key_hash": a hash of the public verification key that can be used to verify -the signature. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then +- "key_hash": a hash of the log's public verification key, which can be used to verify +`signature`. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then hashed using SHA256. The hash value is hex-encoded. There is exactly one `signature` and one `key_hash` field. The -`key_hash` refers to the log's signing key. +`key_hash` refers to the log's public verification key. ### get-tree-head-latest -Returns the latest tree head, signed only by the log. Used for debug. +Returns the latest tree head, signed only by the log. Used for debugging purposes. ``` GET /st/v0/get-tree-head-latest @@ -202,12 +209,13 @@ Output on success: - "tree_size": `tree_head.tree_size` ASCII-encoded decimal number. - "root_hash": `tree_head.root_hash` hex-encoded. - "signature": hex-encoded Ed25519 signature over `tree_head` serialzed as described in section `Merkle tree head`. -- "key_hash": a hash of the public verification key that can be used to verify -the signature. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then -hashed using SHA256. The hash value is hex-encoded. +- "key_hash": a hash of the log's public verification key that can be +used to verify `signature`. The key is encoded as defined in +[RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), +and then hashed using SHA256. The hash value is hex-encoded. There is exactly one `signature` and one `key_hash` field. The -`key_hash` refers to the log's signing key. +`key_hash` refers to the log's public verification key. ### get-proof-by-hash @@ -317,9 +325,10 @@ POST /st/v0/add-cosignature Input: - "signature": an Ed25519 signature over `tree_head`. The result is hex-encoded. -- "key_hash": a hash of the public verification key that can be used to verify -the signature. The public verification key is serialized as in RFC 8032, then -hashed using SHA256. The result is hex-encoded. +- "key_hash": a hash of the witness' public verification key that can be used +to verify the signature. The key is encoded as defined in [RFC 8032, +section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and +then hashed using SHA256. The hash value is hex-encoded. Output on success: - None -- cgit v1.2.3 From a30eb85272010ffbcfd3fb2c6932dc2f15d596c1 Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 4 May 2021 14:53:00 +0200 Subject: get rid of the underspecified term "ordinary users" --- doc/api.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index 0fa445a..976c167 100644 --- a/doc/api.md +++ b/doc/api.md @@ -148,7 +148,8 @@ human-readable value that describes what went wrong. For example, `error=unknown+leaf+hash`. ### get-tree-head-cosigned -Returns the latest cosigned tree head. Used by ordinary users of the log. +Returns the latest cosigned tree head. Used together with +`get-proof-by-hash` and `get-consistency-proof` for verifying the log. ``` GET /st/v0/get-tree-head-cosigned -- cgit v1.2.3 From c163da97d32a291a1c913800c926d7758c641c6e Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 4 May 2021 14:53:36 +0200 Subject: specify serialization of key --- doc/api.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index 976c167..362dc46 100644 --- a/doc/api.md +++ b/doc/api.md @@ -296,8 +296,7 @@ submitter selected. - "checksum": the cryptographic checksum that the submitter wants to log in hex. note: fixed length 64 bytes, validated by the server somehow - "signature": the submitter's signature over `tree_leaf.message`. The result is hex-encoded. -- "verification_key": the submitter's public verification key. It is serialized -as described in the corresponding RFC. The result is hex-encoded. +- "verification_key": the submitter's public verification key. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2). The result is hex-encoded. - "domain_hint": a domain name that indicates where `tree_leaf.key_hash` can be retrieved as a DNS TXT resource record in hex. -- cgit v1.2.3 From 9ee06539685bdcaea84b3daede5354d83264c1e4 Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 4 May 2021 16:08:22 +0200 Subject: explain how input and output data are sent This is the "header in, body out" idea written up. We might change to a "POST body in, receive body out" scheme with "Content-Type: application/stfe" if we can decide that POST is not a terrible idea after all. --- doc/api.md | 10 ++++++++++ 1 file changed, 10 insertions(+) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index 362dc46..c747aa2 100644 --- a/doc/api.md +++ b/doc/api.md @@ -141,6 +141,16 @@ that it must be a valid HTTP(S) URL that can have the `/st/v0/` suffix appended. For example, a complete endpoint URL could be `https://log.example.com/2021/st/v0/get-signed-tree-head`. +Input data (in requests) is sent as ASCII key/value pairs as HTTP +entity headers, with their keys prefixed with the string +`stlog-`. Example: For sending `treee_size=4711` as input a client +would send the HTTP header `stlog-tree_size: 4711`. + +Output data (in replies) is sent in the HTTP message body in the same +format as the input data, i.e. as ASCII key/value pairs on the format +`Key: Value`. Example: For sending `tree_size=4711` as output a log +would send an HTTP message body consisting of `stlog-tree_size: 4711`. + The HTTP status code is 200 OK to indicate success. A different HTTP status code is used to indicate failure. The log should set the "error" key to a human-readable value that describes what went wrong. For example, -- cgit v1.2.3 From 044dc540aaf3950695602e849906969aed7e6a46 Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 4 May 2021 16:15:59 +0200 Subject: be consistent with "request" vs "entity" headers --- doc/api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index c747aa2..638b753 100644 --- a/doc/api.md +++ b/doc/api.md @@ -12,7 +12,7 @@ The log implements an HTTP(S) API: - Requests to the log use the HTTP GET method. - Input data (in requests) and output data (in responses) are expressed as ASCII-encoded key/value pairs. -- Requests use HTTP request headers for input data while responses use +- Requests use HTTP entity headers for input data while responses use the HTTP message body for output data. - Binary data is hex-encoded before being transmitted. -- cgit v1.2.3 From 3b5b4429d94e142ee12af7eb5f89b49997b72237 Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 4 May 2021 16:22:32 +0200 Subject: whitespace changes --- doc/api.md | 323 ++++++++++++++++++++++++++++++++++--------------------------- 1 file changed, 181 insertions(+), 142 deletions(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index 638b753..5b7cb19 100644 --- a/doc/api.md +++ b/doc/api.md @@ -1,7 +1,9 @@ # System Transparency Logging: API v0 -This document describes details of the System Transparency logging API, -version 0. The broader picture is not explained here. We assume that you have -read the System Transparency Logging design document. It can be found [here](https://github.com/system-transparency/stfe/blob/design/doc/design.md). +This document describes details of the System Transparency logging +API, version 0. The broader picture is not explained here. We assume +that you have read the System Transparency Logging design document. +It can be found +[here](https://github.com/system-transparency/stfe/blob/design/doc/design.md). **Warning.** This is a work-in-progress document that may be moved or modified. @@ -17,24 +19,28 @@ The log implements an HTTP(S) API: - Binary data is hex-encoded before being transmitted. The motivation for using a text based key/value format for request and -response data is that it's simple to parse. Note that this format is not being -used for the serialization of signed or logged data, where a more -well defined and storage efficient format is desirable. -A submitter may distribute log responses to their end-users in any +response data is that it's simple to parse. Note that this format is +not being used for the serialization of signed or logged data, where a +more well defined and storage efficient format is desirable. A +submitter may distribute log responses to their end-users in any format that suits them. The (de)serialization required for _end-users_ is a small subset of Trunnel. Trunnel is an "idiot-proof" wire-format in use by the Tor project. ## Primitives ### Cryptography -The log uses the same Merkle tree hash strategy as [RFC 6962, §2](https://tools.ietf.org/html/rfc6962#section-2). -The hash functions must be [SHA256](https://csrc.nist.gov/csrc/media/publications/fips/180/4/final/documents/fips180-4-draft-aug2014.pdf). -The log must sign tree heads using [Ed25519](https://tools.ietf.org/html/rfc8032). -The log's witnesses must also sign tree heads using Ed25519. - -All other parts that are not Merkle tree related also use SHA256 as the hash -function. Using more than one hash function would increases the overall attack -surface: two hash functions must be collision resistant instead of one. +The log uses the same Merkle tree hash strategy as +[RFC 6962,§2](https://tools.ietf.org/html/rfc6962#section-2). +The hash functions must be +[SHA256](https://csrc.nist.gov/csrc/media/publications/fips/180/4/final/documents/fips180-4-draft-aug2014.pdf). +The log must sign tree heads using +[Ed25519](https://tools.ietf.org/html/rfc8032). The log's witnesses +must also sign tree heads using Ed25519. + +All other parts that are not Merkle tree related also use SHA256 as +the hash function. Using more than one hash function would increases +the overall attack surface: two hash functions must be collision +resistant instead of one. ### Serialization Log requests and responses are transmitted as ASCII-encoded key/value @@ -45,32 +51,36 @@ encoding. Using hex as opposed to base64 is motivated by it being simpler, favoring ease of decoding and encoding over efficiency on the wire. -We use the [Trunnel](https://gitweb.torproject.org/trunnel.git) [description language](https://www.seul.org/~nickm/trunnel-manual.html) +We use the +[Trunnel](https://gitweb.torproject.org/trunnel.git) [description language](https://www.seul.org/~nickm/trunnel-manual.html) to define (de)serialization of data structures that need to be signed or inserted into the Merkle tree. Trunnel is more expressive than the [SSH wire format](https://tools.ietf.org/html/rfc4251#section-5). -It is about as expressive as the [TLS presentation language](https://tools.ietf.org/html/rfc8446#section-3). -A notable difference is that Trunnel supports integer constraints. The Trunnel -language is also readable by humans _and_ machines. "Obviously correct code" -can be generated in C and Go. +It is about as expressive as the +[TLS presentation language](https://tools.ietf.org/html/rfc8446#section-3). +A notable difference is that Trunnel supports integer constraints. +The Trunnel language is also readable by humans _and_ machines. +"Obviously correct code" can be generated in C and Go. A fair summary of our Trunnel usage is as follows. -All integers are 64-bit, unsigned, and in network byte order. Fixed-size byte -arrays are put into the serialization buffer in-order, starting from the first -byte. Variable length byte arrays first declare their length as an integer, -which is then followed by that number of bytes. These basic types are -concatenated to form a collection. You should not need a general-purpose -Trunnel (de)serialization parser to work with this format. If you have one, you -may use it though. The main point of using Trunnel is that it makes a simple -format explicit and unambiguous. +All integers are 64-bit, unsigned, and in network byte order. +Fixed-size byte arrays are put into the serialization buffer in-order, +starting from the first byte. Variable length byte arrays first +declare their length as an integer, which is then followed by that +number of bytes. These basic types are concatenated to form a +collection. You should not need a general-purpose Trunnel +(de)serialization parser to work with this format. If you have one, +you may use it though. The main point of using Trunnel is that it +makes a simple format explicit and unambiguous. #### Merkle tree head -Tree heads are signed by the log and its witnesses. It contains a timestamp, a -tree size, and a root hash. The timestamp is included so that monitors can -ensure _liveliness_. It is the time since the UNIX epoch (January 1, 1970 -00:00:00 UTC) in seconds. The tree size specifies the current number of -leaves. The root hash fixes the structure and content of the Merkle tree. +Tree heads are signed by the log and its witnesses. It contains a +timestamp, a tree size, and a root hash. The timestamp is included so +that monitors can ensure _liveliness_. It is the time since the UNIX +epoch (January 1, 1970 00:00:00 UTC) in seconds. The tree size +specifies the current number of leaves. The root hash fixes the +structure and content of the Merkle tree. ``` struct tree_head { @@ -80,14 +90,16 @@ struct tree_head { }; ``` -The serialized tree head must be signed using Ed25519. A witness must not -cosign a tree head if it is inconsistent with prior history or if the timestamp -is backdated or future-dated more than 12 hours. +The serialized tree head must be signed using Ed25519. A witness must +not cosign a tree head if it is inconsistent with prior history or if +the timestamp is backdated or future-dated more than 12 hours. #### Merkle tree leaf -The log supports a single leaf type. It contains a shard hint, a checksum over whatever the submitter wants to log a checksum for, -a signature that the submitter computed over the shard hint and the checksum, and a hash of the -submitter's public verification key, that can be used to verify the signature. +The log supports a single leaf type. It contains a shard hint, a +checksum over whatever the submitter wants to log a checksum for, a +signature that the submitter computed over the shard hint and the +checksum, and a hash of the submitter's public verification key, that +can be used to verify the signature. ``` struct message { @@ -102,23 +114,26 @@ struct tree_leaf { } ``` -Unlike X.509 certificates which already have validity ranges, a checksum does not -carry any such information. Therefore, we require that the submitter selects a -_shard hint_. The selected shard hint must be in the log's _shard interval_. A -shard interval is defined by a start time and an end time. Both ends of the -shard interval are inclusive and expressed as the number of seconds since -the UNIX epoch (January 1, 1970 00:00 UTC). - -Sharding simplifies log operations because it becomes explicit when a log can be -shutdown. A log must only accept logging requests that have valid shard hints. -A log should only accept logging requests during the predefined shard interval. -Note that _the submitter's shard hint is not a verified timestamp_. The -submitter should set the shard hint as large as possible. If a roughly verified -timestamp is needed, a cosigned tree head can be used. - -Without a shard hint, the good Samaritan could log all leaves from an earlier -shard into a newer one. Not only would that defeat the purpose of sharding, but -it would also become a potential denial-of-service vector. +Unlike X.509 certificates which already have validity ranges, a +checksum does not carry any such information. Therefore, we require +that the submitter selects a _shard hint_. The selected shard hint +must be in the log's _shard interval_. A shard interval is defined by +a start time and an end time. Both ends of the shard interval are +inclusive and expressed as the number of seconds since the UNIX epoch +(January 1, 1970 00:00 UTC). + +Sharding simplifies log operations because it becomes explicit when a +log can be shutdown. A log must only accept logging requests that +have valid shard hints. A log should only accept logging requests +during the predefined shard interval. Note that _the submitter's +shard hint is not a verified timestamp_. The submitter should set the +shard hint as large as possible. If a roughly verified timestamp is +needed, a cosigned tree head can be used. + +Without a shard hint, the good Samaritan could log all leaves from an +earlier shard into a newer one. Not only would that defeat the +purpose of sharding, but it would also become a potential +denial-of-service vector. The signed message is composed of the chosen `shard_hint` and the submitter's `checksum`. It must be possible to verify @@ -136,9 +151,10 @@ verifier to locate the appropriate key and make an explicit trust decision. ## Public endpoints -Every log has a base URL that identifies it uniquely. The only constraint is -that it must be a valid HTTP(S) URL that can have the `/st/v0/` suffix -appended. For example, a complete endpoint URL could be +Every log has a base URL that identifies it uniquely. The only +constraint is that it must be a valid HTTP(S) URL that can have the +`/st/v0/` suffix appended. For example, a complete endpoint +URL could be `https://log.example.com/2021/st/v0/get-signed-tree-head`. Input data (in requests) is sent as ASCII key/value pairs as HTTP @@ -151,11 +167,11 @@ format as the input data, i.e. as ASCII key/value pairs on the format `Key: Value`. Example: For sending `tree_size=4711` as output a log would send an HTTP message body consisting of `stlog-tree_size: 4711`. -The HTTP status code is 200 OK to indicate success. A different HTTP status -code is used to indicate failure. The log should set the "error" key to a -human-readable value that describes what went wrong. For example, -`error=invalid+signature`, `error=rate+limit+exceeded`, or -`error=unknown+leaf+hash`. +The HTTP status code is 200 OK to indicate success. A different HTTP +status code is used to indicate failure. The log should set the +"error" key to a human-readable value that describes what went wrong. +For example, `error=invalid+signature`, `error=rate+limit+exceeded`, +or `error=unknown+leaf+hash`. ### get-tree-head-cosigned Returns the latest cosigned tree head. Used together with @@ -169,17 +185,22 @@ Input: - None Output on success: -- "timestamp": `tree_head.timestamp` ASCII-encoded decimal number, seconds since the UNIX epoch. +- "timestamp": `tree_head.timestamp` ASCII-encoded decimal number, + seconds since the UNIX epoch. - "tree_size": `tree_head.tree_size` ASCII-encoded decimal number. - "root_hash": `tree_head.root_hash` hex-encoded. -- "signature": hex-encoded Ed25519 signature over `tree_head` serialzed as described in section `Merkle tree head`. -- "key_hash": a hash of the public verification key (belonging to either the log or to one of its witnesses), which can be used to verify -the most recent `signature`. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then -hashed using SHA256. The hash value is hex-encoded. +- "signature": hex-encoded Ed25519 signature over `tree_head` + serialzed as described in section `Merkle tree head`. +- "key_hash": a hash of the public verification key (belonging to + either the log or to one of its witnesses), which can be used to + verify the most recent `signature`. The key is encoded as defined + in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), + and then hashed using SHA256. The hash value is hex-encoded. The "signature" and "key_hash" fields may repeat. The first signature -corresponds to the first key hash, the second signature corresponds to the -second key hash, etc. The number of signatures and key hashes must match. +corresponds to the first key hash, the second signature corresponds to +the second key hash, etc. The number of signatures and key hashes +must match. ### get-tree-head-to-sign Returns the latest tree head to be signed by log witnesses. Used by @@ -193,20 +214,24 @@ Input: - None Output on success: -- "timestamp": `tree_head.timestamp` ASCII-encoded decimal number, seconds since the UNIX epoch. +- "timestamp": `tree_head.timestamp` ASCII-encoded decimal number, + seconds since the UNIX epoch. - "tree_size": `tree_head.tree_size` ASCII-encoded decimal number. - "root_hash": `tree_head.root_hash` hex-encoded. -- "signature": hex-encoded Ed25519 signature over `tree_head` serialzed as described in section `Merkle tree head`. -- "key_hash": a hash of the log's public verification key, which can be used to verify -`signature`. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then -hashed using SHA256. The hash value is hex-encoded. +- "signature": hex-encoded Ed25519 signature over `tree_head` + serialzed as described in section `Merkle tree head`. +- "key_hash": a hash of the log's public verification key, which can + be used to verify `signature`. The key is encoded as defined in + [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), + and then hashed using SHA256. The hash value is hex-encoded. There is exactly one `signature` and one `key_hash` field. The `key_hash` refers to the log's public verification key. ### get-tree-head-latest -Returns the latest tree head, signed only by the log. Used for debugging purposes. +Returns the latest tree head, signed only by the log. Used for +debugging purposes. ``` GET /st/v0/get-tree-head-latest @@ -216,14 +241,16 @@ Input: - None Output on success: -- "timestamp": `tree_head.timestamp` ASCII-encoded decimal number, seconds since the UNIX epoch. +- "timestamp": `tree_head.timestamp` ASCII-encoded decimal number, + seconds since the UNIX epoch. - "tree_size": `tree_head.tree_size` ASCII-encoded decimal number. - "root_hash": `tree_head.root_hash` hex-encoded. -- "signature": hex-encoded Ed25519 signature over `tree_head` serialzed as described in section `Merkle tree head`. +- "signature": hex-encoded Ed25519 signature over `tree_head` + serialzed as described in section `Merkle tree head`. - "key_hash": a hash of the log's public verification key that can be -used to verify `signature`. The key is encoded as defined in -[RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), -and then hashed using SHA256. The hash value is hex-encoded. + used to verify `signature`. The key is encoded as defined in + [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), + and then hashed using SHA256. The hash value is hex-encoded. There is exactly one `signature` and one `key_hash` field. The `key_hash` refers to the log's public verification key. @@ -235,21 +262,22 @@ POST /st/v0/get-proof-by-hash ``` Input: -- "leaf_hash": a hex-encoded leaf hash that identifies which `tree_leaf` the -log should prove inclusion for. The leaf hash is computed using the RFC 6962 -hashing strategy. In other words, `SHA256(0x00 | tree_leaf)`. -- "tree_size": a human-readable tree size of the tree head that the proof should -be based on. +- "leaf_hash": a hex-encoded leaf hash that identifies which + `tree_leaf` the log should prove inclusion for. The leaf hash is + computed using the RFC 6962 hashing strategy. In other words, + `SHA256(0x00 | tree_leaf)`. +- "tree_size": a human-readable tree size of the tree head that the + proof should be based on. Output on success: - "tree_size": human-readable tree size that the proof is based on. -- "leaf_index": human-readable zero-based index of the leaf that the proof is -based on. +- "leaf_index": human-readable zero-based index of the leaf that the + proof is based on. - "inclusion_path": a node hash in hex. -The "inclusion_path" may be omitted or repeated to represent an inclusion proof -of zero or more node hashes. The order of node hashes follow from our hash -strategy, see RFC 6962. +The "inclusion_path" may be omitted or repeated to represent an +inclusion proof of zero or more node hashes. The order of node hashes +follow from our hash strategy, see RFC 6962. ### get-consistency-proof ``` @@ -258,19 +286,19 @@ POST /st/v0/get-consistency-proof Input: - "new_size": human-readable tree size of a newer tree head. -- "old_size": human-readable tree size of an older tree head that the log should -prove is consistent with the newer tree head. +- "old_size": human-readable tree size of an older tree head that the + log should prove is consistent with the newer tree head. Output on success: -- "new_size": human-readable tree size of a newer tree head that the proof -is based on. -- "old_size": human-readable tree size of an older tree head that the proof is -based on. +- "new_size": human-readable tree size of a newer tree head that the + proof is based on. +- "old_size": human-readable tree size of an older tree head that the + proof is based on. - "consistency_path": a node hash in hex. -The "consistency_path" may be omitted or repeated to represent a consistency -proof of zero or more node hashes. The order of node hashes follow from our -hash strategy, see RFC 6962. +The "consistency_path" may be omitted or repeated to represent a +consistency proof of zero or more node hashes. The order of node +hashes follow from our hash strategy, see RFC 6962. ### get-leaves ``` @@ -282,18 +310,21 @@ Input: - "end_size": human-readable index of the last leaf to retrieve. Output on success: -- "shard_hint": `tree_leaf.message.shard_hint` as a human-readable number. +- "shard_hint": `tree_leaf.message.shard_hint` as a human-readable + number. - "checksum": `tree_leaf.message.checksum` in hex. -- "signature_scheme": human-readable number that identifies a signature scheme. +- "signature_scheme": human-readable number that identifies a + signature scheme. - "signature": `tree_leaf.signature` in hex. - "key_hash": `tree_leaf.key_hash` in hex. -All fields may be repeated to return more than one leaf. The first value in -each list refers to the first leaf, the second value in each list refers to the -second leaf, etc. The size of each list must match. +All fields may be repeated to return more than one leaf. The first +value in each list refers to the first leaf, the second value in each +list refers to the second leaf, etc. The size of each list must +match. -The log may return fewer leaves than requested. At least one leaf must be -returned on HTTP status code 200 OK. +The log may return fewer leaves than requested. At least one leaf +must be returned on HTTP status code 200 OK. ### add-leaf ``` @@ -301,31 +332,38 @@ POST /st/v0/add-leaf ``` Input: -- "shard_hint": human-readable decimal number in the log's shard interval that the -submitter selected. -- "checksum": the cryptographic checksum that the submitter wants to log in hex. note: fixed length 64 bytes, validated by the server somehow -- "signature": the submitter's signature over `tree_leaf.message`. The result -is hex-encoded. -- "verification_key": the submitter's public verification key. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2). The result is hex-encoded. -- "domain_hint": a domain name that indicates where `tree_leaf.key_hash` can be -retrieved as a DNS TXT resource record in hex. +- "shard_hint": human-readable decimal number in the log's shard + interval that the submitter selected. +- "checksum": the cryptographic checksum that the submitter wants to + log in hex. note: fixed length 64 bytes, validated by the server + somehow +- "signature": the submitter's signature over `tree_leaf.message`. + The result is hex-encoded. +- "verification_key": the submitter's public verification key. The + key is encoded as defined in + [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2). The result is hex-encoded. +- "domain_hint": a domain name that indicates where + `tree_leaf.key_hash` can be retrieved as a DNS TXT resource record + in hex. Output on success: - None -The submitted entry will not be accepted if the signature is invalid or if the -downloaded verification-key hash does not match. The submitted entry may also -not be accepted if the second-level domain name exceeded its rate limit. By -coupling every add-leaf request with a second-level domain, it becomes more -difficult to spam the log. You would need an excessive number of domain names. -This becomes costly if free domain names are rejected. +The submitted entry will not be accepted if the signature is invalid +or if the downloaded verification-key hash does not match. The +submitted entry may also not be accepted if the second-level domain +name exceeded its rate limit. By coupling every add-leaf request with +a second-level domain, it becomes more difficult to spam the log. You +would need an excessive number of domain names. This becomes costly +if free domain names are rejected. -The log does not publish domain-name to key bindings because key management is -more complex than that. +The log does not publish domain-name to key bindings because key +management is more complex than that. -Public logging should not be assumed until an inclusion proof is available. An -inclusion proof should not be relied upon unless it leads up to a trustworthy -signed tree head. Witness cosigning can make a tree head trustworthy. +Public logging should not be assumed until an inclusion proof is +available. An inclusion proof should not be relied upon unless it +leads up to a trustworthy signed tree head. Witness cosigning can +make a tree head trustworthy. ### add-cosignature ``` @@ -334,25 +372,26 @@ POST /st/v0/add-cosignature Input: - "signature": an Ed25519 signature over `tree_head`. The result is -hex-encoded. -- "key_hash": a hash of the witness' public verification key that can be used -to verify the signature. The key is encoded as defined in [RFC 8032, -section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and -then hashed using SHA256. The hash value is hex-encoded. + hex-encoded. +- "key_hash": a hash of the witness' public verification key that can + be used to verify the signature. The key is encoded as defined in + [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), + and then hashed using SHA256. The hash value is hex-encoded. Output on success: - None -The key-hash can be used to identify which witness signed the log's tree head. -A key-hash, rather than the full verification key, is used to force the verifier -to locate the appropriate key and make an explicit trust decision. +The key-hash can be used to identify which witness signed the log's +tree head. A key-hash, rather than the full verification key, is used +to force the verifier to locate the appropriate key and make an +explicit trust decision. ## Summary of log parameters -- **Public key**: an Ed25519 verification key that can be used to verify the -log's tree head signatures. +- **Public key**: an Ed25519 verification key that can be used to + verify the log's tree head signatures. - **Log identifier**: the hashed public verification key using SHA256. -- **Shard interval**: the time during which the log accepts logging requests. -The shard interval's start and end are inclusive and expressed as the number of -seconds since the UNIX epoch. -- **Base URL**: where the log can be reached over HTTP(S). It is the prefix -before a version-0 specific endpoint. +- **Shard interval**: the time during which the log accepts logging + requests. The shard interval's start and end are inclusive and + expressed as the number of seconds since the UNIX epoch. +- **Base URL**: where the log can be reached over HTTP(S). It is the + prefix before a version-0 specific endpoint. -- cgit v1.2.3 From d13da7fd14c9050a70313f00b71955beb4276132 Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 4 May 2021 16:25:36 +0200 Subject: seconds, not milliseconds --- doc/api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index 5b7cb19..c9d3db9 100644 --- a/doc/api.md +++ b/doc/api.md @@ -78,7 +78,7 @@ makes a simple format explicit and unambiguous. Tree heads are signed by the log and its witnesses. It contains a timestamp, a tree size, and a root hash. The timestamp is included so that monitors can ensure _liveliness_. It is the time since the UNIX -epoch (January 1, 1970 00:00:00 UTC) in seconds. The tree size +epoch (January 1, 1970 00:00 UTC) in seconds. The tree size specifies the current number of leaves. The root hash fixes the structure and content of the Merkle tree. -- cgit v1.2.3 From 866320e7cb3f8eee21f464cbc56d518f6eb66c72 Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 4 May 2021 16:33:01 +0200 Subject: move long description of sharding to the design doc --- doc/api.md | 49 ++++++++++++++----------------------------------- 1 file changed, 14 insertions(+), 35 deletions(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index c9d3db9..3a595ee 100644 --- a/doc/api.md +++ b/doc/api.md @@ -114,41 +114,20 @@ struct tree_leaf { } ``` -Unlike X.509 certificates which already have validity ranges, a -checksum does not carry any such information. Therefore, we require -that the submitter selects a _shard hint_. The selected shard hint -must be in the log's _shard interval_. A shard interval is defined by -a start time and an end time. Both ends of the shard interval are -inclusive and expressed as the number of seconds since the UNIX epoch -(January 1, 1970 00:00 UTC). - -Sharding simplifies log operations because it becomes explicit when a -log can be shutdown. A log must only accept logging requests that -have valid shard hints. A log should only accept logging requests -during the predefined shard interval. Note that _the submitter's -shard hint is not a verified timestamp_. The submitter should set the -shard hint as large as possible. If a roughly verified timestamp is -needed, a cosigned tree head can be used. - -Without a shard hint, the good Samaritan could log all leaves from an -earlier shard into a newer one. Not only would that defeat the -purpose of sharding, but it would also become a potential -denial-of-service vector. - -The signed message is composed of the chosen `shard_hint` and the -submitter's `checksum`. It must be possible to verify -`signature_over_message` using the submitter's public verification -key. - -Note that the way `shard_hint` and `checksum` are serialized with -regards to signing differs from how they're being transmitted to the -log. - -A `key_hash` of the key used for signing `message` is included in -`tree_leaf` so that the leaf can be attributed to the submitter. A -hash, rather than the full public key, is used to motivate the -verifier to locate the appropriate key and make an explicit trust -decision. +`message` is composed of the `shard_hint`, chosen by the submitter to +match the shard interval for the log, and the submitter's `checksum` +to be logged. + +`signature_over_message` is a signature over `message`, using the +submitter's verification key. It must be possible to verify the +signature using the submitter's public verification key, as indicated +by `key_hash`. + +`key_hash` is a hash of the submitter's verification key used for +signing `message`. It is included in `tree_leaf` so that the leaf can +be attributed to the submitter. A hash, rather than the full public +key, is used to motivate verifiers to locate the appropriate key and +make an explicit trust decision. ## Public endpoints Every log has a base URL that identifies it uniquely. The only -- cgit v1.2.3 From 78c68e528517f157f29784f9dc87b3246f046e52 Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 4 May 2021 16:43:31 +0200 Subject: no need for encoding SPACE --- doc/api.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index 3a595ee..d75fe6f 100644 --- a/doc/api.md +++ b/doc/api.md @@ -147,10 +147,10 @@ format as the input data, i.e. as ASCII key/value pairs on the format would send an HTTP message body consisting of `stlog-tree_size: 4711`. The HTTP status code is 200 OK to indicate success. A different HTTP -status code is used to indicate failure. The log should set the -"error" key to a human-readable value that describes what went wrong. -For example, `error=invalid+signature`, `error=rate+limit+exceeded`, -or `error=unknown+leaf+hash`. +status code is used to indicate failure. The log should set the value +value for the key `error` to a human-readable string describing what +went wrong. For example, `error: invalid signature`, `error: rate +limit exceeded`, or `error: unknown leaf hash`. ### get-tree-head-cosigned Returns the latest cosigned tree head. Used together with -- cgit v1.2.3 From ee4ad9e1e4be9e969c13a12f5e76a2b439077b6e Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 4 May 2021 17:19:46 +0200 Subject: another pass over the input and output descriptions Mostly replacing "human-readable" with something more well defined. --- doc/api.md | 132 +++++++++++++++++++++++++++++++------------------------------ 1 file changed, 68 insertions(+), 64 deletions(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index d75fe6f..8a46af6 100644 --- a/doc/api.md +++ b/doc/api.md @@ -237,65 +237,69 @@ There is exactly one `signature` and one `key_hash` field. The ### get-proof-by-hash ``` -POST /st/v0/get-proof-by-hash +GET /st/v0/get-proof-by-hash ``` Input: -- "leaf_hash": a hex-encoded leaf hash that identifies which - `tree_leaf` the log should prove inclusion for. The leaf hash is - computed using the RFC 6962 hashing strategy. In other words, - `SHA256(0x00 | tree_leaf)`. -- "tree_size": a human-readable tree size of the tree head that the - proof should be based on. +- "leaf_hash": leaf identifying which `tree_leaf` the log should prove + inclusion of, hex-encoded. +- "tree_size": tree size of the tree head that the proof should be + based on, as an ASCII-encoded decimal number. Output on success: -- "tree_size": human-readable tree size that the proof is based on. -- "leaf_index": human-readable zero-based index of the leaf that the - proof is based on. -- "inclusion_path": a node hash in hex. +- "tree_size": tree size that the proof is based on, as an + ASCII-encoded decimal number. +- "leaf_index": zero-based index of the leaf that the proof is based + on, as an ASCII-encoded decimal number. +- "inclusion_path": node hash, hex-encoded. -The "inclusion_path" may be omitted or repeated to represent an -inclusion proof of zero or more node hashes. The order of node hashes -follow from our hash strategy, see RFC 6962. +The leaf hash is computed using the RFC 6962 hashing strategy. In +other words, `SHA256(0x00 | tree_leaf)`. + +`inclusion_path` may be omitted or repeated to represent an inclusion +proof of zero or more node hashes. The order of node hashes follow +from the hash strategy, see RFC 6962. ### get-consistency-proof ``` -POST /st/v0/get-consistency-proof +GET /st/v0/get-consistency-proof ``` Input: -- "new_size": human-readable tree size of a newer tree head. -- "old_size": human-readable tree size of an older tree head that the - log should prove is consistent with the newer tree head. +- "new_size": tree size of a newer tree head, as an ASCII-encoded + decimal number. +- "old_size": tree size of an older tree head that the log should + prove is consistent with the newer tree head, as an ASCII-encoded + decimal number. Output on success: -- "new_size": human-readable tree size of a newer tree head that the - proof is based on. -- "old_size": human-readable tree size of an older tree head that the - proof is based on. -- "consistency_path": a node hash in hex. +- "new_size": tree size of the newer tree head that the proof is based + on, as an ASCII-encoded decimal number. +- "old_size": tree size of the older tree head that the proof is based + on, as an ASCII-encoded decimal number. +- "consistency_path": node hash, hex-encoded. -The "consistency_path" may be omitted or repeated to represent a +`consistency_path` may be omitted or repeated to represent a consistency proof of zero or more node hashes. The order of node -hashes follow from our hash strategy, see RFC 6962. +hashes follow from the hash strategy, see RFC 6962. ### get-leaves ``` -POST /st/v0/get-leaves +GET /st/v0/get-leaves ``` Input: -- "start_size": human-readable index of the first leaf to retrieve. -- "end_size": human-readable index of the last leaf to retrieve. +- "start_size": index of the first leaf to retrieve, as an + ASCII-encoded decimal number. +- "end_size": index of the last leaf to retrieve, as an ASCII-encoded + decimal number. Output on success: -- "shard_hint": `tree_leaf.message.shard_hint` as a human-readable - number. -- "checksum": `tree_leaf.message.checksum` in hex. -- "signature_scheme": human-readable number that identifies a - signature scheme. -- "signature": `tree_leaf.signature` in hex. -- "key_hash": `tree_leaf.key_hash` in hex. +- "shard_hint": `tree_leaf.message.shard_hint` as an ASCII-encoded + decimal number. +- "checksum": `tree_leaf.message.checksum`, hex-encoded. +- "signature": `tree_leaf.signature_over_message`, hex-encoded. +- "key_hash": `tree_leaf.key_hash`, hex-encoded. All fields may be repeated to return more than one leaf. The first value in each list refers to the first leaf, the second value in each @@ -307,31 +311,32 @@ must be returned on HTTP status code 200 OK. ### add-leaf ``` -POST /st/v0/add-leaf +GET /st/v0/add-leaf ``` Input: -- "shard_hint": human-readable decimal number in the log's shard - interval that the submitter selected. +- "shard_hint": number within the log's shard interval as an + ASCII-encoded decimal number. - "checksum": the cryptographic checksum that the submitter wants to - log in hex. note: fixed length 64 bytes, validated by the server - somehow -- "signature": the submitter's signature over `tree_leaf.message`. - The result is hex-encoded. + log, hex-encoded. +- "signature_over_message": the submitter's signature over + `tree_leaf.message`, hex-encoded. - "verification_key": the submitter's public verification key. The key is encoded as defined in - [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2). The result is hex-encoded. -- "domain_hint": a domain name that indicates where - `tree_leaf.key_hash` can be retrieved as a DNS TXT resource record - in hex. + [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2) + and then hex-encoded. +- "domain_hint": domain name indicating where `tree_leaf.key_hash` + can be found as a DNS TXT resource record. Output on success: - None -The submitted entry will not be accepted if the signature is invalid -or if the downloaded verification-key hash does not match. The -submitted entry may also not be accepted if the second-level domain -name exceeded its rate limit. By coupling every add-leaf request with +The submission will not be accepted if `signature_over_message` is +invalid or if the key hash retrieved using `domain_hint` does not +match a hash over `verification_key`. + +The submission may also not be accepted if the second-level domain +name exceeded its rate limit. By coupling every add-leaf request to a second-level domain, it becomes more difficult to spam the log. You would need an excessive number of domain names. This becomes costly if free domain names are rejected. @@ -339,31 +344,30 @@ if free domain names are rejected. The log does not publish domain-name to key bindings because key management is more complex than that. -Public logging should not be assumed until an inclusion proof is -available. An inclusion proof should not be relied upon unless it -leads up to a trustworthy signed tree head. Witness cosigning can -make a tree head trustworthy. +Public logging should not be assumed to have happened until an +inclusion proof is available. An inclusion proof should not be relied +upon unless it leads up to a trustworthy signed tree head. Witness +cosigning can make a tree head trustworthy. ### add-cosignature ``` -POST /st/v0/add-cosignature +GET /st/v0/add-cosignature ``` Input: -- "signature": an Ed25519 signature over `tree_head`. The result is - hex-encoded. -- "key_hash": a hash of the witness' public verification key that can - be used to verify the signature. The key is encoded as defined in +- "signature": Ed25519 signature over `tree_head`, hex-encoded. +- "key_hash": hash of the witness' public verification key that can be + used to verify `signature`. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), - and then hashed using SHA256. The hash value is hex-encoded. + and then hashed using SHA256. The hash value is hex-encoded. Output on success: - None -The key-hash can be used to identify which witness signed the log's -tree head. A key-hash, rather than the full verification key, is used -to force the verifier to locate the appropriate key and make an -explicit trust decision. +`key_hash` can be used to identify which witness signed the log's tree +head. A key-hash, rather than the full verification key, is used to +motivate verifiers to locate the appropriate key and make an explicit +trust decision. ## Summary of log parameters - **Public key**: an Ed25519 verification key that can be used to -- cgit v1.2.3 From 8301e63f91b023e57b2d7c8b11d3dff4f0056aed Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 4 May 2021 17:22:06 +0200 Subject: use backticks for quoting single words I think this is more markdownish. --- doc/api.md | 78 +++++++++++++++++++++++++++++++------------------------------- 1 file changed, 39 insertions(+), 39 deletions(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index 8a46af6..c6a4569 100644 --- a/doc/api.md +++ b/doc/api.md @@ -164,19 +164,19 @@ Input: - None Output on success: -- "timestamp": `tree_head.timestamp` ASCII-encoded decimal number, +- `timestamp`: `tree_head.timestamp` ASCII-encoded decimal number, seconds since the UNIX epoch. -- "tree_size": `tree_head.tree_size` ASCII-encoded decimal number. -- "root_hash": `tree_head.root_hash` hex-encoded. -- "signature": hex-encoded Ed25519 signature over `tree_head` +- `tree_size`: `tree_head.tree_size` ASCII-encoded decimal number. +- `root_hash`: `tree_head.root_hash` hex-encoded. +- `signature`: hex-encoded Ed25519 signature over `tree_head` serialzed as described in section `Merkle tree head`. -- "key_hash": a hash of the public verification key (belonging to +- `key_hash`: a hash of the public verification key (belonging to either the log or to one of its witnesses), which can be used to verify the most recent `signature`. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then hashed using SHA256. The hash value is hex-encoded. -The "signature" and "key_hash" fields may repeat. The first signature +The `signature` and `key_hash` fields may repeat. The first signature corresponds to the first key hash, the second signature corresponds to the second key hash, etc. The number of signatures and key hashes must match. @@ -193,13 +193,13 @@ Input: - None Output on success: -- "timestamp": `tree_head.timestamp` ASCII-encoded decimal number, +- `timestamp`: `tree_head.timestamp` ASCII-encoded decimal number, seconds since the UNIX epoch. -- "tree_size": `tree_head.tree_size` ASCII-encoded decimal number. -- "root_hash": `tree_head.root_hash` hex-encoded. -- "signature": hex-encoded Ed25519 signature over `tree_head` +- `tree_size`: `tree_head.tree_size` ASCII-encoded decimal number. +- `root_hash`: `tree_head.root_hash` hex-encoded. +- `signature`: hex-encoded Ed25519 signature over `tree_head` serialzed as described in section `Merkle tree head`. -- "key_hash": a hash of the log's public verification key, which can +- `key_hash`: a hash of the log's public verification key, which can be used to verify `signature`. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then hashed using SHA256. The hash value is hex-encoded. @@ -220,13 +220,13 @@ Input: - None Output on success: -- "timestamp": `tree_head.timestamp` ASCII-encoded decimal number, +- `timestamp`: `tree_head.timestamp` ASCII-encoded decimal number, seconds since the UNIX epoch. -- "tree_size": `tree_head.tree_size` ASCII-encoded decimal number. -- "root_hash": `tree_head.root_hash` hex-encoded. -- "signature": hex-encoded Ed25519 signature over `tree_head` +- `tree_size`: `tree_head.tree_size` ASCII-encoded decimal number. +- `root_hash`: `tree_head.root_hash` hex-encoded. +- `signature`: hex-encoded Ed25519 signature over `tree_head` serialzed as described in section `Merkle tree head`. -- "key_hash": a hash of the log's public verification key that can be +- `key_hash`: a hash of the log's public verification key that can be used to verify `signature`. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then hashed using SHA256. The hash value is hex-encoded. @@ -241,17 +241,17 @@ GET /st/v0/get-proof-by-hash ``` Input: -- "leaf_hash": leaf identifying which `tree_leaf` the log should prove +- `leaf_hash`: leaf identifying which `tree_leaf` the log should prove inclusion of, hex-encoded. -- "tree_size": tree size of the tree head that the proof should be +- `tree_size`: tree size of the tree head that the proof should be based on, as an ASCII-encoded decimal number. Output on success: -- "tree_size": tree size that the proof is based on, as an +- `tree_size`: tree size that the proof is based on, as an ASCII-encoded decimal number. -- "leaf_index": zero-based index of the leaf that the proof is based +- `leaf_index`: zero-based index of the leaf that the proof is based on, as an ASCII-encoded decimal number. -- "inclusion_path": node hash, hex-encoded. +- `inclusion_path`: node hash, hex-encoded. The leaf hash is computed using the RFC 6962 hashing strategy. In other words, `SHA256(0x00 | tree_leaf)`. @@ -266,18 +266,18 @@ GET /st/v0/get-consistency-proof ``` Input: -- "new_size": tree size of a newer tree head, as an ASCII-encoded +- `new_size`: tree size of a newer tree head, as an ASCII-encoded decimal number. -- "old_size": tree size of an older tree head that the log should +- `old_size`: tree size of an older tree head that the log should prove is consistent with the newer tree head, as an ASCII-encoded decimal number. Output on success: -- "new_size": tree size of the newer tree head that the proof is based +- `new_size`: tree size of the newer tree head that the proof is based on, as an ASCII-encoded decimal number. -- "old_size": tree size of the older tree head that the proof is based +- `old_size`: tree size of the older tree head that the proof is based on, as an ASCII-encoded decimal number. -- "consistency_path": node hash, hex-encoded. +- `consistency_path`: node hash, hex-encoded. `consistency_path` may be omitted or repeated to represent a consistency proof of zero or more node hashes. The order of node @@ -289,17 +289,17 @@ GET /st/v0/get-leaves ``` Input: -- "start_size": index of the first leaf to retrieve, as an +- `start_size`: index of the first leaf to retrieve, as an ASCII-encoded decimal number. -- "end_size": index of the last leaf to retrieve, as an ASCII-encoded +- `end_size`: index of the last leaf to retrieve, as an ASCII-encoded decimal number. Output on success: -- "shard_hint": `tree_leaf.message.shard_hint` as an ASCII-encoded +- `shard_hint`: `tree_leaf.message.shard_hint` as an ASCII-encoded decimal number. -- "checksum": `tree_leaf.message.checksum`, hex-encoded. -- "signature": `tree_leaf.signature_over_message`, hex-encoded. -- "key_hash": `tree_leaf.key_hash`, hex-encoded. +- `checksum`: `tree_leaf.message.checksum`, hex-encoded. +- `signature`: `tree_leaf.signature_over_message`, hex-encoded. +- `key_hash`: `tree_leaf.key_hash`, hex-encoded. All fields may be repeated to return more than one leaf. The first value in each list refers to the first leaf, the second value in each @@ -315,17 +315,17 @@ GET /st/v0/add-leaf ``` Input: -- "shard_hint": number within the log's shard interval as an +- `shard_hint`: number within the log's shard interval as an ASCII-encoded decimal number. -- "checksum": the cryptographic checksum that the submitter wants to +- `checksum`: the cryptographic checksum that the submitter wants to log, hex-encoded. -- "signature_over_message": the submitter's signature over +- `signature_over_message`: the submitter's signature over `tree_leaf.message`, hex-encoded. -- "verification_key": the submitter's public verification key. The +- `verification_key`: the submitter's public verification key. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2) and then hex-encoded. -- "domain_hint": domain name indicating where `tree_leaf.key_hash` +- `domain_hint`: domain name indicating where `tree_leaf.key_hash` can be found as a DNS TXT resource record. Output on success: @@ -355,8 +355,8 @@ GET /st/v0/add-cosignature ``` Input: -- "signature": Ed25519 signature over `tree_head`, hex-encoded. -- "key_hash": hash of the witness' public verification key that can be +- `signature`: Ed25519 signature over `tree_head`, hex-encoded. +- `key_hash`: hash of the witness' public verification key that can be used to verify `signature`. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), and then hashed using SHA256. The hash value is hex-encoded. -- cgit v1.2.3 From af5b14b9e9f85fe15253fbdb48945a302f0b7bec Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 11 May 2021 14:05:31 +0200 Subject: signatures are 64 octets Spotted by Rasmus. --- doc/api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index c6a4569..4f43d2c 100644 --- a/doc/api.md +++ b/doc/api.md @@ -109,7 +109,7 @@ struct message { struct tree_leaf { struct message; - u8 signature_over_message[32]; + u8 signature_over_message[64]; u8 key_hash[32]; } ``` -- cgit v1.2.3 From 533f683ef1ae999c2fdc0086cbc3de4e675d1e33 Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 25 May 2021 11:26:32 +0200 Subject: use POST for requests with input data The major argument for moving input data from HTTP headers in GET requests to body of POST's is that we define the protocol ourselves without any dependencies on HTTP and can make it even simpler to parse. --- doc/api.md | 52 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 32 insertions(+), 20 deletions(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index 4f43d2c..a998d70 100644 --- a/doc/api.md +++ b/doc/api.md @@ -11,11 +11,9 @@ This is a work-in-progress document that may be moved or modified. ## Overview The log implements an HTTP(S) API: -- Requests to the log use the HTTP GET method. -- Input data (in requests) and output data (in responses) are - expressed as ASCII-encoded key/value pairs. -- Requests use HTTP entity headers for input data while responses use - the HTTP message body for output data. +- Input data in requests and output data in responses are expressed as + ASCII-encoded key/value pairs. +- Requests with input data use POST to send the data to the log. - Binary data is hex-encoded before being transmitted. The motivation for using a text based key/value format for request and @@ -136,21 +134,17 @@ constraint is that it must be a valid HTTP(S) URL that can have the URL could be `https://log.example.com/2021/st/v0/get-signed-tree-head`. -Input data (in requests) is sent as ASCII key/value pairs as HTTP -entity headers, with their keys prefixed with the string -`stlog-`. Example: For sending `treee_size=4711` as input a client -would send the HTTP header `stlog-tree_size: 4711`. +Input data (in requests) is POST:ed in the HTTP message body as ASCII +key/value pairs. Output data (in replies) is sent in the HTTP message body in the same format as the input data, i.e. as ASCII key/value pairs on the format -`Key: Value`. Example: For sending `tree_size=4711` as output a log -would send an HTTP message body consisting of `stlog-tree_size: 4711`. +`Key=Value` The HTTP status code is 200 OK to indicate success. A different HTTP -status code is used to indicate failure. The log should set the value -value for the key `error` to a human-readable string describing what -went wrong. For example, `error: invalid signature`, `error: rate -limit exceeded`, or `error: unknown leaf hash`. +status code is used to indicate failure, in which case the log should +respond with a human-readable string describing what went wrong using +the key `error`. Example: `error=Invalid signature.`. ### get-tree-head-cosigned Returns the latest cosigned tree head. Used together with @@ -237,7 +231,7 @@ There is exactly one `signature` and one `key_hash` field. The ### get-proof-by-hash ``` -GET /st/v0/get-proof-by-hash +POST /st/v0/get-proof-by-hash ``` Input: @@ -260,9 +254,12 @@ other words, `SHA256(0x00 | tree_leaf)`. proof of zero or more node hashes. The order of node hashes follow from the hash strategy, see RFC 6962. +Example: `echo "leaf_hash=241fd4538d0a35c2d0394e4710ea9e6916854d08f62602fb03b55221dcdac90f +tree_size=4711" | curl --data-binary @- localhost/st/v0/get-proof-by-hash` + ### get-consistency-proof ``` -GET /st/v0/get-consistency-proof +POST /st/v0/get-consistency-proof ``` Input: @@ -283,9 +280,12 @@ Output on success: consistency proof of zero or more node hashes. The order of node hashes follow from the hash strategy, see RFC 6962. +Example: `echo "new_size=4711 +old_size=42" | curl --data-binary @- localhost/st/v0/get-consistency-proof` + ### get-leaves ``` -GET /st/v0/get-leaves +POST /st/v0/get-leaves ``` Input: @@ -309,9 +309,12 @@ match. The log may return fewer leaves than requested. At least one leaf must be returned on HTTP status code 200 OK. +Example: `echo "start_size=42 +end_size=4711" | curl --data-binary @- localhost/st/v0/get-leaves` + ### add-leaf ``` -GET /st/v0/add-leaf +POST /st/v0/add-leaf ``` Input: @@ -349,9 +352,15 @@ inclusion proof is available. An inclusion proof should not be relied upon unless it leads up to a trustworthy signed tree head. Witness cosigning can make a tree head trustworthy. +Example: `echo "shard_hint=1640995200 +checksum=cfa2d8e78bf273ab85d3cef7bde62716261d1e42626d776f9b4e6aae7b6ff953 +signature_over_message=c026687411dea494539516ee0c4e790c24450f1a4440c2eb74df311ca9a7adf2847b99273af78b0bda65dfe9c4f7d23a5d319b596a8881d3bc2964749ae9ece3 +verification_key=c9a674888e905db1761ba3f10f3ad09586dddfe8581964b55787b44f318cbcdf +domain_hint=example.com" | curl --data-binary @- localhost/st/v0/add-leaf` + ### add-cosignature ``` -GET /st/v0/add-cosignature +POST /st/v0/add-cosignature ``` Input: @@ -369,6 +378,9 @@ head. A key-hash, rather than the full verification key, is used to motivate verifiers to locate the appropriate key and make an explicit trust decision. +Example: `echo "signature=d1b15061d0f287847d066630339beaa0915a6bbb77332c3e839a32f66f1831b69c678e8ca63afd24e436525554dbc6daa3b1201cc0c93721de24b778027d41af +key_hash=662ce093682280f8fbea9939abe02fdba1f0dc39594c832b411ddafcffb75b1d" | curl --data-binary @- localhost/st/v0/add-cosignature` + ## Summary of log parameters - **Public key**: an Ed25519 verification key that can be used to verify the log's tree head signatures. -- cgit v1.2.3 From 8822e78af9fb67dc9280de08c2758350a862b8ab Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Tue, 25 May 2021 12:14:45 +0200 Subject: replace some of "the log" and other rephrasing --- doc/api.md | 49 ++++++++++++++++++++++++++----------------------- 1 file changed, 26 insertions(+), 23 deletions(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index a998d70..beda293 100644 --- a/doc/api.md +++ b/doc/api.md @@ -9,11 +9,12 @@ It can be found This is a work-in-progress document that may be moved or modified. ## Overview -The log implements an HTTP(S) API: +Logs implement an HTTP(S) API for accepting requests and sending +responses. - Input data in requests and output data in responses are expressed as ASCII-encoded key/value pairs. -- Requests with input data use POST to send the data to the log. +- Requests with input data use HTTP POST to send the data to a log. - Binary data is hex-encoded before being transmitted. The motivation for using a text based key/value format for request and @@ -27,12 +28,12 @@ wire-format in use by the Tor project. ## Primitives ### Cryptography -The log uses the same Merkle tree hash strategy as +Logs use the same Merkle tree hash strategy as [RFC 6962,§2](https://tools.ietf.org/html/rfc6962#section-2). The hash functions must be [SHA256](https://csrc.nist.gov/csrc/media/publications/fips/180/4/final/documents/fips180-4-draft-aug2014.pdf). -The log must sign tree heads using -[Ed25519](https://tools.ietf.org/html/rfc8032). The log's witnesses +Logs must sign tree heads using +[Ed25519](https://tools.ietf.org/html/rfc8032). Log witnesses must also sign tree heads using Ed25519. All other parts that are not Merkle tree related also use SHA256 as @@ -73,7 +74,7 @@ you may use it though. The main point of using Trunnel is that it makes a simple format explicit and unambiguous. #### Merkle tree head -Tree heads are signed by the log and its witnesses. It contains a +Tree heads are signed both by a log and its witnesses. It contains a timestamp, a tree size, and a root hash. The timestamp is included so that monitors can ensure _liveliness_. It is the time since the UNIX epoch (January 1, 1970 00:00 UTC) in seconds. The tree size @@ -93,7 +94,7 @@ not cosign a tree head if it is inconsistent with prior history or if the timestamp is backdated or future-dated more than 12 hours. #### Merkle tree leaf -The log supports a single leaf type. It contains a shard hint, a +Logs support a single leaf type. It contains a shard hint, a checksum over whatever the submitter wants to log a checksum for, a signature that the submitter computed over the shard hint and the checksum, and a hash of the submitter's public verification key, that @@ -113,8 +114,8 @@ struct tree_leaf { ``` `message` is composed of the `shard_hint`, chosen by the submitter to -match the shard interval for the log, and the submitter's `checksum` -to be logged. +match the shard interval for the log it's submitting to, and the +submitter's `checksum` to be logged. `signature_over_message` is a signature over `message`, using the submitter's verification key. It must be possible to verify the @@ -142,13 +143,13 @@ format as the input data, i.e. as ASCII key/value pairs on the format `Key=Value` The HTTP status code is 200 OK to indicate success. A different HTTP -status code is used to indicate failure, in which case the log should +status code is used to indicate failure, in which case a log should respond with a human-readable string describing what went wrong using the key `error`. Example: `error=Invalid signature.`. ### get-tree-head-cosigned Returns the latest cosigned tree head. Used together with -`get-proof-by-hash` and `get-consistency-proof` for verifying the log. +`get-proof-by-hash` and `get-consistency-proof` for verifying the tree. ``` GET /st/v0/get-tree-head-cosigned @@ -306,7 +307,7 @@ value in each list refers to the first leaf, the second value in each list refers to the second leaf, etc. The size of each list must match. -The log may return fewer leaves than requested. At least one leaf +A log may return fewer leaves than requested. At least one leaf must be returned on HTTP status code 200 OK. Example: `echo "start_size=42 @@ -340,11 +341,11 @@ match a hash over `verification_key`. The submission may also not be accepted if the second-level domain name exceeded its rate limit. By coupling every add-leaf request to -a second-level domain, it becomes more difficult to spam the log. You +a second-level domain, it becomes more difficult to spam logs. You would need an excessive number of domain names. This becomes costly if free domain names are rejected. -The log does not publish domain-name to key bindings because key +Logs don't publish domain-name to key bindings because key management is more complex than that. Public logging should not be assumed to have happened until an @@ -373,7 +374,7 @@ Input: Output on success: - None -`key_hash` can be used to identify which witness signed the log's tree +`key_hash` can be used to identify which witness signed the tree head. A key-hash, rather than the full verification key, is used to motivate verifiers to locate the appropriate key and make an explicit trust decision. @@ -382,11 +383,13 @@ Example: `echo "signature=d1b15061d0f287847d066630339beaa0915a6bbb77332c3e839a32 key_hash=662ce093682280f8fbea9939abe02fdba1f0dc39594c832b411ddafcffb75b1d" | curl --data-binary @- localhost/st/v0/add-cosignature` ## Summary of log parameters -- **Public key**: an Ed25519 verification key that can be used to - verify the log's tree head signatures. -- **Log identifier**: the hashed public verification key using SHA256. -- **Shard interval**: the time during which the log accepts logging - requests. The shard interval's start and end are inclusive and - expressed as the number of seconds since the UNIX epoch. -- **Base URL**: where the log can be reached over HTTP(S). It is the - prefix before a version-0 specific endpoint. +- **Public key**: The Ed25519 verification key to be used for + verifying tree head signatures. +- **Log identifier**: The public verification key `Public key` hashed + using SHA256. +- **Shard interval start**: The earliest time at which logging + requests are accepted as the number of seconds since the UNIX epoch. +- **Shard interval end**: The latest time at which logging + requests are accepted as the number of seconds since the UNIX epoch. +- **Base URL**: Where the log can be reached over HTTP(S). It is the + prefix to be used to construct a version 0 specific endpoint. -- cgit v1.2.3 From e374db9e70cd329ff46f1a4443c59a8fa118ddd6 Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Fri, 28 May 2021 11:44:39 +0200 Subject: use a proper endpoint in example --- doc/api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index beda293..92344c5 100644 --- a/doc/api.md +++ b/doc/api.md @@ -133,7 +133,7 @@ Every log has a base URL that identifies it uniquely. The only constraint is that it must be a valid HTTP(S) URL that can have the `/st/v0/` suffix appended. For example, a complete endpoint URL could be -`https://log.example.com/2021/st/v0/get-signed-tree-head`. +`https://log.example.com/2021/st/v0/get-tree-head-cosigned`. Input data (in requests) is POST:ed in the HTTP message body as ASCII key/value pairs. -- cgit v1.2.3 From fe2e20f346e5f8a66c92016d77f32241498b790e Mon Sep 17 00:00:00 2001 From: Linus Nordberg Date: Fri, 28 May 2021 11:44:54 +0200 Subject: clarify what the signature in get-tree-head-* is covering --- doc/api.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) (limited to 'doc/api.md') diff --git a/doc/api.md b/doc/api.md index 92344c5..57ad119 100644 --- a/doc/api.md +++ b/doc/api.md @@ -163,8 +163,9 @@ Output on success: seconds since the UNIX epoch. - `tree_size`: `tree_head.tree_size` ASCII-encoded decimal number. - `root_hash`: `tree_head.root_hash` hex-encoded. -- `signature`: hex-encoded Ed25519 signature over `tree_head` - serialzed as described in section `Merkle tree head`. +- `signature`: hex-encoded Ed25519 signature over `timestamp`, + `tree_size` and `root_hash` serialized into a `tree_head` as + described in section `Merkle tree head`. - `key_hash`: a hash of the public verification key (belonging to either the log or to one of its witnesses), which can be used to verify the most recent `signature`. The key is encoded as defined @@ -192,8 +193,9 @@ Output on success: seconds since the UNIX epoch. - `tree_size`: `tree_head.tree_size` ASCII-encoded decimal number. - `root_hash`: `tree_head.root_hash` hex-encoded. -- `signature`: hex-encoded Ed25519 signature over `tree_head` - serialzed as described in section `Merkle tree head`. +- `signature`: hex-encoded Ed25519 signature over `timestamp`, + `tree_size` and `root_hash` serialized into a `tree_head` as + described in section `Merkle tree head`. - `key_hash`: a hash of the log's public verification key, which can be used to verify `signature`. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), @@ -219,8 +221,9 @@ Output on success: seconds since the UNIX epoch. - `tree_size`: `tree_head.tree_size` ASCII-encoded decimal number. - `root_hash`: `tree_head.root_hash` hex-encoded. -- `signature`: hex-encoded Ed25519 signature over `tree_head` - serialzed as described in section `Merkle tree head`. +- `signature`: hex-encoded Ed25519 signature over `timestamp`, + `tree_size` and `root_hash` serialized into a `tree_head` as + described in section `Merkle tree head`. - `key_hash`: a hash of the log's public verification key that can be used to verify `signature`. The key is encoded as defined in [RFC 8032, section 5.1.2](https://tools.ietf.org/html/rfc8032#section-5.1.2), -- cgit v1.2.3