diff options
| -rw-r--r-- | archive/2022-01-04-proposal-add-leaf-endpoint | 90 | ||||
| -rw-r--r-- | archive/2022-01-04-proposal-domain-hint | 51 | ||||
| -rw-r--r-- | archive/2022-01-04-proposal-get-endpoints | 46 | ||||
| -rw-r--r-- | archive/2022-01-04-proposal-tree-head-endpoints | 118 |
4 files changed, 305 insertions, 0 deletions
diff --git a/archive/2022-01-04-proposal-add-leaf-endpoint b/archive/2022-01-04-proposal-add-leaf-endpoint new file mode 100644 index 0000000..3123e02 --- /dev/null +++ b/archive/2022-01-04-proposal-add-leaf-endpoint @@ -0,0 +1,90 @@ +Proposal: change add-leaf endpoint + +Background +--- +Right now a log returns HTTP status 200 OK if it will "try" to merge a submitted +leaf into its Merkle tree. A submitter should not assume that logging happened +until they see an inclusion proof that leads up to a (co)signed tree head. + +If a submitted leaf does not show up in the log despite seeing HTTP status 200 +OK, the submitter must resubmit it. When a resubmission is required/expected is +undefined. + +The reason for this "try" behavior is that log operations become much easier, +especially in self-hosted environments that do not rely on managed databases. +In other words, it is OK to just be "pretty sure" that a submitted leaf will be +persisted and sequenced, and "100%" sure after sequencing actually happened. + +Proposal +--- +A log should not return HTTP status 200 OK unless: +1. The submitted leaf has been sequenced as part of a persisted database. +2. The next tree head that the log signs will contain the submitted leaf. + +HTTP status 3XX is returned with, e.g., "Error=leaf has not been sequenced yet" +if it is not guaranteed that the submitted leaf has been sequenced. + +This means that logging should be assumed after seeing HTTP status 200 OK. This +assumption will be confirmed when the submitter obtains the next (co)signed tree +head. Further investigation is required if it turns out that this assumption is +false. + +Notes +--- +An earlier draft of this proposal considered if useful debug information should +be returned, such as "leaf index", "leaf hash", and "estimated time until a +cosigned tree head is available". We decided to not go in this direction to +avoid redundant and unsigned output that may be mis-used and tampered with ("not +consistent with design"). + +(Note that it is easy to determine when the next cosigned tree head will be +available. The to-sign tree head has a timestamp, and it is rotated every 300s. +Then it takes an additional 300s before the to-sign tree head is served with +collected cosignatures.) + +An earlier draft of this proposal also considered to have verifiable output: + * Option 1: An inclusion proof and a signed tree head + * Option 2: An inclusion proof and a cosigned tree head + +This could be a worthwhile direction if the submitter can only obtain the +required data by using the add-leaf endpoint, thus "forcing resubmits until the +desired output is obtained". Credit to Al Cutter who proposed this (very nice) +idea to us a while back. + +It is not appropriate to always return an inclusion proof for a signed tree +head. What we want is for submitters to get inclusion proofs that reference +cosigned tree heads. + +There are drawbacks to replace the above signed tree head with a cosigned tree +head: + * A submitter that submits multiple leaves will likely (have to?) retrieve + the same cosigned tree head multiple times via the add-leaf endpoint. That + overhead adds up. + * A submitter will have to be in a "resubmit phase" for several minutes as + the default, because it takes time before a cosigned tree head becomes + available. + * (The most sensible implementation would likely resubmit periodically, + say, once per minute. A clever implementation would look at the + timestamp of the to-sign endpoint to determine when is the earliest time + that a merged may have happened.) + +Moreover, removing the get-inclusion-proof and get-tree-head-cosigned endpoints +to force usage of add-leaf excludes (or makes for wonky) usage patterns of the +log: + * "I just want to download all cosigned tree heads to archive them" -> add + leaves. + * "I just want to debug/know that the log is committed to have the leaf + logged, and rely on other witnesses" -> still forced to observe the log's + cosignatures. + * "I want an inclusion proof to a particular tree head" -> build the Merkle + tree yourself to construct that proof. The log's API chooses tree heads for + you. + * (Keeping these endpoints in addition to any new add-leaf output would to + some degree defeat the purpose of adding output, which is why it is not + considered an option.) + +In gist, we decided to go with a solution that is somewhere in between what we +did before and what Al Cutter proposed. We defined when a resubmission is (not) +expected. As a result, a self-hosted log may return at least one HTTP 3XX for +each leaf request, and a few seconds later return HTTP status 200 OK for the +same input data. diff --git a/archive/2022-01-04-proposal-domain-hint b/archive/2022-01-04-proposal-domain-hint new file mode 100644 index 0000000..322d9cc --- /dev/null +++ b/archive/2022-01-04-proposal-domain-hint @@ -0,0 +1,51 @@ +Proposal: stricter domain hint requirements + +Background +--- +Right now a log is expected to look up a submitter's public key hash via DNS. A +domain hint, say, example.com, specifies the location of a TXT RR that contains +the appropriate key hash in hex-encoding. "Some domain knows about the key". + +Downsides with this: +1. A log can be instructed to look up arbitrary TXT records +2. No versioning + +As far as we know there are no amplification threats with (1), but ideally it +would only be possible to query TXT RRs that are actually relevant for Sigsum. + +Not having any versioning could potentially become a headache. All other log +endpoints are versioned. There is no good reason to not have versioning here, +unless that would imply something like registering many different things with +IANA as a result. + +Proposal +--- +Require that a domain hint is formatted as: + + _sigsum_v0.* + +Examples of valid domain hints: + + _sigsum_v0.com + _sigsum_v0.example.com + _sigsum_v0.sub.example.com + +Examples of invalid domain hints: + + _sigsum_v0hello.example.com + +This change addresses both (1) and (2), without making DNS configs harder. + +Notes +--- +For v1 we need to consider if something should be registered with IANA. Credit +to Patrik Wallström who pointed us towards documentation about labels with +underscores: + * https://www.rfc-editor.org/rfc/rfc8552.html + * https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#underscored-globally-scoped-dns-node-names + +Note also that the dependency on TXT look-ups means that a "hidden log" via Tor +would need help from a resolver that is also available over Tor (preferably an +onion but at minimum reachable over TCP). This is because TXT records cannot be +resolved over Tor. This proposal allows the used resolver to be restricted to +only resolve _sigsum_*. diff --git a/archive/2022-01-04-proposal-get-endpoints b/archive/2022-01-04-proposal-get-endpoints new file mode 100644 index 0000000..cbe3170 --- /dev/null +++ b/archive/2022-01-04-proposal-get-endpoints @@ -0,0 +1,46 @@ +Proposal: change get-* endpoints that use HTTP post + +Background +--- +Right now we HTTP POST ASCII key-value pairs on these endpoints: + * get-leaves + * get-inclusion-proof + * get-consistency-proof + +The original reason was to not have an additional parser, say, input-parameters +with percent-encoding as part of the request URL. + +A major problem with this approach is that it will not be possible to benefit +from HTTP caching. Debugging, with "URLs that reference data" also becomes more +messy. You would have to say "I did printf <stuff> | curl ...". + +Proposal +--- +Change these endpoints so that they use HTTP GET. Encode input params in URL: + + <url>/get-leaves/10/20 # get leaves 10,11,...20 + <url>/get-consistency-proof/10/20 # proof from tree size 10 to 20 + <url>/get-inclusion-proof/10/<leaf hash in hex> # proof for tree size 10 + +This notably avoids percent-encoding which is more messy. + +Notes +--- +We considered if it would be a good idea to re-use our ASCII parser for the +portion of the URL that encodes input data. The basic idea would be that +different "end of key" and "end of value" patterns could be used that are better +suited for a URL. + +For example, instead of (=,\n) one could use ([,]) as ("end of key", "end of value"). + * get-leaves/start_size[10]end_size[12] + * get-consistency-proof/old_size[12]new_size[14] + * get-inclusion-proof/tree_size[10]leaf_hash[ab...ef] + +The reasons why we aborted this direction: + * We can not think of any concrete security risk with the shorter '/' proposal. + * There are very few parameters at play here, hard to confuse and quick + feedback loop if you do. For example, "Error=start size must be smaller or + equal to end size". + * We can be sure that the '/' proposal will not introduce any wonky + interoperability issues; picking a ("end of key", "end of value") would + require much more care. diff --git a/archive/2022-01-04-proposal-tree-head-endpoints b/archive/2022-01-04-proposal-tree-head-endpoints new file mode 100644 index 0000000..b2831bf --- /dev/null +++ b/archive/2022-01-04-proposal-tree-head-endpoints @@ -0,0 +1,118 @@ +Proposal: change tree-head endpoints + +Background +--- +Right now the get-tree-head-to-sign endpoint returns the signed tree head that +witnesses should cosign. It does not return any cosignatures. One needs to +wait until the to-sign tree head is finalized and served via +get-tree-head-cosigned. We also have a get-tree-head-latest endpoint that is +sort of hanging around for "debug purposes". + +It would be nice if a submitter could find required cosignatures without always +having to wait for five minutes. The log will likely have received a majority +of cosignatures after one minute, but a submitter currently needs to wait the +full duration before getting access via the get-tree-head-cosigned endpoint. + +It would also be nice to consider if the get-tree-head-latest endpoint can be +removed. + +Here is a rough break-down of how we think about the sigsum API's usage via +roles: + * Submitter + * add-leaf, until HTTP status 200 OK which should mean "you have been sequenced". + * [fetching an inclusion proof for a signed tree head to "verify sequencing" + is not a recommended usage pattern, and does not prevent DoS. The only + difference is that the submitter would notice that the log has not + included with regards to the latest tree head sooner than with regards + to the cosigned tree head. In both cases, there is no proof that + submitter got 200 OK without getting sequenced.] + * Distributor + * get-tree-head-cosigned + * get-inclusion-proof + * [wants "enough" cosignatures, sooner rather than later is a soft requirement] + * Monitor + * get-leaves + * get-tree-head-cosigned + * might hit get-{consistency,inclusion}-proof depending on implementation + * [wants as many cosignatures as possible, does not care about ~minutes of waiting] + * Witness + * get-consistency-proof + * get-tree-head-to-sign + * add-cosignature + * [does not / should not care about other cosignatures; just that the + log signed and that the tree head is consistent with prior history as + observed by the witness] + * End-user + * [does not hit any of the log's endpoints] + * "The curious" + * the latest signed tree head, as fast as possible for quick debug + probably. "is the thing I'm doing working". + * the latest cosigned tree head, with as many cosignatures as possible + for archiving + +Keep in mind that the below proposal should not introduce the log's key hash as +output on any API endpoint. We removed this and other redundant output because +that reduces the risk of faulty implementations that operate on untrusted input. + +For example, in the same way that a faulty witness could verify "the wrong +consistency proof" if it just verified the proof against the tree sizes that the +log returned redundantly (as opposed to the tree sizes the witness asked for), a +faulty witness could end-up cosigning a tree head with another log's context +because "they just copied the key hash and used it because it was there". + +Note that we cannot add the key_hash and cosignature fields to the output of +get-tree-head-to-sign. Our ASCII parser is so simple that it does not permit +empty lists. So, we will either need a way to handle empty lists, or tweak our +endpoints so that they still do what we want without having any empty list. + +[Both rgdd and ln5 would like to avoid complicating the ASCII parser.] + +Proposal +--- +1. Remove the get-tree-head-latest endpoint. We no longer have any recommended +usage-pattern for this endpoint and so it should be removed. Our strongest +arguments for removal are "don't use a signed tree head, it is sort of like a +promise", and "it does not even help you prove that the log's HTTP status 200 OK +semantics were faulty". +2. The get-tree-head-to-sign endpoint is kept as is, but renamed. + * Purpose: used by witnesses. +3. Add an endpoint that returns the logs "to-cosign" tree head and all +cosignatures that were collected thus far. If no cosignatures were received +yet, return an error to avoid having an empty list as output. + * Purpose: used by distributors, but could also be used by a witness' + internal monitoring setup ("is my witness working, are the signatures really + showing up?"). +4. Keep an endpoint that serves the "finalized" cosigned tree head. + * Purpose: mainly used by monitors, but could also be used by distributor's + that don't mind the additional waiting or by parties that want to archive + cosigned tree heads. + +This proposal currently does not have a name for the above endpoints. Help +wanted. + +Notes +--- +A witness polls the "get-tree-head-to-sign" endpoint as before. Witnesses are +recommended to poll the log at least once per minute at randomly selected times. + +After a successful add-cosignature request, a witness should not attempt to add +the same cosignature again. A log can refresh their "to-sign tree head" to +instruct witnesses to send their cosignatures again for the same tree size. + +A witness operator may check that their cosignatures appear on the +"get-tree-head-cosigned endpoints". Such checking would likely be part of how +the operator monitors that the witness operates correctly (i.e., it would not be +something that the witness software does itself after a successful +add-cosignature request). + +A submitter ("Signer" in Figure 1) that wants a cosigned tree head that +satisifies a given policy as fast as possible can poll the "dynamic cosigned +tree head endpoint". Keep in mind that polling more than a few times per minute +would not let you obtain cosignatures much faster, see the above recommendation +for how often witnesses should provide their cosignatures. + +A helpful reflection with regards to naming: + * "The log's to-sign STH shows up, it gets filled-up with cosignatures; the + previous cosigned tree head is served on a separate endpoint. Then + "prev=curr, curr=new". I.e., there is a time aspect here that might be + helpful for naming, although previous and current would be bad choices." |