# consensus.rodmena.co.uk

> Strongly-consistent distributed key-value store for Rodmena services. It is a
> 3-member **etcd v3.5.28** cluster, distributed across three datacentres, behind TLS: linearizable reads and writes,
> compare-and-swap transactions, TTL leases, and change notification via watch.
> Use it for coordination — leader election, distributed locks, service
> discovery, feature flags, live configuration. Do not use it as a general
> database.

Every statement on this page was verified against the live service through its
public HTTPS interface. Where behaviour is surprising, the surprise is written
down rather than smoothed over.

- **Base URL:** `https://consensus.rodmena.co.uk`
- **Engine:** etcd 3.5.28, 3 voting members in 3 datacentres (London / Gravelines / Limburg), Raft quorum of 2
- **Auth:** required for every key operation. No anonymous access.
- **Credentials:** issued by the operator (farshid@rodmena.co.uk). None are published here.

## Two ways in — pick one

| Interface | Endpoint | Use when |
|---|---|---|
| **gRPC** (native etcd protocol) | `consensus.rodmena.co.uk:443` | You have a real etcd client: `etcdctl`, Go `go.etcd.io/etcd/client/v3`, Java, Rust. Best performance, native watch/lease streams. |
| **HTTP/JSON** (etcd gRPC-gateway) | `https://consensus.rodmena.co.uk/v3/...` | You want plain HTTPS. Works from `curl`, any language, no etcd SDK needed. |

Both hit the same cluster and the same data. TLS is a normal publicly-trusted
Let's Encrypt certificate — no custom CA, no client certificates.

## Authentication

Exchange a username and password for a token, then send that token on every
request.

```bash
TOKEN=$(curl -s -X POST https://consensus.rodmena.co.uk/v3/auth/authenticate \
  -d '{"name":"YOUR_USER","password":"YOUR_PASSWORD"}' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["token"])')
```

Send it as a bare `Authorization` header. **There is no `Bearer ` prefix** — this
is the most common integration mistake:

```bash
curl -H "Authorization: $TOKEN" ...     # correct
curl -H "Authorization: Bearer $TOKEN"  # WRONG, rejected
```

Tokens are **JWTs with a 30 minute TTL**, signed by the cluster. Two consequences
worth building for:

- **Renew.** Treat HTTP 401, or `{"error":"etcdserver: invalid auth token"}`, as
  "re-authenticate once and retry", not as fatal. A long-lived process that
  authenticates only at startup will begin failing after 30 minutes.
- **Restarts do not invalidate your token.** Because the tokens are signed
  rather than held in a single member's memory, any member can verify one. A
  rolling restart of the cluster does not log your application out — verified by
  restarting all three members while holding one token.

An unauthenticated key request fails with:

```json
{"error":"etcdserver: user name is empty","code":3}
```

## The base64 rule

**In the HTTP/JSON API every key and every value is base64-encoded**, in both
requests and responses. etcd keys and values are arbitrary bytes, and JSON
cannot carry bytes. Forgetting this is the second most common mistake: you will
successfully store a key whose literal name is the base64 text.

gRPC clients do not have this problem — they send raw bytes.

## Every integer is a JSON string

etcd's gateway renders all 64-bit fields as **strings**, because JSON numbers
cannot hold int64 safely:

```json
{"header":{"revision":"13","raft_term":"2"},"count":"1",
 "kvs":[{"create_revision":"3","mod_revision":"3","version":"1"}]}
```

So `revision`, `count`, `version`, `create_revision`, `mod_revision`, lease `ID`
and `TTL` all arrive as strings. Cast explicitly (`int(r["header"]["revision"])`)
and never compare them as numbers without casting — `"13" > "9"` is `False` in
most languages. Keep lease IDs as strings; they overflow a JavaScript number.

## Key-value operations (HTTP/JSON)

Helper used below: `b64() { printf "%s" "$1" | base64 -w0; }`

### Put

```bash
curl -s -X POST https://consensus.rodmena.co.uk/v3/kv/put \
  -H "Authorization: $TOKEN" \
  -d "{\"key\":\"$(b64 /myapp/config/timeout)\",\"value\":\"$(b64 30s)\"}"
```

Response carries the cluster revision — a global, monotonically increasing
logical clock you can use for ordering and for resuming watches:

```json
{"header":{"cluster_id":"...","member_id":"...","revision":"7","raft_term":"2"}}
```

### Get one key

```bash
curl -s -X POST https://consensus.rodmena.co.uk/v3/kv/range \
  -H "Authorization: $TOKEN" \
  -d "{\"key\":\"$(b64 /myapp/config/timeout)\"}"
```

A miss returns no `kvs` array at all and `count` is absent — **not** an empty
list and **not** `count: 0`. Use `response.get("kvs", [])`.

### Get a prefix (list)

There is no "prefix" flag. You pass `range_end` = the prefix with its **last
byte incremented**. For `/myapp/` that is `/myapp0` (`/` is 0x2F, `0` is 0x30).

```bash
curl -s -X POST https://consensus.rodmena.co.uk/v3/kv/range \
  -H "Authorization: $TOKEN" \
  -d "{\"key\":\"$(b64 /myapp/)\",\"range_end\":\"$(b64 /myapp0)\"}"
```

To range over **every** key, use `key` = base64 of a single NUL byte and
`range_end` the same.

### Delete

```bash
curl -s -X POST https://consensus.rodmena.co.uk/v3/kv/deleterange \
  -H "Authorization: $TOKEN" \
  -d "{\"key\":\"$(b64 /myapp/config/timeout)\"}"
# -> {"header":{...},"deleted":"1"}
```

## Transactions — the compare-and-swap primitive

This is why you use a consensus store. A txn is `compare` → `success` or
`failure`, applied atomically across the cluster.

Create a key only if nobody else has (`version = 0` means "does not exist"):

```bash
curl -s -X POST https://consensus.rodmena.co.uk/v3/kv/txn \
  -H "Authorization: $TOKEN" \
  -d "{
    \"compare\":[{\"key\":\"$(b64 /myapp/leader)\",\"target\":\"VERSION\",\"result\":\"EQUAL\",\"version\":\"0\"}],
    \"success\":[{\"requestPut\":{\"key\":\"$(b64 /myapp/leader)\",\"value\":\"$(b64 node-a)\"}}],
    \"failure\":[{\"requestRange\":{\"key\":\"$(b64 /myapp/leader)\"}}]
  }"
```

**Critical gotcha.** On success the response contains `"succeeded": true`. On
failure the field is **omitted entirely** rather than being `false`:

```python
# WRONG -- treats "field missing" and "field false" identically only by luck,
# and silently misreads any future shape change
if resp["succeeded"]: ...

# CORRECT
if resp.get("succeeded") is True: ...
```

Other compare targets: `VERSION`, `CREATE` (create_revision), `MOD`
(mod_revision), `VALUE`. Comparing on `MOD` is how you do optimistic
concurrency: read a key, note `mod_revision`, then write only if it has not
changed.

## Leases — keys that expire

A lease is a TTL handle. Keys attached to it are deleted when it expires. This
is the building block for liveness: locks, leader election, service registration.

```bash
# 1. grant a 10 second lease
LID=$(curl -s -X POST https://consensus.rodmena.co.uk/v3/lease/grant \
  -H "Authorization: $TOKEN" -d '{"TTL":10}' \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["ID"])')

# 2. attach a key to it
curl -s -X POST https://consensus.rodmena.co.uk/v3/kv/put \
  -H "Authorization: $TOKEN" \
  -d "{\"key\":\"$(b64 /myapp/nodes/node-a)\",\"value\":\"$(b64 10.0.0.5:8080)\",\"lease\":\"$LID\"}"

# 3. hold it open -- send this at roughly TTL/3
curl -s -X POST https://consensus.rodmena.co.uk/v3/lease/keepalive \
  -H "Authorization: $TOKEN" -d "{\"ID\":\"$LID\"}"
```

Verified behaviour: the key is present while keepalives continue and disappears
within a few seconds of them stopping. Lease IDs are **strings** in JSON (they
are 64-bit ints), so keep them as strings — do not round-trip through a JS
number.

If your process dies, the lease lapses and the key vanishes on its own. That is
the point: no cleanup code, no stale registrations.

## Watch — react to changes

Watch is a long-lived streaming response. Each change arrives as one JSON object
per line.

```bash
curl -sN -X POST https://consensus.rodmena.co.uk/v3/watch \
  -H "Authorization: $TOKEN" \
  -d "{\"create_request\":{\"key\":\"$(b64 /myapp/config/)\",\"range_end\":\"$(b64 /myapp/config0)\"}}"
```

The first object confirms creation (`"created": true`); subsequent objects carry
`events`. Each event has `type` (absent means `PUT`, `"DELETE"` for deletes) and
`kv` with base64 key/value.

Pass `start_revision` to resume without missing anything after a reconnect —
take the `revision` from your last processed event and add 1. This is what makes
watch reliable across restarts; without it you have a gap.

Use `-N` (unbuffered) with curl. The edge does not buffer these responses and
allows streams up to 1 hour.

## Python — no etcd SDK required

Verified working against the live service:

```python
import base64, httpx

BASE = "https://consensus.rodmena.co.uk"

def b64(s: str) -> str:
    return base64.b64encode(s.encode()).decode()

def unb64(s: str) -> str:
    return base64.b64decode(s).decode()

class Consensus:
    def __init__(self, user: str, password: str):
        self.c = httpx.Client(base_url=BASE, timeout=10.0)
        self.token = self.c.post(
            "/v3/auth/authenticate", json={"name": user, "password": password}
        ).json()["token"]
        self.c.headers["Authorization"] = self.token   # no "Bearer"

    def put(self, key: str, value: str) -> int:
        r = self.c.post("/v3/kv/put", json={"key": b64(key), "value": b64(value)})
        r.raise_for_status()
        return int(r.json()["header"]["revision"])

    def get(self, key: str) -> str | None:
        r = self.c.post("/v3/kv/range", json={"key": b64(key)})
        r.raise_for_status()
        kvs = r.json().get("kvs", [])          # absent on a miss, not empty list
        return unb64(kvs[0]["value"]) if kvs else None

    def list_prefix(self, prefix: str) -> dict[str, str]:
        end = prefix[:-1] + chr(ord(prefix[-1]) + 1)
        r = self.c.post("/v3/kv/range",
                        json={"key": b64(prefix), "range_end": b64(end)})
        r.raise_for_status()
        return {unb64(kv["key"]): unb64(kv["value"])
                for kv in r.json().get("kvs", [])}

    def create_if_absent(self, key: str, value: str) -> bool:
        """Atomic create. True if we won the race, False if it already existed."""
        r = self.c.post("/v3/kv/txn", json={
            "compare": [{"key": b64(key), "target": "VERSION",
                         "result": "EQUAL", "version": "0"}],
            "success": [{"requestPut": {"key": b64(key), "value": b64(value)}}],
            "failure": [],
        })
        r.raise_for_status()
        return r.json().get("succeeded") is True   # absent means False
```

## Go — native gRPC client

```go
import (
    "context"
    "time"
    clientv3 "go.etcd.io/etcd/client/v3"
)

cli, err := clientv3.New(clientv3.Config{
    Endpoints:   []string{"consensus.rodmena.co.uk:443"},
    DialTimeout: 5 * time.Second,
    Username:    "YOUR_USER",
    Password:    "YOUR_PASSWORD",
    // TLS is a normal public certificate; the zero value uses the system roots.
    TLS: &tls.Config{},
})
defer cli.Close()

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
_, err = cli.Put(ctx, "/myapp/config/timeout", "30s")
cancel()
```

## etcdctl

```bash
etcdctl --endpoints=https://consensus.rodmena.co.uk:443 \
        --user USER:PASSWORD \
        put /myapp/key value

etcdctl --endpoints=https://consensus.rodmena.co.uk:443 \
        --user USER:PASSWORD \
        get --prefix /myapp/
```

## Consistency model

- Reads are **linearizable by default**: a read always reflects every write that
  completed before it started, cluster-wide. This costs a quorum round trip.
- Add `"serializable": true` to a `range` request for a local, possibly stale
  read that skips the quorum. Faster; use only where staleness is acceptable.
- Writes are linearizable, full stop. A successful write is durable on a
  majority of members.
- `revision` is a global logical clock across the whole keyspace. It increases on
  every write to any key, so it orders unrelated keys against each other.
- A write that TIMES OUT has an UNKNOWN outcome. Without quorum a write fails
  with HTTP 504 `{"error":"context deadline exceeded"}` (observed live
  2026-09-24 with 2 of 3 members stopped), but the proposal may still commit once
  quorum returns. Re-read, or make the write a txn that compares on the value or
  `mod_revision`, before retrying. Never assume "timed out" means "not written".

## Limits and refusals

| Limit | Value | Behaviour past it |
|---|---|---|
| Request body | 1.5 MB (2 MB at the edge) | Rejected |
| Value size | Keep under ~1 MB | Rejected |
| Rate, general API | 100 req/s per IP (burst 200) | HTTP 503 |
| Rate, `/v3/auth/authenticate` | 10 req/s per IP (burst 20) | HTTP 503 |
| Stream lifetime (watch, keepalive) | 1 hour | Stream closed; reconnect with `start_revision` |
| History retention | compacted hourly | Watch from a revision older than ~1h fails |

These endpoints are **blocked at the edge** and return 403 even with valid admin
credentials. They are operator actions, performed on the host:

```
/v3/maintenance/snapshot      /v3/cluster/member/add
/v3/maintenance/defragment    /v3/cluster/member/remove
/v3/maintenance/downgrade     /v3/cluster/member/update
/v3/maintenance/alarm         /v3/cluster/member/promote
/v3/maintenance/transfer-leadership
/v3/auth/disable
```

The equivalent gRPC methods are blocked too. A full-database snapshot over the
public internet is total data exfiltration in a single request, so it is refused
regardless of who asks.

## Health

`GET /health` (and `HEAD /health`, same status, no body) is unauthenticated and safe to poll from a load balancer:

```bash
curl -s https://consensus.rodmena.co.uk/health
# {"health":"true","reason":""}
```

## Operational limits you should know

**Members are in three separate datacentres** — London (UK), Gravelines (FR)
and Limburg (DE). Quorum is 2, so the cluster tolerates losing any one member,
its host, or its entire site, and keeps serving reads and writes. This was
verified on 2026-08-21 by killing the leader outright: the remaining two elected
a new leader, continued accepting writes, and the recovered member reconciled
the writes it had missed.

What it still does NOT survive: losing **two** members simultaneously. At one
surviving member there is no quorum, so the cluster stops accepting writes
rather than risk divergence — it is unavailable, not corrupted, and recovers
when a second member returns.

The public edge (nginx) runs on one host (London) only, so an outage of that
host takes the public endpoint down even while consensus continues among the
members. Member ports are firewalled to the other members and are NOT reachable
by clients, so there is currently no alternative address; a second edge is
planned. Design callers to tolerate the endpoint being unavailable.

Back up anything whose loss you could not tolerate: replication is not backup,
and a mistaken `DELETE` replicates to all three in milliseconds.

Key naming: use a `/` prefix per application (`/myapp/...`). Users can be scoped
to a prefix, so a well-namespaced key layout is what makes per-application
credentials possible later.

## Getting access

Credentials are issued per application by the operator. Ask
farshid@rodmena.co.uk for a user scoped to your prefix. Do not share one
credential between unrelated services — a prefix-scoped user per application
means a leak is contained to that prefix.

## See also

- Human documentation: https://consensus.rodmena.co.uk/docs
- etcd API reference: https://etcd.io/docs/v3.5/learning/api/
- etcd gRPC-gateway notes: https://etcd.io/docs/v3.5/dev-guide/api_grpc_gateway/