Node mode
In node mode roxy has no config file. It enrols with a control plane, leases its policy and secrets from it, and sends its flow log back. The control plane is any server that implements the node protocol.
roxy run --control-plane https://cp.example:8443 \
--enrol-token-file /run/secrets/roxy-enrol-token \
--state-dir /var/lib/roxy/node \
--control-plane-ca /etc/roxy/cp-ca.pemFlags
| flag | meaning |
|---|---|
--control-plane URL | The control plane's https:// URL. Cannot be combined with --config. |
--enrol-token-file PATH | The enrolment token. Only read when the state dir holds no node token yet. |
--state-dir DIR | Where the node keeps its token, its CA and its quarantine marker. |
--control-plane-ca PATH | CA certificates to verify the control plane with. Default: the system roots. |
--bootstrap-bind ADDR | Where the proxy listens before the first lease. Default 0.0.0.0:3128. |
--bootstrap-ca-server ADDR | Where ca_server listens before the first lease. Default 0.0.0.0:3130. |
--interception-ca-cert PATH, --interception-ca-key PATH | A CA to use instead of generating one (the interception CA). |
--replace-interception-ca | Replace a stored CA that differs from the one given. |
The control plane decides how long an enrolment token lasts and whether it can be reused. roxy does not delete the token file.
The state dir
The state dir holds the node token, a flow sequence counter, the CA in
ca/, and a quarantine file while the node is quarantined. The rendered
config and secrets stay in memory only. A copy of the state dir is a copy
of the node's identity, which the control plane can revoke.
The interception CA
The CA always lives in <state-dir>/ca. On first start roxy generates
one, or imports the pair given with --interception-ca-cert and
--interception-ca-key. After that the stored CA is used. If the flags
name a different pair, roxy refuses to start, since workloads already
trust the stored CA. Pass --replace-interception-ca to replace it.
Workloads fetch the CA from ca_server as usual
(CA distribution).
Startup
- The listeners open at once with an empty policy, so every request is
denied and
/readyzanswers503 no_policy. - Without a stored node token, the node enrols with the enrolment token. If the control plane refuses it, the node retries for two minutes and then exits, so an orchestrator sees a crash loop. An unreachable control plane is retried for as long as it takes.
- The node fetches its lease and applies it. Settings that only apply at startup (restart-only settings) are taken from the first lease. A later lease that changes one is logged and needs a restart.
The lease
Every lease is the whole policy, applied in full or not at all. A lease
that roxy check would reject, or that lacks a secret the config needs, is
logged and the running policy stays.
The lease sets the policy's expiry (lease), measured on the node's own clock. The node polls again before it runs out. If the control plane is unreachable, the lease expires and roxy denies everything until the next successful poll, with no restart needed.
A revoked node (410) denies everything at once and stops polling. A
node never enrols again on its own. To re-enrol it, empty the state dir
and start it with a new enrolment token.
A quarantined node keeps applying leases. It reports the quarantine on every poll, and lifts it when the control plane releases that incident in a lease.
The node renews its token before it expires. If the token does expire, the node exits, since only re-enrolment can recover it.
Flow shipping
Every flow-log event goes to the local flow log as usual, and to the
control plane in batches, as the lease's flow settings say
(flow upload). Events wait in
memory until the control plane acknowledges them. When too many are
waiting, the lease decides whether roxy holds traffic (hold) or drops the
oldest events (spool). On shutdown roxy ships what is waiting, for up to
ten seconds.
SIGHUP reopens the local log files. Health checks work as in
operations: ready means an unexpired lease is
in force.