Skip to content

Implement Edge Cookie identity system with KV graph and integration tests#621

Open
ChristianPavilonis wants to merge 52 commits intomainfrom
feature/edge-cookies-final
Open

Implement Edge Cookie identity system with KV graph and integration tests#621
ChristianPavilonis wants to merge 52 commits intomainfrom
feature/edge-cookies-final

Conversation

@ChristianPavilonis
Copy link
Copy Markdown
Collaborator

@ChristianPavilonis ChristianPavilonis commented Apr 7, 2026

Summary

  • Implement the Edge Cookie (EC) identity system — a publisher-owned, first-party cookie-based identity mechanism that replaces the previous Synthetic ID concept. EC enables consent-gated identity generation, a KV-backed identity graph with CAS concurrency, partner sync (pixel, batch, pull), an identify endpoint, and auction bidstream decoration with partner EIDs.
  • Add 7 integration test scenarios covering the full EC lifecycle: generation, consent withdrawal, identify edge cases, concurrent partner syncs, and batch sync flows.
  • Rename "Synthetic ID" → "Edge Cookie" across the entire codebase, configuration, and documentation.

Changes

File / Area Change
crates/trusted-server-core/src/ec/ (new module) Full EC implementation: mod.rs (lifecycle orchestration), generation.rs (HMAC-SHA256), cookies.rs (cookie read/write), consent.rs (consent gating), device.rs (device signal derivation, bot gate), kv.rs + kv_types.rs (KV identity graph with CAS), partner.rs (partner registry + admin), sync_pixel.rs (pixel sync), batch_sync.rs (S2S batch sync), pull_sync.rs (background pull sync), identify.rs (identity lookup with CORS), eids.rs (EID encoding), finalize.rs (response finalization middleware), admin.rs (admin endpoints)
crates/integration-tests/ 7 new EC lifecycle test scenarios in tests/common/ec.rs and tests/frameworks/scenarios.rs; updated integration.rs test runner and Viceroy config fixtures
crates/trusted-server-adapter-fastly/src/main.rs New EC endpoint routes namespaced under /_ts/api/v1/ and /_ts/admin/; send_to_client() pattern with background pull sync dispatch
crates/trusted-server-core/src/auction/ endpoints.rs and formats.rs updated to decorate bid requests with partner EIDs from the KV identity graph (user.id, user.eids, user.consent)
crates/trusted-server-core/src/integrations/prebid.rs Bidder deduplication fix; blank auction EC header handling
crates/js/lib/src/integrations/prebid/index.ts JS-side Prebid integration updates for EC headers
crates/trusted-server-core/src/synthetic.rs Deleted — replaced by the EC module
crates/trusted-server-core/src/cookies.rs Slimmed down; EC-specific cookie logic moved to ec/cookies.rs
crates/trusted-server-core/src/settings.rs + settings_data.rs EC configuration fields, partner config, consent config migration
crates/trusted-server-core/src/consent/ Updated consent module for EC-specific consent checks; KV consent storage
crates/trusted-server-core/src/http_util.rs New HTTP helpers for EC endpoints (CORS, request parsing)
crates/trusted-server-core/src/proxy.rs EC header propagation in proxy layer
crates/trusted-server-core/src/publisher.rs Publisher config updates for EC
crates/trusted-server-core/src/constants.rs New EC-related constants
docs/guide/ec-setup-guide.md (new) End-to-end EC setup documentation
docs/guide/edge-cookies.md (new) EC concept overview
docs/guide/api-reference.md API reference updated with all EC endpoints
docs/guide/configuration.md Configuration docs updated for EC settings
docs/guide/synthetic-ids.md Deleted — replaced by EC docs
docs/ (various) Documentation updates reflecting Synthetic ID → Edge Cookie rename
docs/internal/superpowers/specs/2026-03-24-ssc-technical-spec-design.md EC technical spec updated with schema extensions, device signals, bot gate
fastly.toml New KV store bindings for EC
trusted-server.toml EC configuration section added

Closes

Closes #532
Closes #533
Closes #534
Closes #535
Closes #536
Closes #537
Closes #538
Closes #539
Closes #540
Closes #541
Closes #542
Closes #543
Closes #544
Closes #611
Closes #612

Follow-up

Follow-up: The pull sync mechanism (ec/pull_sync.rs) currently passes ec_id and client_ip as query parameters to partner endpoints. While connections are HTTPS-only, these values can appear in partner server access logs and CDN logs. A future improvement should migrate to POST body or hash/encrypt identifiers before transmission. Tracked separately.

Test plan

  • cargo test --workspace
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo fmt --all -- --check
  • JS tests: cd crates/js/lib && npx vitest run
  • JS format: cd crates/js/lib && npm run format
  • Docs format: cd docs && npm run format
  • Other: 7 EC lifecycle integration test scenarios (full lifecycle, consent withdrawal, identify edge cases, concurrent partner syncs, batch sync)

Checklist

  • Changes follow CLAUDE.md conventions
  • No unwrap() in production code — use expect("should ...")
  • Uses tracing macros (not println!)
  • New code has tests
  • No secrets or credentials committed

@ChristianPavilonis ChristianPavilonis self-assigned this Apr 7, 2026
@ChristianPavilonis ChristianPavilonis force-pushed the feature/edge-cookies-final branch from de0525d to 0940c34 Compare April 7, 2026 19:01
@ChristianPavilonis ChristianPavilonis requested review from aram356 and prk-Jr and removed request for aram356 April 7, 2026 19:04
Comment thread crates/trusted-server-core/src/ec/kv.rs Fixed
Comment thread crates/trusted-server-core/src/ec/kv.rs Fixed
Comment thread crates/trusted-server-core/src/ec/kv.rs Fixed
Comment thread crates/trusted-server-core/src/ec/kv.rs Fixed
Comment thread crates/trusted-server-core/src/storage/kv_store.rs Fixed
Copy link
Copy Markdown
Collaborator

@aram356 aram356 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Comprehensive EC identity subsystem: HMAC-based ID generation, KV identity graph with CAS concurrency, partner sync (pixel, batch, pull), identify endpoint, and auction bidstream decoration. The architecture is well-designed — clean separation of concerns, strong input validation, and solid concurrency model. A few cleartext logging issues and a docs inconsistency need addressing before merge.

Blocking

🔧 wrench

  • Cleartext EC ID logging: Full EC IDs are logged at warn!/error!/trace! levels in 5 locations across ec/mod.rs (lines 128, 211, 296), sync_pixel.rs (line 91), and pull_sync.rs (line 91). The log_id() truncation helper was introduced in this PR for exactly this purpose but is not used consistently. See inline comments for fixes.
  • Stale env var in docs: TRUSTED_SERVER__EDGE_COOKIE__SECRET_KEY still appears in configuration.md (lines 40, 956) and error-reference.md (line 154). The PR renamed the TOML section from [edge_cookie] to [ec] and updated some env var references but missed these. Users following the docs will set a variable that is silently ignored, resulting in a startup failure from the placeholder passphrase. Should be TRUSTED_SERVER__EC__PASSPHRASE.

❓ question

  • HttpOnly omitted from EC cookie (ec/cookies.rs:11): Intentionally omitted for a hypothetical future JS use case. Is there a concrete plan? The identify endpoint already exposes the EC ID. If not, HttpOnly would be cheap XSS defense-in-depth.

Non-blocking

🤔 thinking

  • Uppercase EC ID rejection in batch sync (batch_sync.rs:209): is_valid_ec_id rejects uppercase hex, but batch sync validates before normalizing (line 225). Partners submitting uppercase EC IDs get invalid_ec_id instead of normalization.

♻️ refactor

  • _pull_enabled index lacks CAS (partner.rs:564): Read-modify-write without generation markers. Concurrent partner registrations can overwrite each other's index updates. Self-healing via fallback, but inconsistent with the CAS discipline used elsewhere.

🌱 seedling

  • Integration tests don't verify identify response shape: FullLifecycle and ConcurrentPartnerSyncs only assert uids.<partner>. The ec, consent, degraded, eids, and cluster_size fields from the API reference are never checked.
  • Pixel sync failure paths untested end-to-end: ts_reason=no_ec, no_consent, domain mismatch redirects are unit-tested but not covered in integration tests.

📝 note

  • trusted-server.toml ships banned placeholder: passphrase = "trusted-server" is rejected by reject_placeholder_secrets. Works in CI via env override, but new contributors will hit a confusing startup error. A TOML comment would help.

⛏ nitpick

  • Eid/Uid missing Deserialize: openrtb.rs types derive Serialize only. If the auction ever needs to parse EIDs from provider responses, Deserialize will be needed.

CI Status

  • cargo fmt: PASS
  • clippy: PASS
  • cargo test: PASS
  • vitest: PASS
  • integration tests: FAIL
  • CodeQL: FAIL (likely related to cleartext logging findings above)

Comment thread crates/trusted-server-core/src/ec/mod.rs Outdated
Comment thread crates/trusted-server-core/src/ec/mod.rs
Comment thread crates/trusted-server-core/src/ec/sync_pixel.rs Outdated
Comment thread crates/trusted-server-core/src/ec/pull_sync.rs Outdated
Comment thread crates/trusted-server-core/src/ec/cookies.rs Outdated
Comment thread crates/trusted-server-core/src/ec/batch_sync.rs Outdated
Comment thread crates/trusted-server-core/src/ec/partner.rs Outdated
Copy link
Copy Markdown
Collaborator

@prk-Jr prk-Jr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Comprehensive EC identity subsystem: HMAC-based ID generation, KV identity graph with CAS concurrency, partner sync (pixel, batch, pull), identify endpoint, and auction bidstream decoration. The architecture is well-designed with clean separation of concerns, strong concurrency discipline, and solid security choices throughout. Four new blocking findings join the existing ones — three are input-validation gaps and one is a PII leak in the pull sync URL builder that should be resolved before merge.

Blocking

🔧 wrench

  • PII leak: client IP forwarded to partnerspull_sync.rs:273: build_pull_request_url appends the raw user IP as the ip query parameter on every outbound pull-sync request, exposing it in partner access logs. Contradicts the privacy-preserving design intent and is likely a GDPR Article 5 concern. Remove the ip parameter or gate it behind an explicit per-partner allow_ip_forwarding config flag.
  • Pull sync response body unboundedpull_sync.rs:325: take_body_bytes() with no size cap before JSON parsing. A malicious partner can exhaust WASM memory. Batch sync and admin endpoints both have MAX_BODY_SIZE guards; this path needs one too (64 KiB is generous for a {"uid":"..."} response).
  • Whitespace-only UIDs bypass validationbatch_sync.rs:217 and sync_pixel.rs:41: is_empty() passes " " and "\t", which get stored as garbage EID values in the KV graph. Replace with trim().is_empty() at both sites.
  • rand::thread_rng() WASM compatibility unverifiedgeneration.rs:52: requires getrandom with the wasi feature on wasm32-wasip1. Native cargo test passing does not prove the WASM build is safe; integration tests are already failing so this may not have been caught. Switch to OsRng or add getrandom = { version = "0.2", features = ["wasi"] } explicitly.

Non-blocking

🤔 thinking

  • process_mappings UpsertResult branches untestedbatch_sync.rs:232: NotFound, ConsentWithdrawn, and Stale have zero unit test coverage. Stale is documented as "counted as accepted" with no regression test.
  • Shared error messages make pull sync validation tests non-isolatingpartner.rs:152,165: both missing-URL and missing-token conditions return the identical error string, so the tests asserting on substrings pass even if the wrong branch fires. Use distinct messages per condition.

♻️ refactor

  • ec_consent_granted has no testsconsent.rs:20: the entry-point gate for all EC creation has no #[cfg(test)] section. Add smoke tests for granted and denied paths.
  • KV tombstone path never exercised in finalize testsfinalize.rs:149: all finalize tests pass kv: None, so the tombstone write and the cookie-EC ≠ active-EC case in withdrawal_ec_ids are untested.
  • Missing HMAC prefix stability testgeneration.rs:228: no test asserts that the same IP + passphrase always produces the same 64-char hash prefix, which is the core deduplication guarantee for the KV graph.
  • normalize_ec_id duplicated in integration testsintegration-tests/tests/common/ec.rs:374: reimplements normalize_ec_id_for_kv from core. Re-export and use the canonical implementation.

⛏ nitpick

  • Use saturating_sub for consistencykv.rs:605: the subtraction is safe due to the guard above but inconsistent with saturating_sub used throughout the rest of the module.
  • log_id should encapsulate the suffixmod.rs:51: every call site manually appends "…" in the format string; move it inside the function.

Praises 👍

  • CAS-based optimistic concurrency (kv.rs): bounded retries, graceful ItemPreconditionFailed handling, and MAX_CAS_RETRIES preventing infinite loops — textbook correct for a single-writer KV model.
  • Constant-time API key comparison (partner.rs): subtle::ConstantTimeEq for timing attack prevention on key lookups. Many implementations miss this entirely.
  • KvMetadata fast-path consent check (kv_types.rs): mirroring ok/country/cluster_size in metadata to avoid streaming the full KV body is an excellent performance optimisation with the right constraint test.
  • evaluate_known_browser with OnceLock-cached hash table (device.rs): pre-hashing fingerprints once and caching via OnceLock is the right WASM-compatible lazy-init pattern.
  • HMAC stability explicitly documented (generation.rs:30-33): noting that the output format is a "stable contract" that would invalidate all existing identities if changed is exactly the kind of correctness annotation that prevents future breakage.

CI Status

  • cargo fmt: PASS
  • cargo clippy: PASS
  • cargo test: PASS
  • vitest: PASS
  • integration tests: FAIL
  • CodeQL: FAIL (cleartext logging — covered in prior review)

Comment thread crates/trusted-server-core/src/ec/pull_sync.rs Outdated
Comment thread crates/trusted-server-core/src/ec/pull_sync.rs
Comment thread crates/trusted-server-core/src/ec/batch_sync.rs Outdated
Comment thread crates/trusted-server-core/src/ec/sync_pixel.rs Outdated
Comment thread crates/trusted-server-core/src/ec/generation.rs
Comment thread crates/trusted-server-core/src/ec/finalize.rs
Comment thread crates/trusted-server-core/src/ec/generation.rs
Comment thread crates/integration-tests/tests/common/ec.rs Outdated
Comment thread crates/trusted-server-core/src/ec/kv.rs Outdated
Comment thread crates/trusted-server-core/src/ec/mod.rs Outdated
Comment thread crates/trusted-server-core/src/ec/kv.rs Fixed
Comment thread crates/trusted-server-core/src/ec/kv.rs Fixed
Comment thread crates/trusted-server-core/src/storage/kv_store.rs Fixed
Copy link
Copy Markdown
Collaborator

@aram356 aram356 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR lands a large, well-partitioned Edge Cookie identity subsystem, and the prior blocking feedback (PII logging via log_id(), whitespace UID checks, pull-sync response size cap, dropping raw client IP from pull-sync query strings, constant-time auth comparisons) has been materially addressed. Blocking issues remaining are: a CI-red integration test (deterministic, root cause in the test client, not the server), an unverified rand::thread_rng() on wasm32-wasip1 that CI cannot catch because no wasm build runs, two new CodeQL alerts on ec/kv.rs riding the pre-existing settings-taint false-positive flow, one untested backend-state machine in batch_sync, and two questions on the auction / partner surfaces.

Blocking

🔧 wrench

  • CI red — test_ec_lifecycle_fastly deterministic failure: the test client omits Sec-Fetch-Dest: document and Accept: text/html, so is_navigation_request() correctly returns false and EC never generates (crates/integration-tests/tests/common/ec.rs:93).
  • rand::thread_rng() on wasm32-wasip1 unverified; CI builds no wasm target (crates/trusted-server-core/src/ec/generation.rs:52). Add a wasm build to CI and/or switch to an explicit getrandom with the wasi feature.
  • Two new CodeQL alerts on ec/kv.rs:637 and :789 ride the same settings.reject_placeholder_secrets() taint flow that already produces noise on main. Both are false positives but block the CodeQL gate. Fix with a scoped suppression or a repo-level CodeQL config filter on rust/cleartext-logging for settings-derived values.
  • UpsertResult::{NotFound, ConsentWithdrawn, Stale} untested in batch_sync::process_mappings (crates/trusted-server-core/src/ec/batch_sync.rs:231-244). Prior reviewer flagged this; still unresolved.

❓ question

  • Auction user.id = "" on the no-consent path is emitted into the OpenRTB bid request JSON (crates/trusted-server-core/src/auction/endpoints.rs:60-77, formats.rs:186). Has this been validated against the live Prebid Server target? If not, change UserInfo.id to Option<String> with skip_serializing_if.
  • update_pull_enabled_index race self-heal path untested (crates/trusted-server-core/src/ec/partner.rs:552-604). The documented fallback at partner.rs:350 is the only thing keeping the index correct under concurrent upserts — please add a concurrency test or file a tracked follow-up.

Non-blocking

🤔 thinking

  • MAX_CAS_RETRIES = 3, no backoff — likely to starve under contention on hot prefixes; consider exponential backoff + jitter (crates/trusted-server-core/src/ec/kv.rs:300-339).
  • HashMap<String, KvPartnerId> non-deterministic serialization — breaks future hash-based dedup and byte-diffing of stored values; use BTreeMap or IndexMap (crates/trusted-server-core/src/ec/kv_types.rs:59).
  • seen_domains drop-newest eviction freezes long-lived ECs on their first 50 domains, defeating the last timestamp; switch to LRU or document (crates/trusted-server-core/src/ec/kv.rs:627-642).

♻️ refactor

  • EID gating failures log at debug! — bump to warn! so ad-stack anomalies surface in production (crates/trusted-server-core/src/auction/endpoints.rs:80).
  • ec_consent_granted() is a 1-line pass-through — inline or document the sealing-point intent (crates/trusted-server-core/src/ec/consent.rs:20-22).
  • Unvalidated domain strings in EC graph writes — add a 255-byte length cap and hostname-shape check at the write boundary (crates/trusted-server-core/src/ec/kv.rs update_last_seen, kv_types.rs).

🌱 seedling

  • No integration test for the GPC / consent-denied identify path. Once the Sec-Fetch-Dest fix lands, add ec_full_lifecycle_with_gpc sending Sec-GPC: 1 and asserting /_ts/api/v1/identify returns 403 / no EID payload (crates/integration-tests/tests/frameworks/scenarios.rs:501-565).
  • No wasm-target build in CI — root cause of how the rand::thread_rng() concern reached this point unnoticed. Adding cargo build -p trusted-server-adapter-fastly --target wasm32-wasip1 would catch an entire class of deps-pin regressions.

📌 out of scope

  • Pull-sync EC ID still travels as a URL query parameter — acknowledged in the PR description as deferred. Please file the follow-up issue and reference it from the commit that ships this PR so it is not lost (crates/trusted-server-core/src/ec/pull_sync.rs:260-263).

CI Status

  • cargo fmt: PASS
  • cargo clippy: PASS
  • cargo test (native): PASS
  • vitest: PASS
  • integration tests: FAILtest_ec_lifecycle_fastly (see blocking #1)
  • CodeQL: FAIL — 15 high alerts; 13 are pre-existing settings.reject_placeholder_secrets() taint-flow false positives already present on main, 2 are new on ec/kv.rs riding the same flow (see blocking #3)

Comment thread crates/integration-tests/tests/common/ec.rs
Comment thread crates/trusted-server-core/src/ec/generation.rs
Comment thread crates/trusted-server-core/src/ec/kv.rs
Comment thread crates/trusted-server-core/src/ec/batch_sync.rs
Comment thread crates/trusted-server-core/src/auction/endpoints.rs Outdated
Comment thread crates/trusted-server-core/src/ec/kv_types.rs Outdated
Comment thread crates/trusted-server-core/src/ec/kv.rs
Comment thread crates/trusted-server-core/src/auction/endpoints.rs Outdated
Comment thread crates/trusted-server-core/src/ec/consent.rs
Comment thread crates/trusted-server-core/src/ec/kv.rs
- Rename 'Synthetic ID' to 'Edge Cookie (EC)' across all external-facing
  identifiers, config, internal Rust code, and documentation
- Simplify EC hash generation to use only client IP (IPv4 or /64-masked
  IPv6) with HMAC-SHA256, removing User-Agent, Accept-Language,
  Accept-Encoding, random_uuid inputs and Handlebars template rendering
- Downgrade EC ID generation logs to trace level since client IP and EC
  IDs are sensitive data
- Remove unused counter_store and opid_store config fields and KV store
  declarations (vestigial from template-based generation)
- Remove handlebars dependency

Breaking changes: wire field synthetic_fresh → ec_fresh, response headers
X-Synthetic-ID → X-TS-EC, cookie synthetic_id → ts-ec, query param
synthetic_id → ts-ec, config section [synthetic] → [edge_cookie].

Closes #462
The EC rename commit (984ba2b) accidentally re-introduced the
reject_placeholder_secrets() call and InsecureDefault tests that were
intentionally removed in 4c29dbf. Replace with log::warn() for
placeholder detection and restore the simple smoke test.
…igration

- Add ec/ module with EcContext lifecycle, generation, cookies, and consent
- Compute cookie domain from publisher.domain, move EC cookie helpers
- Fix auction consent gating, restore cookie_domain for non-EC cookies
- Add integration proxy revocation, refactor EC parsing, clean up ec_hash
- Remove fresh_id and ec_fresh per EC spec §12.1
- Migrate [edge_cookie] config to [ec] per spec §14
Implement Story 3 (#536): KV-backed identity graph with compare-and-swap
concurrency, partner ID upserts, tombstone writes for consent withdrawal,
and revive semantics. Includes schema types, metadata, 300s last-seen
debounce, and comprehensive unit tests.

Also incorporates earlier foundation work: EC module restructure, config
migration from [edge_cookie] to [ec], cookie domain computation, consent
gating fixes, and integration proxy revocation support.
Implement Story 4 (#537): partner KV store with API key hashing,
POST /admin/partners/register with basic-auth protection, strict
field validation (ID format, URL allowlists, domain normalization),
and pull-sync config validation. Includes index-based API key lookup
and comprehensive unit tests.
Implement Story 5 (#538): centralize EC cookie set/delete and KV
tombstone writes in finalize_response(), replacing inline mutation
scattered across publisher and proxy handlers. Adds consent-withdrawal
cleanup, EC header propagation on proxy requests, and docs formatting.
Implement Story 8 (#541): POST /api/v1/sync with Bearer API key auth,
per-partner rate limiting, batch size cap, per-mapping validation and
rejection reasons, 200/207 response semantics, tolerant Bearer parsing,
and KV-abort on store unavailability.
Implement Story 9 (#542): server-to-server pull sync that runs after
send_to_client() on organic traffic only. Refactors the Fastly adapter
entrypoint from #[fastly::main] to explicit Request::from_client() +
send_to_client() to enable post-send background work.

Pull sync enumerates pull-enabled partners, checks staleness against
pull_sync_ttl_sec, validates URL hosts against the partner allowlist,
enforces hourly rate limits, and dispatches concurrent outbound GETs
with Bearer auth. Responses with uid:null or 404 are no-ops; valid
UIDs are upserted into the identity graph.

Includes EC ID format validation to prevent dispatch on spoofed values,
partner list_registered() for KV store enumeration, and configurable
pull_sync_concurrency (default 3).
Implement Story 11 (#544): Viceroy-driven E2E tests covering full EC
lifecycle (generation, pixel sync, identify, batch sync, consent
withdrawal, auth rejection). Adds EC test helpers with manual cookie
tracking, minimal origin server with graceful shutdown, and required
KV store fixtures. Fixes integration build env vars.
Consolidate is_valid_ec_hash and current_timestamp into single canonical
definitions to eliminate copy-paste drift across the ec/ module tree. Fix
serialization error variants in admin and batch_sync to use Ec instead of
Configuration. Add scaling and design-decision documentation for partner
store enumeration, rate limiter burstiness, and plaintext pull token storage.
Use test constructors consistently in identify and finalize tests.
- Rename ssc_hash → ec_hash in batch sync wire format (§9.3)
- Strip x-ts-* prefix headers in copy_custom_headers (§15)
- Strip dynamic x-ts-<partner_id> headers in clear_ec_on_response (§5.2)
- Add PartnerNotFound and PartnerAuthFailed error variants (§16)
- Rename Ec error variant → EdgeCookie (§16)
- Validate EC IDs at read time, discard malformed values (§4.2)
- Add rotating hourly offset for pull sync partner dispatch (§10.3)
- Add _pull_enabled secondary index for O(1+N) pull sync reads (§13.1)
…nd cleanup

- Add body size limit (64 KiB) to partner registration
- Validate partner UID length (max 512 bytes) in batch sync and sync pixel
- Replace linear scan with binary search in encode_eids_header
- Use constant-time comparison inline in partner lookup, remove unused verify_api_key
- Remove unused PartnerAuthFailed error variant, fix PartnerNotFound → 404
- Add Access-Control-Max-Age CORS header to identify endpoint
- Tighten consent-denied integration test to expect only 403
- Add stability doc-comment to normalize_ip
- Log warning instead of silent fallback on SystemTime failure
…ror variants

Resolve integration issues from rebasing onto feature/ssc-update:
- Restore prepare_runtime() and validate_cookie_domain() lost in conflict resolution
- Add InsecureDefault error variant and wire reject_placeholder_secrets() into get_settings()
- Add sha2/subtle imports for constant-time auth comparison
- Fix error match arms (Ec → EdgeCookie, remove nonexistent PartnerAuthFailed)
- Fix orchestrator error handling to use send_to_client() pattern
- Remove dead cookie helpers superseded by ec/cookies module
… sync coverage

Fix integration test deterministic failure by adding Sec-Fetch-Dest: document
and Accept: text/html headers to EcTestClient.get() so is_navigation_request()
recognizes organic requests. Update all EC scenarios for the new architecture:
config-based partners, Bearer-authenticated identify with scoped responses,
batch sync instead of pixel sync.

Change UserInfo.id from String to Option<String> with skip_serializing_if to
avoid emitting empty user.id in OpenRTB bid requests on the no-consent path.

Add batch_sync unit tests for UpsertResult::{NotFound, ConsentWithdrawn, Stale}
rejection paths (previously untested).

Add safety comments on ec/kv.rs log sites where CodeQL flags false positives
(log_id() already truncates the EC ID).

Add integration test partners (inttest, inttest2) to trusted-server.toml.
Read the sharedId cookie (set by Prebid's SharedID User ID Module) on
organic requests and store the value as a partner UID in the EC identity
graph. Uses the same debounce and best-effort semantics as the ts-eids
cookie ingestion path.

Add sharedid partner config (source_domain: sharedid.org, atype: 1) to
trusted-server.toml. The backend matches the cookie value to this partner
via PartnerRegistry.find_by_source_domain() during ec_finalize_response.
… module

Origin/main introduced RuntimeServices platform abstraction (PR2-PR6) after
our branch diverged. The rebase left inconsistencies between the trait
definitions from our branch and the implementations from origin/main.

Restore source files to their ORIG_HEAD state to reconcile:
- Remove services: &RuntimeServices from IntegrationProxy::handle
  implementations (datadome, didomi, gpt, lockr, permutive, testlight, gtm)
- Remove services from run_auction, proxy_request, AuctionContext
- Restore publisher, proxy, orchestrator, types to branch-correct state
- Restore streaming/html/rsc_flight modules to branch state
@ChristianPavilonis ChristianPavilonis force-pushed the feature/edge-cookies-final branch from 7009838 to 43df212 Compare April 16, 2026 22:14
Comment thread crates/trusted-server-core/src/ec/kv.rs Fixed
Copy link
Copy Markdown
Collaborator

@aram356 aram356 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Large PR (98 files, +11,974 / −3,099). Architecture and consent plumbing are solid; this review focuses on concrete defects and hardening gaps verified against 109c929c. Three CI checks currently fail (integration tests, format-typescript, CodeQL); two of those are trivially fixable and inline-commented below. The CodeQL failure appears to be a false positive that needs a team decision.

Blocking

🔧 wrench

  • Integration tests fail to compile: crates/integration-tests/Cargo.toml missing trusted-server-core dev-dependency while tests/common/ec.rs:19 and tests/frameworks/scenarios.rs:5 import it. (inline)
  • format-typescript CI failure: crates/js/lib/src/integrations/prebid/index.ts:413 fails prettier — one-line fix via npm run format. (inline)
  • Unbounded cookie payloads: ec/prebid_eids.rs:43 (ts-eids) and ec/prebid_eids.rs:111 (sharedId) decode/clone attacker-controlled cookie values with no size ceiling. (inline)

❓ question

  • Consent-withdrawal split-brain at ec/finalize.rs:56: consent-store delete failures are logged-and-continued while KV tombstones still write. Intentional (KV = source of truth) or should withdrawal fail-closed? (inline)
  • CodeQL cross-cutting: 15 high-severity rust/cleartext-logging alerts all name settings.reject_placeholder_secrets() as the tainted source, pinned to log-call lines that do not actually reference that method (e.g. auction/orchestrator.rs:125 is a timeout warning). This is taint-analyzer propagation through a boolean validator that uses .expose() internally; the logged bytes are field names, not secret material. How would the team like to resolve: (a) #[allow] on the validator with a comment explaining the false positive, (b) rename reject_placeholder_secretsvalidate_secrets_not_placeholders to break the taint-name heuristic, or (c) configure a CodeQL suppression?

Non-blocking

🤔 thinking

  • Bot gate is presence-only (ec/device.rs:76) — attackers with spoofed JA4 pass. Document threat model. (inline)
  • user.id = None when EC not allowed (auction/endpoints.rs:60) — correct for consent, but may depress bidder dedup/yield. Consider a session-scoped ephemeral ID. (inline)
  • Pull-sync only on organic routes (trusted-server-adapter-fastly/src/main.rs:373) — auction-heavy SPA publishers never refresh KV. Document or extend. (inline)

♻️ refactor

  • Duplicated bearer parser in ec/identify.rs:34 and ec/batch_sync.rs:101 — extract to ec/auth.rs. (inline)
  • Rate-limiter hourly→minute conversion is lossy by up to ~2× (ec/rate_limiter.rs:59) — document and test the overshoot bound. (inline)

🌱 seedling

  • Non-deterministic EID order on timestamp ties (ec/eids.rs:64) — add partner_id as secondary sort key. (inline)
  • No post-deserialize bounds on KvEntry (ec/kv.rs:158) — legacy/corrupt oversized entries will round-trip through. (inline)
  • platform/test_support.rs defines stubs (NoopConfigStore, NoopSecretStore, noop_services) that no unit test exercises, so trait drift will silently break them. Either consume in ≥1 unit test or remove.
  • Auction ext metadata unsrialized: auction/formats.rs computes strategy/provider-count/timing into result.metadata but never emits it into ext.trustedServer on the OpenRTB response — a useful debug surface.

⛏ nitpick

  • has_cookie_mismatch naming (ec/mod.rs:398) — ambiguous; prefer header_overrides_cookie. (inline)

📌 out of scope

  • ec_id/client_ip as query parameters in pull sync — PR description acknowledges as follow-up.

CI Status

  • cargo fmt: PASS
  • clippy: PASS
  • cargo test (workspace): PASS
  • vitest: PASS
  • format-typescript: FAIL (see inline)
  • integration tests: FAIL — compile error (see inline)
  • CodeQL: FAIL — 15 high (likely false positives) (see question above)
  • browser integration tests: PASS
  • format-docs: PASS

Comment thread crates/integration-tests/Cargo.toml
Comment thread crates/js/lib/src/integrations/prebid/index.ts Outdated
Comment thread crates/trusted-server-core/src/ec/prebid_eids.rs
Comment thread crates/trusted-server-core/src/ec/prebid_eids.rs
Comment thread crates/trusted-server-core/src/ec/finalize.rs
Comment thread crates/trusted-server-core/src/ec/identify.rs Outdated
Comment thread crates/trusted-server-core/src/ec/rate_limiter.rs
Comment thread crates/trusted-server-core/src/ec/eids.rs Outdated
Comment thread crates/trusted-server-core/src/ec/kv.rs
Comment thread crates/trusted-server-core/src/ec/mod.rs Outdated
ChristianPavilonis and others added 2 commits April 20, 2026 16:02
… sensitive information'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
@ChristianPavilonis
Copy link
Copy Markdown
Collaborator Author

Addressed the remaining merge blockers in 4d2ef30:

  • added the missing trusted-server-core dev-dependency for crates/integration-tests
  • formatted crates/js/lib/src/integrations/prebid/index.ts to clear the Prettier failure
  • added raw size guards for ts-eids and sharedId cookie ingestion, with focused regression tests
  • removed the remaining EC-ID-bearing log fields from the two CodeQL false-positive sites to unblock the cleartext-logging findings

Validation run locally:

  • cargo test --workspace
  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cd crates/js/lib && npx vitest run
  • cd crates/js/lib && npx prettier --check src/integrations/prebid/index.ts
  • cargo test --manifest-path crates/integration-tests/Cargo.toml --target "$(rustc -vV | sed -n 's/^host: //p')" --no-run

Comment thread crates/trusted-server-core/src/ec/kv.rs Fixed
Comment thread crates/trusted-server-core/src/storage/kv_store.rs Fixed
ChristianPavilonis and others added 3 commits April 21, 2026 09:00
… sensitive information'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
… sensitive information'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
@ChristianPavilonis
Copy link
Copy Markdown
Collaborator Author

Follow-up update in 6016c48 to address the remaining open review threads:

  • extracted shared Bearer auth to crates/trusted-server-core/src/ec/auth.rs and removed duplication from identify.rs / batch_sync.rs
  • made EID ordering deterministic on synced ties with a stable partner_id secondary key
  • documented + tested the rate-limiter hourly→per-minute rounding behavior
  • added post-deserialize validation for loaded KvEntry values
  • renamed has_cookie_mismatch to cookie_differs_from_active_ec
  • documented the withdrawal authority model (KV tombstone authoritative, consent-store cleanup best-effort)
  • clarified the browser-heuristic threat model, the no-consent auction identity choice, and the current pull-sync trigger contract

Validation re-run locally:

  • cargo test --workspace
  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --all-features -- -D warnings

I also replied to and resolved the remaining open review threads tied to these changes.

@ChristianPavilonis ChristianPavilonis linked an issue Apr 21, 2026 that may be closed by this pull request
Copy link
Copy Markdown
Collaborator

@prk-Jr prk-Jr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR lands the Edge Cookie identity system end to end, but I found two merge blockers in the request lifecycle.

Blocking

🔧 wrench

  • Identify GET still missing browser-direct CORS support: OPTIONS /_ts/api/v1/identify reflects CORS headers, but GET /_ts/api/v1/identify never applies them, so the documented browser-direct flow still fails after a successful preflight. classify_origin() also accepts matching hosts regardless of scheme, while the spec only allows https origins.

Fix: classify origin in the GET path too, return 403 for denied origins, reflect CORS headers on allowed GET responses, and require origin_url.scheme() == "https" in classify_origin(). (crates/trusted-server-core/src/ec/identify.rs:31, crates/trusted-server-core/src/ec/identify.rs:171)

CI Status

  • cargo fmt: PASS
  • cargo test: PASS
  • vitest: PASS
  • browser integration tests: PASS
  • integration tests: FAIL (test_ec_lifecycle_fastly: organic GET / should set ts-ec cookie)
  • CodeQL: FAIL (12 open alerts on the merge ref)

Comment thread crates/trusted-server-adapter-fastly/src/main.rs Outdated
Comment thread crates/trusted-server-core/src/ec/identify.rs
Copy link
Copy Markdown
Collaborator

@aram356 aram356 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Edge Cookie identity subsystem is materially complete and the prior review's blockers have been addressed: navigation-headers fix unblocks test_ec_lifecycle_fastly, UpsertResult variants are tested, user.id is Option<String> with skip_serializing_if, the partner-registry race is gone (architecture is now config-derived), MAX_CAS_RETRIES is 5 with backoff+jitter, raw client IP is no longer forwarded to pull-sync partners, and the WASM build now runs in CI.

Remaining blocking issues: a plaintext partner pull-token sitting on a Debug-deriving struct, an unintentional-looking read/write asymmetry in the bot gate, and a still-failing CodeQL gate riding the same Settings Debug taint-flow false positive that was flagged on the prior review.

Blocking

🔧 wrench

  • Plaintext ts_pull_token on PartnerConfig (crates/trusted-server-core/src/ec/registry.rs:44, :208)

❓ question

  • Bot-gate read/write asymmetry intentional? (crates/trusted-server-adapter-fastly/src/main.rs:208-212 vs. :272-276)
  • CodeQL CI gate is failing on 12 new rust/cleartext-logging alerts, all riding the settings.reject_placeholder_secrets()log::debug!("Settings {settings:?}") taint flow that was flagged on the prior review. Settings Debug is redacted via Redacted<T>, so the alerts are benign — but they block the gate. Sites: crates/trusted-server-adapter-fastly/src/main.rs:71, crates/trusted-server-core/src/publisher.rs:178/:341/:382, crates/trusted-server-core/src/html_processor.rs:70, crates/trusted-server-core/src/consent/mod.rs:173, crates/trusted-server-core/src/auction/orchestrator.rs:125/:278/:338/:408, crates/trusted-server-core/src/integrations/nextjs/html_post_process.rs:123/:456. Plan to suppress at PR level or add a repo-wide CodeQL filter on Redacted-derived flows before merge?

Non-blocking

🤔 thinking

  • std::thread::sleep in CAS retry loop on wasm32-wasip1 (crates/trusted-server-core/src/ec/kv.rs:343, :426, :471, :558)
  • seen_domains still uses HashMap while ids was converted to BTreeMap (crates/trusted-server-core/src/ec/kv_types.rs:133)
  • MIN_PASSPHRASE_LENGTH = 8 is weak entropy for the HMAC-SHA256 anchor (crates/trusted-server-core/src/settings.rs:353)
  • seen_domains drop-newest eviction freezes long-lived ECs at the 50-domain cap (crates/trusted-server-core/src/ec/kv.rs:667-683) — repeat from prior review

♻️ refactor

  • PartnerRegistry::all() exposes HashMap iteration order to callers (crates/trusted-server-core/src/ec/registry.rs:174-176)
  • RateLimiter::exceeded_per_minute round-trips through hourly math (crates/trusted-server-core/src/ec/rate_limiter.rs:42-46, :49-52)

🌱 seedling

  • No end-to-end integration tests for batch sync 429 / 400 / 413 paths (crates/integration-tests/tests/frameworks/scenarios.rs:761-779)
  • Bot-gate observability hook (crates/trusted-server-core/src/ec/device.rs:84-86) — operators have no metric for what fraction of traffic is excluded

📌 out of scope

  • Repo-level CodeQL filter for the Redacted-derived rust/cleartext-logging taint flow so this stops gating PRs. Second consecutive PR review where the same false-positive flow blocks CI.

⛏ nitpick

  • derive(Debug) on PartnerRegistry / PartnerConfig (crates/trusted-server-core/src/ec/registry.rs:17, :53) — combined with the plaintext token wrench, this is the loaded gun.

CI Status

  • cargo fmt: PASS
  • cargo clippy: PASS
  • cargo test (workspace): PASS
  • vitest: PASS
  • integration tests: PASS
  • browser integration tests: PASS
  • WASM release build: PASS
  • CodeQL: FAIL — 12 new alerts, all on the same Settings Debug taint-flow false-positive flow (see ❓ above)

Comment thread crates/trusted-server-core/src/ec/registry.rs
Comment thread crates/trusted-server-adapter-fastly/src/main.rs
Comment thread crates/trusted-server-core/src/ec/kv.rs
Comment thread crates/trusted-server-core/src/ec/kv_types.rs
Comment thread crates/trusted-server-core/src/settings.rs
Comment thread crates/trusted-server-core/src/ec/registry.rs
Comment thread crates/trusted-server-core/src/ec/rate_limiter.rs
Comment thread crates/integration-tests/tests/frameworks/scenarios.rs
Comment thread crates/trusted-server-core/src/ec/device.rs
Comment thread crates/trusted-server-core/src/ec/registry.rs
Copy link
Copy Markdown
Collaborator Author

@ChristianPavilonis ChristianPavilonis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearing an old pending local review so I can reply to the current threads from the branch update.

Comment thread crates/integration-tests/tests/common/ec.rs
Comment thread crates/trusted-server-core/src/ec/kv.rs Fixed
Comment thread crates/trusted-server-core/src/ec/kv.rs Fixed
Comment thread crates/trusted-server-core/src/storage/kv_store.rs Fixed
Comment thread crates/trusted-server-core/src/ec/generation.rs
Comment thread crates/trusted-server-core/src/ec/eids.rs Outdated
Comment thread crates/trusted-server-core/src/ec/kv.rs
Comment thread crates/trusted-server-core/src/ec/mod.rs Outdated
Comment thread crates/trusted-server-adapter-fastly/src/main.rs Outdated
Comment thread crates/trusted-server-core/src/ec/identify.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

4 participants