Skip to content

Bump clarity to 4.0.0 and cache FieldPath lookups (~18% faster)#84

Open
spheenik wants to merge 2 commits intoodota:masterfrom
spheenik:clarity-4.0.0
Open

Bump clarity to 4.0.0 and cache FieldPath lookups (~18% faster)#84
spheenik wants to merge 2 commits intoodota:masterfrom
spheenik:clarity-4.0.0

Conversation

@spheenik
Copy link
Copy Markdown

Summary

Two independent commits:

  1. Cache FieldPath lookups in getEntityProperty — clarity's getFieldPathForName walks the field tree and allocates a fresh S2ModifiableFieldPath on every call, but the (DTClass, property-name) → FieldPath mapping is stable for the lifetime of a parse. Memoizing it eliminates ~5.6M redundant tree walks per Dota replay.

  2. Bump clarity 3.1.3 → 4.0.0. Source-level changes:

    • DOTA_COMBATLOG_TYPES moved from DOTAUserMessages into a dedicated DOTACombatLog wrapper (clarity-protobuf 5.4 → 6.1) — import swap in 4 files.
    • Custom OnWardKilled / OnWardExpired / OnWardPlaced annotations ported to clarity 4.0's typed-event pattern (nested Listener + Event interfaces, @GenerateEvent, parameterClasses dropped from @UsagePointMarker).
    • Wards processor uses the new single-arg createEvent and typed Event field types.
    • maven-shade plugin: ClassIndexTransformer (atteo) → AppendingTransformer for META-INF/clarity/providers.txt. Clarity 4.0 ships its own EventAnnotationProcessor that writes per-module provider lists to that path; without the new transformer the shaded jar would only carry clarity's own list and the Wards processor wouldn't be discovered at runtime.

Performance

Bench setup: 195 MB modern Dota S2 replay (dota/s2/340/8168882574_1198277651.dem), OpenJDK 21, 2 warmup + 10 timed runs, ByteArrayInputStream → OutputStream.nullOutputStream().

version median vs 3.1.3 baseline
3.1.3 baseline (today) 4.384s
3.1.3 + FieldPath cache 3.953s −9.8%
4.0.0 + FieldPath cache 3.609s −17.7%

Spreads under 4% on all configurations; ranges non-overlapping. Cache hit rate 99.97% over 5.6M getEntityProperty calls per parse.

Cosmetic caveat — event casts

Wards.java now contains lines like:
```java
evKilled = (OnWardKilled.Event) ctx.createEvent(OnWardKilled.class);
```
That cast is unfortunate. Clarity 4.0.0's Context.createEvent returns Event<A> (the base class), but the runtime instance is already the typed subclass. The next clarity release widens the signature to <E extends Event<A>> E createEvent(Class<A>), so the cast disappears with no source change here — just a clarity version bump.

Looking ahead

The next clarity release brings further improvements that are expected to trim parse time on this workload again. Happy to open a follow-up PR once it ships.

Test plan

  • `mvn -DskipTests package` builds clean.
  • Run against a representative modern S2 replay; JSON-lines output should match 3.1.3 modulo whitespace (no semantic Parse.java changes outside the imports / annotation rewrites).
  • If you have a CI corpus, this PR should not regress it; the FieldPath cache is purely additive and the typed-event port preserves the same listener wiring.

🤖 Generated with Claude Code

spheenik and others added 2 commits April 22, 2026 21:26
Memoize per-DTClass FieldPath lookups so each (DTClass, property name)
pair walks the field tree only once. Caller-side hits are 99.97% on a
representative match; ~10% wall-clock parse-time reduction on a 195MB
S2 replay.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bumps the clarity dependency from 3.1.3 to 4.0.0. Source-level
adjustments:

- DOTA_COMBATLOG_TYPES moved out of DOTAUserMessages into its own
  DOTACombatLog wrapper class (clarity-protobuf 5.4 -> 6.1). Imports
  updated in Parse, TrackVisitor, GreevilsGreedVisitor, Wards.

- Custom event annotations (OnWardKilled / OnWardExpired / OnWardPlaced)
  ported to the typed-event pattern: nested Listener and Event
  interfaces, @GenerateEvent, parameterClasses dropped from
  @UsagePointMarker. Wards now holds typed Event fields and uses the
  single-arg createEvent with a cast (cast goes away once clarity ships
  the next release with a generic-return overload).

- maven-shade plugin: ClassIndexTransformer (atteo) replaced by an
  AppendingTransformer for META-INF/clarity/providers.txt, the
  per-module file written by clarity 4.0.0's EventAnnotationProcessor
  to declare local @provides processors. Without this transformer the
  shaded jar would only carry clarity's own providers list and the
  Wards processor would not be discovered at runtime.

Verified end-to-end on dota/s2/340/8168882574_1198277651.dem (195 MB):
the Wards processor is discovered, the parse runs to completion, and
median wall-clock drops from 4.384s on 3.1.3 to 3.609s on 4.0.0
(combined with the FieldPath cache from the previous commit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@spheenik
Copy link
Copy Markdown
Author

spheenik commented Apr 22, 2026

Quick follow-up for the curious: I also pushed a preview branch spheenik/parser:clarity-5.0.0 that bumps to clarity 5.0.0-SNAPSHOT (the upcoming next release, just published to Maven Central Snapshots). The port-forward delta is small:

  • Two call sites switch from dtClass.getFieldPathForName(name) to the Entity convenience method entity.getFieldPathForName(name) (clarity 5 threads EntityState through path resolution).
  • The three (OnWardX.Event) ctx.createEvent(OnWardX.class) casts in Wards.java go away — clarity 5's Context.createEvent gains a typed-return overload, so the typed nested interface is inferred from the assignment target. The cosmetic caveat from the PR description disappears entirely on 5.0.

Same bench protocol as in this PR (2 warmup + 10 timed runs, 195 MB modern Dota S2 replay):

version median vs 3.1.3 baseline
3.1.3 baseline 4.384s
3.1.3 + FieldPath cache 3.953s −9.8%
4.0.0 + FieldPath cache (this PR) 3.609s −17.7%
5.0.0-SNAPSHOT + cache 3.156s −28.0%

Worth noting: clarity 5.0 also drastically reduces allocations (a new flat entity-state representation cuts GC pressure by roughly 65% on this replay family vs 4.0). For a server-style workload like odota's parser that handles many replays back-to-back, that's likely an even bigger win than the wall-clock number above suggests.

One more worth-knowing: clarity 5.0 raises the runtime baseline to Java 21 (the sealed-type refactor uses exhaustive switch, which forces source/target=21). odota's pom is already on 21 so no change needed, but downstream consumers of the parser jar would need to be on 21 too.

Sharing FYI — the diff is right there for whenever 5.0.0 lands on Central proper. That said, if you're feeling adventurous, the snapshot is reasonably well-tested already (clarity-analyzer and the in-tree examples track it), so consuming it ahead of the GA isn't unsafe — just understand it's a moving target until release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant