feat: Add performance observability with tags/metrics separation#52
Open
sumedhsakdeo wants to merge 3 commits intolinkedin:li-0.11from
Open
feat: Add performance observability with tags/metrics separation#52sumedhsakdeo wants to merge 3 commits intolinkedin:li-0.11from
sumedhsakdeo wants to merge 3 commits intolinkedin:li-0.11from
Conversation
sumedhsakdeo
changed the title
3 tasks
Add a pluggable PerfObserver pattern to instrument ArrowScan, manifest reading, delete file resolution, and scan planning with duration tracking and typed attributes split into tags (dimensions) and metrics (values) for compatibility with Prometheus/OTEL/Kafka backends. Includes CompositeObserver with exception isolation, thread-safe global observer, zero-overhead NullPerfObserver fast path, and @timed decorator with _perf_ctx injection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix mypy incompatible-types in LoggingPerfObserver by renaming loop variable to avoid str/int|float conflict - Fix test fixture return type to Iterator[None] for generator fixture - Remove unused type: ignore comment on yield - Restructure test_emits_on_exception to avoid mypy unreachable-statement with pytest.raises - Fix pydocstyle D401: use imperative mood in timed() docstring Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move pyiceberg.observability import after pyiceberg.manifest to satisfy ruff import ordering (I001). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
sumedhsakdeo
force-pushed
the
ssakdeo/perf-observability
branch
from
98b2c56 to
269a0c2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PerfObserverpattern for performance instrumentation of pyiceberg internalspyiceberg/observability.pymodule withPerfEvent,PerfObserverprotocol,NullPerfObserver,LoggingPerfObserver,CompositeObserver,perf_timercontext manager, and@timeddecoratorPerfEventseparatestags(low-cardinality dimensions) frommetrics(measured values) for Prometheus/OTEL/Kafka compatibilityNullPerfObserveris active (skipstime.monotonic()entirely)threading.LockCompositeObserverisolates exceptions so one failing observer doesn't block others@timeddecorator injects_perf_ctxkwarg for runtime attribute settingarrow.read_file,arrow.to_record_batches,arrow.read_delete_files,scan.open_manifest,scan.plan_files,scan.plan_files_local,manifest.fetch_entries,manifest.read_listTest plan
pytest tests/test_observability.py)🤖 Generated with Claude Code