Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
a75604a
feat: define `PreparedWrite` and `SupportsChunkPacking` data structures
d-v-b Apr 7, 2026
a072c31
Merge branch 'main' of https://github.com/zarr-developers/zarr-python…
d-v-b Apr 7, 2026
47a407f
feat: new codec pipeline that uses sync path
d-v-b Apr 7, 2026
3c27e49
feat: complete second codecpipeline
d-v-b Apr 8, 2026
9b834a4
Merge branch 'main' of https://github.com/zarr-developers/zarr-python…
d-v-b Apr 8, 2026
c731cf2
fix: handle rectilinear chunks
d-v-b Apr 8, 2026
9e25150
Merge branch 'main' of https://github.com/zarr-developers/zarr-python…
d-v-b Apr 8, 2026
ae0580c
fixup
d-v-b Apr 9, 2026
053f2ee
fix: fixup
d-v-b Apr 9, 2026
ba393d3
Merge branch 'main' of github.com:zarr-developers/zarr-python into pe…
d-v-b Apr 9, 2026
cfe9539
fix: wire up prototype in setitem
d-v-b Apr 9, 2026
0b2512b
refactor: define chunklayout class
d-v-b Apr 9, 2026
5fb28b9
perf: only fetch the chunks we need
d-v-b Apr 9, 2026
9b620c0
docs: add design spec for PhasedCodecPipeline performance fix
d-v-b Apr 14, 2026
2348fba
docs: update spec — global thread pool for compute, not inline or to_…
d-v-b Apr 14, 2026
41f0fff
docs: rework spec around streaming pipeline architecture
d-v-b Apr 14, 2026
f338449
docs: add partial shard write support to spec
d-v-b Apr 14, 2026
cb42fa2
docs: add implementation plan for PhasedCodecPipeline performance fix
d-v-b Apr 14, 2026
5ff788d
fix: remove duplicate @staticmethod decorator on _scatter
d-v-b Apr 14, 2026
4616238
feat: add global thread pool for codec compute
d-v-b Apr 14, 2026
89ec63a
perf: add SimpleChunkLayout fast path in _transform_read
d-v-b Apr 14, 2026
1d46e93
perf: rewrite PhasedCodecPipeline read/write as streaming pipelines
d-v-b Apr 14, 2026
f56c7fc
refactor: simplify sync read/write with SimpleChunkLayout fast path
d-v-b Apr 14, 2026
304be12
refactor: remove dead code from old phase-based read/write
d-v-b Apr 14, 2026
cdcdfd2
feat: add ByteRangeSetter protocol and implement in memory/local stores
d-v-b Apr 14, 2026
6234ef8
feat: add partial shard write support for fixed-size inner codecs
d-v-b Apr 14, 2026
d163008
perf: inline decode for non-sharded reads — skip executor overhead
d-v-b Apr 14, 2026
5a2fa97
perf: sync fast path in async read/write — bypass event loop for sync…
d-v-b Apr 14, 2026
bdfc0e7
perf: optimize read_sync — inline layout lookup, direct store.get_sync
d-v-b Apr 14, 2026
2e03b11
perf: cache default ArraySpec for regular chunk grids
d-v-b Apr 14, 2026
e5e082d
perf: optimize shard decode — eliminate BasicIndexer, cache prototype
d-v-b Apr 14, 2026
64d2140
feat: add partial shard write support for fixed-size inner codecs
d-v-b Apr 14, 2026
ddaf176
perf: vectorized shard decode for fixed-size inner codecs
d-v-b Apr 14, 2026
7c34878
perf: pre-extract hot-loop references in read_sync
d-v-b Apr 15, 2026
103f93d
refactor: rename ByteRangeSetter to SetsRange
d-v-b Apr 15, 2026
2f01c48
refactor: rename SetsRange to SupportsSetRange, explicit subclassing
d-v-b Apr 15, 2026
da232ae
feat: add SupportsSetRange protocol and store implementations
d-v-b Apr 15, 2026
e57bd5a
Merge branch 'main' into feat/byte-range-setter
d-v-b Apr 15, 2026
579ff16
test: add tests for SupportsSetRange on MemoryStore and LocalStore
d-v-b Apr 15, 2026
2b9d804
docs: changelog
d-v-b Apr 15, 2026
9f8bce7
perf: cache default ArraySpec for regular chunk grids
d-v-b Apr 15, 2026
0cb7197
docs: changelog
d-v-b Apr 15, 2026
4073cff
Merge feat/byte-range-setter into perf/prepared-write-v2
d-v-b Apr 15, 2026
f66a19c
Merge perf/cache-default-chunk-spec into perf/prepared-write-v2
d-v-b Apr 15, 2026
4746601
test: parametrize codec pipeline tests over BatchedCodecPipeline and …
d-v-b Apr 15, 2026
8b8714f
fix: pass chunk_shape for rectilinear grids in read_sync fast path
d-v-b Apr 15, 2026
ccc1436
test: add rectilinear chunk config to codec pipeline tests
d-v-b Apr 15, 2026
5622853
refactor: move decode/encode into ChunkLayout — clean IO/compute sepa…
d-v-b Apr 15, 2026
e097ca5
test: add declarative read plan tests (stub — defines target interface)
d-v-b Apr 15, 2026
2650261
refactor: replace ByteRange with RangeByteRequest in ShardIndex
d-v-b Apr 15, 2026
5f45b7f
refactor: replace ChunkLayout base class with four-phase interface
d-v-b Apr 15, 2026
3d200f6
refactor: implement four-phase model on SimpleChunkLayout, ShardedChu…
d-v-b Apr 15, 2026
c390708
refactor: update async read and write to use four-phase model
d-v-b Apr 15, 2026
64e38d8
test: update expected get count for PhasedCodecPipeline shard reads
d-v-b Apr 15, 2026
52a97b7
refactor: remove dead code from ChunkLayout and PhasedCodecPipeline
d-v-b Apr 15, 2026
2b40e4a
docs: enumerate all IO/compute scenarios for codec pipeline
d-v-b Apr 15, 2026
1913b06
docs: four-phase pipeline design spec
d-v-b Apr 15, 2026
dbb6fda
docs: four-phase pipeline implementation plan
d-v-b Apr 15, 2026
5db2a12
feat: add leaf_transform to ShardIndex
d-v-b Apr 15, 2026
ba60797
feat: add generic fetch_chunks and decode_chunks_from_index functions
d-v-b Apr 15, 2026
638d57f
refactor: write paths use generic merge_and_encode with ShardIndex.le…
d-v-b Apr 15, 2026
f7772cc
feat: recursive resolve_index for nested sharding with leaf_transform
d-v-b Apr 15, 2026
8831672
refactor: remove dead layout methods — ChunkLayout owns only resolve_…
d-v-b Apr 15, 2026
cd64c3d
test: update read/write plan tests for RangeByteRequest and nested sh…
d-v-b Apr 15, 2026
ae48c67
fix: update stale test referencing removed _transform_read method
d-v-b Apr 15, 2026
ed2e230
fix: nested sharding write + leaf_transform property + remove dead un…
d-v-b Apr 15, 2026
5fa3fdf
refactor: simplify PhasedCodecPipeline — remove layout abstraction, u…
d-v-b Apr 16, 2026
01f4445
fix: update tests for simplified pipeline, add n_workers parameter
d-v-b Apr 16, 2026
850bbe4
feat: implement thread-pool parallelism for sync read/write
d-v-b Apr 16, 2026
b4995ab
Merge branch 'main' of github.com:zarr-developers/zarr-python into pe…
d-v-b Apr 16, 2026
b43f229
chore: lint
d-v-b Apr 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .claude/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"permissions": {
"allow": [
"WebFetch(domain:api.github.com)",
"Read(//home/d-v-b/dev/zarr-python/**)",
"Read(//^async def |^def /**)",
"Bash(awk 'NR>=4630 && NR<=4640' src/zarr/core/array.py)",
"Read(//^async def /**)",
"Read(//home/**)",
"Bash(getent passwd:*)",
"Bash(uv --version)",
"Bash(UV_LINK_MODE=copy uv venv /home/d-v-b/dev/zarr-python/.venv --python 3.12)",
"Bash(mount)",
"Bash(findmnt /home/d-v-b)",
"Bash(/usr/bin/uv --version)",
"Bash(/usr/local/bin/uv --version)",
"Bash(command -v uv)",
"Bash(/home/d-v-b/.local/bin/uv --version)",
"Bash(/home/d-v-b/.local/bin/uv sync:*)",
"Bash(uvx pyright:*)",
"Bash(VIRTUAL_ENV= uvx ty check src/zarr/core/metadata/v3.py)",
"Bash(uvx pre-commit:*)",
"Bash(awk '/^ def resolve\\\\\\(/,/^ def [a-z_]/ {print NR\": \"$0}')",
"Bash(pytest tests/test_transforms/test_domain.py -v --tb=short)",
"Bash(jobs -l)",
"Bash(ps -p 843814 -o pid,stat,etime,pcpu,rss --no-headers)",
"Bash(pstree -p 843812)",
"Bash(ps --ppid 843814 --no-headers)",
"Bash(kill 843812)",
"Bash(wait)"
]
}
}
1 change: 1 addition & 0 deletions changes/3907.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add protocols for stores that support byte-range-writes. This is necessary to support in-place writes of sharded arrays.
1 change: 1 addition & 0 deletions changes/3908.misc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Reuse a constant `ArraySpec` during indexing when possible.
Loading
Loading