Skip to content

split block ranges when there's too many proofs returned#95

Merged
n13 merged 1 commit intomainfrom
handle_more_than_1k_block_rewards
Apr 18, 2026
Merged

split block ranges when there's too many proofs returned#95
n13 merged 1 commit intomainfrom
handle_more_than_1k_block_rewards

Conversation

@n13
Copy link
Copy Markdown
Contributor

@n13 n13 commented Apr 18, 2026

Fix for this:

❌ Error: Error: Error: GraphQL errors: Query returned 2103 results, which exceeds the limit of 1000. Please use longer hash prefixes for more specific queries.

Summary of changes:

src/subsquid/client.rs — added SubsquidClient::query_all_transfers_by_prefix. It seeds the range from base_params.after_block/before_block (defaulting to 0 and u32::MAX), calls the existing query_transfers_by_prefix per sub-range with limit=1000, and when the server responds with the "exceeds the limit" error it binary-splits at the midpoint and retries via an explicit Vec stack (no async recursion boxing). Any other error propagates immediately. If a single block legitimately has >1000 matches it fails loudly rather than looping.

src/collect_rewards_lib.rs — swapped the three call sites (collect_rewards, query_pending_transfers, query_pending_transfers_for_address) to use the paginating method. Removed the now-irrelevant with_limit(1000) parameters. Privacy prefix stays at 8 hex chars; SDK function signatures unchanged.

Test — added test_server_limit_error_marker guarding the "exceeds the limit" substring the paginator pattern-matches on, so future server wording changes break a test instead of silently breaking production.

cargo build and all subsquid/collect_rewards unit tests pass.

Regarding your follow-up question on scale: after this fix there's no architectural cap on transfers per collect_rewards call. The true per-transaction cap remains 16 proofs (num_leaf_proofs); 1000s of transfers are submitted as ceil(N/16) unsigned extrinsics back-to-back. The bottleneck becomes serial proof generation time and in-memory buffering of all proof bytes, not the indexer or the chain.

@n13 n13 merged commit 85d9740 into main Apr 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants