Address tokio contention messages #6279
Draft
+257
−78
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…pool metrics Use parking_lot::RwLock instead of std::sync::RwLock for connection pool metric recording. parking_lot::RwLock is faster for short-held locks as it uses efficient spinning before parking, reducing tokio worker thread blocking during metric recording. This change helps reduce tokio threadpool contention when the connection pool is under heavy load, as the metric recording locks are held for only microseconds. Co-Authored-By: Claude Opus 4.5 <[email protected]>
These locks are accessed on every GraphQL query, so using the faster parking_lot::RwLock reduces lock contention in the query path. Co-Authored-By: Claude Opus 4.5 <[email protected]>
…ents Replace std::sync::RwLock with parking_lot::RwLock in the SubscriptionManager to reduce lock contention. parking_lot's RwLock is faster for short-held locks due to efficient spinning before parking, which helps reduce tokio threadpool contention. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Replace std::sync::RwLock with parking_lot::RwLock in the background writer's Request::Write batch handling. This reduces lock contention as parking_lot's RwLock is faster for short-held locks due to efficient spinning before parking. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Replace std::sync::RwLock with parking_lot::RwLock in TimedCache for faster lock acquisition on cache gets and sets. parking_lot's RwLock uses efficient spinning before parking, reducing contention. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Replace std::sync::RwLock with parking_lot::RwLock for the chain stores map in BlockStore. This reduces lock contention when looking up or modifying chain stores. Co-Authored-By: Claude Opus 4.5 <[email protected]>
…egistry Replace std::sync::RwLock with parking_lot::RwLock for the global metrics caches in MetricsRegistry. This reduces lock contention when registering or looking up global metrics. Co-Authored-By: Claude Opus 4.5 <[email protected]>
…eepAlive Replace std::sync::RwLock with parking_lot::RwLock for the alive_map in SubgraphKeepAlive. This reduces lock contention when tracking running subgraph deployments. Co-Authored-By: Claude Opus 4.5 <[email protected]>
lutter
force-pushed
the
lutter/block
branch
2 times, most recently
from
ab17afd to
a6245db
Compare
With many subgraphs, chain_head_ptr() was querying the database on every call, leading to connection pool saturation. This adds an adaptive cache that learns optimal TTL from observed block frequency. The cache uses EWMA to estimate block time and sets TTL to 1/4 of that estimate (bounded by 20ms-2000ms). During warmup (first 5 blocks), it uses the minimum TTL to avoid missing blocks on unknown chains. New metrics: - chain_head_ptr_cache_hits: cache hit counter - chain_head_ptr_cache_misses: cache miss counter (DB queries) - chain_head_ptr_cache_block_time_ms: estimated block time per chain Safety escape hatch: set GRAPH_STORE_DISABLE_CHAIN_HEAD_PTR_CACHE=true to revert to the previous uncached behavior. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replaces std::sync::RwLock with parking_lot::RwLock for pool metrics
Use parking_lot::RwLock instead of std::sync::RwLock for connection pool metric recording. parking_lot::RwLock is faster for short-held locks as it uses efficient spinning before parking, reducing tokio worker thread blocking during metric recording.
This change helps reduce tokio threadpool contention when the connection pool is under heavy load, as the metric recording locks are held for only microseconds.