An index build after a tier-served query fails with "Cannot upgrade RocksDB instance"
Symptom#
In one process, a View query answered from the aggregation tier, then an index build on the same index root without aggregation, throws:
Cannot upgrade RocksDB instance at '<index>/.dftindex' from read-only to read-write while it is still in use
A test can hit it by accident when a later test case reuses a temporary directory path of an earlier one: the failure then shows only when the test cases run in one process, and the case passes alone.
Cause#
RocksDBManager::get_or_open (src/dftracer/utils/index/store/db_manager.cpp) refuses to upgrade a cached read-only handle to read-write when another holder keeps a reference (use_count() != 1). The view tier cache (tier_cache in index/schemas/dft/agg/view_agg_tier.cpp) keeps a read-only handle per index path for reuse. Nothing released it before the build asked for read-write. An aggregated build did not fail, because agg::tier::open (before: EventAggregator::open_with_merge_operator) calls RocksDBManager::reset(path), which runs the reset listeners and so evicts the cache.
Fix#
In get_or_open, on the first upgrade conflict, call the reset listeners for the path (outside mutex_, since a listener may re-enter the manager), then check again once. Throw only if a holder is still there. The tier cache already registers evict_tier_cache as a reset listener, so it drops its handle. The change was made in the agg-extension work (stage 10a) and was not committed when this lesson was written.
Evidence#
- Test
index/test_agg_extension, case "a file without a current entry makes readers scan": build with aggregation,count_reads({a})(tier-served), thenIndexer::open({a, b}).build()without aggregation. - With the old
db_manager.cpp(fromgit show HEAD:...):ERROR: test case THREW exception: Cannot upgrade RocksDB instance ... while it is still in use. - With the fix:
test cases: 1 | 1 passed. Full C++ suite: 284 of 284 passed.