Symptom#

A query that uses an index to skip data returns fewer rows than a full scan. Two cases seen in dftracer-utils:

• phase("metadata").query('name == "thread_name"') returned 0 of 40 records, and a pruned export kept 1 of 40 thread_name records.
• A per-file summary (a plugin extension's file payload) built from only part of a file's chunks could have ruled out the whole file.

Cause#

• The chunk evidence (blooms, zone maps, postings) was built from data events only. Metadata records were not in it, so for a metadata query every chunk looked like it could not match and was skipped.
• In a cross-rank split, one rank sees only a slice of a file's chunks. A file-level summary from that slice does not describe the whole file, so "no match in the summary" does not prove "no match in the file".

Fix#

• For every piece of evidence, record which records it covers, and let it rule out only queries over those records. For records it does not cover, keep the chunk or the file.
• dftracer-utils added dft.metadata: per chunk, the count of metadata and context records and their names and field paths (capped at 64). A metadata query skips only chunks that this evidence rules out; an export also reads chunks with context records.
• Build file-level summaries only from whole files; skip them for file slices.
• Test each pruning path against a full scan with the index off (use_index=False) and require equal rows.

Evidence#

• Regression tests in tests/trace/views/test_metadata_query.cpp reproduced both metadata bugs before the fix. After it, laghos kept all 16 thread_name and 16 process_name records in a pruned export, where it kept 5 of each before.
• The plugin index extensions (stage 9b) build only when pf.slice.members == nullptr in batch_builder.cpp, with the comment that a slice's file payload could rule out the whole file.