Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills

Key points

• Problem: Prior procedural-text extraction assumed prescriptive input such as recipes and wet-lab protocols, while in lab notebooks confirmed observations, uncertain judgments and next-step suggestions often appear in the same passage. The authors argue that losing this distinction causes uncertainty laundering, where an author-qualified statement becomes a stronger processing commitment without adequate authorization and evidence and may lead to discarding data, and directive loss, where a firm directive is dropped or weakened into an opinion so that a supported processing action is missed.
• Input and output: This framework paper proposes a two-stage pipeline and evaluates it on newly annotated corpora and real nanopore experiment records. The input is notebook text segments, and Stage 1 (Epistemic Directive Extraction, EDE) outputs for each segment, as JSON, whether it holds a directive, one of five directive types, and an author certainty label: FACT (an unhedged assertion), JUDGMENT (an uncertain or tentative assessment) or SUGGESTION (optional advice). Stage 2 builds a Markdown skill document (MetaSkill) that attaches the source excerpt and certainty label to each directive, and an agent reading it picks one action per recording file: KEEP_FULL (keep the whole recording), FLAG_FOR_REVIEW (keep it for review), RAISE_THRESHOLD (raise the event-detection threshold), TRUNCATE_AT (cut at a given time) or SKIP_FILE (exclude the file).
• Method: Stage 1 extracts directives and certainty labels with LLM prompting, and in Stage 2 a deterministic compiler with no LLM emits one capsule per directive and checks directive preservation, unchanged labels, source fidelity and schema validity. At execution, Layer 1 turns the instrument trace into a SignalFindings summary, a Claude Sonnet 4.5 agent in Layer 2 proposes a processing decision, and in the verify setting the Layer 3 executor passes strong decisions such as TRUNCATE_AT, RAISE_THRESHOLD and SKIP_FILE only when a capsule authorization path and the signal evidence agree, otherwise applying an action-specific fallback, usually FLAG_FOR_REVIEW. The elevate setting turns a cautious proposal into TRUNCATE_AT only when exactly one stored truncation candidate timestamp matches a calibrated step drop.
• Implementation: The annotated corpora total 461 segments: 201 FreeNotes segments from Chinese-English nanopore notebooks contributed by two senior researchers at two institutions, 155 ONS (Open Notebook Science) segments from openlabnotebooks.org, and 105 WLP (Wet Lab Protocols) segments from a published protocol corpus, and Stage 1 compared GPT-4o, Claude Sonnet 4.5 and Qwen-Max with zero-shot and few-shot prompting at temperature 0 on an 87-segment test set. The downstream evaluation compared seven configurations on 48 nanopore files from three FreeNotes sessions (17 in Saturation-A, 22 in Saturation-B, 9 in Step-drop), with five API calls per configuration and file decided by majority vote. The authors describe a released artifact with compiler and executor code, evaluation scripts and fixed decision records, but the paper gives no repository address, and the raw ABF (Axon Binary Format) current traces stay out of redistribution under the data-sharing agreement.
• Results: In Stage 1, Claude Sonnet 4.5 few-shot reached a directive-detection F1 of 0.737, above 0.682 for the strongest zero-shot setup, and the Stage 2 compiler preserved all 149 fixed input directives and passed all four checks. On the 48 nanopore files, the external LLM and Action-only baselines reached only 0 to 21.4% review recall on the saturation sessions, while the full configuration (verify+elevate) reached 85.7% and 100% review recall and 96.4% and 100% balanced accuracy on Saturation-A and Saturation-B. On the FACT-dominated Step-drop session, the full configuration's accuracy of 77.8% was below the 88.9% of Action-only.
• Limits: The authors state that the downstream evaluation covers only three nanopore sessions and one downstream model, and that only the nanopore domain adapter was built and evaluated. The main comparison used adjudicated EDE inputs, and replacing them with Claude Sonnet 4.5 few-shot predictions dropped directive precision to 34.8% on Saturation-B and 47.9% on Step-drop and pushed Cohen's kappa below chance to -0.07 and -0.03. The executor-enabled comparison also changes the Layer 2 policy text, so the authors note that a matched-policy experiment is needed to isolate the executor's own effect.

Paper links

External research summaries. These are not HDATF publications or measured product results.