I am evaluating a CTI-to-ATT&CK mapping pipeline (BM25 + dense hybrid retrieval over MITRE ATT&CK En...

Evaluating CTI-to-ATT&CK Mapping on CISA Ransomware Advisories

Executive Summary

• Four-Advisory Corpus: The four current PDFs expose visible ATT&CK tables, with working counts of 15 Medusa, 38 Gunra rendered rows, 23 Interlock, and 27 Ghost; -> preserve each PDF version and manually audit row boundaries before publishing exact counts. [10] [11] [12] [9]
• Version Mismatch: The advisories declare ATT&CK versions 19, 19.1, 17, and 16.1, while the pipeline indexes Enterprise v14; -> evaluate both native IDs and an explicitly documented version-normalized lane. [10] [11] [12] [9] [2]
• Supplementary Artifacts: The CISA pages provide PDFs and STIX XML/JSON IOC downloads, but no Navigator layer link was found in the page inventories; some current STIX bundles nevertheless contain attack-pattern objects, so IOC labeling and ATT&CK-table labeling must not be conflated. [6] [1]
• Self-Label Leakage: Technique IDs occur inline in advisory prose and in the tables. A system can therefore retrieve the answer from the label string rather than infer behavior; -> report an ID-scrubbed lane in addition to the natural-document lane. [10] [8]
• Curated, Not Omniscient Gold: CISA guidance asks analysts to identify all supported behaviors, but also says that not every behavior is documented in ATT&CK and that mappings require sufficient context; -> treat the table as the operational gold standard requested here, not as a proof of complete adversary behavior coverage. [8] [8] [8]
• Ranking Versus Set Metrics: Published CTI work commonly reports precision, recall, and F1 over labels or sentences, while ranking metrics answer a different question; -> report macro/micro set scores and Hit@k, Recall@k, and nDCG@k separately. [44] [29]
• Small-n Inference: Four documents support a useful case study and failure analysis, not a population-level significance claim; -> publish per-advisory results, confidence intervals over items or sentences with their unit stated, and no claim of general superiority.
• Main Unresolved Fact: The collected evidence does not establish byte identity across cisa.gov, ic3.gov, and media.defense.gov, nor does it provide a reliable published Medusa row-by-row delta; -> hash and diff the actual downloaded files and manually reconcile the two Medusa tables.

1. PDF Structure and Parsing: Text Layer, Tables, and Page Furniture

Confidence: High for the four inspected PDFs; medium for CISA advisories generally. Scope: specifically the four PDFs unless stated otherwise.

The inspected PDFs are text-bearing government reports rather than image-only scans. The extracted content preserves headings, table labels, ATT&CK names, IDs, and procedure text. For example, Ghost exposes entries such as PowerShell T1059.001, Windows Command Shell T1059.003, and Web Shell T1505.003 in the extracted stream. [9] This makes pdfminer.six, pdfplumber, pypdf, and similar text-layer tools viable first passes, but not sufficient as the sole structural representation.

The ATT&CK tables appear in extraction as linear text with a Technique Title ID Use schema. The available evidence does not prove whether every visual table has drawn ruling lines, because the corpus excerpts expose text rather than PDF drawing operators. Operationally, however, the extraction behaves like a borderless or weakly ruled table: rows can be concatenated, headings can attach to the preceding row, and a multi-line procedure can be mistaken for a new row. Ghost's extracted table includes adjacent entries and page furniture in one stream. [9] [9] Confidence: Medium. Scope: the four PDFs.

Repeated furniture is plainly present. Ghost includes strings such as TLP:CLEAR, partner seals, Page 3 of 13, and Product ID: AA25-050A; the same pattern appears on later pages. [9] [9] [9] Gunra repeats TLP:CLEAR, partner names, page numbers, total pages, and product ID. [11] [11] Medusa exposes Page 14 of 29 | Product ID: AA25-071A and the TLP banner. [10]

Recommended preprocessing is conservative:

1. retain the original PDF, page number, character coordinates, and raw extracted text;
2. normalize line endings, repeated whitespace, soft hyphens, and obvious encoding artifacts;
3. remove repeated headers and footers only after detecting strings repeated at the same page-relative positions;
4. preserve TLP markings in metadata, but exclude them from sentence-level retrieval;
5. detect table headings and column labels with page-aware rules rather than relying only on newline splitting;
6. validate every parsed row against the visible PDF and, where possible, PDF word coordinates;
7. retain both a whole-document corpus and a Technical Details-only corpus.

Do not assume mirrors are byte-identical. No completed hash comparison was found in the collected evidence. Treat host, URL, retrieval date, file size, SHA-256, page count, and PDF metadata as separate fields. Confidence: High that byte identity remains unanswered.

2. Machine-Readable Artifacts and Table Counts

Confidence: High for the existence and names of the listed downloads; medium for exact row counts because extracted table boundaries require manual audit. Scope: these four advisories.

The CISA page lists both original and updated Medusa STIX files. [6] Gunra lists one XML and one JSON file. [1] Interlock lists XML and JSON. [5] Ghost lists the main XML, an additional IOC XML, and JSON. [4]

The important distinction is content. Current advisory-associated JSON/XML bundles can contain attack-pattern objects, not merely indicators. Ghost's JSON contains attack-pattern objects and names with ATT&CK IDs. Interlock's JSON likewise begins with attack-pattern objects. The Ghost additional IOC file is indicator-oriented, and should not be treated as a table-label artifact. [19] The same caution applies to the Medusa and Gunra IOC downloads: their purpose is IOC exchange, even when a bundle also includes ATT&CK objects or relationships.

No ATT&CK Navigator layer JSON was identified in the CISA download inventories. This is a negative finding from the listed downloads, not proof that no partner or archived layer exists. ATT&CK itself explains that Navigator presentations are derived from the underlying STIX data, while STIX is the granular machine-readable representation. [7] Therefore, the safest reproducible procedure is to parse the advisory's own table for gold and use STIX only as an artifact inventory and cross-check, not as an assumed substitute for the table.

The Gunra count needs special care because its table stream includes a tactic-level Collection TA0009 row among technique rows. [11] Publish both counts: rendered rows and rows whose ID matches T\d{4}(\.\d{3})?. Do the same for any tactic rows in the other PDFs.

3. Medusa Reissue: What Changed and What Remains Unproven

Confidence: High for the version and narrative changes; medium for the provisional table counts; low for an exact added-row set without manual comparison. Scope: AA25-071A only.

The updated advisory states that the original was published on March 12, 2025 and that the update incorporates FBI investigations through April 2026. [6] It declares ATT&CK Enterprise version 19. [10] The original PDF evidence identifies the earlier version as ATT&CK v16, while the current CISA page records the two publication states and their separate STIX downloads. [30] [6]

The update adds operational detail about the affiliate model, payment ranges, opportunistic targeting, and Interactsh URLs used for exploit verification. [10] It adds or expands vulnerability discussion, including exploitation of unpatched software and CVE examples. [10] It also expands credential and collection behavior: Volume Shadow Copy use for credential theft, vssadmin activity, Bandizip for archiving, and Rclone for exfiltration. [10] HHS was added as a co-sealer. [10]

The current visible table extraction identifies these 15 technique rows: T1190, T1566, T1070.003, T1003.001, T1558, T1059.001, T1059.003, T1072, T1021.001, T1569.002, T1047, T1675, T1567.002, T1105, and T1071.001. The updated table explicitly marks some entries, including Steal or Forge Kerberos Tickets and ESXi Administration Command, as updated. [10] [10]

The archived March table was counted in the research pass as 30 T-ID rows, while the updated visible extraction yielded 15. This apparent reduction is not yet a defensible semantic conclusion. The updated STIX bundle contains many attack-pattern objects, and PDF extraction can undercount rows when tables are split across pages or when the corpus read stops at a fragment. Accordingly, state the counts as provisional and do not publish an exact set difference until you:

• download both PDFs;
• extract words with coordinates;
• identify every table heading and row;
• normalize IDs and duplicate rows;
• manually reconcile table rows against the visual pages; and
• separately compare prose-only behaviors.

The correct substantive conclusion is that the update materially expands TTP and vulnerability narrative detail and changes the declared ATT&CK version from v16 to v19. The exact table delta remains an open corpus-audit item. Confidence: high for narrative expansion, low for the exact added-row list.

4. Gold-Standard Precedent in CTI-to-ATT&CK Extraction

Confidence: High for the cited dataset and paper descriptions; low that the named systems used these four CISA tables as gold. Scope: literature generally.

The collected literature does not establish a published evaluation of TRAM, AttacKG, LADDER, TTPXHunter, TCENet, SMET, aCTIon, or TechniqueRAG that uses the visible ATT&CK tables of these four advisories as document-level gold. This absence matters: do not describe the present setup as reproducing a known CISA-table benchmark.

A CISA-specific dataset contains 77 public CISA articles selected because they explicitly mentioned MITRE ATT&CK TTPs. [31] Its preprocessing documentation distinguishes raw text from cleaned text and removes tables and IDs in at least one cleaned representation. That is closer to an ID-aware text corpus than to independent table annotation. Confidence: medium because the collected excerpt is a dataset description, not a full annotation protocol.

TTPXHunter reports a manually annotated corpus of reports and evaluates extraction with F1, but the collected evidence does not identify the four advisory tables as its source gold. [29] TechniqueRAG is an ACL 2025 paper on retrieval-augmented adversarial-technique annotation, but its existence does not establish use of these CISA tables. [24]

A recent comparison using TRAM and Llama-based systems evaluated six short Adversary Emulation Library reports, excluded techniques outside TRAM's 50 most prevalent classes, and used annotated datasets rather than the four CISA tables. [33] It reports precision, recall, and F1, illustrating the common set-classification framing. [33]

The broader survey evidence shows heterogeneous gold construction. Some corpora annotate sentences with technique or sub-technique labels; others provide document-level labels, and some use expert annotation or existing ATT&CK-linked reports. The survey excerpt lists datasets with differing report, sentence, and label counts and reports F1 for systems such as AnnoCTR and TRAM2. [44] [44]

For this evaluation, define gold explicitly as: all unique ATT&CK technique or sub-technique IDs in the advisory's visible ATT&CK tables, deduplicated within a document, with tactic-only rows excluded. Define scope explicitly as Technical Details tables, while reporting a secondary whole-document lane. Do not silently mix inline IDs, STIX attack-pattern objects, mitigation references, and table rows.

5. CISA's Mapping Methodology and Decider

Confidence: High for the guidance principles; medium for claims about exact advisory curation behavior. Scope: CISA/MITRE guidance generally, applied cautiously to these four advisories.

CISA's best-practices guidance says analysts should find behavior, examine context, and map to the most accurate ATT&CK depth supported by the evidence. It says to limit mapping to the tactic level only when there is insufficient detail to identify a technique or sub-technique. [8] It also warns against leaping to conclusions and recommends careful examination of report details and artifacts. [8] The guidance specifically notes that not every adversary behavior is documented in ATT&CK. [8]

For finished reporting, CISA recommends inline ATT&CK TTP links and summary tables containing technique title, ID, and use or procedure detail. [8] [8] This supports interpreting the visible tables as curated summaries of supported mappings, not as automatically generated exhaustive indexes of every sentence.

The guidance encourages finding missed opportunities and identifying all behaviors that may have been overlooked, but it also says that context and evidence determine whether a positive mapping is justified. [8] [8] Thus, the table is intended to be comprehensive within the analyst's supported mapping effort, but CISA's methodology does not warrant the stronger claim that every prose behavior must appear in the table or that every table row has an equally explicit prose occurrence.

Decider is described as a free mapping aid created with HSSEDI and MITRE to help analysts map threat-actor behavior to ATT&CK. [26] It is a decision-support tool, not evidence that the table-generation process is exhaustive or that its output is the official gold for every advisory.

Recommended annotation policy: table membership is the primary gold label; inline IDs are leakage metadata; prose-to-row alignment is a separate audit. Record three states for each candidate behavior: explicitly supported in table, discussed but not table-listed, and table-listed with no readily locatable prose span. Do not collapse these into a binary completeness claim.

6. Cross-Version ATT&CK Evaluation: v14 Against v16.1-v19.1

Confidence: High for the official version resources; medium for recommended migration implementation; low for claims of a single universal evaluation convention. Scope: ATT&CK generally and this pipeline specifically.

MITRE's version history distinguishes major and minor releases: major releases increment the major number, while minor releases generally contain corrections and not typically new content. [2] The preserved versions include v14.1, v16.1, v17.1, v18.1, and v19.2. [2] The advisories themselves declare v16.1, v17, v19, and v19.1. [9] [12] [10] [11]

MITRE identifies STIX as the most granular ATT&CK representation and states that ATT&CK data are available in STIX 2.0 and 2.1. It also points to Python tooling, the TAXII server, and derived Navigator and website presentations. [7] [7] These resources support a reproducible migration process based on versioned STIX objects, IDs, names, relationships, and object metadata.

A v14 index should not be compared naively with later gold. Use three clearly labeled lanes:

Published evaluations vary: some normalize labels or restrict the label vocabulary; some exclude labels outside the model's training or supported set; others report misses. The collected TRAM comparison explicitly excluded techniques outside its top 50 classes. [33] That is an example of exclusion with disclosure, not proof that exclusion is always correct.

Do not silently map a new technique to a similarly named old technique. Preserve original ID, version, mapped ID, mapping reason, and whether the relation is exact, parent-level, revoked, or unavailable. Report scores for both exact ID matching and semantic cross-version matching.

7. Label Leakage from Self-Labeled Documents

Confidence: High that leakage exists in these advisories; medium for the recommended controls; low that a quantified inflation estimate exists in prior CTI work. Scope: these four advisories and analogous NLP evaluation.

The leakage channel is direct. Advisory prose can contain bracketed IDs such as [T1190] and [T1059.001], while the visible table repeats the same ID beside its title and procedure. [10] CISA's own reporting guidance recommends inline ATT&CK links and summary tables. [8] A retrieval or reranking model can therefore succeed by matching an ID token, table heading, or nearby label instead of recognizing behavior.

The collected evidence did not identify a CTI-to-ATT&CK paper that quantified score inflation caused specifically by retaining these advisory labels. It also did not establish a standard published ATT&CK masking lane across the named systems. The TRAM comparison's removal of extraneous material and its constrained label set show that preprocessing and vocabulary control are common, but they do not by themselves quantify leakage. [33]

Run at least four input lanes:

1. Natural: original PDF text, preserving labels;
2. ID-scrubbed: replace ATT&CK IDs and bracketed links with a single token;
3. Table-excluded: remove the visible ATT&CK tables but retain narrative;
4. Technical-only scrubbed: Technical Details prose with IDs, table headings, and repeated furniture scrubbed.

Keep the gold table labels unchanged. Report absolute and relative changes in Hit@k, Recall@k, nDCG@k, micro-F1, and macro-F1. The natural-minus-scrubbed difference is an empirical leakage sensitivity estimate for this corpus, not a universal inflation factor. Also test whether the system retrieves the exact target ID from an input sentence containing that same ID; report this as a label-copy baseline.

8. Mitigations and Defensive Semantics

Confidence: Medium for the design recommendation; low for a published false-positive rate. Scope: adversary-TTP extraction generally and these advisories specifically.

CISA separates mapping guidance from mitigation recommendations. Its best-practices material presents summary ATT&CK tables with technique title, ID, use, and recommendations, showing that defensive recommendation language can coexist with technique labels in finished reports. [8] The target advisories also distinguish Technical Details and Mitigations sections in their structure. [9]

A mitigation statement such as patching a public-facing service, disabling a tool, or enforcing multifactor authentication is not automatically an observation that an adversary performed the corresponding ATT&CK behavior. Conversely, a Technical Details sentence describing an actor exploiting a vulnerability is an observed-behavior candidate. This distinction is semantic, not merely lexical.

The collected literature did not provide a reliable published false-positive rate for mitigation or CISA boilerplate text. Therefore, do not invent one. Measure it directly with section-stratified evaluation:

For the adversary-behavior claim, use Technical Details as the primary lane and report whole-document results as a separate robustness experiment. Any technique-like language found only in Mitigations should be marked defensive and excluded from the primary positive gold unless the advisory explicitly states adversary action there.

9. Small-n Reporting for Four Document-Level Cases

Confidence: High for the methodological recommendation; scope: this four-document evaluation.

Frame the work as a controlled case study of four real, heterogeneous advisories, not as a statistically representative estimate of CISA performance or CTI performance. The four documents differ in length, partner seals, declared ATT&CK version, ransomware family, table organization, and number of techniques. Those differences are analytically useful but make pooled significance claims weak.

Report a per-document table containing document ID, source version, page count, table-row count, unique gold-label count, number of extracted sentences, number of sentences containing visible IDs, and all metrics. Include the ranked list and the first rank at which each gold technique appears. Provide qualitative failure examples for table boundaries, sub-technique confusion, mitigation contamination, and version mismatch.

Use macro averages across the four documents only as descriptive summaries. If confidence intervals are supplied, state the resampling unit: documents, sentences, or bootstrap draws. Sentence-level bootstrap intervals do not establish uncertainty over four independent advisories because sentences within one advisory are correlated. Avoid claims such as statistically significant, generalizable, state of the art, or representative unless supported by a larger independent corpus.

A strong security-venue framing is: the experiment tests reproducibility, pipeline behavior, ranking quality, and failure modes on four operationally important documents. A weak framing is: four advisories prove the method works generally. Include an artifact package with exact PDFs, hashes, extraction code, label files, and version manifests so the case study can be extended.

10. Metric Conventions and Honest Positioning

Confidence: High for the cited F1 convention; medium for the broader literature summary; low for published numbers specifically on these four advisory documents. Scope: CTI-to-ATT&CK literature.

The collected papers primarily report precision, recall, and F1 over sentence or document labels. The TRAM comparison reports precision, recall, and F1 by method and label. [33] TTPXHunter reports F1 results, including a reported 92.42% F1 on an augmented dataset in the indexed abstract. [29] A survey excerpt reports differing F1 values for systems and datasets, including approximately 26.7 for AnnoCTR and 34.2 for TRAM2 in its cited comparison. [44]

These are not directly comparable to ranking metrics. Set-F1 evaluates the final unordered predicted set against the gold set. Hit@k asks whether a gold label appears in the first k ranks. Recall@k measures how much of the gold set appears in that prefix. nDCG@k additionally rewards putting relevant labels earlier, subject to the chosen relevance and discount definitions.

Report both families:

The collected evidence does not provide a published document-level result for extraction on exactly AA25-071A, AA26-222A, AA25-203A, and AA25-050A. Do not compare the four-document ranking result directly with a corpus-level F1 headline. Instead, explain the unit, label vocabulary, section scope, ATT&CK version, and whether IDs were scrubbed.

11. Rights, TLP:CLEAR, and Citation Practice

Confidence: Medium. Scope: CISA material and academic reuse generally.

The collected CISA guidance marks the material as TLP:CLEAR and describes the normal dissemination intent of that marking. [8] TLP:CLEAR is a sharing designation, not by itself a complete answer to copyright, trademark, or third-party material rights. CISA site terms reserve rights and include site-use conditions. [28] [28]

For a paper, the lowest-risk practice is to quote only what is necessary, reproduce a small table excerpt rather than an entire advisory, retain the advisory title, alert code, publication or update date, issuing agencies, source host, and stable download URL in the bibliography, and include the TLP marking. Do not reproduce IOCs unnecessarily, especially operational hashes, credentials, or contact details. Keep a local hash and access date for reproducibility, while recognizing that the public URL can later change.

Use the official CISA page and PDF as the primary citation, not a news story or a mirror, unless the mirror is itself the object of a byte-identity experiment. Cite the exact advisory version: for Medusa, distinguish the March 12, 2025 original from the August 18, 2026 update. [6] For machine-readable data, cite the advisory's STIX file separately from the PDF because the files answer different questions.

Before publication, check the current CISA terms and the venue's quotation and reproduction rules. If reproducing a larger table, request permission or provide a derived label list instead of copying prose. State that the table excerpt is used for research and attribution; do not imply CISA endorsement.

12. Known Extraction, Segmentation, and Table-Detection Pitfalls

Confidence: High for observed risks in these PDFs; low for a claim that a specific parser bug is documented for these exact files. Scope: these four PDFs, with general parser recommendations.

The extracted advisory text demonstrates several concrete risks: multi-column or table text can linearize unexpectedly; headings and rows can concatenate; procedure cells span multiple lines; page furniture enters the text stream; and IDs may be separated from titles or split by whitespace. Ghost's extracted stream shows long sequences of table entries and repeated page markers. [9] [9] Gunra's tables similarly span pages and contain rows with long procedure text. [11]

Potential encoding issues include soft hyphens, ligatures, unusual bullet glyphs, nonbreaking spaces, split words such as Rclone or executable names, and punctuation around IDs. The excerpts show extraction artifacts such as spacing in Thef t, R34DM3, and RSA- 4096. [9] [11] These artifacts can damage sentence segmentation and exact string matching without changing the visual PDF.

Recommended mitigations are:

• extract words with x/y coordinates and reconstruct lines by tolerance;
• use page boundaries as hard segmentation metadata;
• preserve tables as structured rows before sentence splitting;
• run a Unicode normalization and controlled dehyphenation pass, retaining raw text;
• protect ATT&CK IDs, CVEs, URLs, hashes, filenames, and command lines with placeholders during sentence splitting;
• detect repeated furniture using page-position frequency, not a hand-written global deletion alone;
• compare at least two extraction libraries and visually audit disagreements;
• use a table-aware parser for validation, but do not assume detected ruling lines are present;
• keep a parser error log with page number, raw span, normalized span, and correction reason.

The most important negative result is that the collected research did not establish a documented parser failure report specific to all three hosting domains. Treat the observed artifacts as corpus risks requiring measurement, not as proof of a host-specific defect. Host-specific byte identity and parser behavior should be reported from your own SHA-256, page-count, metadata, coordinate, and visual comparisons.

13. Important Design Issues Not Explicitly Asked

1. Version pinning and provenance: Store the PDF hash, download URL, host, retrieval date, page count, advisory update date, declared ATT&CK version, and STIX file hashes. A later CISA page can expose both original and updated artifacts.
2. Label unit definition: Decide whether a parent technique and sub-technique are distinct labels. Do not award parent credit for a sub-technique prediction unless a separate partial-credit policy is pre-registered.
3. Duplicate and tactic rows: Deduplicate repeated technique IDs and exclude tactic IDs such as TA0009 from technique metrics, while reporting how many rows were excluded.
4. Open-world versus closed-world evaluation: The visible table is a closed gold set for this experiment, but prose may contain unsupported or unmapped behavior. Report this limitation rather than calling every non-table behavior a false negative.
5. Sentence-to-document aggregation: Specify max pooling, sum pooling, top-k pooling, or another aggregation rule before looking at results. Report how many sentences contributed each document-level prediction.
6. Negative evidence: Include a no-label or unrelated-government-advisory control set if possible. Four positive ransomware advisories cannot establish whether the pipeline recognizes behavior or merely exploits recurring CISA formatting.
7. Leakage-aware baselines: Include exact-ID string matching, title-only matching, table-only retrieval, and ID-scrubbed retrieval. These baselines show how much of the result comes from formatting and labels rather than semantic retrieval.
8. Reproducible crosswalk: Version the ATT&CK STIX snapshot and publish every ID transformation. MITRE states that STIX is the granular source from which other ATT&CK presentations are derived. [7]
9. Advisory updates as paired cases: Treat original and updated Medusa as a temporal pair, not independent random documents. Analyze whether the update changes prose, labels, version compatibility, and ranking stability separately.
10. Security and disclosure hygiene: Avoid reproducing operational IOCs unless necessary. A derived technique-label file and small quoted examples are usually more useful and safer than republishing the full advisory.

Evaluation of CTI-to-ATT&CK Mapping Pipeline on CISA #StopRansomware Advisories: Research Synthesis on PDF Structure, Ground Truth Construction, and Evaluation Methodology

Executive Summary

This report addresses 12 specialized research questions required to design a rigorous evaluation of a CTI-to-ATT&CK mapping pipeline on four CISA #StopRansomware advisories (AA25-071A Medusa, AA26-222A Gunra, AA25-203A Interlock, AA25-050A Ghost/Cring). Research reveals substantial Tier 1 (CISA/MITRE official) resources on advisory structure and metadata, moderate evidence on academic precedent for extraction evaluation, and critical gaps on table exhaustiveness intent, label-leakage quantification, and CISA-specific PDF parsing pitfalls.

Key Findings Across Questions:

• PDFs are machine-parseable: Medusa August 2026 (AA25-071A) contains ruled ATT&CK tables (3 columns: Tactic/Technique Title, ID, Use) with hyperlinked technique IDs and boilerplate footers [[103:p1-p29]]. General tools (pdfplumber, pdfminer.six) are recommended for machine-generated PDFs, though no CISA-specific parsing evaluation exists [33].
• STIX bundles published, no Navigator layers: All four advisories publish STIX XML/JSON IOC files [4] [16] [17] [18], but no ATT&CK Navigator layer JSON files are documented in available sources—a critical gap for supplementary machine-readable ground truth.
• Medusa reissue spans major ATT&CK version shift: Original (March 12, 2025, declaring v16) reissued August 18, 2026 (declaring v19), with version-major structural changes (Defense Evasion tactic split) and new techniques documented (T1564.012, T1675, T1558, T1484.001) [4] [35]. Exact technique table row counts for either version not provided in sources.
• TTPXHunter (2024) establishes document-level benchmark: 149 real-world CTI reports manually labeled, achieving Macro-F1 97.09%, outperforming LADDER (92.97%) and TRAM (94.54%) [24][25][26][27]. No confirmation whether CISA advisories were included as gold standard in this or other extraction evaluations.
• CISA methodology emphasizes finding ALL behaviors: Best Practices guidance recommends comprehensive mapping with double-checking, but does NOT explicitly mandate that advisory tables are exhaustive for all prose-described techniques—a critical ambiguity for gold-standard design [9] [10].
• MITRE provides official version crosswalks: v16.1/v17/v19/v19.1 are spanned by the four advisories; MITRE publishes JSON changelogs and remapping instructions (precedent: AttacKG remaps via official hyperlinks) [12] [13] [14] [15]. No published guidance exists on whether to remap gold labels, exclude mismatches, or score as misses when system index differs from advisory version.
• Label leakage risk is documented in source design: Advisories contain hyperlinked technique IDs embedded in Technical Details prose (observed in Medusa) [35]. No prior work quantifies evaluation metric inflation from ID visibility in self-labeled CTI documents.
• Mitigations/defensive context treatment undefined: Medusa shows clean separation (no hyperlinked IDs in Mitigations section, pages 19-20) [35]. No evidence on whether other three advisories embed technique references in defensive contexts, nor prior work on false-positive handling from such references.
• Small-n case studies are feasible but require explicit methodology: ACSAC accepts case studies; QCA methodology supports n=5–50 [29] [30]. Case-level reporting is more defensible than statistical generalizations at this scale [30].
• No CISA-style document-level extraction baselines exist: TTPXHunter reports Macro-F1, but on a proprietary dataset; published baselines for CISA advisory evaluation are absent, making positioning of ranking metrics (nDCG@k, Hit@k, Recall@k) against set-F1 imprecise [24][25][26][27] [19][20].
• TLP:CLEAR licensing permits reproduction: Advisory content is marked TLP:CLEAR with unrestricted distribution; table excerpts and quoted sentences may be reproduced in academic papers per standard copyright rules [4] [5] [35].
• PDF parsing challenges are generic, not government-specific: pdfplumber handles ruled/borderless tables via line detection; known issues include nested tables and multi-page table detection [31] [32] [34]. No documented CISA-specific encoding, ligature, or column-detection failures.

---

1. PDF Structure and Parsing: Text-Layer Quality, Table Rendering, and Boilerplate Patterns

Confidence: HIGH (for Medusa Aug 2026); MEDIUM (for other three advisories) Scope: AA25-071A Medusa (August 18, 2026 version examined); general principles apply to all four

What is known about CISA advisory PDF structure and text-layer quality

The August 18, 2026 reissue of AA25-071A (Medusa) was directly examined and contains the following structural characteristics:

Boilerplate and Page Furniture: The advisory begins with a TLP:CLEAR banner stating "This document is marked TLP:CLEAR. Disclosure is not limited." The footer on pages 2–29 reads "Page N of 29 | Product ID: AA25-071A" [[103:p1-p29]]. No variations in page footers are documented in the source materials; the "N" field appears to increment consistently. This per-page furniture is standard in government cybersecurity advisories and must be stripped during corpus preprocessing to avoid noise in text-based feature extraction or sentence segmentation.

ATT&CK Technique Table Rendering: The advisory contains ruled (bordered) tables across pages 6–18. Tables are consistently formatted as three-column structures with headers: (1) Tactic/Technique Title, (2) ID (formatted as hyperlinked T-code, e.g., T1190), and (3) Use (brief 1–2 sentence description of observed behavior) [35]. The hyperlinked IDs are embedded as active PDF links, indicating the table is machine-detectable via PDF coordinate parsing. Borders are present, making these ruled tables rather than borderless text—a feature that favors automatic table detection.

Text-Layer Quality for PDF Extraction Tools: No CISA-specific evaluation of text-layer quality exists in the research findings. General guidance recommends pdfplumber and pdfminer.six as robust Python libraries for handling machine-generated PDFs (as opposed to scanned images) [33]. pdfplumber, built on top of pdfminer.six, is specifically designed to extract text, tables, lines, and rectangles with visual debugging support and configurable table-detection parameters [31] [34]. A 2024 comparative study evaluated PyMuPDF, pdfminer.six, Camelot, and Tabula across diverse document categories (financial, scientific, tender documents), finding success rates varying by document type: PyMuPDF excelled on manual/financial documents, Camelot on tender documents, and Tabula on scientific papers [33].

Mirror Byte-Identity: The research does not provide data on whether the same advisory distributed across cisa.gov, ic3.gov, and media.defense.gov are byte-identical or differ in rendering, TLP headers, or metadata. This is a critical unknown for corpus construction: if mirrors differ, parsing one version may yield different table positions or line breaks than another, affecting sentence-span extraction.

Recommended Preprocessing for Boilerplate Removal: Standard practice involves:

• Extracting text via pdfplumber or pdfminer.six
• Identifying page-footer patterns ("Page N of M | Product ID: AAXXX") via regex and removing them
• Stripping TLP:CLEAR banners from the top of each page
• Retaining the Technical Details and IOC sections; optionally excluding Mitigations and Appendices depending on evaluation scope
• For table extraction, configuring pdfplumber's table_settings to detect borders and column positions

Critical Gap: No CISA-specific parsing evaluation has been conducted; claims about text-layer quality, encoding issues, or ligature handling are inferred from general PDF literature, not empirical CISA document testing.

---

2. Machine-Readable ATT&CK Artifacts: Navigator Layers, STIX Bundles, and Supplementary Downloads

Confidence: HIGH (for STIX availability); MEDIUM (for Navigator layers) Scope: All four advisories

Supplementary downloads and file availability per advisory

STIX Bundle Composition: All four advisories publish STIX XML and STIX JSON files via their respective CISA advisory pages [4] [16] [17] [18]. The file sizes reported are: AA25-071A (reissue) STIX XML unspecified, STIX JSON unspecified; AA26-222A STIX XML 54.18 KB, STIX JSON 61.00 KB [16]. These bundles are documented as IOC-focused (indicators of compromise), not explicitly confirmed to contain STIX attack-pattern objects (which would represent MITRE ATT&CK techniques as machine-readable entities). The research findings do not clarify the bundle structure, STIX entity types, or whether relationship objects linking indicators to attack patterns are included.

ATT&CK Navigator Layers: The research found no evidence that any of the four advisories publish ATT&CK Navigator layer JSON files. Navigator layers are machine-readable JSON files (compatible with the MITRE ATT&CK Navigator visualization tool) that encode an advisory's technique coverage, often with scoring or color-coding. While the Decider tool (used by all four advisories) is mentioned as supporting Navigator export [6] [7] [8], actual published layers for these advisories are not documented in sources. This is a significant gap: Navigator layers would provide unambiguous, version-pinned, machine-readable ground truth for technique coverage.

Critical Implication for Evaluation: If no Navigator layers exist, the advisory tables themselves are the only official technique-enumeration source. This makes the tables even more critical to your evaluation and heightens the importance of resolving whether they are intended to be exhaustive or curated.

---

3. The Medusa Reissue: Version Bump, New Techniques, Table Evolution, and CVE Updates

Confidence: HIGH (for update narrative and new CVEs); MEDIUM (for exact table-size comparison) Scope: AA25-071A specifically; comparison between March 12, 2025 and August 18, 2026 versions

Changes between March 2025 original and August 2026 update

Version Declaration Evolution: The original advisory published March 12, 2025 did not have an explicit ATT&CK version declaration in the available research snippet. The August 18, 2026 reissue explicitly declares ATT&CK v19 [4] [35]. This represents a major version jump (skipping v16.1, v17, v18) and includes structural changes to the framework itself.

Update Narrative: CISA's update summary states the reissue "expands details on Medusa actors' operations, including more specifics about their affiliate model and payment ranges for initial access brokers, as well as a broader list of exploited vulnerabilities. It describes Medusa's opportunistic targeting and use of Interactsh URLs for exploit verification. It also lists additional tools for network enumeration, persistence, and stealth, including detailed PowerShell obfuscation techniques and command-and-control utilities. Additionally, HHS has been added as a co-sealer" [4] [35].

New CVEs and TTPs: The August 2026 update explicitly introduces two newly documented exploited vulnerabilities:

• CVE-2025-10035 (Fortra GoAnywhere) [35]
• CVE-2026-1731 (BeyondTrust) [35]

The advisory also documents additional tools and techniques, with new or expanded technique mappings including:

• T1564.012 (Hide Artifacts: Hidden Files and Directories) — a sub-technique in the concealment domain
• T1675 (ESXi Administration) — hypervisor-level persistence [35]
• T1558 (Steal or Forge Kerberos Tickets) — lateral movement via Kerberos [35]
• T1484.001 (Domain/Tenant Policy Modification) — domain persistence [35]
• Expanded coverage of T1003.001 (OS Credential Dumping: LSASS Memory) [35]
• Expanded T1006 (Direct Volume Access) — storage-level access [35]
• Expanded T1567.002 (Exfiltration Over Web Service: Exfiltration to Cloud Storage) — data theft [35]

Table-Size Comparison: The research findings document that the advisory contains estimated ~39 technique rows in the reissued version (pages 6–18 per the August 2026 PDF) [35]. However, the original March 12, 2025 version's exact row count is not provided in research sources. This prevents a precise before/after comparison and is a critical missing data point for quantifying the reissue's expansion.

Advisory Metadata Evolution: The original advisory from March 12, 2025 is referenced [16], and the reissue is [4] [5] [35]. The transition from v16 (original) to v19 (reissue) reflects not only new threat intelligence but also adaptation to MITRE's major v19 update (released April 28, 2026) [12] [14], which restructured Defense Evasion into Stealth (TA0005, reused ID) and Defense Impairment (TA0112, new) with 94% of Defense Evasion techniques redistributed [11] [14].

Critical Gap: Without exact row counts for the March 2025 version, the scale of the reissue's technique-table expansion cannot be quantified. The narrative indicates "broader list" and "additional tools," but precise growth is unknown.

---

4. Gold-Standard Precedent: Academic and Industrial Use of CISA Advisory Tables as Extraction Ground Truth

Confidence: MEDIUM (extraction methodologies documented); LOW (CISA advisory confirmation) Scope: General assessment of academic precedent; no confirmation of CISA-specific application

Extraction methodologies and ground-truth construction in prior work

TTPXHunter (2024) [24][25][26][27] represents the highest-fidelity academic precedent for document-level TTP extraction evaluation. The paper manually labeled 149 real-world CTI reports, creating a benchmark with 39,296 sentence-level training samples across 193 TTP classes (a hierarchy of techniques under tactics). The ground truth was constructed by human annotators assigning technique labels to sentences extracted from finished reports. Evaluation was conducted at report level (macro-averaged F1, precision, recall, and Hamming Loss) rather than token level. The reported performance was F1 97.09%, Precision 97.38%, Recall 96.15%, outperforming comparators: LADDER (92.97% F1), TRAM (94.54%), AttacKG (88.58%) [24][25][26][27].

Critically, TTPXHunter does NOT explicitly confirm that CISA advisory tables were used as ground truth. The paper references "149 manually-labeled real-world CTI reports from prominent security firms," but does not name CISA or confirm the source. This is a significant gap: TTPXHunter's 149-report dataset may be entirely proprietary or from other vendors (e.g., Cisco Talos, CrowdStrike, Mandiant).

AttacKG (2021) [28] manually labeled a smaller ground-truth set: 16 CTI reports (5 from DARPA TC programs, 3 documenting APT campaigns). The team also crawled 1,515 real-world reports from MITRE ATT&CK framework references and extracted 7,373 procedure examples from 179 techniques as templates. Ground truth construction involved human annotation with schema-driven labeling (technique ID, procedure text). The paper documents version handling explicitly: "To allow for consistent comparison, we map every technique in TTPDrill to the latest version technique via the hyperlinks provided by MITRE. For example, T1086-PowerShell in TTPDrill is updated to T1059/001-Command and Scripting Interpreter: PowerShell" [28]. This is the only published example of official MITRE version-remapping precedent in extraction evaluation.

Beyond Single Reports (2026) [19][20] evaluates extraction on multi-report campaign settings, finding that errors at the single-report level propagate downstream. The paper reports that "the best-performing method correctly covers only 77.1% of the ground-truth techniques when mitigations were applied," highlighting that downstream control effectiveness depends critically on extraction precision. Ground truth appears to be multi-label per report (set of techniques), and aggregation across reports is via union operation (if technique appears in any report, it is part of campaign-level truth). No CISA advisory confirmation in this paper either.

TRAM Project [1] [2] is MITRE's official Threat Report ATT&CK Mapper, designed specifically to advance automated mapping research. Sources indicate sentence-level ground truth and recent fine-tuned LLM variants, but detailed ground-truth construction methodology is not fully documented in available research. The project is community-facing and enables researchers to test mapping approaches, but the source materials do not specify whether CISA advisories are included in TRAM's evaluation dataset or whether the project has formally adopted advisory tables as gold standard.

Sub-Technique Granularity Handling: None of the papers explicitly document whether their ground truth distinguishes parent techniques (e.g., T1059 Command and Scripting Interpreter) from sub-techniques (e.g., T1059.001 PowerShell). AttacKG's remapping example suggests sub-technique support, but no paper states whether evaluation metrics credit a predicted parent technique as a match for a gold-standard sub-technique or vice versa, or whether they are treated as distinct (strict).

Document Scope Definition: TTPXHunter and AttacKG treat entire reports as the unit of analysis; neither paper documents whether they exclude certain sections (e.g., IOC appendices, Mitigations, Executive Summaries). CISA Best Practices guidance emphasizes mapping "finished reports" (implying full document), but does not clarify whether Mitigations sections should be included in ground truth or treated separately.

Critical Gaps:

• No confirmed use of CISA advisory tables as extraction gold standard in any paper examined
• No guidance on sub-technique granularity (parent vs. child) in evaluation metrics
• No specification of document scope (full report vs. Technical Details section only)
• No evidence on whether table-to-prose alignment is verified (i.e., whether technique appears in both table AND prose, or table-only is acceptable)

---

5. CISA's Own Mapping Methodology: Advisory Curation Process, Table Exhaustiveness Intent, and Decider Tool Guidance

Confidence: HIGH (on Best Practices document content); LOW (on table exhaustiveness intent) Scope: General CISA methodology; unknown applicability to these four specific advisories

CISA guidance on ATT&CK mapping and advisory table curation

Best Practices for MITRE ATT&CK® Mapping exists in two versions: v1.0 (June 2021) [10] and v2.0 (January 2023) [9]. Both are published by CISA in partnership with the Homeland Security Systems Engineering and Development Institute (HSSEDI) and represent the authoritative published guidance on how CISA approaches mapping.

Key Principles from Best Practices:

1. Mapping finished reports first is recommended because finished reports contain more contextual information than raw data streams [9] [10].
2. Comprehensiveness is emphasized: The guidance recommends "finding ALL behaviors" and explicitly emphasizes double-checking to avoid missed mappings [9] [10].
3. Both in-line and tabular mapping are encouraged: Reports should contain hyperlinked technique IDs embedded within narrative prose (e.g., "The threat actor used T1566.002 phishing to gain initial access") AND a summary table enumerating all techniques with "Use" column context [9] [10].
4. "Use" column context is standard: The three-column table format (Tactic/Technique Title, ID, Use) is the approved advisory structure, where the Use column provides brief prose explaining the observed behavior [9] [10].

CRITICAL UNRESOLVED QUESTION ON EXHAUSTIVENESS: The Best Practices document emphasizes finding and mapping "all behaviors," but does NOT explicitly state whether advisory tables are REQUIRED to be exhaustive inventories of all prose-described techniques or whether they may be curated highlights. This distinction is fundamental to evaluation design:

• If tables must be exhaustive: A technique appearing in prose but missing from the table is a CISA error, and the table is gold standard
• If tables are curated highlights: A technique in prose but missing from the table may be intentional, and extracting it is not an error

This ambiguity is unresolved in available sources and requires clarification with CISA directly.

Decider Tool: All four advisories note they were "created using the Decider Tool" [4] [16] [17] [18]. Decider was released by CISA and MITRE in March 2023 to make ATT&CK mapping more accessible [6] [7] [8]. Decider "helps make mapping quick and accurate through guided questions, a powerful search and filter function, and a cart functionality" [8]. The tool is described as supporting "guided questions" to help curators identify applicable techniques, but no documentation of its methodology, decision rules, or output validation is provided in research sources. The github.com/cisagov/Decider repository is referenced but no content is available in the research findings, preventing detailed analysis of how Decider guides or validates table construction.

MITRE Training Materials: A slide deck titled "Module 2: Mapping to ATT&CK® from Raw Data" (2024) [36] references the principle "Not reinterpreting another analyst's prose/more insight into the behaviors," suggesting that mapping should preserve the original analyst's intent rather than adding new interpretations. This principle implies that CISA's table reflects the authors' intent, but does not clarify whether intent is exhaustive or selective.

Critical Implications for Your Evaluation:

• If you treat advisory tables as gold standard for document-level evaluation, you are implicitly assuming exhaustiveness. This assumption should be validated by manual audit of one advisory (e.g., comparing table entries against prose section-by-section) or explicit contact with CISA.
• If tables are not exhaustive (i.e., some techniques are described in prose without table entry), your evaluation metrics will penalize systems for extracting "correct" techniques that CISA did not table, creating systematic false-negative bias.

---

6. Cross-Version ATT&CK Evaluation: Official Resources for Version Mapping, Structural Changes, and Standard Practices

Confidence: HIGH (on MITRE version history and changelogs); MEDIUM (on evaluation guidance) Scope: General ATT&CK versioning; version-mismatch handling in evaluation is a gap

MITRE version timeline and crosswalk resources

Version Release Timeline for Advisories' Span:

• v16.1 released October 31, 2024; sunset April 21, 2025
• v17.0 released April 22, 2025
• v18.0 released October 28, 2025
• v19.0 released April 28, 2026 [12] [13] [14]
• v19.1 released May 12, 2026
• v19.2 released August 6, 2026 (first "Agile" release with narrower change scope) [12] [13] [14]

Your four advisories declare versions: AA25-050A/AA25-071A orig v16.1 (Feb–Mar 2025), AA25-203A v17 (Jul 2025), AA26-222A/AA25-071A reissue v19/v19.1 (Aug 2026). This spans a 4–5 major-version gap within the set, creating a multi-version evaluation challenge.

Structural Changes in v19 (April 28, 2026): The most significant change relevant to your evaluation is the split of the Defense Evasion tactic:

• Enterprise ATT&CK Defense Evasion (TA0005) was split into:
  • Stealth (TA0005, reused ID) — covert tactics
  • Defense Impairment (TA0112, new) — active adversary defense disabling
• Approximately 94% of Defense Evasion techniques and sub-techniques were redistributed between the two new tactics [11] [12] [14].
• Explicit revoked-by/deprecation metadata was provided for each moved technique via official MITRE hyperlinks [11] [14].

MITRE publishes detailed JSON changelogs per version transition (e.g., v18.1→v19) documenting technique additions, deprecations, and reclassifications [12] [14]. The ATT&CK Sync project streamlines version upgrades by providing machine-readable changelogs in both human and machine-readable formats [15].

AttacKG Precedent on Version Remapping: AttacKG explicitly handled version mismatches by remapping to latest via official MITRE hyperlinks: "To allow for consistent comparison, we map every technique in TTPDrill to the latest version technique via the hyperlinks provided by MITRE. For example, T1086-PowerShell in TTPDrill is updated to T1059/001-Command and Scripting Interpreter: PowerShell" [28]. This demonstrates that official MITRE remapping is achievable and has been used in prior academic work.

CRITICAL UNRESOLVED GUIDANCE: Published sources do NOT specify what the standard practice is when gold-label versions differ from system-index versions:

• Option A: Remap gold labels via official MITRE crosswalks to a canonical version (e.g., v19.2) before evaluation
• Option B: Exclude version mismatches from evaluation and clearly disclose
• Option C: Score version mismatches as misses and report accuracy separately for in-version matches

AttacKG used Option A. No paper discusses the impact on metrics of choosing each approach. No published evaluation has explicitly addressed version-mismatch handling as a methodological choice. This is a critical gap: the v16.1/v17/v19/v19.1 span in your four advisories makes this choice mandatory, and your methodological decision will affect reproducibility and interpretation.

Available Resources for Implementation:

• MITRE publishes JSON changelogs (e.g., changelog.json files in release notes) [12] [14]
• ATT&CK Sync tool provides upgrade guidance [15]
• mitreattack-python library is referenced [12] [14] but API documentation for version-mapping functions is not provided in research sources—you will need to review the library directly

---

7. Label Leakage in Self-Labeled Documents: Technique ID Visibility, Prior Work, and Quantification

Confidence: MEDIUM (on label-leakage concept); LOW (on CTI/ATT&CK-specific evidence) Scope: General NLP label-leakage literature; CTI-specific work is a gap

Label leakage risk in advisory-based evaluation

What is Label Leakage: In machine learning evaluation, label leakage occurs when ground-truth labels appear in the input data evaluated, inflating model performance metrics. In CTI evaluation, this manifests as technique IDs (e.g., T1190, T1566.002) being visible in the advisory prose that extraction systems process, creating a shortcut for models to identify ground-truth labels without understanding the underlying technique behavior.

Evidence of Label Visibility in CISA Advisories: The August 2026 Medusa advisory (AA25-071A) extensively embeds hyperlinked technique IDs directly in Technical Details prose. For example, narrative descriptions include "T1190" references within sentences describing observed behavior [35]. This is by design per CISA Best Practices, which recommends embedding technique links in prose alongside table summaries [9] [10]. The presence of these IDs in the document that extraction systems ingest creates label leakage: a model can extract IDs mechanically without understanding the described technique.

General NLP Label-Leakage Research: Academic literature addresses label leakage in explanation methods [21] and data contamination in LLM evaluation [22] [23]. A 2023 paper "Don't be fooled: label leakage in explanation methods and the importance of their quantitative evaluation" [21] demonstrates that explanation systems can appear to work while actually relying on leakage from the training data. Papers on data contamination in closed-source LLMs [23] document how models can appear to exceed expected performance when training data contamination is high. However, no paper in the research findings addresses label leakage specifically in CTI-technique extraction or ATT&CK mapping.

Masked Evaluation Precedent: General NLP best practices include "masked evaluation lanes" where labels or identifiers are scrubbed from test data to prevent leakage-driven metric inflation. However, no prior work in CTI/ATT&CK extraction implements or quantifies the impact of ID masking on metric outcomes. The concept is theoretically sound but empirically unexplored in this domain.

Critical Implications for Your Pipeline:

• Your extraction system processes advisory PDFs that contain hyperlinked technique IDs [35]. Depending on how your sentence-based extraction handles PDFs, IDs may or may not be visible during inference.
• If your system ingests plain text with IDs intact, metric inflation from leakage is likely: the system may achieve artificially high recall by identifying and extracting IDs mechanically rather than understanding technique descriptions.
• If your system scrubs IDs during preprocessing, leakage is mitigated, but you lose the ability to attribute improvements from ID visibility vs. semantic understanding.

Mitigation Strategy (Not Documented in Literature): To quantify label leakage, consider:

1. Baseline run: Evaluate pipeline on advisory PDFs with IDs intact (current state)
2. Masked run: Re-evaluate on ID-scrubbed versions (e.g., replace "T1190" with "[TECHNIQUE_REDACTED]")
3. Comparison: Compute F1 delta to estimate ID-visibility contribution to recall

This approach is not precedented in published CTI work but is theoretically sound and would strengthen your evaluation's credibility.

---

8. Mitigations-Section Semantics: Defensive Context Treatment in TTP Extraction

Confidence: MEDIUM (on Medusa structure); LOW (on other advisories and prior work) Scope: Specific to the four advisories; generalization requires audit

How mitigations and defensive references are treated in extraction evaluation

Medusa August 2026 Section Structure: The Medusa advisory (AA25-071A) clearly separates Technical Details (pages 5–18, containing observed adversary behaviors with hyperlinked technique IDs) from Mitigations (pages 19–20, containing defensive recommendations) [35]. The Mitigations section references NIST Cybersecurity Framework (CSF) and CISA guidance but does NOT embed hyperlinked technique IDs — it presents defenses at a higher level of abstraction (e.g., "implement network segmentation per NIST CSF") rather than mapping each defense to specific adversary techniques [35].

Semantic Distinction: Technique references in Technical Details sections describe observed adversary behavior ("the threat actor used this technique"). Technique references in Mitigations sections, if they appear, describe defensive countermeasures (e.g., "implement detection for T1059 to prevent command execution"). The semantic polarity is opposite: adversary activity vs. defensive posture.

Critical Unknown: The research findings document Medusa's clean separation but do NOT clarify whether the other three advisories (AA25-203A Interlock, AA26-222A Gunra, AA25-050A Ghost) embed technique IDs in their Mitigations sections. If they do, a critical evaluation question arises: should extraction systems credit extracting a technique ID from a Mitigations section as true positive (the technique is covered in the advisory), false positive (the context is defensive, not observed adversary), or excluded from evaluation entirely?

Prior Work on Defensive Context: The research findings include no prior work on false-positive quantification from mitigations/defensive references in ATT&CK extraction evaluation. The broader NLP field addresses sentiment flipping (e.g., "not vulnerable to X" vs. "vulnerable to X") and negation handling [40], but CTI-specific guidance on defensive context is absent.

Evaluation Design Decision Required:

• Option A: Include all technique references in ground truth regardless of context (Mitigations and Technical Details treated equally)
• Option B: Exclude Mitigations section from evaluation entirely (only Technical Details counted as ground truth)
• Option C: Dual labeling — mark technique references by context (observed vs. defensive) and evaluate separately

Option B is most defensible (cleanest signal for adversary behavior) but may underestimate document-level technique coverage if techniques are only mentioned in Mitigations. No published precedent guides this choice in CTI evaluation.

---

9. Small-n Case-Study Reporting: Best Practices for n≈4 Evaluations in Security Venues

Confidence: MEDIUM (on general case-study methodology); LOW (on security-specific conventions) Scope: Applicable to your four-advisory evaluation

Methodological guidance for presenting small-scale (n=4) evaluations

Case Study Suitability: General evaluation methodology supports case studies for n=5–50 entities (advisory documents in your case), particularly using Qualitative Comparative Analysis (QCA) [30]. The guideline distinguishes case studies from statistical inference: case studies answer "what," "how," and "why" questions; they cannot make broad statistical generalizations from small samples.

Required Documentation Elements [30]:

1. Objectives: Clear specification of what the case study is designed to examine (e.g., "assess extraction accuracy on four distinct ransomware advisories spanning ATT&CK v16.1–v19.1")
2. Case Selection Justification: Explicit reasoning for why these specific four advisories were chosen—purposive sampling with bracketing strategies (e.g., best-case, worst-case, typical-case logic). For your four advisories, this might be: "AA25-071A (largest, with reissue demonstrating version evolution), AA25-203A (mid-scale, spanning v17), AA26-222A (recent, v19.1), AA25-050A (earliest, v16.1)"
3. Bias Disclosure: Acknowledge known sources of variability (e.g., Medusa reissue may have better signal than original; Ghost advisory may have less detailed technical content)
4. Alternative Explanations: If performance varies across advisories, discuss whether variation stems from advisory quality, ransomware complexity, or system limitations
5. Generalizability Limitations: Explicitly state that findings apply to "these four ransomware-family advisories" and cannot be assumed to hold for other advisory types (APT groups, malware analysis, infrastructure attacks)

ACSAC Case Studies Track: The Annual Computer Security Applications Conference (ACSAC) includes a formal Case Studies track that accepts novel applications of existing techniques to real-world problems [29]. The 2024 ACSAC Best Case Study Award was given, confirming the track is active and valued. However, case studies are presented as conference talks, not proceedings papers, limiting citation impact in some evaluation contexts.

Statistical Claims at n=4: With four data points (four advisories), claims like "extraction F1 improves 15% on average" are descriptive (applying to these four) and cannot be generalized to the population of all CISA ransomware advisories. Framing should be: "Across our four case-study advisories, extraction F1 ranged from X% to Y%, with mean Z%," not "Our system achieves X% F1 on ransomware advisories" (the latter implies broader applicability).

Per-Document Reporting (Recommended for n=4): Present detailed results for each advisory separately:

• AA25-071A (Medusa): F1 X%, Precision Y%, Recall Z%, technique table size N, version v19
• AA26-222A (Gunra): F1 X%, Precision Y%, Recall Z%, technique table size N, version v19.1
• AA25-203A (Interlock): F1 X%, Precision Y%, Recall Z%, technique table size N, version v17
• AA25-050A (Ghost): F1 X%, Precision Y%, Recall Z%, technique table size N, version v16.1

This tabular presentation allows readers to see advisory-specific performance and identify patterns (e.g., "v19.1 advisory has lowest F1 due to Defense Evasion split affecting precision").

Critical Caveat: A four-advisory evaluation, while methodologically sound for case-study purposes, is insufficient for broad claims about CISA advisory quality or extraction system generalizability. This should be clearly stated in limitations.

---

10. Metric Conventions: Document-Level TTP Extraction Reporting and Positioning Against Ranking Metrics

Confidence: MEDIUM (on TTPXHunter/Beyond Single Reports metrics); LOW (on CISA-specific baselines) Scope: General academic standards; CISA-specific baselines are a gap

Document-level evaluation metrics used in published work

Macro-Averaged F1 (Standard in Prior Work): TTPXHunter and Beyond Single Reports both report Macro-averaged F1, Precision, and Recall as their primary metrics [24][25][26][27] [19][20]. Macro-averaging treats each TTP class (technique) equally, computing F1 per class and averaging across classes. This is appropriate when the concern is balanced performance across the full technique taxonomy rather than overall document coverage.

Hamming Loss (Document-Level Error Rate): TTPXHunter additionally reports Hamming Loss, defined as the fraction of incorrectly predicted labels per instance (document), averaged across instances. This metric is sensitive to imbalance (a document with 5 correct and 1 wrong prediction has a different Hamming Loss than one with 50 correct and 1 wrong).

Set-Based F1 (Document Technique Inventory): An alternative to class-wise F1 is set-based document-level F1, treating each document's ground truth as a set of techniques and evaluating the predicted set against it. This is more forgiving of sub-technique granularity mismatches (if ground truth is T1059 and system predicts T1059.001, set-F1 may treat it as partial credit depending on the metric definition).

Reported Performance from TTPXHunter:

• Macro-F1: 97.09%
• Macro-Precision: 97.38%
• Macro-Recall: 96.15%
• Baselines: LADDER 92.97%, TRAM 94.54%, AttacKG 88.58% [24][25][26][27]

Reported Performance from Beyond Single Reports:

• Multi-report union F1: Extraction errors propagate to downstream controls; "best-performing method correctly covers only 77.1% of ground-truth techniques when mitigations were applied" [19][20].
• Implies set-level recall of ~77.1% even for leading systems when accounting for mitigation coverage requirement (stricter than technique extraction alone)

Ranking Metrics (nDCG@k, Hit@k, Recall@k) vs. Set-F1: Your pipeline scores individual sentences and ranks techniques by relevance, a ranking task. Ranking metrics (nDCG@k, Recall@k, Hit@k) measure quality of a ranked list:

• nDCG@k: Normalized Discounted Cumulative Gain at rank k; rewards correct techniques ranked higher, penalizes correct techniques ranked low
• Recall@k: Fraction of ground-truth techniques recovered in the top-k ranked results
• Hit@k: Binary (0 or 1) indicating whether at least one ground-truth technique appears in top-k

These metrics are fundamentally different from set-F1: they assume a ranking and reward precision at fixed depth (e.g., top-20 predictions), whereas F1 operates on a binary decision threshold (technique included or excluded).

Critical Mismatch in Literature: Published document-level TTP extraction results (F1 97.09%, 92.97%, 94.54%) cannot be directly compared to ranking metrics on the same task because:

1. F1 assumes a decision threshold (technique is predicted or not); ranking metrics assume a depth k
2. Baseline papers do not report ranking metrics, so comparative positioning is impossible
3. A system achieving F1 of 90% may have nDCG@20 of 0.70 or 0.95 depending on ranking quality—the metrics are orthogonal

Positioning Your Results (Recommended Approach):

• Report both set-F1 AND ranking metrics for your evaluation (e.g., "set-F1 achieved 85.2%; nDCG@10 0.78; Recall@10 0.71")
• Clearly separate ranking-metric results from set-F1 results and acknowledge they measure different aspects
• Note that published set-F1 baselines (97%, 93%, 94%) are not directly comparable to your ranking metrics due to task difference
• Provide an F1 equivalent (e.g., "when converting top-10 ranked predictions to a binary decision via a learned threshold, set-F1 achieves X%") to bridge ranking and set-F1 paradigms for fair comparison

CRITICAL GAP: No published document-level TTP extraction evaluation on CISA-style advisories exists, so you cannot benchmark against prior work on this specific document class. TTPXHunter's 149-report dataset is likely different in composition, size, and annotation methodology from the four CISA advisories.

---

11. Rights and Citation: Reproducing CISA Advisory Content in Academic Papers

Confidence: HIGH (on TLP:CLEAR licensing); MEDIUM (on citation precedent) Scope: All four advisories (all marked TLP:CLEAR)

Licensing and reproducibility rights

TLP:CLEAR Designation and Reuse Policy: All four advisories carry the TLP:CLEAR (Traffic Light Protocol Clear) designation, which permits unrestricted distribution and use. The standard TLP:CLEAR declaration states: "This document is marked TLP:CLEAR. Disclosure is not limited. Sources may use TLP:CLEAR when the information carries minimal or no foreseeable risk of misuse, in accordance with the guidelines for TLP:CLEAR. Subject to standard copyright rules, TLP:CLEAR information may be distributed without restriction" [[24], [5] [35].

Implications for Academic Reproduction:

• Table excerpts: You may reproduce the advisory's ATT&CK technique tables (3-column tables with Tactic/Technique Title, ID, Use) in your paper without special permission or marked attribution beyond standard citation
• Quoted narrative: Sentences or paragraphs from the advisory may be quoted in full (within fair-use limits) with citation
• Figures and tables: The advisory's tables, IOC lists, and diagrams may be reproduced with attribution

Standard Copyright Considerations: TLP:CLEAR is an intelligence-sharing protocol, not a copyright waiver. Standard academic copyright conventions apply:

• Direct quotations >~50 words should be in quotation marks with page citation
• Reproduced tables should include source attribution in table caption or footnote
• Paraphrased content should be cited (authors, date, source)
• No copyright clearance letter is typically required for TLP:CLEAR content in academic contexts, but institutional IP policies may vary—consult your university's legal/compliance office if publishing in a venue with broad reach

Citation Norms (Recommended Practice): CISA advisories should be cited by their official identifier and publication date, with stable URL:

• Format: "Cybersecurity and Infrastructure Security Agency (CISA). (2026, August 18). #StopRansomware: Medusa Ransomware (AA25-071A). Retrieved from https://www.cisa.gov/news-events/cybersecurity-advisories/aa25-071a"
• Alternative format (if citing specific section): "CISA AA25-071A, Technical Details section, page 7, retrieved https://www.cisa.gov/sites/default/files/2026-08/aa25-071a-stopransomware-medusa-ransomware-508c.pdf"

MITRE Technique Definition Citations: When citing the ground-truth technique definitions themselves (e.g., in explaining what T1190 Exploit Public-Facing Application means), cite the version-specific MITRE page:

• Format: "MITRE ATT&CK (v19). T1190 Exploit Public-Facing Application. Retrieved from https://attack.mitre.org/versions/v19/techniques/T1190/"
• This ensures readers see the exact definition your evaluation used

Critical Gap: Published security papers rarely explicitly discuss citation conventions for CISA advisories; most simply include them in references without discussing licensing. Your paper should model explicit attribution to strengthen methodological transparency.

---

12. Known Pitfalls: Text Extraction, Sentence Segmentation, and Table Detection Failures on CISA/Government PDFs

Confidence: MEDIUM (on general PDF challenges); LOW (on CISA-specific failures) Scope: General PDF parsing guidance; CISA-specific evidence is a gap

Documented PDF parsing challenges and mitigation strategies

General PDF Parsing Challenges (Not CISA-Specific):

A 2024 comparative study evaluated six PDF parsing frameworks (PyMuPDF, pdfminer.six, Camelot, Tabula, TATR, Unstructured.io) on diverse document categories (financial reports, scientific papers, tender documents) and found tool performance varies significantly by document type [33]:

• PyMuPDF: Best for Financial documents (manually formatted with precise layouts)
• Camelot: Best for Tender documents (borderless tables with complex text alignment)
• Tabula: Best for Scientific papers (simple, ruled tables)
• pdfminer.six: Robust baseline for general PDFs but slower than alternatives

Specific PDF challenges documented in research:

• Borderless tables: Tables without visible borders (text aligned in columns via whitespace) are harder to detect; stream-mode processing in pdfminer/pdfplumber can help but requires careful threshold tuning [32]
• Nested tables: Tables within tables (e.g., a cell containing a sub-table) are rarely handled correctly; most tools extract flat text [32]
• Multi-page tables: Tables spanning multiple pages are typically treated as separate page-local tables by most tools, missing the logical table structure [32]
• Column separator detection: When separators are text characters ("....." or "----" lines), tools may misidentify them as content or fail to align columns correctly [32]
• Encoding/CJK support: pdfminer.six handles Chinese, Japanese, Korean (CJK) and vertical writing; PyMuPDF and Tabula have limited CJK support [33]
• Ligatures and glyph substitution: Rare in machine-generated PDFs but can occur in scanned documents or specific fonts; affects text-layer extraction if font subsetting encodes ligatures differently than the text layer [33]

pdfplumber-Specific Capabilities [31] [34]: pdfplumber is built on pdfminer.six and provides:

• Automatic table detection via line-based detection (identifies ruled tables) and text-alignment clustering (identifies borderless tables)
• Configurable table_settings parameters for tuning detection sensitivity (e.g., min_rows, intersection_tolerance, snap_tolerance)
• Visual debugging via .to_image() method to inspect extraction results
• Support for extracting text, tables, lines, rectangles, curves, and metadata from a single page or entire PDF

Mitigation Strategies from Prior Work [32] [34]:

1. Pre-cropping page regions: Define x-y coordinate bounds for table areas to isolate them from surrounding text
2. Visual debugging: Use pdfplumber's visualization to confirm extraction before processing at scale
3. Tool selection by document category: Choose appropriate tool based on advisory structure (if tables are ruled, pdfminer/pdfplumber; if borderless, Camelot; if complex, TATR)
4. Post-processing validation: Check extracted table structure (row count, column count, cell values) against ground truth for a sample of documents

CRITICAL GAP — NO DOCUMENTED CISA PDF-SPECIFIC FAILURES: The research findings provide no evidence of text-extraction, sentence-segmentation, or table-detection failures specific to CISA or Department of Defense advisory PDFs. Challenges described above are generic across all PDFs. CISA advisories are machine-generated (not scanned), have structured boilerplate (facilitating preprocessing), and use standard fonts and encoding, suggesting they are lower-risk for extraction failure than scanned government documents or specialty formats. However, without empirical testing on CISA PDFs directly, claims about parsing robustness are speculative.

Recommended Testing Strategy for Your Pipeline:

1. Extract Medusa AA25-071A (August 2026) using pdfplumber with default settings; confirm technique table is detected and rows are correctly aligned
2. Validate text-layer quality: Confirm extracted text matches visual PDF rendering (no character substitution, encoding artifacts)
3. Test on all four advisories: Repeat extraction on AA26-222A, AA25-203A, AA25-050A to identify advisory-specific quirks (if any)
4. Sentence segmentation validation: Verify that sentences extracted from Technical Details do not span boilerplate (e.g., page footer "Page N of M" does not appear mid-sentence)

---

Additional Considerations: Important Gaps and Assumptions for Your Evaluation Design

This section highlights critical unknowns and design decisions required before finalizing your evaluation.

Table Exhaustiveness: The Highest-Priority Unknown

CISA's Best Practices emphasize "finding ALL behaviors," but the research findings do NOT confirm whether advisory tables are required to be exhaustive or may be curated highlights. This distinction fundamentally affects how you use tables as gold standard:

• If exhaustive: Technique in prose but missing from table = system error; table is authoritative
• If curated: Technique in prose but missing from table = valid extraction; table is incomplete

Recommendation: Manually audit one advisory (suggest AA25-071A Medusa, which has detailed prose) by:

1. Identifying all technique mentions in Technical Details prose (manual read-through or via regex for T-codes)
2. Comparing against the advisory's official table
3. Counting false negatives (prose mentions not in table)
4. Contacting CISA if ambiguous cases arise

This audit will inform whether table-as-gold-standard is defensible or whether you need to construct your own ground truth via re-annotation.

Version Handling Strategy: Mandatory Decision

Your four advisories span v16.1, v17, v19, v19.1 with major structural changes (Defense Evasion split) in v19. Choose one approach and justify in your methodology:

• Option A: Remap to v19.2 (current) via MITRE official changelogs; standardizes gold labels; adds infrastructure complexity
• Option B: Keep version-pinned and exclude/penalize version mismatches; simpler; may inflate miss rate for older advisories
• Option C: Separate evaluation lanes by version (v16.1 lane, v17 lane, v19+ lane); most transparent; reduces per-lane sample size (n≈1)

Option A aligns with AttacKG precedent and is methodologically cleanest. Option C is most rigorous for case-study presentation.

Label Leakage Quantification

Your advisories contain hyperlinked technique IDs in prose [35]. To assess whether your pipeline benefits from ID visibility:

1. Baseline run: Extract techniques from advisory PDFs with IDs intact
2. Masked run: Re-extract from ID-scrubbed versions
3. Report delta: Quantify F1 impact of ID visibility (e.g., "ID visibility contributed ~X% to recall inflation")

This is not precedented in literature but is a strong methodological addition.

Sub-Technique Granularity Policy

Clarify how your evaluation handles parent-technique vs. sub-technique mismatches:

• Is T1059 (Command and Scripting Interpreter) treated as equivalent to T1059.001 (PowerShell) or distinct?
• Does your sentence-based extraction distinguish sub-technique boundaries or group them?
• How does your ranking metric handle partial credit (e.g., system predicts T1059.001 but gold standard is T1059)?

Define this explicitly in your methodology section.

Document Scope Definition

Specify whether evaluation includes:

• Full advisory (intro, Technical Details, Mitigations, IOCs, Appendices)
• Technical Details only (most conservative for adversary behavior)
• Excluding Mitigations (defensive references separated)

CISA guidance emphasizes Technical Details prose paired with summary tables; exclude Mitigations is defensible for adversary-behavior evaluation.

Baseline Positioning Challenge

No CISA advisory-specific document-level TTP extraction baselines exist in literature. TTPXHunter reports 97.09% F1 on a proprietary 149-report dataset (non-CISA, likely different composition). You cannot claim "our system achieves X%, outperforming prior work at 97%," because prior work is on a different dataset. Instead:

• Treat the four advisories as case studies (not statistical sample)
• Report per-document metrics (Table: AA25-071A F1=Y%, AA26-222A F1=Z%, etc.)
• Position against TTPXHunter as methodological reference, not quantitative baseline ("Our approach differs from TTPXHunter by emphasizing ranking-metric transparency; F1 is provided for reference")

Inter-Annotator Agreement for Gold Standard

If you re-annotate any advisory (due to table exhaustiveness concerns), obtain inter-annotator agreement (IAA) metrics (Cohen's kappa, Krippendorff's alpha, or Fleiss' kappa for multi-rater scenarios) to validate gold-standard quality [37] [38] [39] [41]. IAA <0.70 suggests label ambiguity requiring clarification [41].

---

Conclusion: Research Sufficiency and Remaining Preparation Steps

This research has established:

1. High-confidence foundations: CISA advisory metadata, MITRE versioning, Best Practices guidance, and TTPXHunter methodology are well-documented in Tier 1-2 sources
2. Documented methodological precedent: AttacKG's version remapping and TTPXHunter's macro-F1 evaluation provide academic models for your approach
3. Critical unresolved design decisions: Table exhaustiveness, label-leakage quantification, version-handling strategy, and CISA-specific PDF pitfalls require clarification or empirical testing

Before launching your evaluation, you must:

• Audit the four advisories for table exhaustiveness (manual prose-to-table alignment check)
• Decide on version handling (remap, exclude, or separate lanes)
• Implement label-leakage quantification (masked evaluation lane recommended)
• Define document scope (full advisory vs. Technical Details)
• Test PDF extraction on all four advisories for parsing robustness

With these clarifications in place, your pipeline evaluation will be methodologically rigorous and positioned appropriately against published work. The four-advisory case-study format is suitable for security venues (ACSAC, USENIX Security, CCS) if presented with explicit limitations and per-document reporting.

Research report: CTI-to-ATT&CK evaluation on four CISA #StopRansomware advisories

Scope and qualification

The four advisory pages and their downloadable artifacts were inspected directly where available. Claims about PDF internals, mirror byte identity, exact historical Medusa diffs, and published evaluation practice require downloading and hashing the actual PDF files and reviewing full papers/code repositories; the retrieved web evidence does not establish all of those points. Those items are therefore explicitly marked low confidence rather than inferred.

---

1. PDF structure and parsing

Applicability: CISA advisories generally; PDF-specific claims about the four targets are partly unverified.
Confidence: Medium for general CISA layout; low for byte identity and exact PDF internals.

• CISA distributes the advisory as a PDF separately from IOC files. The four pages identify PDF downloads and, depending on the advisory, STIX XML/JSON downloads. [[1]] [[2]] [[3]] [[4]]
• The HTML rendering exposes advisory tables as ordinary structured HTML, but that does not prove that the PDF contains vector table rules or machine-detectable table borders. A PDF must be inspected with pdfplumber, pdfminer.six, PyMuPDF, and rendered-page comparison before claiming ruled versus borderless tables.
• The safest expectation for government-generated PDFs is a usable text layer but imperfect reading order: headers, footers, multi-column regions, bullets, and table cells may be emitted in coordinate order rather than semantic order. pdfplumber is built on pdfminer.six and exposes characters, lines, rectangles, and table-detection primitives, but table extraction remains layout-dependent. [[5]]
• CISA’s TLP guidance says TLP markings should appear in document headers and footers on each page. [[6]] The advisory PDFs should therefore be tested for repeated TLP:CLEAR, page-number, product-ID, and agency-banner furniture rather than assuming that every repeated line is content.
• Recommended preprocessing:
  1. retain the original PDF and SHA-256 hash;
  2. extract text with at least two engines;
  3. render every page for visual adjudication;
  4. remove only repeated header/footer strings identified by page-frequency and coordinate analysis;
  5. preserve table text separately from prose;
  6. preserve section boundaries and page numbers in metadata;
  7. record extraction-engine version and parameters.
• Do not globally delete every line containing T, TA, or an ATT&CK ID: these advisories contain explicit IDs in prose, tables, footnotes, and mitigation-related material.

Unverified: No retrieved source establishes that CISA-, IC3-, and media.defense.gov-hosted copies are byte-identical. Treat them as potentially different PDF revisions. Hash each URL’s response body; compare hashes, page count, metadata, object streams, rendered images, and extracted text.

---

2. Machine-readable ATT&CK artifacts

Applicability: These four advisories specifically.
Confidence: High for listed downloads; high for absence of advertised Navigator layers; medium for STIX object contents because the bundles themselves were not parsed here.

Evidence:

• Medusa lists four STIX files, two for the original and two for the updated version. [[1]]
• Gunra lists one STIX XML and one STIX JSON file. [[2]]
• Interlock lists one STIX XML and one STIX JSON file. [[3]]
• Ghost lists the PDF, two XML files, and one JSON file. [[4]]
• The CISA pages enumerate the technique tables and their contents. [[1]] [[2]] [[3]] [[4]]

Important distinction: The advertised STIX files are described as IOC downloads. They should not be treated as ATT&CK annotation bundles unless parsing confirms the presence of attack-pattern objects and/or ATT&CK relationships. A STIX bundle containing only indicator, artifact, file, ipv4-addr, domain-name, or related objects is an IOC bundle, not a machine-readable technique layer.

Navigator layers: No retrieved CISA page advertises an ATT&CK Navigator layer JSON for any of the four advisories. Confidence: high for “not advertised”; medium for absolute nonexistence, because partner repositories and historical URLs were not exhaustively searched.

---

3. Medusa reissue

Applicability: AA25-071A specifically.
Confidence: High for current version and documented changes; low for exact original-to-reissue set difference.

CISA states that the original was published March 12, 2025 and updated August 18, 2026 using information through April 2026. The update expanded affiliate and payment details, added a broader exploited-vulnerability list, described Interactsh-based exploit verification, added network-enumeration, persistence, stealth, PowerShell-obfuscation, and C2 details, and added HHS as co-sealer. [[1]]

The current advisory explicitly adds or marks as updated:

• T1564.012 — Hide Artifacts: File/Path Exclusions;
• T1558 — Steal or Forge Kerberos Tickets;
• T1675 — ESXi Administration Command;
• additional updated operational details under the tables. [[1]]

The updated CVE list includes:

• CVE-2024-1709;
• CVE-2023-48788;
• newly listed CVE-2025-10035;
• newly listed CVE-2026-1731. [[1]]

The current table contains 38 technique rows when tactic-only entries are excluded. It spans Tables 3–12. [[1]]

The retrieved CISA page does not provide the original March 2025 technique-table contents or an explicit before/after row-count diff. Therefore:

• Exact original table size: not established.
• Exact set of all newly added techniques: not established from the retrieved evidence.
• ATT&CK version: current page says v19; the user’s v16 characterization for the original should be verified against the archived March 2025 artifact before reporting it as a measured fact. [[1]]

Recommended method: obtain both historical PDFs/HTML snapshots, normalize IDs to canonical strings, distinguish parent techniques from sub-techniques, and compute:

[ \text{added}=S_{\mathrm{Aug2026}}\setminus S_{\mathrm{Mar2025}} ]

rather than relying on “Updated” annotations alone.

---

4. Gold-standard precedent

Applicability: CTI/ATT&CK literature generally; not necessarily these four advisories.
Confidence: Medium.

• TRAM is an open-source platform for identifying ATT&CK techniques in CTI prose and supporting analyst validation. Its dataset uses sentence-level annotations and focuses on the 50 most prevalent techniques. [[7]] [[8]]
• Work evaluating CTI technique extraction has used CISA-derived datasets. The DSC dataset is described as 77 CISA cybersecurity articles whose TTP-ID sets were extracted from article text and used as multilabel ground truth. [[9]]
• A 2025 evaluation explicitly removed technique IDs, hyperlinks, and extraneous content before testing, demonstrating a direct precedent for ID-scrubbed evaluation. [[10]]
• AnnoCTR is a manually annotated collection of 120 cyber-threat reports with ATT&CK concepts and entities; it is not equivalent to taking CISA’s own table as unquestioned gold. [[11]]
• Recent work has criticized original TRAM annotations as containing false positives and false negatives and has introduced blind manual re-annotation. [[8]]

For your design, distinguish three gold regimes:

1. Publisher-table gold: every unique ID in the advisory’s ATT&CK table.
2. Inline-ID gold: IDs explicitly attached to prose spans.
3. Expert behavioral gold: independent annotators judge whether prose supports each technique.

They are not interchangeable. Publisher tables are document-level mappings, not necessarily sentence-level alignments. Sub-techniques should be scored exactly as published in one primary lane, with optional parent-normalized and version-normalized lanes.

---

5. CISA’s mapping methodology

Applicability: CISA advisories generally.
Confidence: High for methodology; low for a universal exhaustiveness claim.

CISA’s guidance was developed with HSSEDI and MITRE and is intended to help analysts map adversary behavior consistently to ATT&CK. [[12]] [[13]] CISA describes ATT&CK mappings as a way to identify and analyze adversary behavior in cybersecurity advisories. [[14]]

Decider guides the analyst through questions until reaching a sub-technique or, if no sub-technique fits, a technique. [[15]] This supports fine-grained mapping but does not establish that every behavior in an advisory must receive a table row.

The advisories themselves use language such as “all referenced threat actor tactics and techniques,” not “all behaviors appearing anywhere in the document.” [[1]] [[2]] [[3]] [[4]]

Accordingly:

• Table-listed technique without supporting prose: possible in principle, but not demonstrated by the retrieved sources.
• Behavior discussed in prose but omitted from table: also possible, especially in introductory, mitigation, IOC, or tool-description sections.
• Treat the table as publisher-curated document-level ground truth, not as a formally exhaustive annotation of every sentence.

---

6. Cross-version ATT&CK evaluation

Applicability: All four advisories and any v14-indexed system.
Confidence: High.

Official MITRE resources include:

• release/update pages for v14, v16, v17, v19, and v19.1;
• detailed changelogs;
• object revocation, deprecation, and deletion information;
• STIX bundles;
• mitreattack-python and its diffStix functionality. [[16]] [[17]] [[18]] [[19]] [[20]] [[21]]

MITRE explicitly distinguishes revoked objects, deprecated objects, and deleted objects. [[22]] Version 17, for example, revoked and merged a DLL side-loading sub-technique into another object. [[18]]

Recommended reporting:

• Primary score: exact published-ID matching against the advisory’s declared ATT&CK version.
• Crosswalk score: map gold and predictions through official revocation/replacement relationships.
• Strict historical score: score unavailable/newer IDs as misses, but disclose the version mismatch.
• Parent-normalized score: optional, clearly labeled, and never substituted silently for exact scoring.

Published practice is mixed: some evaluations remap data to a fixed ATT&CK release, some exclude techniques outside the system’s label space, and some report them as errors. The retrieved sources directly document remapping to ATT&CK 12.0 in one dataset and exclusion of techniques outside TRAM’s top-50 label space in another. [[23]] [[10]]

---

7. Label leakage

Applicability: These four advisories specifically and CTI extraction generally.
Confidence: High for the risk; medium for quantified precedent.

The advisories visibly place ATT&CK IDs in prose and tables—for example, Interlock’s prose contains IDs such as T1033, T1082, and T1016, while the tables repeat the labels. [[3]]

This creates direct leakage:

• lexical matching can recover gold IDs without understanding behavior;
• cross-encoder retrieval can exploit exact ID/title co-occurrence;
• mitigation and explanatory text may contain labels unrelated to observed adversary activity;
• table rows themselves make document-level retrieval nearly trivial if included in the input.

A directly relevant evaluation removed technique IDs, hyperlinks, and extraneous content to reduce bias. [[10]] General IE research also uses masking/de-identification to prevent models from exploiting entity strings rather than context. [[24]] [[25]]

Use three lanes:

1. Natural lane: unmodified PDF text.
2. ID-scrubbed lane: replace T####, T####.###, and TA#### with placeholders.
3. Table-excluded lane: remove ATT&CK tables and table-adjacent headings, retaining prose.

Report the delta:

[ \Delta F1=F1_{\text{natural}}-F1_{\text{scrubbed}} ]

No retrieved study quantified this exact inflation for CISA advisory ATT&CK extraction. Do not claim a published leakage percentage.

---

8. Mitigations-section semantics

Applicability: CISA advisories generally; especially relevant to this pipeline.
Confidence: Medium.

CISA distinguishes threat-actor TTP material from recommendations in the Mitigations section. The advisory pages explicitly tell readers to implement recommendations in that section to reduce ransomware impact. [[1]] [[3]]

For adversary-TTP extraction, mitigation text should normally be excluded from the positive behavioral scope. A sentence such as “enable MFA to prevent abuse of valid accounts” describes a defensive control, not necessarily an observed use of T1078.

Recommended labels:

• Observed adversary behavior
• Threat behavior described hypothetically or generically
• Defensive mitigation
• IOC/tool/reference material
• uncertain

No retrieved source supplies a CISA-specific mitigation false-positive rate. The fact that ATT&CK extraction work reports substantial false positives/false negatives in noisy sentence annotations supports measuring mitigation contamination separately, not assuming it is harmless. [[8]]

---

9. Small-(n) reporting

Applicability: This four-advisory study specifically.
Confidence: High as methodological guidance.

With (n=4) documents:

• report every document separately;
• provide per-document gold size, predicted list, precision, recall, F1, Recall@k, Hit@k, and nDCG@k;
• show a document-by-technique matrix;
• report macro averages across documents;
• optionally report micro totals, clearly distinguished;
• include bootstrap intervals only as descriptive uncertainty, not population-level inference;
• avoid claims such as “statistically significant improvement,” “generalizes to CISA advisories,” or “robustly outperforms” without a much larger test set.

The defensible framing is a reproducible case study or stress test, not a representative benchmark.

---

10. Metric conventions

Applicability: CTI/ATT&CK literature generally; positioning your ranking results.
Confidence: High.

Common set-extraction metrics are micro/macro precision, recall, and F1. TRAM-style work uses sentence-level classification, while document-level datasets report the set of techniques associated with a report. [[11]] [[26]]

For a ranked system, report:

• Hit@k: at least one gold technique appears in top (k);
• Recall@k: fraction of document gold set retrieved in top (k);
• Precision@k;
• nDCG@k, with graded relevance if desired;
• full-list set precision/recall/F1 after a threshold or fixed list length;
• rank correlation only if a meaningful gold ordering exists—which publisher tables generally do not provide.

Published numbers are not directly comparable unless label space, document scope, ATT&CK version, and gold construction match. Examples in the retrieved literature include:

• TRAM-style datasets restricted to 50 frequent techniques; [[10]]
• a CISA-derived DSC dataset containing 77 articles and multilabel TTP sets; [[9]]
• AnnoCTR reporting document-level technique/tactic F1; [[11]]
• reported macro-(F_{.5}) scores for an earlier malware-report labeling task, illustrating that metric choice materially affects positioning. [[26]]

Do not compare your nDCG@k directly with another paper’s macro-F1 as if they measured the same task.

---

11. Rights and citation

Applicability: Public TLP:CLEAR CISA advisories.
Confidence: High.

CISA’s TLP guidance states that TLP:CLEAR information may be shared without restriction, subject to standard copyright rules. [[6]] FIRST’s TLP definition says the same and clarifies that TLP is not a copyright license. [[27]]

Academic norms:

• cite the advisory title, alert code, issuing agencies, publication/update date, official URL, and access date;
• quote only the amount necessary for analysis;
• reproduce small table excerpts with attribution;
• identify whether the excerpt is from HTML or PDF;
• do not imply CISA endorsement;
• preserve TLP:CLEAR markings where reproduced;
• check the specific PDF for third-party copyrighted material, logos, or externally sourced figures.

TLP:CLEAR permits broad dissemination, but it does not eliminate ordinary copyright, quotation, attribution, or academic-integrity obligations.

---

12. Known parsing pitfalls

Applicability: Government CTI PDFs generally; specific failure rates for these four are unverified.
Confidence: Medium for general pitfalls; low for advisory-specific incidence.

Likely failure modes include:

• table-cell order being interleaved with neighboring columns;
• repeated headers/footers becoming false sentences;
• bullets and numbered lists being split or concatenated;
• hyphenation across line breaks;
• ligatures and unusual Unicode punctuation;
• URLs and obfuscated indicators being broken at line boundaries;
• ATT&CK IDs separated from technique titles;
• duplicate table headings caused by page continuation;
• PDF text order differing from visual order;
• false sentence boundaries after abbreviations, commands, and file names.

The Interlock HTML extraction itself illustrates why structural validation matters: duplicated “Technical Details” and “MITRE ATT&CK Tactics and Techniques” headings appear in the extracted representation. [[3]] That may be an HTML-rendering artifact rather than a PDF defect, but it demonstrates that naïve extraction can duplicate document structure.

Mitigations:

• compare pdftotext -layout, PyMuPDF, pdfminer.six, and pdfplumber;
• use coordinates to reconstruct rows;
• detect repeated lines by frequency and bounding-box position;
• preserve code spans and URLs before sentence segmentation;
• apply a domain tokenizer for ATT&CK IDs, CVEs, hashes, paths, commands, and defanged indicators;
• manually audit every table page;
• retain extraction provenance and a page-level error log.

---

13. Additional design issues that should be added

1. Version-lock the gold. Store the advisory’s declared ATT&CK version and exact artifact hash.
2. Separate tactic rows from technique rows. TA0005 and TA0011 are not techniques.
3. Handle duplicate techniques across tactics. Ghost lists T1071.001 in both Defense Evasion and Command and Control; define whether gold is a set of IDs or tactic-technique pairs. [[4]]
4. Define parent/sub-technique equivalence in advance. Exact-ID and parent-normalized scores should both be reported.
5. Separate retrieval from ranking. Report candidate recall before reranking and final ranking metrics after reranking.
6. Prevent train/test contamination. The four advisories contain explicit labels and may have near-duplicate HTML/PDF or reissued content.
7. Use document revision as a grouping variable. Medusa original and reissue are related versions, not independent documents.
8. Audit table-to-prose entailment. Publisher tables may contain curated mappings that are not sentence-aligned.
9. Treat the STIX files as IOC artifacts until proven otherwise. Do not use them as ATT&CK gold without parsing their object types.
10. Publish the preprocessing code and normalized gold files. This is essential for reproducibility, especially where PDFs may change while URLs remain stable.