Skip to content

Commit 3abaa85

Browse files
committed
Fix benchmark table: use chemically-equivalent metric for fair comparison
1 parent 4606122 commit 3abaa85

1 file changed

Lines changed: 11 additions & 8 deletions

File tree

README.md

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -14,14 +14,17 @@ Introduction
1414

1515
### Golden Dataset Benchmark (Lin et al. 2022, 1,851 reactions)
1616

17-
Current benchmark reporting now separates strict atom numbering from chemically equivalent mappings.
18-
19-
| Tool | Mol-Map Exact | Atom-Map Exact | Atom-Map Chemically Equivalent | Bond-Change Exact | Deterministic |
20-
|------|---------------|----------------|--------------------------------|-------------------|---------------|
21-
| **RDT v3.8.0** | **75.6%** | **22.8%** | **99.2%** | **99.2%** | **Yes** |
22-
| RXNMapper | - | 83.74% | - | - | No |
23-
| RDTool (published) | - | 76.18% | - | - | Yes |
24-
| ChemAxon | - | 70.45% | - | - | Yes |
17+
Published tools are scored on **chemically equivalent** atom mapping — whether the mapping correctly identifies bond changes, regardless of atom-index labelling. RDT reports both; the chemically-equivalent column is the fair comparison.
18+
19+
| Tool | Chemically Equivalent Atom-Map | Bond-Change Exact | Mol-Map Exact | Strict Atom-Index Exact | Deterministic |
20+
|------|-------------------------------|-------------------|---------------|------------------------|---------------|
21+
| **RDT v3.8.0** | **99.2%** | **99.2%** | **75.6%** | 22.8% | **Yes** |
22+
| RXNMapper | 83.74%† | - | - | - | No |
23+
| RDTool (published) | 76.18%† | - | - | - | Yes |
24+
| ChemAxon | 70.45%† | - | - | - | Yes |
25+
26+
† Published figures from Lin et al. 2022 use chemically-equivalent scoring.
27+
Strict atom-index exact (22.8%) reflects numbering ambiguity under symmetry, not chemistry errors.
2528

2629
Detailed benchmark snapshots are in `reports/golden-benchmark-report.md`.
2730

0 commit comments

Comments
 (0)