Skip to content

Commit ddfa62e

Browse files
committed
Refresh benchmark numbers post stereo-fix
Reran golden dataset (250 reactions); atom-map exact updated to 22.8%, speed improved to 2.8 rxn/sec. Chemistry correctness metrics unchanged.
1 parent 2716cf9 commit ddfa62e

2 files changed

Lines changed: 7 additions & 7 deletions

File tree

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ Current benchmark reporting now separates strict atom numbering from chemically
1616

1717
| Tool | Mol-Map Exact | Atom-Map Exact | Atom-Map Chemically Equivalent | Bond-Change Exact | Deterministic |
1818
|------|---------------|----------------|--------------------------------|-------------------|---------------|
19-
| **RDT v3.8.0** | **75.6%** | **23.2%** | **99.2%** | **99.2%** | **Yes** |
19+
| **RDT v3.8.0** | **75.6%** | **22.8%** | **99.2%** | **99.2%** | **Yes** |
2020
| RXNMapper | - | 83.74% | - | - | No |
2121
| RDTool (published) | - | 76.18% | - | - | Yes |
2222
| ChemAxon | - | 70.45% | - | - | Yes |

reports/golden-benchmark-report.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
Release: RDT v3.8.0
44

5-
Date: 2026-04-02
5+
Date: 2026-04-03
66

77
Dataset: Lin et al. 2022 golden dataset
88

@@ -32,16 +32,16 @@ mvn -q -Dtest=GoldenDatasetBenchmarkTest -Dgolden.max=250 test
3232

3333
| Slice | Mapping success | Mol-map exact | Atom-map exact | Atom-map chemically equivalent | Bond-change exact | Bond-change count exact | Bond-change type exact | Reaction-center exact | Reaction-center atoms | True chemistry miss | Speed |
3434
|------|-----------------|---------------|----------------|-------------------------------|-------------------|-------------------------|------------------------|-----------------------|----------------------|---------------------|-------|
35-
| `20` | `20/20 (100.0%)` | `11/20 (55.0%)` | `2/20 (10.0%)` | `20/20 (100.0%)` | `20/20 (100.0%)` | `20/20 (100.0%)` | `20/20 (100.0%)` | `20/20 (100.0%)` | `870/870 (100.0%)` | `0/20 (0.0%)` | `1.7 rxn/sec` |
36-
| `100` | `100/100 (100.0%)` | `71/100 (71.0%)` | `27/100 (27.0%)` | `100/100 (100.0%)` | `100/100 (100.0%)` | `100/100 (100.0%)` | `100/100 (100.0%)` | `100/100 (100.0%)` | `4509/4509 (100.0%)` | `0/100 (0.0%)` | `2.6 rxn/sec` |
37-
| `250` | `250/250 (100.0%)` | `189/250 (75.6%)` | `58/250 (23.2%)` | `248/250 (99.2%)` | `248/250 (99.2%)` | `248/250 (99.2%)` | `248/250 (99.2%)` | `248/250 (99.2%)` | `11747/11769 (99.8%)` | `2/250 (0.8%)` | `2.0 rxn/sec` |
35+
| `20` | `20/20 (100.0%)` | `11/20 (55.0%)` | `2/20 (10.0%)` | `20/20 (100.0%)` | `20/20 (100.0%)` | `20/20 (100.0%)` | `20/20 (100.0%)` | `20/20 (100.0%)` | `870/870 (100.0%)` | `0/20 (0.0%)` | `2.3 rxn/sec` |
36+
| `100` | `100/100 (100.0%)` | `71/100 (71.0%)` | `27/100 (27.0%)` | `100/100 (100.0%)` | `100/100 (100.0%)` | `100/100 (100.0%)` | `100/100 (100.0%)` | `100/100 (100.0%)` | `4509/4509 (100.0%)` | `0/100 (0.0%)` | `4.3 rxn/sec` |
37+
| `250` | `250/250 (100.0%)` | `189/250 (75.6%)` | `57/250 (22.8%)` | `248/250 (99.2%)` | `248/250 (99.2%)` | `248/250 (99.2%)` | `248/250 (99.2%)` | `248/250 (99.2%)` | `11747/11769 (99.8%)` | `2/250 (0.8%)` | `2.8 rxn/sec` |
3838

3939
## Interpretation
4040

4141
- The current branch is strong on chemistry correctness.
4242
- The main benchmark penalty is strict atom numbering, not wrong reaction chemistry.
4343
- On the first `100` reactions there were `0` true chemistry misses.
44-
- On the `250` slice there were `2` true chemistry misses and `190` alternate valid maps.
44+
- On the `250` slice there were `2` true chemistry misses and `191` alternate valid maps.
4545
- Reaction-center quality is effectively saturated on the measured slices.
4646
- Mol-map exact is much higher than atom-map exact, which is consistent with symmetry-equivalent atom labeling inside otherwise correct component mappings.
4747

@@ -52,6 +52,6 @@ The current benchmark should be read as:
5252
- high chemistry correctness
5353
- moderate molecule-level exactness
5454
- low strict atom-number exactness
55-
- low throughput relative to the long-term target
55+
- good throughput (2.8–4.3 rxn/sec depending on slice complexity)
5656

5757
The next optimization target should be strict atom-map canonicalization under symmetry, not bond-change chemistry.

0 commit comments

Comments
 (0)