Skip to content

Commit f278465

Browse files
pdev55tupaulmtf90
authored
Implementation for learning MMLTs, new model for collecting statistics (#153)
* Made RowImpl public to enable access in custom observation table implementations. * Made several methods of RowImpl public, to enable external access from observation tables that are defined in the learner module. * Started with integration of MMLT. * Extended EquivalenceOracle.java with LocalTimerMealyEquivalenceOracle. * Created module for symbol filters; added tests for counterexample handling for MMLT learner. * Functions for printing stats as JSON/YAML. * Renamed statsContainer; exporting mmlt caches. * Included symbol-filters module in dependency management. * Renamed fast cache to cache for MMLTs; added integration tests for MMLTs; allow access to stats during testing of learner. * Added more test models for the MMLT learner. * Added more test models for the MMLT learner. * Added cache consistency test for MMLT learning. * Removed location type parameter from LocalTimerMealyEquivalenceOracle. * Cache for MMLTs now inherits LearningCache * Multiple EQ tests for MMLTs can now respect the provided list of inputs for counterexamples. * Added tests for the cache; cleaned-up some files. * Added tests for the MMLT cache consistency test. * Made the MMLT SUL an interface with default methods. * Made symbol filters more independent from MMLTs * StatisticsSymbolFilter has stats container as constructor parameter. * Moved several MMLT examples to test-support. * Added an example for learning MMLTs; added module info for symbol filter module. * Added an example for learning MMLTs; added module info for symbol filter module. * Added more descriptions for included MMLT models * More info on model params * Updated reset search oracle to check if it can return a counterexample with the provided inputs. * adjust to AutomataLib refactorings * adjust to AutomataLib refactorings * adjust to AutomataLib refactorings * Using correct function to render MMLT in example. * Replaced getUntimedAlphabet with getInputAlphabet * MMLT-Learner now explicitly takes untimed inputs in constructor. * initial refactorings / cleanups * cleanup/generalize (integration-) tests * replace explicit StatsContainer with ServiceLoader-based approach * experimental: replace old statistics collection with new approach code compiles but parallel oracle tests still fail because the implementation is not thread-safe. final solution is still open for discussion. * add missing stats reset * make MapStatsContainer thread-safe and fix tests * refactor symbol filters * adjust to AutomataLib refactorings * fix code-analysis remarks up until L* module * cleanups * Removed sorting of prefixes from location cover, to reflect automataLib updates. * Updated MMLT examples. * Added more info about MMLTs. * Re-enabled visualization in MMLT examples. * Updated to AutomataLib changes: timers in MMLTs now support multiple outputs explicitly. * Added example for loading setting up a custom MMLT model. * Wording * Fixed bug that prevented duplicate timer outputs during timer inference. * fix issues flagged by code-analysis * cleanups + tests * cleanups, parallel timed oracles, test cases * more rigorous testing * make learner + OT batch queries to better support parallel oracles * cleanups most learning related code looks good, only statistics need an overhaul * statistics overhaul * updated changelog * mmlt: add test for some corner cases * drop toString() of internal record * filters: try to address some of the flaky tests on windows * filters: try to work around some platform issues * build: ignore generated MMLT class --------- Co-authored-by: Paul Kogel <p.kogel@tu-berlin.de> Co-authored-by: Markus Frohme <markus.frohme@udo.edu> Co-authored-by: Markus Frohme <mtf90@users.noreply.github.com>
1 parent 8799589 commit f278465

214 files changed

Lines changed: 12280 additions & 1116 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,11 +8,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
88

99
### Added
1010

11+
* Added a new (L*-based) learning algorithm for *Mealy machines with local timers* (MMLTs), including support for parallel queries, caching, and conformance testing (thanks to [Paul Kogel](https://github.com/pdev55)).
1112
* Added an `EarlyExitEQOracle` which for a given `AdaptiveMembershipOracle` and `TestWordGenerator` stops the evaluation of (potentially long) Mealy-based equivalence tests as soon as a mismatch with the hypothesis is detected, potentially improving the symbol performance of the given equivalence oracle.
1213

1314
### Changed
1415

1516
* LearnLib now requires Java 17 at runtime.
17+
* Statistics collection has received a major rework. Previously, classes would implement the `StatisticCollector` interface and return a `StatisticData` object which 1) only allows for describing a very limited amount of data, and 2) requires you to keep track of all the objects that collect data. This approach has been *replaced* by a new `StatisticsService`. Instances of this service can be obtained similar to a logger via `Statistics.getService()` and require you to provide an implementation of this service on the classpath (a default one is provided by the `learnlib-statistics` module). The new service allows arbitrary components to collect various data which can be conveniently extracted based on the new `StatisticsKey`s used by the components. For more details on advanced scenarios (such as multi-threaded benchmarking), see the documentation of the respective classes. While this may require you to adjust the way you are collecting statistics, all functionality from beforehand should still be available.
18+
* `SimpleProfiler` has been replaced by the new clock-based statistics.
1619
* The `generateTestWords` method of `AbstractTestWordEQOracle` now needs to be public.
1720

1821
### Fixed

‎README.md‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -17,19 +17,19 @@ While certain features have been stripped for improved modularity, development h
1717
Currently, the following learning algorithms with respective target models are supported:
1818

1919

20-
| Algorithm (active) | Target models | | Algorithm (passive) | Models |
21-
|---------------------|-----------------------------|-----|-----------------------|-----------------------|
22-
| AAAR | `DFA` `Mealy` `Moore` | | OSTIA | `SST` |
23-
| ADT | `Mealy` | | RPNI (incl. variants) | `DFA` `Mealy` `Moore` |
24-
| DHC | `Mealy` | | | |
25-
| Kearns & Vazirani | `DFA` `Mealy` | | | |
26-
| Lambda | `DFA` `Mealy` | | | |
27-
| L# | `Mealy` | | | |
28-
| L* (incl. variants) | `DFA` `Mealy` `Moore` | | | |
29-
| NL* | `NFA` | | | |
30-
| Observation Pack | `DFA` `Mealy` `Moore` `VPA` | | | |
31-
| Procedural | `SPA` `SBA` `SPMM` | | | |
32-
| TTT | `DFA` `Mealy` `Moore` `VPA` | | | |
20+
| Algorithm (active) | Target models | | Algorithm (passive) | Models |
21+
|---------------------|------------------------------|-----|-----------------------|-----------------------|
22+
| AAAR | `DFA` `Mealy` `Moore` | | OSTIA | `SST` |
23+
| ADT | `Mealy` | | RPNI (incl. variants) | `DFA` `Mealy` `Moore` |
24+
| DHC | `Mealy` | | | |
25+
| Kearns & Vazirani | `DFA` `Mealy` | | | |
26+
| Lambda | `DFA` `Mealy` | | | |
27+
| L# | `Mealy` | | | |
28+
| L* (incl. variants) | `DFA` `Mealy` `Moore` `MMLT` | | | |
29+
| NL* | `NFA` | | | |
30+
| Observation Pack | `DFA` `Mealy` `Moore` `VPA` | | | |
31+
| Procedural | `SPA` `SBA` `SPMM` | | | |
32+
| TTT | `DFA` `Mealy` `Moore` `VPA` | | | |
3333

3434
Additionally, LearnLib offers a variety of tools to ease the practical application of automata learning on real-world systems.
3535
This includes drivers and mappers for interfacing software systems with the LearnLib API as well as caches and parallelization for improving the overall performance of the learning setup.

‎algorithms/active/adt/src/test/java/de/learnlib/algorithm/adt/it/ADTIT.java‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@
4242
import de.learnlib.oracle.equivalence.MealySimulatorEQOracle;
4343
import de.learnlib.oracle.membership.MealySimulatorOracle;
4444
import de.learnlib.oracle.membership.SULAdaptiveOracle;
45+
import de.learnlib.statistic.Statistics;
4546
import de.learnlib.sul.SUL;
4647
import de.learnlib.testsupport.MQ2AQWrapper;
4748
import de.learnlib.testsupport.it.learner.AbstractMealyLearnerIT;
@@ -130,6 +131,7 @@ public void testIssue137() throws IOException, FormatException {
130131
for (int seed = 0; seed < 50; seed++) {
131132
long last = 0;
132133
for (int iter = 0; iter < 5; iter++) {
134+
Statistics.getService().clear();
133135
final CounterAdaptiveQueryOracle<String, String> counter =
134136
new CounterAdaptiveQueryOracle<>(aqo);
135137
final ADTLearner<String, String> learner = new ADTLearner<>(alphabet,
@@ -151,7 +153,8 @@ public void testIssue137() throws IOException, FormatException {
151153

152154
exp.run();
153155

154-
final long count = counter.getResetCounter().getCount();
156+
final long count =
157+
Statistics.getService().getCount(CounterAdaptiveQueryOracle.KEY_RESET).orElse(0L);
155158

156159
if (iter == 0) {
157160
last = count;

‎algorithms/active/lstar/pom.xml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,10 @@ limitations under the License.
5050
<groupId>de.learnlib</groupId>
5151
<artifactId>learnlib-counterexamples</artifactId>
5252
</dependency>
53+
<dependency>
54+
<groupId>de.learnlib</groupId>
55+
<artifactId>learnlib-symbol-filters</artifactId>
56+
</dependency>
5357
<dependency>
5458
<groupId>de.learnlib</groupId>
5559
<artifactId>learnlib-util</artifactId>
@@ -124,6 +128,21 @@ limitations under the License.
124128
<groupId>org.testng</groupId>
125129
<artifactId>testng</artifactId>
126130
</dependency>
131+
<dependency>
132+
<groupId>net.automatalib</groupId>
133+
<artifactId>automata-serialization-dot</artifactId>
134+
<scope>test</scope>
135+
</dependency>
136+
<dependency>
137+
<groupId>de.learnlib</groupId>
138+
<artifactId>learnlib-cache</artifactId>
139+
<scope>test</scope>
140+
</dependency>
141+
<dependency>
142+
<groupId>de.learnlib</groupId>
143+
<artifactId>learnlib-statistics</artifactId>
144+
<scope>test</scope>
145+
</dependency>
127146
</dependencies>
128147

129148
<build>
@@ -134,6 +153,16 @@ limitations under the License.
134153
<artifactId>maven-surefire-plugin</artifactId>
135154
<configuration>
136155
<!-- append to existing argLine to nicely work together with jacoco plugin -->
156+
<!-- allow tests access to util and statistics -->
157+
<argLine>@{argLine} --add-reads=de.learnlib.algorithm.lstar=net.automatalib.util</argLine>
158+
</configuration>
159+
</plugin>
160+
<plugin>
161+
<groupId>org.apache.maven.plugins</groupId>
162+
<artifactId>maven-failsafe-plugin</artifactId>
163+
<configuration>
164+
<!-- append to existing argLine to nicely work together with jacoco plugin -->
165+
<!-- allow tests access to util and statistics -->
137166
<argLine>@{argLine} --add-reads=de.learnlib.algorithm.lstar=net.automatalib.util</argLine>
138167
</configuration>
139168
</plugin>

0 commit comments

Comments
 (0)