Improved CI and started generating some screenshots that look close

Author: shai-almog

Ports/JavaScriptPort/STATUS.md

@@ -3,121 +3,54 @@
 JavaScript Port Status (ParparVM)
 =================================
 
-Last updated: 2026-04-07
+Last updated: 2026-04-09
 
 Current State
 -------------
 
-- **WORKAROUND IN PLACE**: `ensureDisplayEdt()` in port.js creates synthetic EDT if missing.
-- **FIX**: `initImpl` shim in port.js guards against `getClass().getName()` failures during `Display.init()`.
-  - In the ParparVM JS translation, `Object.getClass()` may return null or `Class.getName()` may return an underscore-separated name,
-    causing `String.lastIndexOf('.')` to return -1 and a subsequent `substring(0, -1)` to throw.
-  - The shim catches these errors, calls `init(m)` directly, and sets `packageName` from the bootstrap object's class metadata.
-- **FIX**: Simplified `ParparVMBootstrap.bootstrap()` to match `JavaScriptPortBootstrap.bootstrap()` structure.
-  - Removed intermediate `hasNativeTheme()`/`installNativeTheme()` calls between `Display.init()` and `bootstrap.run()`.
-  - These calls were triggering additional failures in the partially-initialized Display state.
-- **Removed**: Bogus root-level `ParparVMBootstrap.java` that had incorrect imports and structure.
-- **Remaining issue**: `IllegalStateException` still caught in fallback handlers even after EDT is set. May be unrelated to EDT (different code path).
-- The separate ParparVM Java test pipelines that were failing in CI (`job-logs2.txt`, `job-logs3.txt`) are now reproduced and fixed locally.
-
-## Diagnostic Evidence (Updated 2026-04-06)
-
-After EDT workaround, browser logs show:
-```
-PARPAR:DIAG:EDT_ENSURE:reusedMainThread=1
-PARPAR:DIAG:POST_EDT_ENSURE_formCtorTitleLayoutGlobal:displayClassExists=1:instance=present:edt=present:edtThreadName=main
-PARPAR:DIAG:PRE_formCtorLayout:displayClassExists=1:instance=present:edt=present:edtThreadName=main
-PARPAR:DIAG:ERR_formCtorLayout:displayClassExists=1:instance=present:edt=present:edtThreadName=main
-PARPAR:DIAG:FALLBACK:formCtorLayout:bypassIllegalState=1:detail=java_lang_IllegalStateException
-```
-
-**Key observations**:
-- EDT is successfully set (`edt=present:edtThreadName=main`)
-- Tests pass (suite completes)
-- `IllegalStateException` still triggered but caught by fallback handler
-- Exception may originate from code path other than `Display.setCurrent()` EDT check
-
-## Previous Issues (Historical)
-
-What Was Fixed In This Round
-----------------------------
-
-1. **Added EDT initialization workaround** in `port.js`:
-   - `checkDisplayInitState()` -checks Display.INITANCE and EDT state
-   - `ensureDisplayEdt()` - creates synthetic EDT thread or reuses main thread
-   - Form constructor bypasses call `ensureDisplayEdt()` before construction
-   - Verified workaround works: logs show `EDT_ENSURE:reusedMainThread=1`
-
-2. **Rebuilt JavaScript bundle** with updated port.js:
-   - Built with `SKIP_PARPARVM_BUILD=1 ./scripts/build-javascript-port-hellocodenameone.sh`
-   - Bundle contains updated `port.js` with workaround
-
-3. **Tests now pass**:
-   - Suite completes with `CN1SS:SUITE:FINISHED`
-   - Screenshots generated for MainActivity, graphics-draw-line, graphics-draw-rect, graphics-fill-rect, kotlin
-
-1. Restored native categorization for JavaScript translation.
-   - Reintroduced `vm/ByteCodeTranslator/src/com/codename1/tools/translator/JavascriptNativeRegistry.java`.
-   - Restored registry-driven behavior in `JavascriptMethodGenerator.appendNativeStubIfNeeded(...)`:
-     - runtime-implemented natives: no generic "Missing javascript native method" stubs.
-     - host-hook natives: generated `jvm.invokeHostNative(...)` stubs.
-     - unsupported natives: explicit unsupported error reasons.
-
-2. Expanded host-hook coverage to the JavaScript port boundary.
-   - Host-hook prefix now covers `cn1_com_codename1_impl_platform_js_`.
-   - This fixes `JavaScriptPortHost` bridge symbols used by JavaScript port smoke tests.
-
-3. Fixed `String.format(...)` runtime coercion bug.
-   - File: `vm/ByteCodeTranslator/src/javascript/parparvm_runtime.js`.
-   - Changed format token conversion to use `runtimeFormatTokenValue(...)`.
-   - This resolves the deterministic `JsJavaApiCoverageApp` mismatch (`expected 511, got 495`; missing bit `16`).
-
-4. Added explicit PolyForm header text to missing JavaScriptPort boundary files.
-   - Added `"PolyForm Noncommercial License 1.0.0"` markers to all Java/Markdown files previously flagged by `JavaScriptPortSmokeIntegrationTest`.
-
-5. Reverted risky JS `INVOKESPECIAL` remap in codegen.
-   - Removed the recent `Util.resolveInvokeSpecialOwner(...)` injection from JS invoke emission paths for now.
-   - The broader regressions observed in CI are fixed without that change.
-
-6. **Enhanced Form constructor error diagnostics**.
-   - Added `checkDisplayInitState()` function to report Display.INSTANCE and EDT state.
-   - Added `emitDisplayInitDiag()` calls before and after Form constructor execution.
-   - Enhanced `stringifyThrowable()` to capture `messageOnly` separately.
-   - Bypass handlers now log `PRE_` and `ERR_` Display state along with exception details.
-
-Validated Locally
------------------
-
-- `mvn -pl tests -am "-Dtest=JavascriptRuntimeSemanticsTest,JavascriptTargetIntegrationTest,JavaScriptPortSmokeIntegrationTest" -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false test`
-- Result: `Tests run: 313, Failures: 0, Errors: 0, Skipped: 0` (BUILD SUCCESS, finished 2026-04-06 10:47:59 +03:00).
+- Startup/protocol is no longer the primary blocker. Main remaining blockers are in screenshot correctness and screenshot pipeline throughput.
+- `ensureDisplayEdt()` and diagnostics remain active in `port.js`.
+- Form constructor `IllegalStateException` bypass now attempts recovery instead of returning `null`:
+  - calls default `Form` constructor
+  - reapplies layout
+  - reapplies title (for title+layout constructor)
+  - emits `PARPAR:DIAG:FALLBACK:formCtor*:recoverApplied=1` markers
+- CI artifact behavior and fresh local rebuild behavior are currently diverged (details below).
 
 Next Steps
 ----------
 
-1. **Investigate remaining `IllegalStateException`**:
-   - Exception occurs even after EDT is set
-   - May be from different code path (not Display.setCurrent)
-   - Add exception message logging to `ERR_` diagnostics
-
-2. **Optional**: Investigate why Display.init() doesn't create EDT in JavaScript port:
-   - Trace `Display.initNative()` call sequence
-   - Check if thread creation is happening but not persisting
-
-3. **Commit workaround**: The `ensureDisplayEdt()` fix allows tests to pass; consider committing as a workaround while root cause is investigated.
+1. Unify build provenance first:
+   - Confirm CI bundle contains expected translated `Cn1ssDeviceRunner` variant.
+   - In generated `translated_app.js`, verify whether runner uses `TEST_CLASSES` (old list path) or `DEFAULT_TEST_CLASSES/prependedTest` (new array path).
+   - This directly changes failure mode and must be deterministic before further triage.
+2. Reduce EDT starvation in screenshot emission:
+   - Current fresh local run advances to 11 tests, then stalls after `DrawImage` due very large `CN1SS:<name>:<chunk>` emissions.
+   - Add bounded chunk/preview strategy for extremely large screenshots and move heavy conversion off hot EDT path where possible.
+3. Keep form-constructor fallback bounded:
+   - `IllegalStateException` still occurs frequently.
+   - Recovery now preserves object state better, but we need to reduce recursive/looping constructor retry behavior and make one-shot recovery per form instance.
+4. After throughput fix, run 3 repeated local runs and one CI run:
+   - Exit gate is `CN1SS:SUITE:FINISHED` + 33 screenshot streams + no repeated stream collapse.
 
 Important Notes
 --------------
 
-- The CI breakage in the ParparVM Java test pipelines was real and not just screenshot-noise; it is now addressed by the translator/runtime fixes above.
-- The screenshot problem remains a separate rendering/lifecycle issue after startup/protocol recovery.
-- Error messages are now extracted separately from the full throwable detail for easier parsing.
+- Fresh local rebuild (`/tmp/cn1-js-fresh*.zip`) now translates current runner code and shows 11 unique tests before timeout.
+- Existing CI artifact in `javascript-ui-tests/HelloCodenameOne-js/translated_app.js` still shows old list-based runner (`TEST_CLASSES`) and produces only 5 screenshot streams with suite completion.
+- This means there are at least two active failure modes:
+  1. old-runner path: suite finishes with only 5 streams (wrong collapse)
+  2. new-runner path: progresses further but times out during heavy screenshot emission
 
 Known Important Context
 -----------------------
 
-- This file supersedes older status notes that referenced initial JSBody/static-wrapper bootstrap issues as the primary blocker.
-- Current bottleneck is no longer "suite timeout"; it is "suite passes but screenshots are wrong".
-- Existing local tree also includes ongoing debug-oriented changes in:
+- Useful diagnostics to grep:
+  - `PARPAR:DIAG:FALLBACK:lambdaBridge:capturedTest=...:capturedIndex=...`
+  - `PARPAR:DIAG:FALLBACK:formCtorLayout:bypassIllegalState=1`
+  - `PARPAR:DIAG:FALLBACK:formCtorLayout:recoverApplied=1`
+  - `CN1SS:INFO:suite starting test=...`
+  - `CN1SS:SUITE:FINISHED`
+- Current local patch set touches:
   - `Ports/JavaScriptPort/src/main/webapp/port.js`
-  - `Ports/JavaScriptPort/src/main/java/com/codename1/impl/html5/HTML5Implementation.java`
-  These are part of the active CI recovery/debugging workflow.
+  - `scripts/hellocodenameone/common/src/main/java/com/codenameone/examples/hellocodenameone/tests/Cn1ssDeviceRunner.java`

Ports/JavaScriptPort/src/main/webapp/port.js

@@ -1620,6 +1620,69 @@ const formCtorLayoutMethodId = "cn1_com_codename1_ui_Form___INIT___com_codename1
 const formCtorTitleLayoutMethodId = "cn1_com_codename1_ui_Form___INIT___java_lang_String_com_codename1_ui_layouts_Layout";
 const formCtorLayoutOriginal = typeof global[formCtorLayoutMethodId] === "function" ? global[formCtorLayoutMethodId] : null;
 const formCtorTitleLayoutOriginal = typeof global[formCtorTitleLayoutMethodId] === "function" ? global[formCtorTitleLayoutMethodId] : null;
+const formDefaultCtorMethodId = "cn1_com_codename1_ui_Form___INIT__";
+const formSetTitleMethodId = "cn1_com_codename1_ui_Form_setTitle_java_lang_String";
+const containerSetLayoutMethodId = "cn1_com_codename1_ui_Container_setLayout_com_codename1_ui_layouts_Layout";
+
+function* recoverFormCtorIllegalState(self, title, layout, marker) {
+  if (!self || !self.__class) {
+    emitDiagLine("PARPAR:DIAG:FALLBACK:" + marker + ":recoverSkipped=noSelf");
+    return null;
+  }
+  if (self.__cn1FormCtorRecovering) {
+    emitDiagLine("PARPAR:DIAG:FALLBACK:" + marker + ":recoverSkipped=reentry");
+    return null;
+  }
+  self.__cn1FormCtorRecovering = true;
+  let ctorApplied = false;
+  try {
+    const defaultCtor = global[formDefaultCtorMethodId + "__impl"] || global[formDefaultCtorMethodId];
+    if (typeof defaultCtor === "function") {
+      try {
+        yield* defaultCtor(self);
+        ctorApplied = true;
+      } catch (ctorErr) {
+        emitDiagLine("PARPAR:DIAG:FALLBACK:" + marker + ":recoverCtorError=" + String(ctorErr && ctorErr.__class ? ctorErr.__class : ctorErr));
+      }
+    }
+    if (layout && layout.__class) {
+      let layoutApplied = false;
+      try {
+        const setLayout = jvm.resolveVirtual(self.__class, containerSetLayoutMethodId);
+        yield* setLayout(self, layout);
+        layoutApplied = true;
+      } catch (_setLayoutErr) {
+        // Fall through to direct field patch.
+      }
+      if (!layoutApplied) {
+        self["cn1_com_codename1_ui_Container_layout"] = layout;
+      }
+    }
+    if (title && title.__class === "java_lang_String") {
+      let titleApplied = false;
+      try {
+        const setTitle = jvm.resolveVirtual(self.__class, formSetTitleMethodId);
+        yield* setTitle(self, title);
+        titleApplied = true;
+      } catch (_setTitleErr) {
+        // Fall through to direct field patch.
+      }
+      if (!titleApplied) {
+        self["cn1_com_codename1_ui_Form_title"] = title;
+      }
+    }
+    self.__cn1FormCtorRecovered = true;
+  } finally {
+    self.__cn1FormCtorRecovering = false;
+  }
+  emitDiagLine(
+    "PARPAR:DIAG:FALLBACK:" + marker + ":recoverApplied=1"
+    + ":ctor=" + (ctorApplied ? "1" : "0")
+    + ":layout=" + (layout && layout.__class ? "1" : "0")
+    + ":title=" + (title && title.__class === "java_lang_String" ? "1" : "0")
+  );
+  return null;
+}
 
 function installGlobalIllegalStateBypass(symbol, marker) {
   const original = global[symbol];
@@ -1705,7 +1768,7 @@ bindCiFallback("Form.layoutCtorIllegalStateBypass", [
       if (messageOnly) {
         emitDiagLine("PARPAR:DIAG:FALLBACK:formCtorLayout:messageOnly=" + messageOnly);
       }
-      return null;
+      return yield* recoverFormCtorIllegalState(__cn1ThisObject, null, layout, "formCtorLayout");
     }
     throw err;
   }
@@ -1749,7 +1812,7 @@ bindCiFallback("Form.titleLayoutCtorIllegalStateBypass", [
       if (messageOnly) {
         emitDiagLine("PARPAR:DIAG:FALLBACK:formCtorTitleLayout:messageOnly=" + messageOnly);
       }
-      return null;
+      return yield* recoverFormCtorIllegalState(__cn1ThisObject, title, layout, "formCtorTitleLayout");
     }
     throw err;
   }
@@ -1762,6 +1825,7 @@ const cn1ssCompleteMethodId = "cn1_com_codenameone_examples_hellocodenameone_tes
 const cn1ssEmitChannelMethodId = "cn1_com_codenameone_examples_hellocodenameone_tests_Cn1ssDeviceRunnerHelper_emitChannel_byte_1ARRAY_java_lang_String_java_lang_String";
 const baseTestRegisterReadyCallbackMethodId = "cn1_com_codenameone_examples_hellocodenameone_tests_BaseTest_registerReadyCallback_com_codename1_ui_Form_java_lang_Runnable";
 const cn1ssRunnerClassId = "com_codenameone_examples_hellocodenameone_tests_Cn1ssDeviceRunner";
+const cn1ssRunnerListGetMethodId = "cn1_java_util_List_get_int_R_java_lang_Object";
 function collectCn1ssRunnerLambdaMethodIds() {
   const ids = [];
   if (!jvm || !jvm.classes || !jvm.classes[cn1ssRunnerClassId] || !jvm.classes[cn1ssRunnerClassId].methods) {
@@ -1866,6 +1930,33 @@ bindCiFallback("Cn1ssDeviceRunner.lambdaRunNextTestBridge", cn1ssLambdaBridgeMet
     }
   }
   const effectiveIndex = capturedIndex != null ? (capturedIndex | 0) : (index | 0);
+  // Prefer runner static test list over lambda captures when available. Captured
+  // lambda arguments have been observed to drift after initial indices in ParparVM.
+  let indexedTestObject = null;
+  if (runner && runner.__class === cn1ssRunnerClassId) {
+    try {
+      const runnerClass = jvm.classes[cn1ssRunnerClassId];
+      const testClasses = runnerClass && runnerClass.staticFields
+        ? runnerClass.staticFields["TEST_CLASSES"]
+        : null;
+      if (testClasses && testClasses.__class) {
+        const listGetMethod = jvm.resolveVirtual(testClasses.__class, cn1ssRunnerListGetMethodId);
+        indexedTestObject = yield* listGetMethod(testClasses, effectiveIndex);
+      }
+    } catch (_err) {
+      indexedTestObject = null;
+    }
+  }
+  if (jvm.instanceOf(indexedTestObject, "com_codenameone_examples_hellocodenameone_tests_BaseTest")) {
+    if (!effectiveTestObject || !effectiveTestObject.__class || effectiveTestObject.__class !== indexedTestObject.__class) {
+      emitLambdaBridgeDiag(
+        "PARPAR:DIAG:FALLBACK:lambdaBridge:indexedOverride:index=" + String(effectiveIndex)
+        + ":from=" + (effectiveTestObject && effectiveTestObject.__class ? effectiveTestObject.__class : "null")
+        + ":to=" + indexedTestObject.__class
+      );
+    }
+    effectiveTestObject = indexedTestObject;
+  }
   emitLambdaBridgeDiag(
     "PARPAR:DIAG:FALLBACK:lambdaBridge:capturedRunner="
     + (runner && runner.__class ? runner.__class : "null")

scripts/hellocodenameone/common/src/main/java/com/codenameone/examples/hellocodenameone/tests/Base64NativePerformanceTest.java

@@ -52,23 +52,22 @@ public boolean runTest() {
             return false;
         }
 
-        int encodedLen = ((payloadBytes.length + 2) / 3) * 4;
-        byte[] cn1EncodedBytes = new byte[encodedLen];
-        int encodedWritten = Base64.encodeNoNewline(payloadBytes, cn1EncodedBytes);
-        if (encodedWritten != encodedLen) {
-            fail("CN1 preallocated Base64 encode returned unexpected length");
+        byte[] cn1EncodedBytes;
+        try {
+            cn1EncodedBytes = cn1Encoded.getBytes("UTF-8");
+        } catch (Exception ex) {
+            fail("Failed to encode CN1 Base64 output to UTF-8 bytes: " + ex);
             return false;
         }
-        byte[] cn1DecodedBuffer = new byte[payloadBytes.length];
 
         if (!isIos()) {
-            warmup(nativeBase64, payload, payloadBytes, nativeEncoded, cn1EncodedBytes, cn1DecodedBuffer);
+            warmup(nativeBase64, payload, payloadBytes, nativeEncoded, cn1EncodedBytes);
         }
 
         long nativeEncodeMs = measureNativeEncode(nativeBase64, payload);
-        long cn1EncodeMs = measureCn1Encode(payloadBytes, cn1EncodedBytes);
+        long cn1EncodeMs = measureCn1Encode(payloadBytes);
         long nativeDecodeMs = measureNativeDecode(nativeBase64, nativeEncoded);
-        long cn1DecodeMs = measureCn1Decode(cn1EncodedBytes, cn1DecodedBuffer);
+        long cn1DecodeMs = measureCn1Decode(cn1EncodedBytes);
 
         double encodeRatio = cn1EncodeMs / Math.max(1.0, (double) nativeEncodeMs);
         double decodeRatio = cn1DecodeMs / Math.max(1.0, (double) nativeDecodeMs);
@@ -85,12 +84,12 @@ public boolean runTest() {
         return true;
     }
 
-    private static void warmup(Base64Native nativeBase64, String payload, byte[] payloadBytes, String nativeEncoded, byte[] cn1EncodedBytes, byte[] cn1DecodedBuffer) {
+    private static void warmup(Base64Native nativeBase64, String payload, byte[] payloadBytes, String nativeEncoded, byte[] cn1EncodedBytes) {
         for (int i = 0; i < 40; i++) {
             nativeBase64.encodeUtf8(payload);
-            Base64.encodeNoNewline(payloadBytes, cn1EncodedBytes);
+            Base64.encodeNoNewline(payloadBytes);
             nativeBase64.decodeToUtf8(nativeEncoded);
-            Base64.decode(cn1EncodedBytes, cn1DecodedBuffer);
+            Base64.decode(cn1EncodedBytes);
         }
     }
 
@@ -102,10 +101,10 @@ private static long measureNativeEncode(Base64Native nativeBase64, String payloa
         return System.currentTimeMillis() - start;
     }
 
-    private static long measureCn1Encode(byte[] payloadBytes, byte[] outputBuffer) {
+    private static long measureCn1Encode(byte[] payloadBytes) {
         long start = System.currentTimeMillis();
         for (int i = 0; i < ITERATIONS; i++) {
-            Base64.encodeNoNewline(payloadBytes, outputBuffer);
+            Base64.encodeNoNewline(payloadBytes);
         }
         return System.currentTimeMillis() - start;
     }
@@ -118,10 +117,10 @@ private static long measureNativeDecode(Base64Native nativeBase64, String encode
         return System.currentTimeMillis() - start;
     }
 
-    private static long measureCn1Decode(byte[] encoded, byte[] outputBuffer) {
+    private static long measureCn1Decode(byte[] encoded) {
         long start = System.currentTimeMillis();
         for (int i = 0; i < ITERATIONS; i++) {
-            Base64.decode(encoded, outputBuffer);
+            Base64.decode(encoded);
         }
         return System.currentTimeMillis() - start;
     }