Falsified "CI Text" test

[felixonmars]

  CI Text:            FAIL
    *** Failed! Falsified (after 77 tests and 6 shrinks):
    "\43929"
    "\43929" /= "\5065"
    Use --quickcheck-replay=93264 to reproduce.

[phadej]

Which GHC this is (the toUpper and friends change based on Unicode spec and different base use different ones)

[phadej]

Ok, I see:

-- on GHC-8.6
Prelude Data.Char> generalCategory '\43929'
NotAssigned

-- on GHC-8.8
Prelude Data.Char> generalCategory '\43929'
LowercaseLetter

This will be fixed when https://github.com/nick8325/quickcheck/pull/274/files change makes it to Hackage.

[felixonmars]

It's GHC 8.8.2 in case it still matters :P

[phadej]

And it seems that behavior is "wrong" in Text:

Data.Text.Lazy> toCaseFold  "\43929" :: Text
"\5065"

I guess what happens for unassigned characters is undefined, so I don't know what should/shouldn't happen (one could agree that doing nothing is what should happen)

[phadej]

Hmm,

• https://www.fileformat.info/info/unicode/char/ab99/index.htm
• https://www.fileformat.info/info/unicode/char/13c9/index.htm

CHEROKEE LETTER QUO and CHEROKEE SMALL LETTER QUO

[phadej]

Ok, found an issue:

Cherokee letters should fold to upper case, but now the don't converge:

Prelude Data.Char Data.Text> toCaseFold "\43929" 
"\5065"
Prelude Data.Char Data.Text> toCaseFold "\5065"
"\43929"

---

The docs say:

toCaseFold :: Text -> Text

O(n) Convert a string to folded case. Subject to fusion.

This function is mainly useful for performing caseless (also known as case insensitive) string comparisons.

A string x is a caseless match for a string y if and only if:

toCaseFold x == toCaseFold y

---

https://unicode.org/faq/casemap_charprop.html

Says

Q: What happens if the uppercase letter is the one that is already encoded?
A: That situation is more complicated. When the existing encoded letter is an uppercase letter and the proposal is to encode a new lowercase letter case pair for it, that is normally disallowed. The case folding for the existing uppercase letter would change, and that is blocked by the requirement for case folding stability. In exceptional situations, if a lowercase letter must be added, it would need to be case-folded to the existing uppercase letter, rather than changing the case folding for that existing letter. Such an exceptional situation did, in fact, apply for the addition of Cherokee lowercase syllables in Version 8.0. Cherokee case folding rules were specified to map to the old uppercase syllables, to preserve case folding stability for them.

---

Could you report this to text repository? Something wrong is happening with cherokee stuff: Folding of cherokee letters is not correct.

[felixonmars]

@phadej Sure. Thanks for investigating!

[felixonmars]

  CI Text:            FAIL
    *** Failed! Falsified (after 72 tests and 6 shrinks):
    "\5115"
    "\5115" /= "\5107"
    Use --quickcheck-replay=960636 to reproduce.

There is yet another CI Text failure with tasty-golden 2.3.3.2, should be a different thing though.

[phadej]

It is the same issue:

Unicode Character 'CHEROKEE SMALL LETTER YU' (U+13FB) -- 5115

[felixonmars]

Hrm, weird that it's only triggered after a tasty-golden update though. Sorry for the noise.

[trofi]

This text bug makes binary-instances tests non-deterministic: sometimes they pass, sometimes they fail.

I suggest making tests always fail by adding counterexample explicitly. Or always pass by filtering out known problematic cases.