Use UTF-8 as default with ScriptEngine (#962)

* Use UTF-8 as default with ScriptEngine

* Restore changes in PythonContext

Author: BCSharp

Src/IronPythonTest/EngineTest.cs

@@ -423,17 +423,22 @@ public void ScenarioCodePlex20472() {
 #if NETCOREAPP
             Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
 #endif
-
+            // This test file is encoded in Windows codepage 1251 (Cyrilic) but lacks a magic comment (PEP-263)
+            string fileName = Path.Combine(Path.Combine(Common.ScriptTestDirectory, "encoded_files"), "cp20472.py");
             try {
-                string fileName = Path.Combine(Path.Combine(Common.ScriptTestDirectory, "encoded_files"), "cp20472.py");
-                _pe.CreateScriptSourceFromFile(fileName, Encoding.GetEncoding(1251)).Compile();
+                _pe.CreateScriptSourceFromFile(fileName).Compile();
 
-                //Disabled. The line above should have thrown a syntax exception or an import error,
-                //but does not.
+                // The line above should have thrown a syntax exception or an import error
+                // because the default source file encoding is UTF-8 and there are decoding errors
 
-                //throw new Exception("ScenarioCodePlex20472");
+                throw new Exception("ScenarioCodePlex20472");
             }
-            catch (IronPython.Runtime.Exceptions.ImportException) { }
+            catch (SyntaxErrorException) { }
+
+            // Opening the file with explicitly specifying the correct encoding should work
+            CompiledCode prog = _pe.CreateScriptSourceFromFile(fileName, Encoding.GetEncoding(1251)).Compile();
+            prog.Execute();
+            Assert.AreEqual(prog.DefaultScope.GetVariable("s"), "\u041F\u0436\u0451");
         }
 
         [Test]

Tests/encoded_files/cp20472.py

@@ -1,2 +1,3 @@
 pi = 3.14 #This module is completely broken because of:  Ïæ¸
-          #AKA encoding 1251
+          #AKA encoding 1251
+s = "Ïæ¸"

WhatsNewInPython30.md

@@ -36,7 +36,7 @@ Text Vs. Data Instead Of Unicode Vs. 8-bit
 - [ ] Filenames are passed to and returned from APIs as (Unicode) strings. This can present platform-specific problems because on some platforms filenames are arbitrary byte strings. (On the other hand, on Windows filenames are natively stored as Unicode.) As a work-around, most APIs (e.g. `open()` and many functions in the `os` module) that take filenames accept bytes objects as well as strings, and a few APIs have a way to ask for a `bytes` return value. Thus, `os.listdir()` returns a list of `bytes` instances if the argument is a `bytes` instance, and `os.getcwdb()` returns the current working directory as a `bytes` instance. Note that when `os.listdir()` returns a list of strings, filenames that cannot be decoded properly are omitted rather than raising `UnicodeError`.
 - [ ] Some system APIs like `os.environ` and `sys.argv` can also present problems when the bytes made available by the system is not interpretable using the default encoding. Setting the `LANG` variable and rerunning the program is probably the best approach.
 - [x] [PEP 3138][]: The `repr()` of a string no longer escapes non-ASCII characters. It still escapes control characters and code points with non-printable status in the Unicode standard, however.
-- [ ] [PEP 3120][]: The default source encoding is now UTF-8.
+- [x] [PEP 3120][]: The default source encoding is now UTF-8.
 - [x] [PEP 3131][]: Non-ASCII letters are now allowed in identifiers. (However, the standard library remains ASCII-only with the exception of contributor names in comments.)
 - [x] The `StringIO` and `cStringIO` modules are gone. Instead, import the `io` module and use `io.StringIO` or `io.BytesIO` for text and data respectively.