Run tests for installed packages

[Thom1729]

Currently, UnitTesting cannot run tests that are inside installed packages.

This would allow package authors to verify that their tests run when the package is installed. End users could also more easily troubleshoot their installed packages.

This is mostly because it inherits the discover method of unittest.TestLoader, which uses the filesystem directly. We could reimplement discover in UnitTestingLoader to use the resource system instead. In principle, this should work for our use case, because every test suite that UnitTesting can find should be accessible as a resource.

There are also a couple of minor issues, such as the way that unittesting.json files are loaded.

I'm working on an implementation.

[randy3k]

Thank for the initiative work.

[Thom1729]

How would you feel about adding sublime_lib as a dependency? sublime_lib.ResourcePath would substantially simplify the implementation. (We could use other parts of it as well, but those would be separate PRs.)

[randy3k]

Sure. As a note, we will also need to modify the bootstrap code for the CI services so sublime_lib is installed automatically.

[Thom1729]

I'm new to the CI stuff, but shouldn't Package Control install all of the dependencies automatically?

It looks like there's some pre-ST3 code in there. I wonder if it could be simplified a bit to take advantage of new features like the builtin "Install Package Control" command.

[randy3k]

Installation of Package Control is optional. If a user didn't install Package Control manually via bootstrap install_package_control, the script will download the current only dependency coverage from the github repo.

[gerardroche]

Just to note that many packages don't include the tests in production packages delivered by Package Control. Package Control uses git to create the packages when installing. This allows developers to take advantage of the export-ignore rule in .gitattribute files (files and directories with the attribute export-ignore won’t be added to archive files. See http://git-scm.com/docs/gitattributes for details).

For example:

https://github.com/NeoVintageous/NeoVintageous/blob/master/.gitattributes

[Thom1729]

Good point. In that case, I would expect UnitTesting to behave as it normally would for a package with no tests.

Is there a good tutorial somewhere on setting up this sort of CI on a local machine for testing?

[randy3k]

I used Travis docker machine for testing.