Servo: None of the support crates are being tidy-checked or license-checked

the platform directory has a similar problem

Looking at the list of ignored files, this might be done

A bunch of "our" code (which might be defined as non-forks under https://github.com/servo/) is in separate repositories and used through cargo, and so not visible to tidy.py. So this is not done, and now harder than before we switched to Cargo and everything was a submodule.

Can't we just put tidy in each repo with a different config? Those repos should all be in the CI system, so that would achieve the same end.

Each repos has its own separate CI that could have a copy of tidy.py, but that means many copies to update when we want to add something.

Move tidy to a submodule? Download it and run it during CI? There are a few approaches to reduce that problem.

An option to consider: uploading tidy to PyPi as servo_tidy and just download the latest version whenever we want to tidy

EDIT: jack beat me to it

@frewsxcv But I like Corey's more concrete version of the suggestion best.

Alright. I have thought about this for a little and think I am up for the challenge. I will report back with my thoughts before implementing anything; I might be able to fix #6999 simultaneously with this.

After some thinking, here is what I propose:

1. Virtual environments (bleh?). Upon running any mach command, Python will create/activate a virtual environment (which will live somewhere like .pyenv/ or python/.pyenv/). Then we will install/update our dependencies as per a python/requirements.txt file). This will allow us to remove the following from our tree and add them in as PyPi requirements:
   
   
   
   • python/mozdebug
   
   
   • python/mozinfo
   
   
   • python/mozlog
   
   
   • dependencies/*
   
   
   • python/toml
   
   

This will also fix #6999. Depending on whether the builders blow away the working directory after every build, we might need to add some sort of caching option or a mach parameter for specifying a Python virtualenv.

1. Package tidy.py. This could mean just creating python/tidy/tidy.py and python/tidy/setup.py.
2. Incorporate tidy.py into the other repos. There are a couple ways to do this:
   
   
   
   1. Release it on PyPi. Unless we created a system for automating publishing the Python package whenever it changes, releasing it could be a pain.
   
   
   2. Having all the other repositories just do:
   
   

pip install -e git+https://github.com/servo/servo.git#egg=servo_tidy&subdirectory=python/tidy

Let me know if anyone has any other ideas or thoughts

We already have a virtualenv for web-platform-tests, maybe it could also be used for this.

We already have a virtualenv for web-platform-tests, maybe it could also be used for this.

Good idea. I was about to suggest that we also move tests/wpt/harness (wptrunner) out of the tree (and make it a pip dependency), but it looks like you just made some edits to our copy in-tree :P

The upstream for that is https://github.com/w3c/wptrunner , and changes made in our copy are supposed to be upstreamed. I don’t know why it’s not a submodule or installed from PyPI. But that’s kinda off topic, feel free to open another issue.

I was thinking of tests/wpt/_virtualenv (which is created when you run ./mach test-wpt), not tests/wpt/harness.

Also if that _virtualenv is going to do more duties (which is fine), it should probably move higher up the tree.

I was thinking of tests/wpt/_virtualenv (which is created when you run ./mach test-wpt), not tests/wpt/harness.

Right, sounds good. The Python code to generate/activate a new virtual environment is not very much, so in case we ever want to separate WPT requirements and tidy requirements in the future (due to incompatibilities or whatever) it should not be that difficult.

I am also not opposed to moving the virtualenv path to a more generic place like python/_virtualenv

And yet again, jack beat me to it

python/_virtualenv seems like a good spot.

mach now uses python/_virtualenv, thanks to the resolution of #6999.

The benefit to publishing the package from where it lives in servo tree is that we wouldn't have to change all the other repos if we eventually split it out; the drawback is that we have to manage yet another set of credentials for pypi and make sure the requisite changes get pushed.

To publish on pypi, someone would have a copy of the .pypirc that holds the pypi account's username and password, then run the commands to register and upload the package. If the "someone" in this case is the buildmaster host, we could automate the process of updating the package.

With that said, I'm fine with either pypi or direct installation. Pypi is more work now, direct installation is a bigger hassle at an indeterminate point in the future, and both require making tidy into a package.

@shinglyu, do you want to move tidy and set it up as a Python package, or shall I?

@edunham : I can do that. :)

My colleague @askeing is very interested in this issue, so I'll leave this to him.

Hi @edunham ,
I try to set up it as the Python package (with tests) here: https://github.com/askeing/servo_tidy

@askeing Thank you! It looks like you've done most of the hard work already. My only nitpick is that it'd be helpful to have setup() in setup.py include something like

entry_points={
        'console_scripts': [
            'servo-tidy=servo_tidy:scan',
        ],
    },

so that we can just invoke servo-tidy directly in the script section of another project's .travis.yml once the package has been installed.

@frewsxcv @larsbergstrom @metajack What are your thoughts on tidy living in the Servo tree vs in its own repo? How important to the project is it to keep the tidy git history from the current tree? I personally prefer to keep history whenever possible, but that's got more to do with opinion than necessity in this case.

No strong preference from me. Worth mentioning that tidy.py seems to be a starting point for some new contributors, and having that file live in a separate repository might increase the barrier those contributors.

I think the most important concern is that it's possible to modify tidy.py and see how those changes affect ./mach test-tidy with the fewest possible intermediate steps.

Oops, saying "fixes" in that PR was going a bit far. The crates aren't actually using it in their CI yet.

I'm pretty sure this has been down by now, or that other issues superseded this one.