pip 19.1 and develop mode (with pyproject.toml - build-backend present)

While I'm not trying to apportion blame here, it should be noted that a much simpler fix would be to simply delete pyproject.toml. The reason this isn't possible appears to be because projects like black chose to use that file for their general configuration, which is arguably allowed by PEP 518 (it depends on how you define "build tool", basically) but might be considered a bit premature, when the intended use of that file (PEPs 517 and 518 were still in development).

We need to find a way forward, and not look back at how we got here, but it might be worth someone looking at the possibility of black adding support for configuraion in setup.cfg. I don't know if other tools have pyproject.toml-only configuration - if the problem is significantly more widespread than just black, then this may not be a practical solution.

@pfmoore it's not important how we got here or who is to blame.

The workflow that this breaks is the following: you want to distribute your code as PEP-517/518. And you're testing with pure PEP-517/518 in your CI/tox. However, while developing the codebase it's helpful to install the code in developer mode (to avoid needing to reinstall the package after every code-change to run your tests). In such case, you do want to install the code in editable mode (but also to keep the pyprojec.toml). Deleting the file is not possible in this case. Specifying no pep 517 builds also doesn't work.

Ah, OK. There's so many cases being discussed across various issues that I'd missed that this was talking about a different situation¹. Sorry.

That's definitely a fair comment then - although it's probably worth being clear when documenting this that you would be doing your development with what is essentially a completely different build system than the one you're going to deploy with. As an example, console script wrappers are handled in a completely different way depending on whether you use the legacy direct install method or the PEP 517 install via wheel method.

¹ It might be worthwhile for someone to collect together the various use cases we're seeing reported here and document them, as a starting point for the inevitable "how to extend PEP 517 to support editable mode" discussion. If nothing else, this whole debate makes it clear that the need for a standard covering editable mode isn't something we can put off too long.

@pfmoore that's okay though, editable mode always was a close enough approximation; the non-editable test suits are the ones that confirm 100% the code is good. For example, if one would load resources they can behave differently. But this is already the case. As such needing to pass some flags is alright, however, for now, there's no way to do this. Can we get a fix? --really-ignore-pep517-for-develop-mode is fine by me.

Specifying no pep 517 worked for me. For anyone who hit this when using tox, adding a custom install_command line to tox.ini is my current workaround (I only have pyproject.toml for setting the line length for black).

[testenv]
install_command =
    python -m pip install --no-use-pep517 {opts} {packages}

That's a different problem, as you don't have build-backend and requires specified.

@gaborbernat Can you do something like have a simple wrapper script you use when developing locally that moves the file prior to installing in editable mode? Or copying the pyproject.toml file into place when running in CI, etc? I'm not sure the exact situation, but it seems like you'd have a fair amount of flexibility in coming up with a workflow.

Moving around the files is a really bad workaround and would generate a lot more trouble. This now breaks tox for example. tox it's no longer able to create a develop environment.

While moving the pyproject.toml out and back would workaround the issue (in a scary horrible way), it would break parallel mode. Imagine if you want to run 5 tox environments in parallel, 4 of them use PEP 517/518 and one wants an editable install (for example is the only way cython code coverage can be done). Again feels to me like a bad place to be, now one would need some lock to hold why altering the work tree, just to get an editable install through. On this point, I either downgrade to pip < 19.1, or use python setup.py develop... I would really prefer a true PEP-517/PEP-518 disable behind an obscure flag. Wrapper scripts at that point can pass through that, rather than altering the source directory in-place.

Imagine if you want to run 5 tox environments in parallel, 4 of them use PEP 517/518 and one wants an editable install

Okay, but that's a different scenario from what you mentioned above. I was responding to what you wrote. Above you just said you wanted to install in develop mode when developing locally, and run CI/tox in pure PEP-517/518.

Like I said above, I don't know your exact situation. It just feels to me like you would have a lot of options given that you can structure things however you want in your repo and local environment.

Well, I never said that I do install in develop mode when developing locally manually (I use tox for that too) or not in parallel with some other operation on the same repo. pip installs at the moment are parallel safe, your suggestion would break that constant, so it's not a true workaround.

Can we get a fix?

Even assuming someone pulls together a proposal that covers all the various issues that are getting raised, and gets agreement from everyone, it'll still need implementing. We're still only at the stage of everyone contributing their use case, and we haven't got much clarity on what is the best way forward yet. Stage one is probably for someone to step up and manage the various threads and discussions going on here (we really need one proposal to handle all the use cases, not multiple "temporary" solutions).

Also, the next pip release isn't for 3 months, so there's no huge rush. Let's think things through and make sure we have a good solution before rushing into someone writing a PR.

But yes, we can get a fix, in due course.

PS It might actually be possible to get a PEP 517 extension supporting editable mode sorted in the 3 months before pip's next release - let's not waste all of the energy here on temporary solutions if we can harness it to address the underlying problem!

One other question - is this something new in pip 19.1? The PEP 517 support was introduced in 19.0, a few months ago - is there something in 19.1 that makes it worse? If not, then how come this issue has suddenly flared up now? Just coincidence?

Also, the next pip release isn't for 3 months, so there's no huge rush. Let's think things through and make sure we have a good solution before rushing into someone writing a PR.

This breaks a lot of CIs and workflows. Notably mypy, tox, virtualenv (but a lot-lot more within my workplace). I consider this to be a bug-fix request, so waiting for three months feels to me not the right way to go.

pip 19.0.0 allowed develop mode (without satisfying the requires section) when pyproject.toml was present. pip 19.1 via https://github.com/pypa/pip/pull/6370 decided to hard error instead.

So #6370 says to use --no-use-pep517. Are you saying that the PR is broken, and it doesn't (as stated) "Get pip install -e working again for the PEP 517-is-optional case (provided --no-use-pep517 was passed)"?

If the PR is not delivering what it claims to deliver, then let's fix that (and yes, such a fix might warrant a 19.1.1 release), not add yet more options and workarounds. But we'd need instructions on how to reproduce the breakage in that case.

see https://github.com/pypa/pip/pull/6370#issuecomment-486156720

The simplest work-around for people should be to add --no-use-pep517 to any invocation that requires editable mode (which I believe any error message you're seeing should say)

That fails with:

ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml

This seems to imply that PIP refuses to opt-out of PEP 517 if there is a build-backend in the pyproject.toml even if the environment allows to just run setup.py directly.

@pfmoore This issue is about the first of the two PR's: https://github.com/pypa/pip/pull/6331 rather than the second.

The first is the case where PEP 517 mandates that pyproject.toml-style be used and so --no-use-pep517 isn't permitted.

I think PEP-517/518 does not cover/handle editable install (on purpose, TBD) so there's nothing mandating things actually once someone asks editable install. But at the very least we should be able to disable entirely PEP-517 in case of editable installs.

@pfmoore Here are the two main comments from you where we talked about this before the two PR's were implemented:

• https://github.com/pypa/pip/issues/6314#issuecomment-469176276

• https://github.com/pypa/pip/pull/6331#issuecomment-473231822

@cjerdonek Thanks. This still seems correct - PEP 517 doesn't support editable mode, so projects shouldn't use it if they want to support editable mode.

This does emphasise the need for someone to reopen the standardisation process for editable mode under PEP 517. And when it's been agreed, pip will (of course) follow it.

The decision for pip to switch to PEP 517 mode when a PEP 517-style source layout was detected was made right back in the beginning of the implementation. I pushed for it (I believed then, and still do, that it's important to exercise the new code path where possible, and writing your source code to use the new features seems like a pretty strong opt-in), but it was generally agreed to be the correct choice. And I still believe it is - if we'd held back on using the new code path, how would we have found out that the lack of editable mode was such a significant issue? We'd just have been delaying the point when this issue arose.

Moving forward, though, we do need to work out what to do next.

@pfmoore in the meantime we agree on editable installs do we still plan to offer the legacy editable install mode (behind some weird flag) or pip 19.0.0 has dropped editable install support for now, and worked only by chance pre 19.1?

I don't agree this is a good approach. Basically, you're saying whoever wants to use editable install for development needs to drop pyproject.toml PEP-517/518 for deployment too.

@gaborbernat When asking these questions, for clarity you should distinguish between the two cases of whether "build-backend" is present in pyproject.toml or not because the answer is different in the two cases. If you don't distinguish, it can create confusion because people won't know which case is being referred to.

Changed name of the issue to reflect scope.

I have proposed discussing this at the Packaging Mini-summit, to the extent that "likes" are considered votes, liking it will help get it prioritized.

Either way, maybe we can get the new standard jump-started at PyCon next week? I'm thinking an open space to flesh out ideas among people interested would be good, if there's time.

Now that the issue name sets the scope, I'll re-ask:

In the meantime we agree on editable installs do we still plan to offer the legacy editable install mode (behind some weird flag) or pip 19.0.0 has dropped editable install support for now, and worked only by chance pre 19.1?

I don't agree this is a good approach. Basically, you're saying whoever wants to use editable install for development needs to drop pyproject.toml PEP-517/518 for deployment too.

@pfmoore in the meantime we agree on editable installs do we still plan to offer the legacy editable install mode (behind some weird flag) or pip 19.0.0 has dropped editable install support for now, and worked only by chance pre 19.1?

Huh? I'm not sure what you're asking here. And I'm not sure who "we" is in this context. But my personal view is:

1. Editable installs are still supported exactly as prior to pip 19 - for non-PEP 517 projects (those that don't have build-backend in pyproject.toml). You may need --no-use-pep517.
2. PEP 517 has never supported editable installs, although pip 19.0 did have some cases where it fell into a mode that mixed PEP 517 and legacy processing. Those cases had problems in 19.0, we fixed them in line with the "PEP 517 does not support editable installs" principle, by being clearer that editable mode wasn't supported by PEP 517.

Apparently (remember, this is just my opinion here!) there are a lot of people who opted into PEP 517 without worrying too much about the "doesn't support editable installs" issue, who ended up relying on pip's sort-of support. They are now struggling because of (2) above, and it would be good if pip had an answer for them. I'd prefer that answer to be standards-based, but I appreciate that's not a short-term fix. If someone offers a short-term fix (a PR) I'll take a view on it at that stage, but I'm not at all keen on any sort of --make-it-go-away style of option - I'd rather whatever gets done fits better into what pip already has.

Full disclosure: I won't personally have any free time to develop a solution for this in the short term (next few weeks). And even if I did, I'd rather review solutions from someone with practical experience of the problem than try to work out what's best myself.

PS One possible option (@cjerdonek - I don't know how practical this would be, maybe you can comment?) would be to alter the places where we raise make_editable_error to instead simply report the message as a (strong) warning, saying that using editable with PEP 517 projects is undefined, and you use it at your own risk - but then carry on without raising. If that falls through back to the old behaviour, the user has been warned but we don't stop their build. I would expect that if we do this, we don't offer support for users experiencing weird behaviour if this message was displayed... (I also wouldn't bother adding tests for this behaviour, or anything like that - make it truly unsupported).

I would be just fine with a big red warning written on stderr - PEP-517 detected but editable install requested, this combination is not supported, will fallback to pre PEP-518/7 mode; @cjerdonek ?

PS. We as in PyPa, maintainers of the PyPa packages.

Note that I'm not committing to "fallback to pre PEP-517/8 mode" - the reality is that I think it's more "fall back to some weird frankenstein mode that's mostly legacy with some PEP 518 in there" :-) But the point is that I don't actually know what it'd do.

Not sure how you'd get an opinion from "PyPA" as a whole, the only formal method for doing that is standards. I can only offer my personal opinion, and most of the PyPA people don't follow the pip tracker. We're not even close to seeing a consensus opinion from the pip maintainers yet...

@pfmoore wouldn't it fallback to same as without the pyproject.toml, meaning the same state it did pre PEP-517/PEP-518 - probably meaning just invoking python setup.py develop until we get a standardized develop mode.

Note that I'm not committing to "fallback to pre PEP-517/8 mode" - the reality is that I think it's more "fall back to some weird frankenstein mode that's mostly legacy with some PEP 518 in there" :-) But the point is that I don't actually know what it'd do.

I realize that last time we had a lot of trouble getting the pip ~=18 PEP 518 behavior after the PEP 517 changes went in, but I think the ideal situation would be to fall back to pip's PEP 518 behavior (i.e. building the isolated build environment), then invoke setup.py develop.

Hi all,
spent the entire morning trying to figure out why my packages stopped building today. This is my configuration at work:

pipenv 2018.11.26 create a new virtualenv which (since today) has pip 19.1 installed by default. There is a pyproject.toml file in the project root dir with the following section:

[build-system]
requires = [
    "setuptools",
    "wheel",
]

which means there is a "build-backend" defined, I suppose. The package is installed in edit-mode for convenience reasons (tests, and so on). pyproject.toml also contains settings for black and tidypy. @pfmoore can't remember where you mentioned that just black is using pyproject.toml for its settings, but apparently also tidypy is doing that.

Since I use pipenv (not pip directly) I was not able to make --no-use-pep517 work (because version 2018.11.26 doesn't support that). The only way I was able to get my environment back to work was to export PIP_USE_PEP517=false. If there is a better way to get it done (at least for the time being please feel free to chime in).

Earlier this morning (before this isssue was opened) I've also tried to remove the [build-system] section from the pyproject.toml file without any success, so I'm not sure if the distinction between with/without "build-backend" is real or it is just me getting something wrong.

In my projects, I've been specifying PEP517/518 as a matter of course in order to (a) prove the behavior, (b) opt-in to best practices, and (c) evangelize the functionality.

In many of the projects, the 'editable' workflow is the one that works most consistently (avoiding issues relating to consistency and duplication inherent in the copy that a built install necessarily creates).

If the recommendation is that PEP517 is essentially non-viable for projects that need editable installs, that's going to mean a major transition for hundreds, maybe thousands of projects.

Options our dev team are considering:

• Block pip 19.1 and later at the package index
• Stop using editable projects and deal with the fallout
• Hack tox to pin to older pip

I'd much prefer an approach like --really-ignore-pep517-for-develop-mode that Bernát proposed.

Whatever workaround we come up here tox will swiftly adopt it, as long it makes developer mode work without removing the pyproject.toml. And yes part of that very well can be print out a big red message that FYI you're in unsupported land.

@pfmoore wouldn't it fallback to same as without the pyproject.toml, meaning the same state it did pre PEP-517/PEP-518 - probably meaning just invoking python setup.py develop until we get a standardized develop mode.

@pfmoore wouldn't it fallback to same as without the pyproject.toml, meaning the same state it did pre PEP-517/PEP-518 - probably meaning just invoking python setup.py develop until we get a standardized develop mode.

I don't know, is the honest answer. The code is very different now than it was before PEP 517 support was added.

Let's find it out and create a solution is my proposal. Can we agree on that while we do get PEP-517 editable mode in place doing in the meantime:

I think the ideal situation would be to fall back to pip's PEP 518 behavior (i.e. building the isolated build environment), then invoke setup.py develop.

is the right way to be; with a big red message saying this behaviour will change as soon as we get proper PEP-517 editable mode done. If so I can try creating a PR doing this, and release it as 19.1.1.

This would allow people to follow PEP-517/8 best practices for distribution but fallback to something sane for development (which will change at some point).

@gaborbernat I believe @ncoghlan took a crack at restoring the PEP 518 behavior when we were first scrambling to deal with the relative imports issue and eventually abandoned it. He may have thoughts as to how feasible this is.

Given that we've really wanted a "fall back to PEP 518 only" option twice now, it seems to me that it's worth putting in the effort.

Edit To clarify: I do not mean "fall back to PEP 518" as a pip option, just as a code path. My understanding is that when PEP 517 support was added, it was added in place of the PEP 518 support, so the two are inextricably linked. Having a code path that pip can take that falls back to PEP 518 in certain scenarios is something we've needed twice now to implement "quick fix" patches.

I would not make it myself disable-able PEP-517 as such. But instead, disable only in case of editable install.

_(this is probably a topic for a separate discussion but I'll state it here)_

It looks like major changes keep unexpectedly break dists in the wild.
One solution to this would be introducing testing of the downstream libs: these could be libs with known "weird configs" and/or random things on PyPI. At least running different cases against those once a week should raise a red flag faster...
(This doesn't negate the need for those downstreams to run tests against pip's master branch on their side as well)

PS One possible option (@cjerdonek - I don't know how practical this would be, maybe you can comment?) would be to alter the places where we raise make_editable_error to instead simply report the message as a (strong) warning,

This would be straightforward (essentially just changing the raise to logging a warning in the appropriate cases, and updating the test cases). The relevant code starts here:
https://github.com/pypa/pip/blob/9816d17ccaf33dc5450b74ad310b6827915e2fe7/src/pip/_internal/pyproject.py#L136
Someone just needs to decide what the behavior should be. Personally, I'm okay with whatever @pfmoore decides and will defer to him.

I think @pfmoore's suggestion is quite good for immediate firefighting...

I'll see if I can work something up within the next couple days and then we can review what the warning text should be, etc.

Hey @cjerdonek - each day that passes with this release means more projects that require a patched release process. I'd gladly put a $50 bounty on a bugfix release if it can be made in the next 36 hours.

@pfmoore To clarify, was your proposal in this comment that if editable mode is requested (for both the "build-backend" and without-"build-backend" cases), then it automatically opts out of PEP 517 processing with appropriate warnings? Or would --no-use-pep517 still be required in both cases? My understanding of your suggestion (whether or not --no-use-pep517 is required) is that there should be a warning in both cases that the legacy behavior might not work the same, but the wording would be stronger in the case that "build-backend" is present (saying the behavior is currently undefined in the spec).

On the above, it would seem weird to me if we required --no-use-pep517 only when "build-backend" isn't present. That would mean that adding "build-backend" when editable mode is being used would have the same effect as passing --no-use-pep517 (only the warning text would be different).

I don't have a suggestion as such - it just occurred to me that replacing "raise" with "print the message and continue" might do more or less what the code used to do (the PR that added the raise pretty much just did "if then raise").

But I've no feel at all for what would be a "good" option here. If the situation is serious enough, reverting #6314 and #6331 might be warranted. If not, then someone needs to decide how to preserve the spirit of those two patches, while fixing the current breakage. But beyond not being happy with a "switch off PEP 517" blunt instrument option, I haven't had time to think the issue through at all.

My thoughts are that I don't think we should revert because it would only push the problems further down the road and in the process make them even bigger and harder to deal with later (and maybe even intractably so). The purpose of the change was in part to deal with this issue while we still had more of a chance.

Regarding the path forward, my feeling is that --no-use-pep517 should be needed in either both cases or neither (with or without the "build-backend" key). That choice results in a tradeoff when it comes to future changes in behavior. If we don't require the flag now, people will complain when PEP 517 adds support for editable and things behave differently, and maybe limit us in terms of what choices we can make when considering editable for PEP 517. If we do require the flag now, the problem is that people may complain that it's inconvenient to have to use the option and maybe it doesn't work correctly in all cases (e.g. the issue with nested requirements file).

I agree with your analysis. My preference would be to require --no-use-pep517 to make it explicit. If I understand the comments above, that essentially makes --no-use-pep517 act similar to the --make-it-just-work flag people were asking for, so I think that should be an acceptable inconvenience.

I just posted an initial PR (https://github.com/pypa/pip/pull/6442) for review that implements what @pfmoore and I described above.

To everyone looking for a workaround - you could consider:

1. Using pip 19.0;
2. export PIP_USE_PEP517=false (credit: @ariciputi );
3. if pyproject.toml is there only because of black, you can rename it and use black --config black_config.toml .;
4. not using editable mode;

Augmenting @dxxxdk's list, here's a quick workaround for tox:

1. https://github.com/ansible/molecule/pull/2009 / https://github.com/jaraco/skeleton/commit/123b0b2 (credit: @jaraco);

Thanks! Now we just need a release, @dstufft

which means there is a "build-backend" defined, I suppose.

@ariciputi The pyproject.toml contents you showed in your comment here, only has a "requires" key in the "build-system" section and not a "build-backend" key. When the message mentions the "build-backend" key, it means that literal string would need to occur in the toml file.

@pfmoore @cjerdonek I was hoping to look at the original issues a bit more closely before weighing in, but I'm not sure if I'm going to have time to do that at the moment, so I'm just going to shoot from the hip here. I do think that there may be some value in trying to keep pip -e working at least as well as it was working, without forcing people to opt out of PEP 517.

At the end of the day, I'm pretty sure that we will need editable installs or something else in order to get PEP 517 adoption, so migrating people away from editable installs temporarily while we come up with a standard seems counter-productive. I think that the fallback to setup.py develop should not require --no-use-pep517 in either case, just issue the warning automatically.

Somewhat late to the discussion here, but most of my projects fall exactly under the category that @gaborbernat describes--I have -e . in my requirements-dev.txt, and when setting up a new dev environment for them I have just done $ pip install -r requirements-dev.txt.

I don't know how general the python setup.py develop solution is, but in the one project I've tried it on thus far it worked fine. (I'm using the src config, and I in particular need the project working directory to be installed in the environment so that the test suite can import the package.)

If setup.py develop could be a general stopgap solution for most of these problem cases (no idea whether it would or not), it might suffice just to augment the "editable installs not supported with PEP517" error message, with something like:

.... Until editable install support is defined and implemented for PEP517 setuptools projects, running 'python setup.py develop' may be a suitable workaround.

FWIW, I just tested python setup.py develop as an alternative to -e . in my requirements files, and my test suite passes locally, my CI passed just fine, and tox (v3.9, up to date) was perfectly content:

This may not be a feasible workaround for all the breakages, but it looks like it should work fine for my development workflow -- and even if projects have to refactor to a src layout, it's not that onerous a transition. All of the key pieces of configuration info are laid out in my blog post on the topic, which I will be updating shortly!

If it helps, I just posted PR #6447 to fix #6438 and permit using --no-use-pep517 (and --use-pep517) in requirements files.

@pganssle IMO, it emphasises the need for us to start working on a PEP 517 extension to support editable mode as a matter of some priority.

We definitely need to do it, but no need to break current existing workflows to bump priority. Anyways let's talk about it at the packaging summit.

I'm a little late on this train but I'd say that ideally specifying -e on a requirement should currently imply --no-use-pep517 only for this requirement.

@pfmoore Yeah, I agree that we need to start working on it, I actually had proposed discussing it at the Packaging mini-summit before any of this happened, and breaking a ton of workflows has only exacerbated the need to get this done.

But in the meantime, I think we should go back to the (substantive) behavior of pip 19.0.3. This is not one of those situations where there's pain that has to happen and we might as well bite the bullet now before it gets worse. We're going to make editable installs possible, so making people explicitly disable PEP 517 just to have them turn it back on later is just creating a lot of busywork.

This is probably naive, and may already have been covered in prior discussion on editable installs, but: could enabling editable installs be as simple ("simple") as adding an editable_install field to [build-system] in pyproject.toml? It would hold the (series of) command(s) to execute in order to achieve the editable install; and, just like the defaults for the other [build-system] fields assume setuptools as the build backend, the default would be editable_install=python setup.py develop?

In theory, the build system and the (editable) install system should be fully decoupled, right? If so, the build system should only really need to know what command to call in the/an install system to achieve the editable install, if one is specified to it.

This probably would require revisions to PEP517/518 first, though...?

@bskinn Editable installs are a backend feature, so it doesn't make sense to require every project using a particular backend to copy a particular incantation into their pyproject.toml file - instead their needs to be a way for frontends to ask backends to create a local editable install.

We've shifted discussion to https://discuss.python.org/t/pip-19-1-and-installing-in-editable-mode-with-pyproject-toml/1553.

Hi everyone,

I'm interested in this thread because unfortunately this also broke pytest. I appreciate everyone participating and that a solution was found so fast.

While reading through it though I am under the impression that editable install workflows are somewhat uncommon for some people, while I thought the exact opposite, that they were the standard way of working.

How projects with a pyproject.toml (with and/or a build-backend section) develop using virtual environments (other than setting PYTHONPATH of course)?

Personally, just by using tox to install the project into a virtualenv for testing. And for adhoc testing I do a build and install cycle. It's not that onerous, and I feel more comfortable with that than with editable installs (and the path hacking that they employ). That's for all projects, not just ones that use PEP 517.

That's purely my personal view and workflow, of course.

Just noticed that pip install -e failed with pip itself when using 19.1, but works now after installing it from master non-editable first.
I also typically use editable installs, because it easily allows to edit the version-controlled files for patches.

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.