Pipenv: Dependencies could not be resolved. Regression?

It looks like pipenv adds a strict pin to the current round candidates.

This means that if in a next round, another packages requires a version not matching the current candidate, it will fail (like this) instead of trying to find another candidate. That's how the resolving algorithm works: Full-re-compute with the new constraints until it stabilises.

I'm not sure of what is causing this.
I noticed that to reproduce this, I have to clear the cache (--clear) if I used pip-tools alone before.
I can tell this is not a pip-tools bug, and its not directly related to the patches of pip-tools (I removed them all to try, and this still occurs).
I hopes this helps.

Allright, found the culprit:
https://github.com/kennethreitz/pipenv/blob/master/pipenv/patched/pip/req/req_set.py#L752

self.requirements.values() contains the req_to_install itself, pinned, which is wrong to return as a dependency, and can make the dependency resolution fail, like here, if a subsequent package requires a version that does not fit the current candidate. Doing this assumes that our first candidate must be the right one, or fail..

@kennethreitz From what I understand, that was to do "deep extras resolving". I'm not sure I understand what's that supposed to mean exactly. I'm ready to keep stabbing at this, but I'd need some info to not go in the wrong direction.

I believe I am seeing the same issue:

$ pipenv --version
pipenv, version 8.2.7

$ pipenv install
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches django==2.0a1,>=1.8,>=1.8.0
Tried: 1.1.3, 1.1.4, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.2.6, 1.2.7, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.4.10, 1.4.11, 1.4.12, 1.4.13, 1.4.14, 1.4.15, 1.4.16, 1.4.17, 1.4.18, 1.4.19, 1.4.20, 1.4.21, 1.4.22, 1.5, 1.5.1, 1.5.2, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.8, 1.5.9, 1.5.10, 1.5.11, 1.5.12, 1.5.12, 1.6, 1.6, 1.6.1, 1.6.1, 1.6.2, 1.6.2, 1.6.3, 1.6.3, 1.6.4, 1.6.4, 1.6.5, 1.6.5, 1.6.6, 1.6.6, 1.6.7, 1.6.7, 1.6.8, 1.6.8, 1.6.9, 1.6.9, 1.6.10, 1.6.10, 1.6.11, 1.6.11, 1.7, 1.7, 1.7.1, 1.7.1, 1.7.2, 1.7.2, 1.7.3, 1.7.3, 1.7.4, 1.7.4, 1.7.5, 1.7.5, 1.7.6, 1.7.6, 1.7.7, 1.7.7, 1.7.8, 1.7.8, 1.7.9, 1.7.9, 1.7.10, 1.7.10, 1.7.11, 1.7.11, 1.8a1, 1.8b1, 1.8b2, 1.8rc1, 1.8, 1.8, 1.8.1, 1.8.1, 1.8.2, 1.8.2, 1.8.3, 1.8.3, 1.8.4, 1.8.4, 1.8.5, 1.8.5, 1.8.6, 1.8.6, 1.8.7, 1.8.7, 1.8.8, 1.8.8, 1.8.9, 1.8.9, 1.8.10, 1.8.10, 1.8.11, 1.8.11, 1.8.12, 1.8.12, 1.8.13, 1.8.13, 1.8.14, 1.8.14, 1.8.15, 1.8.15, 1.8.16, 1.8.16, 1.8.17, 1.8.17, 1.8.18, 1.8.18, 1.9a1, 1.9b1, 1.9rc1, 1.9rc2, 1.9, 1.9, 1.9.1, 1.9.1, 1.9.2, 1.9.2, 1.9.3, 1.9.3, 1.9.4, 1.9.4, 1.9.5, 1.9.5, 1.9.6, 1.9.6, 1.9.7, 1.9.7, 1.9.8, 1.9.8, 1.9.9, 1.9.9, 1.9.10, 1.9.10, 1.9.11, 1.9.11, 1.9.12, 1.9.12, 1.9.13, 1.9.13, 1.10a1, 1.10a1, 1.10b1, 1.10b1, 1.10rc1, 1.10rc1, 1.10, 1.10, 1.10.1, 1.10.1, 1.10.2, 1.10.2, 1.10.3, 1.10.3, 1.10.4, 1.10.4, 1.10.5, 1.10.5, 1.10.6, 1.10.6, 1.10.7, 1.10.7, 1.10.8, 1.10.8, 1.11a1, 1.11b1, 1.11rc1, 1.11rc1, 1.11, 1.11, 1.11.1, 1.11.1, 1.11.2, 1.11.2, 1.11.3, 1.11.3, 1.11.4, 1.11.4, 1.11.5, 1.11.5, 1.11.6, 1.11.6, 2.0a1

where 2.0a1 is clearly one of the available options.

I am unable to test dependency resolution in any older versions of pipenv due to https://github.com/kennethreitz/pipenv/issues/786. In my opinion, critical bug fixes should be applied to at least a few of the most recent major versions.

Same issue - it's complaining about requests specifically.

This is a bug I believe as related to @vphilippon's comment. I do not think it is related to #909 however.

Exact, not related to #909.
Like I said, I'll need the input from one of the @maintainers to understand what "deep extras resolving" exactly means here. I have a fix, but I'd like to know more to not break anything first.

Or I could go ahead and open a PR with the fix and discuss there. I'll see with my free time.

@vphilippon Since this isn't in an organization but rather a personal repo for Kenneth, we unfortunately don't have a handy @ tag to get all of us. I'd suggest just pinging @erinxocon and myself.

The patch to pip here is unfortunately locked away somewhere in Kenneth's head. I don't think I understand, or have had time to look at, what's going on here. If you find some time to compile what you think may be going on, I can try to help verify assumptions. Until then though, this is something we'll have to tread lightly around before making changes.

@nateprewitt Allright, I'll add as much as I can. Grab a chair.

What's wrong?

In https://github.com/kennethreitz/pipenv/issues/875#issuecomment-335570812, I noted the source of the error: selected candidates in a resolving round are added as pinned dependency of the candidate itself directly, which is wrong (more on that coming up)

In https://github.com/kennethreitz/pipenv/issues/875#issuecomment-336609268, I found the source of that pin: the self.requirements.values() which was added to the return value includes a pin to the current package. In other word, _prepare_file should be returning the dependencies of a package (or "A list of additional InstallRequirements to also install"), but now it includes itself, pinned, as its own dependency.

To help demonstrate why having the package itself, pinned, as its own dependency, is really wrong, here's how the dependency resolving algorithm works, in short:

1. Get the initial_constraints_set, a list of InstallRequirements (ex: requests>=2.18).
2. Define additional_constraints and candidate as empty set.
3. Select a set of candidate (ex: requests==2.18.4) that respect the union of initial_constraints_set and additional_constraints
4. For each candidate, get and combine the set of dependencies of all candidate, as additional_constraints (ex: certifi>=2017.4.17)
5. If additional_constraints has changed, clear candidate and go back to 3.
6. Return the set of candidate, done.

Now, lets reproduce the issue With the above in mind.

To reproduce the issue, you need this Pipfile (because `django-cms 3.4.5 with Django 1.11 support was released):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
django-cms = "==3.4.4"
django = "*"

And now pipenv lock --verbose --clear:

Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project.
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple

                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done
Locking [packages] dependencies…
Using pip: -i https://pypi.python.org/simple

                          ROUND 1
Current constraints:
  django
  django-cms==3.4.4

Finding the best candidates:
  found candidate django==1.11.6 (constraint was <any>)
  found candidate django-cms==3.4.4 (constraint was ==3.4.4)

Finding secondary dependencies:
  django-cms==3.4.4 not in cache, need to check index
  django-cms==3.4.4         requires django-classy-tags>=0.7.2, django-cms==3.4.4, django-formtools>=1.0, django-sekizai>=0.7, django-treebeard>=4.0.1, Django<1.11,>=1.8, djangocms-admin-style>=1.0
  django==1.11.6 not in cache, need to check index
  django==1.11.6            requires django==1.11.6, pytz

New dependencies found in this round:
  adding [u'django', '<1.11,==1.11.6,>=1.8', '[]']
  adding [u'django-classy-tags', '>=0.7.2', '[]']
  adding [u'django-cms', '==3.4.4', '[]']
  adding [u'django-formtools', '>=1.0', '[]']
  adding [u'django-sekizai', '>=0.7', '[]']
  adding [u'django-treebeard', '>=4.0.1', '[]']
  adding [u'djangocms-admin-style', '>=1.0', '[]']
  adding [u'pytz', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  django<1.11,==1.11.6,>=1.8
  django-classy-tags>=0.7.2
  django-cms==3.4.4
  django-formtools>=1.0
  django-sekizai>=0.7
  django-treebeard>=4.0.1
  djangocms-admin-style>=1.0
  pytz

Finding the best candidates:
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches django<1.11,==1.11.6,>=1.8
Tried: 1.1.3, 1.1.4, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.2.6, 1.2.7, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.4.10, 1.4.11, 1.4.12, 1.4.13, 1.4.14, 1.4.15, 1.4.16, 1.4.17, 1.4.18, 1.4.19, 1.4.20, 1.4.21, 1.4.22, 1.5, 1.5.1, 1.5.2, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.8, 1.5.9, 1.5.10, 1.5.11, 1.5.12, 1.5.12, 1.6, 1.6, 1.6.1, 1.6.1, 1.6.2, 1.6.2, 1.6.3, 1.6.3, 1.6.4, 1.6.4, 1.6.5, 1.6.5, 1.6.6, 1.6.6, 1.6.7, 1.6.7, 1.6.8, 1.6.8, 1.6.9, 1.6.9, 1.6.10, 1.6.10, 1.6.11, 1.6.11, 1.7, 1.7, 1.7.1, 1.7.1, 1.7.2, 1.7.2, 1.7.3, 1.7.3, 1.7.4, 1.7.4, 1.7.5, 1.7.5, 1.7.6, 1.7.6, 1.7.7, 1.7.7, 1.7.8, 1.7.8, 1.7.9, 1.7.9, 1.7.10, 1.7.10, 1.7.11, 1.7.11, 1.8a1, 1.8b1, 1.8b2, 1.8rc1, 1.8, 1.8, 1.8.1, 1.8.1, 1.8.2, 1.8.2, 1.8.3, 1.8.3, 1.8.4, 1.8.4, 1.8.5, 1.8.5, 1.8.6, 1.8.6, 1.8.7, 1.8.7, 1.8.8, 1.8.8, 1.8.9, 1.8.9, 1.8.10, 1.8.10, 1.8.11, 1.8.11, 1.8.12, 1.8.12, 1.8.13, 1.8.13, 1.8.14, 1.8.14, 1.8.15, 1.8.15, 1.8.16, 1.8.16, 1.8.17, 1.8.17, 1.8.18, 1.8.18, 1.9a1, 1.9b1, 1.9rc1, 1.9rc2, 1.9, 1.9, 1.9.1, 1.9.1, 1.9.2, 1.9.2, 1.9.3, 1.9.3, 1.9.4, 1.9.4, 1.9.5, 1.9.5, 1.9.6, 1.9.6, 1.9.7, 1.9.7, 1.9.8, 1.9.8, 1.9.9, 1.9.9, 1.9.10, 1.9.10, 1.9.11, 1.9.11, 1.9.12, 1.9.12, 1.9.13, 1.9.13, 1.10a1, 1.10a1, 1.10b1, 1.10b1, 1.10rc1, 1.10rc1, 1.10, 1.10, 1.10.1, 1.10.1, 1.10.2, 1.10.2, 1.10.3, 1.10.3, 1.10.4, 1.10.4, 1.10.5, 1.10.5, 1.10.6, 1.10.6, 1.10.7, 1.10.7, 1.10.8, 1.10.8, 1.11a1, 1.11b1, 1.11rc1, 1.11rc1, 1.11, 1.11, 1.11.1, 1.11.1, 1.11.2, 1.11.2, 1.11.3, 1.11.3, 1.11.4, 1.11.4, 1.11.5, 1.11.5, 1.11.6, 1.11.6

• In a single resolving round, we detect that we have to install django==* and django-cms==3.4.4.
• We select candidate: django==1.11.6, django-cms==3.4.4.
• We get the dependencies (I'll focus on the important part):
  
  
  
  • django-cms==3.4.4 requires django<1.11,>=1.8
  
  
  • django requires django==1.11.6 (This is not true, django does not require itself! No package requires itself!)
  
  
• We merge the dependencies specifier together:
  
  
  
  • We now require django <1.11,==1.11.6,>=1.8 (You can see where this is going...)
  
  
• Constraints have changed, back to selecting candidate.
• Try to find a candidate for django that is <1.11,==1.11.6,>=1.8:
  
  
  
  • Could not find a version that matches django<1.11,==1.11.6,>=1.8 [...]
  
  

What should have happened (like it does in pip-tools, with an unpatched pip, with --rebuild to clear the cache):

• In a single resolving round, we detect that we have to install django==* and django-cms==3.4.4.
• We select candidate: django==1.11.6, django-cms==3.4.4.
• We get the dependencies (I'll focus on the important part):
  
  
  
  • django-cms==3.4.4 requires django<1.11,>=1.8
  
  
  • django requires pytz, but not django==1.11.6
  
  
• We merge the dependencies specifier together:
  
  
  
  • We now require django <1.11,>=1.8 (Much better...)
  
  
• Constraints have changed, back to selecting candidate.
• Try to find a candidate for django that is <1.11,>=1.8:
  
  
  
  • Selects a new django candidate, likely django==1.10.8
  
  
• Computes dependencies, keeps going and resolves happily.

What to do from here?

Well, removing the self.requirements.values() from the return in _prepare_file fixes this issue, I can confirm it. Unfortunately, I haven't figured exactly why it was added there.

Maybe Kenneth wanted to return an InstallRequirement object for the candidate itself after performing some operation on it in order to obtain information related to "deep extras resolving" (I'm really going on a stretch here). I could try to patch the patch to keep that additional InstallRequirement object in the returned value, but unpinning it first. I'm pretty sure that wasn't intentional, and even if it was, its broken.

I think that's it. ☕️

@vphilippon, thank you so much for putting such a detailed breakdown! This definitely clears things up for me.

Digging back through the commit log, this was added in ae4591b2 and as you've said it's pretty unclear why this was added. I think the next step is to put together a failing test case for this, and a working test case for installing "deep dependencies". There aren't any tickets bringing up this issue and the commit messages aren't helpful so we're just kind of guessing at this point.

Looking at the code, I think this is trying to solve having a declaration like requests[security] which may have a dependency certifi[some-extra] which also needs to be resolved. That's the only thing that I can think of at least. I'm betting there's something in the Django universe that's doing this, and if we can find an example that'll let us write a test.

So! If anyone would like to work on putting these tests together, ensuring the first fails and the "deep dependency" works, we can look at removing the self.requirements.values() declaration.

@nateprewitt Here's something that I think describe what you meant:
https://github.com/vphilippon/testdeepextra

But, the "foo[a] depends on bar[b]" deep resolution seems to be already ok in pip-tools, as it can be tested with that repo.
So that patch might not be exactly that, as I would assume its for something that was not originally working in pip-tools. Unless its for some specific case I was not able to test here.

Anyway, we can still test the extra's resolving behavior of pipenv with that repo (which I don't have the time to do right away), with and without that patch, and see if it has any effect, and/or compare with the output of pip-tools. At least we have a starting point.

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches sanic-plugins-framework==0.5.0.dev20171225,>=0.5.0.dev20171225
Tried: 0.2.0.dev20171102, 0.2.0.dev20171102, 0.3.0.dev20171102, 0.3.0.dev20171102, 0.3.1.dev20171102, 0.3.1.dev20171102, 0.3.2.dev20171102, 0.3.2.dev20171102, 0.3.3.dev20171102, 0.3.3.dev20171102, 0.4.0.dev20171103, 0.4.0.dev20171103, 0.4.1.dev20171103, 0.4.1.dev20171103, 0.4.2.dev20171106, 0.4.2.dev20171106, 0.4.4.dev20171107, 0.4.4.dev20171107, 0.4.5.dev20171113, 0.4.5.dev20171113, 0.5.0.dev20171225, 0.5.0.dev20171225, 0.5.2.dev20180201, 0.5.2.dev20180201

1. I got this error when the packages's version equal to '*'，but when I change it to the known version, such as '==0.1.2', it works well.
2. Unfortunately, when I type pipenv install sanic-plugins-framework==0.5.0.dev20171225, I got back this big error message upon. Maybe all of the version above is not greater then '0.5.0.dev20171225'
3. How should I do to cover this

pipenv lock —pre —clear

Thank you @Jasonsey for sharing your bit and to @techalchemy for your answer.
I had a problem specifically with sanic-plugins-framework, and adding the --pre (_Allow pre-releases._ flag) helped me install the package 🙌

Was this ever fixed? I'm still getting the same erroneous behavior.
https://github.com/pypa/pipfile/issues/114