Virtualenv: Problem with non ASCII car in path

Hello,
Got the exact same issue here with :
python v2.7.5
Win7 64bit
Can be reproduced at will

Looks like #453

This issue occurs also on Mac OS X 10.8.4 with Python v2.7.2:

New python executable in venv/bin/python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 16: ordinal not in range(128)
ERROR: The executable venv/bin/python is not functioning
ERROR: It thinks sys.prefix is u'/Users/RSChiang/\u539f\u59cb\u78bc/repo' (should be u'/Users/RSChiang/\u539f\u59cb\u78bc/repo/venv')
ERROR: virtualenv is not compatible with this system or executable

Pull request #187 mentioned something about FS encoding differs from default encoding. Probably system configuration would solve it?

atykhonov's fix works for me.

Bump. Any progress? @atykhonov How about sending your patch as a pull requests?

Ditto. Problem also occurs on Linux, here with Chinese characters:

$ virtualenv /tmp/中文
New python executable in /tmp/中文/bin/python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 5: ordinal not in range(128)
ERROR: The executable /tmp/中文/bin/python is not functioning
ERROR: It thinks sys.prefix is u'/home/vinay/projects/scratch' (should be u'/tmp/\u4e2d\u6587')
ERROR: virtualenv is not compatible with this system or executable
vinay@ubuntu:~/projects/scratch$ virtualenv --version
13.1.2
vinay@ubuntu:~/projects/scratch$ /tmp/中文/bin/python
Python 2.7.10 (default, Oct 14 2015, 16:09:02) 
[GCC 5.2.1 20151010] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

I have a WIP patch for this problem, while things can't be done before the bug in https://bitbucket.org/pypa/distlib/pull-requests/27 is fixed.

@yan12125 - I'm not sure if your PR is correct, see my comments on it. Also, it's not clear that the console dump above relates to the distlib issue - it appears to be a different bug in virtualenv which is invoked when testing that the venv was created correctly.

There are lots of pull requests trying to solve this:

• #900 (#875): opened by me, all tests passed
• #902 (#894): opened by @aodag
• #907 (#453): opened by @magres
• #908 (#492): opened by @atykhonov
• #912 (#630): opened by @dholth

I guess a Windows testing environment like Appveyor is also necessary. pypa/pip is already using it.

FI, I got the same error on Debian GNU/Linux 9.2, Python 2.7.14.

% pwd
/home/tiger-222/Téléchargements

Python 2:

% python2 -m pip install --user --upgrade virtualenv
Collecting virtualenv
  Using cached virtualenv-15.1.0-py2.py3-none-any.whl
Installing collected packages: virtualenv
Successfully installed virtualenv-15.1.0

% python2 -m virtualenv test-python2   
New python executable in /home/tiger-222/Téléchargements/test-python2/bin/python2
Also creating executable in /home/tiger-222/Téléchargements/test-python2/bin/python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 17: ordinal not in range(128)
ERROR: The executable /home/tiger-222/Téléchargements/test-python2/bin/python2 is not functioning
ERROR: It thinks sys.prefix is u'/home/tiger-222/T\xe9l\xe9chargements' (should be u'/home/tiger-222/T\xe9l\xe9chargements/test-python2')
ERROR: virtualenv is not compatible with this system or executable

Python 3:

% python3 -m pip install --user --upgrade virtualenv
Collecting virtualenv
  Using cached virtualenv-15.1.0-py2.py3-none-any.whl
Installing collected packages: virtualenv
Successfully installed virtualenv-15.1.0

% python3 -m virtualenv test-python3
Using base prefix '/usr'
New python executable in /home/tiger-222/Téléchargements/test-python3/bin/python3
Also creating executable in /home/tiger-222/Téléchargements/test-python3/bin/python
Installing setuptools, pip, wheel...done.

For whatever it's worth, I can reproduce this issue on Windows 7, Python 3.6.4, virtualenv 15.1.0, when working under a user account with username starting with $.

PS C:\Users\$scuthbertson\Desktop\bugtest> python -m virtualenv .
Using base prefix 'C:\\Program Files\\Python36'
New python executable in C:\Users\$scuthbertson\Desktop\bugtest\Scripts\python.exe
Installing setuptools, pip, wheel...
  Complete output from command C:\Users\$scuthberts...t\Scripts\python.exe - setuptools pip wheel:
  Collecting setuptools
  Using cached setuptools-39.0.1-py2.py3-none-any.whl
Collecting pip
  Using cached pip-9.0.3-py2.py3-none-any.whl
Collecting wheel
  Using cached wheel-0.30.0-py2.py3-none-any.whl
Installing collected packages: setuptools, pip, wheel
Exception:
Traceback (most recent call last):
  File "C:\Program Files\Python36\Lib\distutils\util.py", line 202, in subst_vars
    return re.sub(r'\$([a-zA-Z_][a-zA-Z_0-9]*)', _subst, s)
  File "C:\Users\$scuthbertson\Desktop\bugtest\lib\re.py", line 191, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "C:\Program Files\Python36\Lib\distutils\util.py", line 199, in _subst
    return os.environ[var_name]
  File "C:\Users\$scuthbertson\Desktop\bugtest\lib\os.py", line 669, in __getitem__
    raise KeyError(key) from None
KeyError: 'scuthbertson'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\site-packages\virtualenv_support\pip-9.0.1-py2.py3-none-any.whl\pip\basecommand.py
", line 215, in main
    status = self.run(options, args)
  File "C:\Program Files\Python36\lib\site-packages\virtualenv_support\pip-9.0.1-py2.py3-none-any.whl\pip\commands\insta
ll.py", line 342, in run
    prefix=options.prefix_path,
  File "C:\Program Files\Python36\lib\site-packages\virtualenv_support\pip-9.0.1-py2.py3-none-any.whl\pip\req\req_set.py
", line 784, in install
    **kwargs
  File "C:\Program Files\Python36\lib\site-packages\virtualenv_support\pip-9.0.1-py2.py3-none-any.whl\pip\req\req_instal
l.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "C:\Program Files\Python36\lib\site-packages\virtualenv_support\pip-9.0.1-py2.py3-none-any.whl\pip\req\req_instal
l.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "C:\Program Files\Python36\lib\site-packages\virtualenv_support\pip-9.0.1-py2.py3-none-any.whl\pip\wheel.py", lin
e 247, in move_wheel_files
    prefix=prefix,
  File "C:\Program Files\Python36\lib\site-packages\virtualenv_support\pip-9.0.1-py2.py3-none-any.whl\pip\locations.py",
 line 153, in distutils_scheme
    i.finalize_options()
  File "C:\Program Files\Python36\Lib\distutils\command\install.py", line 307, in finalize_options
    self.expand_basedirs()
  File "C:\Program Files\Python36\Lib\distutils\command\install.py", line 475, in expand_basedirs
    self._expand_attrs(['install_base', 'install_platbase', 'root'])
  File "C:\Program Files\Python36\Lib\distutils\command\install.py", line 469, in _expand_attrs
    val = subst_vars(val, self.config_vars)
  File "C:\Program Files\Python36\Lib\distutils\util.py", line 204, in subst_vars
    raise ValueError("invalid variable '$%s'" % var)
ValueError: invalid variable '$'scuthbertson''
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Program Files\Python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python36\lib\site-packages\virtualenv.py", line 2328, in <module>
    main()
  File "C:\Program Files\Python36\lib\site-packages\virtualenv.py", line 713, in main
    symlink=options.symlink)
  File "C:\Program Files\Python36\lib\site-packages\virtualenv.py", line 945, in create_environment
    download=download,
  File "C:\Program Files\Python36\lib\site-packages\virtualenv.py", line 901, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "C:\Program Files\Python36\lib\site-packages\virtualenv.py", line 797, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command C:\Users\$scuthberts...t\Scripts\python.exe - setuptools pip wheel failed with error code 2
PS C:\Users\$scuthbertson\Desktop\bugtest>

@philosophicles: that looks like a different issue than Chinese characters or spaces in path. Could you open a new issue?

BTW, venv is also affected, and it would be great if an issue is opened on https://bugs.python.org/ for venv as well.

@philosophicles This may not be a bug - it appears to be happening because distutils is seeing $cuthbertson as a placeholder and trying to replace it with a variable, which isn't found. See this part of the documentation.

I will raise separate issues (in both virtualenv and venv) for my problem presently.

@vsajip - my opinion is that it is definitely a bug. I have looked at the documentation you reference, but that's an API reference for somebody developing using distutils. I'm just trying to run a simple command like > python -m virtualenv . under Windows, and encountering a fatal error that prevents virtualenv from doing what it's meant to do. I shouldn't have to care at all about the internal implementation of virtualenv, or how distutils works, or even what my full directory path is. It's legitimate for Windows directory paths to contain $ so they should, ideally, be supported.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Just add a comment if you want to keep it open. Thank you for your contributions.

@stale closed this .... but ping. This is still a bug... but I am ok if this is not fixed on 2.7.x ... I just need to know

This now has been fixed, as long as the file system codec can handle it, we'll handle it too 👍 if not we'll raise appropriate error message 👍 On windows pre python3.6 this implies mcbs codec 👍 in all later cases we're talking about utf-8.

@gaborbernat Thank you ++

Hi @gaborbernat , is this problem fixed in the latest master branch?
I used the master branch and got errors similar to issue#1375.
if not, I can create a PR to fix this problem.

Not in the master branch, but the rewrite branch that will soon replace master.

OK , thanks for your reply.