Psutil: Cygwin Support

_From [email protected] on March 19, 2010 18:21:45_

I looked briefly at this and it appears to be more complex than I had initially
hoped. Cygwin acts like a POSIX environment, but the system level code still needs to
be Windows-specific. Unfortunately this means that most of the code is similar to the
Windows codebase, but that the Cygwin version of Python doesn't know about things
like PyErr_SetFromWindowsErr which we use all over the place. Conversely, the /proc
system and sysctl() don't appear to be available under Cygwin so we can't use the
code from other unix-like systems. 

It's likely we'd need a separate cygwin C module specific to working with Cygwin,
based on the Windows code but with workarounds for things like
PyErr_SetFromWindowsErr() and any other issues that crop up. Opening this ticket as
an enhancement for a placeholder if/when we have a chance to look at this more in depth.

_From [email protected] on May 12, 2010 09:29:40_

Issue 87 has been merged into this issue.

_From g.rodola on June 19, 2010 11:14:39_

I'm closing this out for now as it's unlikely we'll find time to look into this in the near future.

Status: WontFix

Would you accept a patch to add Cygwin support?

I must admit I've never used it and don't know much about it. What would it take to add support? What benefits would it bring exactly (e.g. why would you want to use psutil from cygwin)?

I must admit I've never used it and don't know much about it.

I know a decent amount about it--have spent some time (more than I would like to admit) in its internals.

What would it take to add support?

From my initial experiments, the easiest way to add support would be to explicitly add it as a platform. It's too different either from Linux, or plain Windows, to try to shoehorn support for it into either the existing Windows or Linux platform-specific modules. However, it would reuse a decent amount of code from both, which might result in (a hopefully _small_) amount of refactoring.

Fortunately not a lot of new code would need to be written. Most methods from the Linux module work--those that don't can be made to work with existing code from the Windows module. It's just a matter of pulling the right bits in from the right places.

Additionally, a configuration would have to be added to the AppVeyor builds that includes Cygwin. I haven't tried using Cygwin in AppVeyor yet, but it's something I'm working on.

What benefits would it bring exactly (e.g. why would you want to use psutil from cygwin)?

The same benefits psutil has on any other platform :)

Ok, to be less glib, my primary motivation is work on SageMath (e.g. https://trac.sagemath.org/ticket/21598). I'm working on getting Sage working on Windows, which for some time at least will continue to require Cygwin. Sage is in the process of switching to psutil for all process-inspection related functionality (a move I strongly supported!) However, I did not realize cygwin support was missing, so that permanently breaks my Cygwin builds.

That said, I think Cygwin is still a useful platform to support in general, if I add a support module for it I'm willing to maintain it.

Thanks for the info. OK, if you want to go for it you have my blessing. Do you have any idea about where you might start already? Can I help you in any way?

if I add a support module for it I'm willing to maintain it

That is encouraging as there are a lot of platforms to test and keep under control already. Adding cygwin to appveyor CI (assuming it's possible) would be great in this sense.
As part of this work I would ask you to update the INSTALL / DEVGUIDE documents describing how to setup / install psutil on cygwin (because I don't know basically =)).

@giampaolo Thanks for your blessing--that alone is a big help; knowing that if I can get this working to your satisfaction I won't have to maintain a fork forever is encouraging. Will let you know if I have any questions.

As far as setup goes, this doesn't require much beyond what's already there for Linux, though it will be necessary to document what packages are needed in Cygwin. Beyond that I just made a virtualenv and ran make setup-dev-env and that's about all that's needed.

Just to update, I'm actually nearly done a prototype Cygwin port of psutil. Been working for it off and on over the last several months. The only hold up right now is I'm trying to get the tests to pass on AppVeyor. Stay tuned...

@embray that sounds cool; I would be very interested in seeing cygwin support for psutil. Are there some functionalities you weren't able to implement? How "beta" is this? Let me know if I can help in any way.

@giampaolo All the tests pass on Python 2 and 3 (except on AppVeyor where I am still having some problems, especially with Python 3, for some reason).

There are a couple bits of functionality that don't/can't work. Most notably, reading the environment of arbitrary processes doesn't work (or at least didn't, which actually led me to submit a patch, now accepted, to Cygwin to support /proc/<pid>/environ. So actually that functionality will be available with newer Cygwin versions. ) I'm also working off and on on another patch to Cygwin to support more /proc/net/ endpoints, which will make the networking components much easier, but I don't know how long that will take. In the meantime there are lots of bits borrowed from the Windows module. I need to do a lot of refactoring so that there is less duplicated code.

I'll try to have a PR up before long.

...so what file will contain the implementation? _pswindows.py or _pslinux.py? Are you working in a public accessible branch I can look at?
As for appveyor: I don't think it's necessary to test cygwin on all python versions. E.g. Python 2.7 only or 2.7 + 3.5 would also be fine. I don't want to introduce slow downs as appveyor builds are already pretty slow.

@giampaolo Regarding Cygwin, only Python 2.7 and 3.4 are available anyway (Cygwin support on Python 3 has not been maintained well, which is something I'm also working to fix).

There is a new _pscygwin.py because it has to take bits from Linux and/or POSIX, and Windows, as well as a few of its own unique bits.

Cool, thanks for clarifying.

FYI ... current cygwin says:

$ uname -a
CYGWIN_NT-10.0 Beren 2.8.0(0.309/5/3) 2017-04-01 20:47 x86_64 Cygwin
$ python3 --version
Python 3.6.1
$ python --version
Python 2.7.13
$ ls /proc
15020/   cygdrive@    meminfo  partitions   self@  sysvipc/
16148/   devices      misc     registry/    stat   uptime
236/     filesystems  mounts@  registry32/  swaps  version
cpuinfo  loadavg      net/     registry64/  sys/
$ ls /proc/$$
cmdline  environ  fd/   mountinfo  ppid   stat    uid
ctty     exe@     gid   mounts     root@  statm   winexename
cwd@     exename  maps  pgid       sid    status  winpid

Is this enough to treat Cygwin as a pseudo-standard port?

@binkley Just having some nodes in /proc is not enough by a long shot. Please see some of the discussion in my commits in #998 to understand more of the intricacies of this porting effort :)

@embray I did look through that -- it was depressing. :) Thanks!

Hi. Thanks @embray for your work. I'm also quite interested in this, I divide my work environment between plain linux and my workstation also running windows.

Question: I did notice you sent patches to cygwin, which is great ! How much more of your port is actually working around limitations of Cygwin, instead of trying to integrate better with Windows when possible ? Briefly looking at the patches seems you're trying to do fully windows compatible psutil running on cygwin. I would expect the cygwin psutil to just be aware of the cygwin environemnt as if it was a linux port.

Thank you again.

@joaoe I haven't worked on it in a while but I haven't dropped it entirely--I'm still working on some refactoring effort. I'm also working on moving some code out to a separate cygwin module for Python that provides a wrapper around the Cygwin DLL's few external functions (such as for path conversion).

As to your question, I think the port is trying to use as much as possible from the POSIX-compliant interfaces in Cygwin. But it should not be said that Cygwin is a "Linux port". While Cygwin does adapt some features and inspiration from Linux (such as some of the layout of /proc) it isn't necessarily guaranteed to ever be 1-to-1 with Linux-specific interfaces (it is more focused simply on POSIX-compliance, but even there there are a handful of places where compromises are necessary).

For my port, there are parts that go directly through Windows since those features aren't available in Cygwin. This is particularly true in the case of enumerating network interfaces. An alternative would be to simply disable those features of psutil on Cygwin, and I'm not totally opposed to that either. But my experience so far has been that it's worth keeping those features until and unless it's possible to port those features to Cygwin (something I would like to do, for example, is fill out more of /proc/net on Cygwin, but that would be a much more time-intensive project).

So does this just need refactoring? Is it complete in terms of features?

I'm not sure--I think there have been features added to psutil since I first started on my port that now aren't supported. But that could be handled separately if necessary--i.e. simply document those features as unsupported on Cygwin for now, and add support if/when there's time or enough demand. I'm thinking in particular of sensors.

I also need to get the AppVeyor build for Cygwin working. I was trouble with a few tests IIRC.

There's no reason to deny the install. It doesn't have to work properly and can throw an exception when used but installation should be allowed. This would allow its use with Ansible. My local system is Windows and I'm managing remote Linux nodes with Ansible. Ansible copies the module on the target node for execution, which means it should work properly.

@embray this is still relevant. If you have time, please, contribute.

I need to use psutil on cygwin, im waiting since 2014 im getting mad at you, last person i got mad with, died.

Would also love this.

The aws cli is horrifically slow on all platforms, including cygwin, and sadly, aws is behind much of the internet now, including my little corner of development.

It turns out there's a nifty little python script that will use the python libs to access aws services much more quickly than one can from the command line, but (you guessed it) it requires psutil:

https://github.com/janakaud/aws-cli-repl/blob/master/awsr

For continuous integration, might GitHub Actions replace Appveyor?