Aiohttp: Websocket client hanging/blocking?

GitMate.io thinks possibly related issues are https://github.com/aio-libs/aiohttp/issues/2200 (background Client websockets hangs on SIGINT), https://github.com/aio-libs/aiohttp/issues/1002 (Websocket response .close() can hang indefinitely), https://github.com/aio-libs/aiohttp/issues/376 (ProactorEventLoop hangs ), https://github.com/aio-libs/aiohttp/issues/3027 (ws_connect hang), and https://github.com/aio-libs/aiohttp/issues/265 (Implement client-side websocket).

it looks like client_ws.py#L204 is not returning... any ideas?

The server is a SkyQ set-top box, so might not be entirely RFC-compliance implementation of WS. For example, it doesn't honour ping/poings... And there is that weird �~� in the payload...Not sure if that's an opcode or some other quirk...

Websocket is a binary protocol �~� encodes a message type and length.
Sorry, I cannot help you without your participation in decoding it and comparing to actual payload size.

�~� is 001111110111111000111111 in binary, which is: 3F7E3F in hex.

those bytes look like this end-to-end...
00111111
01111110
00111111

I am not an expert on th RFC, but what is this saying in terms of type and length?

Do you think the problem is with that? Ie, it not agreeing to the actual payload length, and the thing waiting for an extra frame, for example?

Interestingly, this web socket cli client, written in go, handles the output fine, including that binary stuff at the top:

```
ws ws://skyq:9006/as/system/status
< {
"camessage" : {
"reason" : "no message",
"state" : "unavailable"
},
"drmstatus" : {
"state" : "available"
},
"entitlements" : [
"ANALYTICS",
"BIGBASIC",
"ETHAN_APP_1",
"HD",
"PDL",
"SKY_DRM_CE",
"SKY_DRM_MR",
"SKY_IPPV",
"ULTRA+",
"SKY+",
"GATEWAYENABLER",
"SIDELOAD"
],
"epginfobits" : {
"epginfobits" : "0xFDE5FFC0",
"mask" : "0x5FFA003F",
"state" : "available"
},
"gatewayservices" : {
"state" : "available"
},
"hdmi" : {
"2160p10bitCapable" : false,
"authenticatedHDCP" : "NONE",
"reason" : "HDMI output port is disabled",
"sinkHDCP" : "NONE",
"sinkHLG" : false,
"sinkUHD" : false,
"state" : "unavailable",
"uhdConfigured" : false
},
"network" : {
"state" : "available"
},
"nssplayback" : {
"state" : "available"
},
"pvr" : {
"state" : "available"
},
"schedule" : {
"lastdate" : "20181003",
"state" : "available"
},
"servicelist" : {
"state" : "available"
},
"smartcard" : {
"active" : true,
"bouquet" : "4101",
"countryCode" : "GBR",
"currency" : "GBP",
"cwe" : true,
"householdid" : "10947783",
"paired" : true,
"state" : "available",
"subbouquet" : "1",
"transactionlimit" : 65535,
"viewingCardNumber" : "725 325 260"
},
"swupdate" : {
"reason" : "IDLE",
"state" : "unavailable"
},
"systemupdates" : {
"entitlements" : 2,
"install" : 1,
"servicegenres" : 1,
"smartcard" : 1
},
"updatetask" : {
"reason" : "no update",
"state" : "unavailable"
}
}

^C
Interrupt
✔ [brad@bradmac:~] $```

The protocol says: the first 3F byte in WS message is forbidden (reserved for future use).
Are you sure that you didn't miss something?

I assume you are referring to the 2d, 3rd and 4th bit in the base framing protocol. In which case yes, maybe I did get it wrong. It could be a unicode/ASCII issue. I'll take another look.

I've tried a few other online converters... I'm getting this now:
hex = ef bf bd 7e ef bf bd
binary = 11101111 10111111 10111101 01111110 11101111 10111111 10111101

If according the RFC 6455

5.2.  Base Framing Protocol

   This wire format for the data transfer part is described by the ABNF
   [RFC5234] given in detail in this section.  (Note that, unlike in
   other sections of this document, the ABNF in this section is
   operating on groups of bits.  The length of each group of bits is
   indicated in a comment.  When encoded on the wire, the most
   significant bit is the leftmost in the ABNF).  A high-level overview
   of the framing is given in the following figure.  In a case of
   conflict between the figure below and the ABNF specified later in
   this section, the figure is authoritative.

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-------+-+-------------+-------------------------------+
     |F|R|R|R| opcode|M| Payload len |    Extended payload length    |
     |I|S|S|S|  (4)  |A|     (7)     |             (16/64)           |
     |N|V|V|V|       |S|             |   (if payload len==126/127)   |
     | |1|2|3|       |K|             |                               |
     +-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
     |     Extended payload length continued, if payload len == 127  |
     + - - - - - - - - - - - - - - - +-------------------------------+
     |                               |Masking-key, if MASK set to 1  |
     +-------------------------------+-------------------------------+
     | Masking-key (continued)       |          Payload Data         |
     +-------------------------------- - - - - - - - - - - - - - - - +
     :                     Payload Data continued ...                :
     + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
     |                     Payload Data continued ...                |
     +---------------------------------------------------------------+

   FIN:  1 bit

      Indicates that this is the final fragment in a message.  The first
      fragment MAY also be the final fragment.

   RSV1, RSV2, RSV3:  1 bit each

      MUST be 0 unless an extension is negotiated that defines meanings
      for non-zero values.  If a nonzero value is received and none of
      the negotiated extensions defines the meaning of such a nonzero
      value, the receiving endpoint MUST _Fail the WebSocket
      Connection_.

Then it looks like RSV1 and RSV2 are being set to 1, as you suggest, unless some etxension is being negotiated? I have no idea.

The number of the characters in the JSON doc is 1703

The RFC says this about payload length:

Payload length:  7 bits, 7+16 bits, or 7+64 bits

      The length of the "Payload data", in bytes: if 0-125, that is the
      payload length.  If 126, the following 2 bytes interpreted as a
      16-bit unsigned integer are the payload length.  If 127, the
      following 8 bytes interpreted as a 64-bit unsigned integer (the
      most significant bit MUST be 0) are the payload length.  Multibyte
      length quantities are expressed in network byte order.  Note that
      in all cases, the minimal number of bytes MUST be used to encode
      the length, for example, the length of a 124-byte-long string
      can't be encoded as the sequence 126, 0, 124.  The payload length
      is the length of the "Extension data" + the length of the
      "Application data".  The length of the "Extension data" may be
      zero, in which case the payload length is the length of the
      "Application data".

So, given all that, where do we go next? If the server is brain-dead and not RFC compliant, then I need to work around it as other client's clearly can do... Can you offer any suggestions?

Still looks suspicious.
The only known extension is WebSocket compression which utilizes RSV1. I don't know what extension uses RSV2.
Moreover, opcode FF is reserved too and should be not used.

Unless you don't know what the server sends there is no way to "fix" a client.

@asvetlov I tried the websockets library and it works... So I'm going to go with that, as I cannot spend more time on debugging this library with you...

aoihttp -- doesn't work

import asyncio
import logging
import sys

import aiohttp

LOGGER = logging.getLogger(__name__)
logformat = "[%(asctime)s] %(levelname)s:%(name)s:%(message)s"
logging.basicConfig(level=logging.DEBUG, stream=sys.stdout,
                    format=logformat, datefmt="%Y-%m-%d %H:%M:%S")


async def main():
    async with aiohttp.ClientSession() as session:
        async with session.ws_connect('http://skyq:9006/as/system/status') as ws:
            payload = await ws.receive()
            print(payload.text)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

websockets -- works

import asyncio
import websockets
import logging
import sys

LOGGER = logging.getLogger(__name__)
logformat = "[%(asctime)s] %(levelname)s:%(name)s:%(message)s"
logging.basicConfig(level=logging.DEBUG, stream=sys.stdout,
                    format=logformat, datefmt="%Y-%m-%d %H:%M:%S")


async def hello():
    async with websockets.connect(
            'ws://skyq:9006/as/system/status') as websocket:

        payload = await websocket.recv()
        print(f"{payload}")

asyncio.get_event_loop().run_until_complete(hello())

Fee free to close this ticket if you wish to, but you might want to keep it open as it does appear to be a bug... Or at least a situation which does not obey Postel's law.

I appreciate your help in trying to address this.

Still looks suspicious.
The only known extension is WebSocket compression which utilizes RSV1. I don't know what extension uses RSV2.
Moreover, opcode FF is reserved too and should be not used.

Hi @asvetlov

Unless you don't know what the server sends there is no way to "fix" a client.

I think you meant "unless, you do know what a server sends, you cannot fix a client."

In which case, I completely disagree with this point.

You are familiar with the Robustness Principle?

"Be conservative in what you send, and liberal in what you receive."

Regardless of MUSTs and SHOULDs and MUST NOTs, in the RFC, I think you are interpreting and expecting entities to comply with the RFC _completely_... Clearly they do not, and many many internet servers and clients out there, of various types, have horrendous bastardised half-implementations of protocols.

A good library/protocol implementation must be forgiving of these to the extent that it can. See RFC1122. It is completely possible to write a protocol client to be forgiving of (some) protocol violations without having access to the other participant in the communication.

I feel this is not the case here, you seem to be taking the hard-line view that if a host is not 100% compliant then too bad for it, we will not talk to it. This is too bad. It's like saying: 'I only speak the Queen's English, and if you come talking to me in some weird dialect of English, I won't talk to you.'

Clearly this is not a good approach if the whole point of the library is to facilitate communication.

My suggestion, therefore, is to consider where you can relax stringent requirements of the protocol (maybe via method kwargs, that turn on the "relaxed" handling), and see if you can make things talk even when the other side is not behaving properly.

Aside from this critique, I'd like to thank you and the contributors from aio-libs for all your work for the community. 👍 🥇

@asvetlov Just an update on this... I figured out that my (non-compliant) server requires an uppercase Upgrade: header before it will work. Just thought I'd mention it in case the issue of Title Case headers is being considered by you guys... This is a real case of a protocol violation from a poorly implemented server which causes the websocket client not to work..