ERROR:ssl_client_socket_impl.cc handshake failed on Ubuntu 22.04

[markakis-sch]

Hello,

I'm getting the following error when i press the "Start Searching" button in Scratch 3 with pyscrlink running on Ubuntu 22.04:
[10348:0306/122147.012576:ERROR:ssl_client_socket_impl.cc(981)] handshake failed; returned -1, SSL error code 1, net_error -101

In the terminal window running pyscrlink i get no error messages:
$ /usr/local/bin/scratch_link -d
Print debug messages
2023-03-06 12:21:07,006 set scan_seconds: 10.0
2023-03-06 12:21:07,025 Certificate is ready in FireFox NSS DB: /home/administrator/.mozilla/firefox/qj7kn4vq.default-release
2023-03-06 12:21:07,036 Certificate is ready for Chrome
2023-03-06 12:21:07,078 Started scratch-link

I have the following packages installed on my system:
sudo apt install bluez libbluetooth-dev libnss3-tools libcap2-bin python3-pip libglib2.0-dev python3-bluez python3-websockets
sudo pip3 install pyscrlink

and i have run the command:
/usr/local/bin/bluepy_helper_cap

[kawasaki]

Hello @markakis-sch , thanks for the report, but I'm not sure how to resolve it. The "net_error = -101" should mean something, but I have no idea what it is.

Instead, I found a stackoverflow discussion. If you are using Chromium (or Chrome), adding the command options '--ignore-ssl-errors' and/or '--ignore-certificate-errors-spki-list' may help. But even if it avoids the failure, it does not explain the cause of the failure.

One possible cause is certificate that pyscrlink generated might have expired (it should work for 10 years, so it is too early for you to expire...). If this is the case, it may worth to remove scratch-device-manager.cer and scratch-device-manager.key at ~/.local/share/pysrclink. The scratch_link will regenerate those files, then it should be ok to remove them. After that, please try to run scratch_link to see if affects the error or not.

[kawasaki]

In the issue #34, it was reported that FireFox certification DB path changed for Ubunut 22.04. @markakis-sch , if you are using FireFox, please try to edit /.local/lib/python3.X/site-packages/pyscrlink/gencert.py, and replace the string ".mozilla/firefox/" with "/snap/firefox/common/.mozilla/firefox". I'm not sure about the exact path on Ubuntu 22.04, then the correct path string might be slightly different.

[alkisg]

On Ubuntu 22.04, I tested the following, which indicate the problem is in pyscrlink itself:

# Clear existing global and local python packages.
# WARNING, this may erase other packages that you need:
sudo rm -rf /usr/local/lib/python*
rm -rf ~/.local/lib/python*
# Install most prerequisites from apt repositories
sudo apt install bluez libbluetooth-dev libnss3-tools \
 libcap2-bin python3-pip libglib2.0-dev python3-bluez \
 python3-websockets
# Install the rest from pip
sudo pip3 install pyscrlink
# Run the helper
sudo /usr/local/bin/bluepy_helper_cap
# Run pyscrlink
/usr/local/bin/scratch_link
# And check if it's listening
wget --no-check-certificate https://127.0.0.1:20110 -O-
--2023-03-25 08:44:34--  https://127.0.0.1:20110/
Connecting to 127.0.0.1:20110... connected.
Unable to establish SSL connection.

So the problem seems to be that pyscrlink doesn't register a proper https service and even wget is unable to contact it.

[kawasaki]

Hi @alkisg , thanks for the detailed report. I tried your command lines on Ubuntu 22.04 on my old laptop. And I observed a bit different output by wget.

$ LANG=C --no-check-certificate https://127.0.0.1:20110 -O-
--2023-03-26 11:11:05--  https://127.0.0.1:20110/
Connecting to 127.0.0.1:20110... connected.
WARNING: cannot verify 127.0.0.1's certificate, issued by 'CN=device-manager.scratch.mit.edu':
  Self-signed certificate encountered.
    WARNING: certificate common name 'device-manager.scratch.mit.edu' doesn't match requested host name '127.0.0.1'.
HTTP request sent, awaiting response... 426 Upgrade Required
2023-03-26 11:11:05 ERROR 426: Upgrade Required.

Also, the command /usr/local/bin/scratch_link worked in my environment. My micro:bit can be controlled from Scratch on my firefox via the /usr/local/bin/scratch_link command. The https service is working in my environment.

I think these observations indicates that there should be a difference between your environment and my environment. But I'm not sure what it is...

I googled with the keyword "Unable to establish SSL connection." and found a discussion in stackoverflow. It advises to try "openssl" command for debug. I tried,

$ LANG=C openssl s_client -debug 127.0.01:20110

It printed certain amount of logs but they do not look to reporting any errors.

Could you try the command above and share the output on your system? I'm not SSL expert, but will try to find out something in it.

[alkisg]

Hello, here's the output:

$ LANG=C openssl s_client -debug 127.0.01:20110
CONNECTED(00000003)
write to 0x55824872afa0 [0x55824873c260] (293 bytes => 293 (0x125))
0000 - 16 03 01 01 20 01 00 01-1c 03 03 6e 83 32 45 ef   .... ......n.2E.
0010 - 58 39 a9 69 d7 b9 7a f8-c4 1c 94 34 08 0f 44 e2   X9.i..z....4..D.
0020 - e6 67 e9 68 0f 47 76 3f-cf 8c 95 20 21 fd 8e a2   .g.h.Gv?... !...
0030 - 57 db 0a 4a af 39 25 fa-32 fc 5e 14 a7 79 8d e3   W..J.9%.2.^..y..
0040 - e4 e4 c8 7c a9 8d 2a f7-30 a3 1e 7f 00 3e 13 02   ...|..*.0....>..
0050 - 13 03 13 01 c0 2c c0 30-00 9f cc a9 cc a8 cc aa   .....,.0........
0060 - c0 2b c0 2f 00 9e c0 24-c0 28 00 6b c0 23 c0 27   .+./...$.(.k.#.'
0070 - 00 67 c0 0a c0 14 00 39-c0 09 c0 13 00 33 00 9d   .g.....9.....3..
0080 - 00 9c 00 3d 00 3c 00 35-00 2f 00 ff 01 00 00 95   ...=.<.5./......
0090 - 00 0b 00 04 03 00 01 02-00 0a 00 16 00 14 00 1d   ................
00a0 - 00 17 00 1e 00 19 00 18-01 00 01 01 01 02 01 03   ................
00b0 - 01 04 00 23 00 00 00 16-00 00 00 17 00 00 00 0d   ...#............
00c0 - 00 2a 00 28 04 03 05 03-06 03 08 07 08 08 08 09   .*.(............
00d0 - 08 0a 08 0b 08 04 08 05-08 06 04 01 05 01 06 01   ................
00e0 - 03 03 03 01 03 02 04 02-05 02 06 02 00 2b 00 05   .............+..
00f0 - 04 03 04 03 03 00 2d 00-02 01 01 00 33 00 26 00   ......-.....3.&.
0100 - 24 00 1d 00 20 7a ea 36-47 04 44 ca 36 3a 98 7c   $... z.6G.D.6:.|
0110 - 4c f9 ca e4 cb c7 a9 ff-49 67 b0 af 14 10 3f 20   L.......Ig....? 
0120 - 5c ee 1d af 2e                                    \....
read from 0x55824872afa0 [0x558248733043] (5 bytes => 0)
write to 0x55824872afa0 [0x55824873c260] (7 bytes => -1)
40D78EB23B7F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:308:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 293 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
read from 0x55824872afa0 [0x558248685650] (8192 bytes => 0)

Since it's working for you but not for us, could you please send us the output of this command in your PC?

find /usr/local -ls

[kawasaki]

Thanks for the openssl command output. It shows the sudden read failure, but it just say "unexpected eof" which is not helpful for me...

Here's the find command output:
find_user_local.log

[alkisg]

I think the difference is that you're using packages from pip, while we are using the distribution packages.

Could you please temporarily move aside your local directories and try again?

sudo mv /usr/local /usr/local.old
mv ~/.local/lib ~/.local/lib.old
sudo pip3 install pyscrlink

...or try the commands I wrote above using a live Ubuntu USB stick...

[kawasaki]

That operation looks dangerous :) So I tried differently:

$ sudo pip uninstall websockets
$ sudo apt install python3-websockets

After the commands above, I observe the same symptom as yours!

$ LANG=C wget --no-check-certificate https://127.0.0.1:20110 -O- 
--2023-03-26 17:39:02--  https://127.0.0.1:20110/
Connecting to 127.0.0.1:20110... connected.
Unable to establish SSL connection.

The openssl command printed the same message. Thank you very much. One step forward :)

So as you guessed, the python-websockets version difference is the cause, most likely. Ubuntu package version is 9.1-1, and the PyPI pip version is 10,4.

I did not see the failure when I followed your commands. I checked my command history, and found that I made a typo: "python3-websocket" which should be "python3-websockets", with the last 's'. Then the Ubuntu package was not installed at that time. It is surprising that these packages have such very similar names...

Assuming the Ubuntu old package is the cause, it may help you or other Ubuntu users to try:

$ sudo apt remove python3-websockets (added the last 's' afterwards)
$ sudo pip install websockets

Another left question is that why the websockets version 9.1 cause the failure. I will try to understand it, and think if pyscrlink can cover the both old and new versions.

[markakis-sch]

@kawasaki you made a typo again! :)
The correct command to remove the distribution package is:

sudo apt remove python3-websockets

Or even better:

sudo apt purge python3-websockets

I can confirm that when i removed the distro (Ubuntu 22.04) package python3-websockets and installed the websockets package via pip, i was able to connect a LEGO WeDo 2.0 Hub, using Pyscrlink and:

• the desktop application Scratch 3
• the online Scratch 3 with Mozilla Firefox
• the online Scratch 3 with Google Chrome

I think using the pip package websockets is an acceptable solution for now, until the distribution package is fixed/upgraded.

Thank you very much @kawasaki and @alkisg!

[kawasaki]

@markakis-sch My bad, sorry for the typo again. I edited my comment to fix it.
Good to know that the issue was resolved for you :) Thanks @alkisg for your support.
I'll keep this issue open for a while I check the old version of websocket_s_.