new version of paho-mqtt breaks mqtt messaging within pywws

[rick165]

It seems the new version of paho-mqtt (v2.0.0) breaks the mqtt messaging within pywws.

pywws.service.mqtt:Unsupported callback API version: version 2.0 added a callback_api_version, see migrations.md for details
Work around is to install v1.6.1 of paho-mqtt

sudo pip install paho-mqtt==1.6.1

[GeekyTim]

I'll put that on my list to fix, alongside the self-cert issue.

[GeekyTim]

I've got a REALLY simple fix to this paho-mqtt version issue, as well as allowing self-certs. I'll send a pull request once I've tested a few more things.

[cromemcos]

I just installed 24.8.0 and I was so hopeful ... but nothing changed. mqtt still broken. :-(

1:22:54:pywws.service.metoffice:data: {'dateutc': '2024-09-22 20:22:51', 'winddir': '315', 'windspeedmph': '0.00', 'windgustmph': '1.57', 'humidity': '97', 'tempf': '61.5', 'baromin': '29.7898', 'rainin': '0.0000', 'dailyrainin': '0.0000', 'dewptf': '60.7', 'softwaretype': 'pywws-24.8.0', 'siteid': 'a4f1ade3-29b2-ed11-9ac4-201642ba4e0a', 'siteAuthenticationKey': '535251'} 21:22:54:pywws.weatherstation:delay 0, pause 27.4161 21:22:57:pywws.service.mqtt:Unsupported callback API version: version 2.0 added a callback_api_version, see docs/migrations.rst for details Traceback (most recent call last): File "/home/m/.local/pipx/venvs/pywws/lib/python3.11/site-packages/pywws/service/__init__.py", line 180, in run OK = self.upload_batch()

[GeekyTim]

I will take a look and see if there are issues.

[GeekyTim]

Curious. I have that version installed on Python 3.9 with paho-mqtt 2.1, and it works fine for me.

Could you please take a look at /home/m/.local/pipx/venvs/pywws/lib/python3.11/site-packages/pywws/service/mqtt.py and check that the first line after 'def def session(self):' is:

session = mosquitto.Client(
            mosquitto.CallbackAPIVersion.VERSION1, self.params['client_id'], protocol=mosquitto.MQTTv31)

Thanks

[cromemcos]

So yes I checked the file and mine looks a little different Here is a numbered copy paste from a vi edit

`

240 for line in template.splitlines():

241 if line:

242                 result.append(pprint.pformat(line, width=256))
243         return '(\n' + '\n'.join(result) + '\n)'
244 
245     @contextmanager
246     def session(self):
247         session = mosquitto.Client(
248             self.params['client_id'], protocol=mosquitto.MQTTv31)
249         if self.params['password']:
250             session.username_pw_set(
251                 self.params['user'], self.params['password'])
252         elif self.params['user']:
253             session.username_pw_set(self.params['user'])
254         logger.debug(('connecting to host {hostname:s}:{port:d} '
255                       'with client_id "{client_id:s}"').format(**self.params))
256         if self.params['tls_cert']:
257             selfcert = False

"mqtt.py" line 246 of 301 --81%-- col 5
`

`

m@domo:/.local/pipx/venvs/pywws/lib/python3.11/site-packages/pywws/service $ grep -i Callback *.py
m@domo:/.local/pipx/venvs/pywws/lib/python3.11/site-packages/pywws/service $ grep -i mosquitto *.py

mqtt.py:* Mosquitto (a lightweight broker): http://mosquitto.org/
mqtt.py:See https://mosquitto.org/man/mosquitto-tls-7.html for information on how to
mqtt.py:http://www.steves-internet-guide.com/mosquitto-tls/ for a step-by-step
mqtt.py:This has been tested with the Mosquitto Open Source MQTT broker, running
mqtt.py:import paho.mqtt.client as mosquitto
mqtt.py: session = mosquitto.Client(
mqtt.py: self.params['client_id'], protocol=mosquitto.MQTTv31)
`

I do apologise that I am crucifying the formatting, inserting code in the tool seems not to do what I expected.

[GeekyTim]

I can see there that it is the old version of the file, so needs to be updated.

I think @jim-easterbrook wold be better at answering why it didn't get upgraded correctly.

[cromemcos]

I just redid the installation and show the output here for clarity

pipx install pywws==24.8.0
'pywws' already seems to be installed. Not modifying existing installation in '/home/m/.local/pipx/venvs/pywws'. Pass '--force' to
force installation.
m@domo:~/.local/pipx/venvs/requests $ pipx install pywws==24.8.0 --force
Installing to existing venv 'pywws'
⚠️  Note: pywws-hourly was already on your PATH at /usr/local/bin/pywws-hourly
⚠️  Note: pywws-livelog was already on your PATH at /usr/local/bin/pywws-livelog
⚠️  Note: pywws-livelog-daemon was already on your PATH at /usr/local/bin/pywws-livelog-daemon
⚠️  Note: pywws-reprocess was already on your PATH at /usr/local/bin/pywws-reprocess
⚠️  Note: pywws-setweatherstation was already on your PATH at /usr/local/bin/pywws-setweatherstation
⚠️  Note: pywws-testweatherstation was already on your PATH at /usr/local/bin/pywws-testweatherstation
⚠️  Note: pywws-version was already on your PATH at /usr/local/bin/pywws-version
  installed package pywws 24.8.0, installed using Python 3.11.2
  These apps are now globally available
    - pywws-hourly
    - pywws-livelog
    - pywws-livelog-daemon
    - pywws-reprocess
    - pywws-setweatherstation
    - pywws-testweatherstation
    - pywws-version
done! ✨ 🌟 ✨

pywws-version
24.8.0

[GeekyTim]

Does it work now?

[cromemcos]

Correct me if I am wrong but on Github it does not have the API V1 call ??

[cromemcos]

Does it work now?

So when I reinstalled then no it still does not work and this is presumably because the mqtt.py file is as the github shows the code that does not include the V1 API call.

[GeekyTim]

You are absolutely correct. It seems to have been reverted somehow.

I'll have to ask Jim.

[jim-easterbrook]

The commit history of mqtt.py doesn't show anything odd.
https://github.com/jim-easterbrook/pywws/commits/master/src/pywws/service/mqtt.py

The version on my local machine, from which the packages uploaded to PyPI were made, has these lines:

       session = mosquitto.Client(
            self.params['client_id'], protocol=mosquitto.MQTTv31)

I just redid the installation and show the output here for clarity

Note: pywws-hourly was already on your PATH at /usr/local/bin/pywws-hourly

Does pipx have write access to /usr/local/bin? I suspect the pywws stuff there is a hangover from an earlier installation, and pipx is warning you that the ones it has installed somewhere else won't get used. (I don't use pipx so don't know much about it.)

[cromemcos]

Ah okay, thanks Jim for replying. So your comment relates to stuff in /usr/local/bin. Indeed I checked and that was the result of an old installation.

so it was a

`
sudo rm -fv /usr/local/bin/pyww**

and then a reinstallation
which now looks like this`

`

pipx install pywws==24.8.0

'pywws' already seems to be installed. Not modifying existing installation in
'/home/m/.local/pipx/venvs/pywws'. Pass '--force' to force installation.
m@domo:/usr/local/bin $ pipx install pywws==24.8.0 --force
Installing to existing venv 'pywws'
installed package pywws 24.8.0, installed using Python 3.11.2
These apps are now globally available
- pywws-hourly
- pywws-livelog
- pywws-livelog-daemon
- pywws-reprocess
- pywws-setweatherstation
- pywws-testweatherstation
- pywws-version
done! ✨ 🌟 ✨
`

BUT and BUT the issue still remains pywws is unable to talk to mqtt

and I check the logs ... still bused

[GeekyTim]

Somehow the change I submitted was there, then wasn't. I'm not sure why (since Jim did the pull request). I need to get some time to spend on this; I had it working here until I took certs off my MQTT server because of issues with other software!

We'll get it sorted.