TLS/SSL handshakes exhausts heap memory kills voice assistant

[lone-baggie]

I wanted to test the Intercom features of the Lovelace Intercom card, so I enabled SSL on my local Home Assistant host, as this is a requirement for the card. This proved to be a disaster. Immediately the voice pipeline broke down; my Waveshare audio became dumb, unable to reply to anything but the shortest sentences. Error logs build up, showing -

[10:30:25.365][I][esp-idf:000][ann_read]: E (174390) esp-aes: Failed to allocate memory
[10:30:25.365][I][esp-idf:000][ann_read]: E (174391) esp-tls-mbedtls: read error :-0x0001
[10:30:25.365][I][esp-idf:000][ann_read]: E (174391) transport_base: esp_tls_conn_read error, errno=Success
[10:30:25.365][E][speaker_media_player:364]: The announcement pipeline's file reader encountered an error.
[10:30:25.877][E][speaker_media_player:364]: The announcement pipeline's file reader encountered an error.
[10:30:25.879][I][esp-idf:000][ann_read]: E (174853) esp-aes: Failed to allocate memory
[10:30:25.880][I][esp-idf:000][ann_read]: E (174854) esp-tls-mbedtls: read error :-0x0001
[10:30:25.880]

It appears the overhead for encrypting and decrypting packets is too much for the ESP32 to handle; as soon as I removed SSL from my HA host, everything worked fine. Is there any way to force the satellite to stop using SSL?

[n-IA-hane]

Hi, this is a problem I'm familiar with. I optimized the audio as much as possible to free up resources for the HA and Esphome applications. Despite this, Esphome's SSL system consumes a lot of RAM. I've tried enabling SDK options in the past that force SSL components to operate in PSRAM, but everything starts to malfunction after that. This is how I solved the problem. In the HA settings, if you declare a non-SSL URL in the lowest slot, you can still reach your HA via SSL, but the HA communicates with the ESPs (sends them non-SSL streams) in this way. As I mentioned, I've tried enabling SDK options in the ESP that force SSL streaming in PSRAM, but that's not enough. I've also tried disabling some SSL options, such as hardware decryption (which still consumed the same amount of RAM), effectively forcing everything to run in PSRAM, but I've subsequently encountered slowdowns and instability. The only solution I found is the one you see in the screenshot. Let me know if this solves your problem. In my locations, adopting this solution solved every problem. If it works for you, I'll add it to the readme as a user requirement.

[lone-baggie]

Yes, I only set up SSL to test the Lovelace intercom card. I use my custom domain at Nabu Casa for remote access so I can leave local Home Assistant on HTTP.

Memory is still tight; I have had the odd drop-out when streaming music. I need to do more testing; there might be issues with my music assistant.

Is it possible to remove the intercom features and leave just the voice assistant? The interconnection of all the packages is too much for me to edit out.

Again, this is a remarkable piece of software. Looking forward to seeing how it develops.

[lone-baggie]

Just discovered there is a flag in Chrome. Insecure origins treated as secure. This will allow your http page (http://homeassistant.local) to more trusted. This will allow your intercom lovelace card to work . Result

[n-IA-hane]

Sure, you can. You just need to build your own YAML without intercom, around esp_audio_stack.

but before taking everything apart, I’d recommend waiting a bit.

With the GMF libraries and the WS3, which has two microphones, I’ve actually noticed some heap shortage too. Strangely, in my case it showed up in a less catastrophic way, but I found some SDK options for IDF that move fairly heavy tasks into PSRAM, such as lwIP, mDNS, and others.

I’m testing this right now. With those options enabled, I managed to keep internal RAM at a reasonable level even with MWW + Intercom + music + TTS response all running together.

I’m fixing one small remaining issue, but I think in a few hours I’ll be able to push the updated YAMLs to main with these SDK options, which save RAM.

I’ll write to you later when I’ve finished testing. For you, it should just mean downloading the new YAML with the updated SDK options and recompiling it. As I said, I’m already seeing clear improvements.

Talk later.

[n-IA-hane]

https://raw.githubusercontent.com/n-IA-hane/esphome-intercom/refs/heads/dev/yamls/full-experience/single-bus/waveshare-s3-full-afe-tcp.yaml

Hi, try compiling this.

I think I found the balance I was talking about. We fixed several issues on this ESP: one important discovery was that Wi-Fi disable_recovery_mode was forcing some
unwanted behavior, and we also made GMF/audio pipeline changes to stabilize it.

With the current setup, this device now handles MWW, FD profile, full duplex, VAD and AEC enabled, intercom calls, background music, and TTS responses running
together.

Keep me updated, so if everything works well for you too, we can consider pushing it to main.

[lone-baggie]

Compile OK. Running fine, logs are clean. Been playing music for an hour with no issues. VA running fine. Struggles to hear the wake word in a very noisy environment, but I feel this is a limitation of the hardware. no different to the previous versions.

Ducking works fine. Copes with ambient noise so much better than anything else I have tried on this hardware. Can cope with voice radio in the background; never seems to pick up the speech.

A couple of things:-

Cannot get the Lovelace Intercom card to work; it does not see the device. I have been running the UDP version previously. Still running without SSL on home assistant

Don't have any other devices to test with, so don't know about the intercom.

There's a minor screen overrun on Intercom Endpoint sensor on the dashboard

Will keep testing, but so far so good.

[n-IA-hane]

Hi, that's strange. I have the same native intercom and the same device, and the compiled yaml doesn't have any problems with intercom. Could something be stuck in the cache? Try removing the device from the esphome integration, and it should rediscover it. Re-adding it sometimes unlocks entity name updates. Yes, I also noticed that attribute going out of bounds, see if I can avoid it. If it still doesn't unlock, try redoing the complete native intercom setup. These inexplicable errors usually also have to do with the component cache.

[lone-baggie]

The intercom is an issue at my end. Nothing to do with the firmware or Lovelace card.

Also, I have noticed the Voice Assistant Wizard to set up a new voice assistant works correctly on the new firmware; it always failed before.

[n-IA-hane]

⚠️ Heads-up: major breaking changes are coming in the next version.

I know it can be frustrating to keep up with every big architectural change, and before you send me to hell: this one is not just another internal refactor.

The next release makes ESPHome Intercom compatible with SIP and VoIP. Yes, really. This is already working on the dev branch.

What does it mean in practice?

It means that, if you want, you can have a simple Home Assistant / ESPHome-centric PBX without having to carry Asterisk along with it.

• Your ESP intercom can still stay simple: one button, one destination, call, answer, talk, hang up.
• Under the hood, ESP devices are becoming real SIP/VoIP phones.
• ESPs can still call each other directly when direct SIP routing is available.
• ESPs can still call Home Assistant, and Home Assistant can still call ESPs.
• Home Assistant / Intercom Native is now also a SIP softphone, router/B2BUA, RTP bridge/resampler, optional local registrar and optional trunk client.
• You can register a real softphone such as Zoiper, Linphone, baresip or similar directly to Intercom Native.
• Registered softphones can appear in the central phonebook, so ESPs and Home Assistant can call them by name.
• Home Assistant itself can receive calls as a VoIP softphone and make its Lovelace card ring.
• If you configure a SIP trunk, Home Assistant and ESP devices can make and receive external calls without requiring Asterisk beside the integration.
• You can call landline/mobile numbers from Home Assistant or from an ESP through the trunk.
• External inbound calls can ring Home Assistant by default.
• I am also working on DTMF-based routing/IVR: for example, call your home number, press nothing to ring Home Assistant, or dial an optional internal extension such as 200 to ring a specific ESP.
• Internal extension numbers are optional: contacts can still be resolved by name, SIP URI, direct IP/port, HA routing, softphone registration, or trunk routing depending on your setup.
• Audio quality is negotiated per direction, so a device can receive the best compatible speaker format while transmitting whatever its microphone path can actually produce.

This is a breaking migration because the old project-specific call-control model is being replaced by SIP/SDP/RTP primitives. The goal is to make the project simpler conceptually and much more interoperable: ESPs, Home Assistant, browser card, softphones and trunks all speak the same VoIP model.

If you want to test early, use the dev branch. If your install is critical, wait for the release notes and migration guide before upgrading.

Current testing status: I am actively testing with Spotpear, Waveshare S3, Home Assistant, Zoiper and baresip. I have not completed testing across every YAML/profile yet, but if you want to start experimenting now, these are the devices and clients I can already say are working.