I have also thought about this solution, but I think that it is not comfortable. If I am not mistaken, the same behaviour is also true for the Mycroft mark1. How is it behaviour solved with this device?
First make sure you’re using precise. (ask what your listener is) https://mycroft-ai.gitbook.io/docs/using-mycroft-ai/troubleshooting/wake-word-troubleshooting
For the second thing, make sure your mic and speakers are far enough apart to minimize interference. If you can isolate them from each other as well through whatever means you have (a case? positioning? sound proofing material?) that could be helpful.