I will add that the biggest hinderence is still a strong speech recognition technology that can be run locally. While many tout CMU Sphinx/PocketSphinx or Kaldi as options, the experience of many is that they really aren’t adequate. Mozilla’s DeepSpeech has the brightest future, IMHO, but practically speaking it is still not quite ready for prime time for most users.
It is a bit of a chicken-and-the-egg situation, but I don’t think there is a strong case for creating an easy offline setup before the core components are ready. It is close, but I suspect it will be at least another year before we (the open source community) have all those pieces ready.