Display custom GUI elements on Picroft

Vangrel · March 31, 2020, 2:57pm

I’ve been trying to get a Picroft rasbpi installation to run something like what is found in https://github.com/MycroftAI/mycroft-gui (though this only works on KDE Neon) that would allow rendering of custom misc elements when using mycroft, specifically images, or gifs on a desktop environment. Have searched thoroughly and haven’t found any readily available solution, any help or direction would be greatly appreciated!

JarbasAl · March 31, 2020, 6:04pm

Hi vangrel

Mycroft-core supports any numbers of GUIs, even connecting at the same time to a single mycroft instance. it is completely implementation agnostic.

However the gui can be considered alpha, there are no readily available examples yet.

In essence mycroft-core opens a websocket that anything can connect to and know what needs to be displayed, if you are confortable with reading source code you can find the API here

github.com

MycroftAI/mycroft-core/blob/dev/mycroft/enclosure/gui.py

# Copyright 2019 Mycroft AI Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
""" Interface for interacting with the Mycroft gui qml viewer. """
from os.path import join

from mycroft.configuration import Configuration
from mycroft.messagebus.message import Message
from mycroft.util import resolve_resource_file

This file has been truncated. show original

the mycroft-gui repo is the first implementation available, but you can implement any GUI that you would like. @forslund recently shared a picture of a proof of concept javascript based gui in the chat

Keep in mind this is in alpha stage so may things can change in the GUI api, a couple things are also placeholders to get the initial qml implementation going, those will likely be the trickiest ones to get around, but have a look at the link above and you should have a general idea of the current state of things

JarbasAl · April 1, 2020, 12:45am

I don’t think i was very clear in the past post, so i’m leaving a few more reference links

In here you can see the original GUI Pull Request

Most relevant is probably the text based GUI (used for debugging)

If you run mycroft in debug mode and while in the CLI press ctrl+G you should see something like this

wikipedia skill will send data to display, in this case summary and a picture

weather skill sends weather data

this GUI example only displays the fields, you can however just listen to the websocket parse the json and display whatever way you want

JarbasAl · April 1, 2020, 5:20am

i made a quick and dirty command line GUI interface, you can check it here

it’s a simple command line interface, useful for debugging but not a real GUI. It is meant to illustrate how the GUI protocol works

you will want to watch out for this bug, you might need to restart enclosure for the GUI to properly connect

Vangrel · April 1, 2020, 6:09am

Thank you for this @JarbasAl, had been banging my head on this for a while, going through these resources now

JarbasAl · April 1, 2020, 7:00am

at some point i may revisit my old display service and maybe resurrect it and integrate with the current GUI, here is a video of my early experiments

Vangrel · April 1, 2020, 6:25pm

This hits it on the nail, thank you so much @JarbasAl I !