Hi
I updated the intent test runner: This version does not just log events on the bus, it actually checks that the tests succeed. It can be found here: https://github.com/CarstenAgerskov/Mycroft-intent-test-runner
It is, however, still work in progress.
That’s great work @CarstenA, I’m going to flag @forslund and @steve.penrod so they’re aware of this.
Best, Kathy
Very nice! It looks much cleaner than the old one in integrationtests. (https://github.com/MycroftAI/mycroft-core/tree/dev/test/integrationtests/skills)
Can it be integrated with the discovery_tests?
Thanks. Actually, I wasn’t aware of discovery_tests or skill_tester. Don’t know why I didn’t find them myself. I understood that something had been made, but was unfinished/abandoned. Thats why i started out coding this 
What is the status of discovery_tests and skill_tester? Are they used?
Anyway, a cursory look tells me that the skill_tester actually loads the skills one by one, where my code depends on an already running Mycroft. I could probably borrow some code there. I will look into this in more detail.
I pushed the latest version of my test runner, with support for the “internal format” and example of testing removal of context and spoken response from Mycroft.
I looked into discovery_tests and skill_tester. Modifying the SkillTest class to run my code seems easy enough.
However, when debugging it seems that the current SkillTest does not get the emitted messages on the bus. Also, it seems that loading of some skills fails initially.
So even if modify SkillTest, I guess I will end up in a situation where I can’t send messages to skills. Remember my current code depends on a running Mycroft instance.
I will continue to look into this, I just wanted to let you know my initial findings. And I would like to know the status of discovery_tests and skill_tester?
The status is that no-one has looked at it since this summer when an intern did an update to add checking of the skill’s response and I thing about 75% of the then existing skills were passing the tests and the others had changed so the test cases weren’t still valid.
We’d like to have the tests running without a mycroft instance if possible just using a mocked emitter.
If I recall correctly the discovery_test should be working, it just finds skills to test and then calls skill_test to check the skill. I’d love to have these updated and will try to help debug the existing code to integrate with yours.
Hi Ă…ke
You are right, best not to depend on a Mycroft instance running. It was easier to start that way, but it also create problems, like orchestrating test and missing isolation between skills.
Yes, from what I can see discovery_tests does what I expect, I will dive deeper into skill_tester and let you know if I hit the wall.