Help a college in Brazil!!

Wolner

Hi Javi,

Nice to meet you!! I am a research student at IFPE (Instituto Federal de Pernambuco), a college in Brazil.

I saw the help you provided to students at a university here in Brazil (https://community.handsfreeforweb.com/d/24-help-our-university-br-project) and would like to know if you can help me too?

I am developing a plugin for Moodle, which is a virtual learning platform. The plugin I want to create is for my research, whose deadline is November this year and which consists of a Moodle plugin that activates your plugin.

What I intend to do is create a plugin that generates a button. This button when pressed, triggers a URL that opens a file (html, php or other) and this file itself calls Handsfree for Web. That is, it is a way to install it as a Moodle plugin, not a plugin from Chrome.

I know it only works on Chrome and I want to keep it that way, but I would like this functionality to be possible in Moodle too.

What did I plan make sense?
I don't want to change your code due to copyright. I respect: D

Any questions can ask me what I explain better.

Good weekend!!
Bye

javi

Hi Wolner ,

I'm not sure of following what you want to accomplish. There are some limits that I think they will prevent you from making Handsfree for Web a Moodle plugin as you have described it.

Let me describe the current scenario

Handsfree for Web at this moment works exclusively as a Google Chrome extension. Some of the supported features require permissions that only the browser can bring. i.e. tabs management

It is not my intention to change your project, as an example, here what I think you can implement using the current state of Handsfree for Web.

Create a plugin for moodle that turns on a voice command interface
When installing the plugin, if the extension is not installed on the user's browser, show the link or open the chrome extension page of Chrome Store
After installing the plugin, if the extension is running, allow the user to browse Moodle or execute some action of it via voice commands
Voice command definitions will be loaded into the Moodle template by the new plugin

At this moment I'm working on making Handsfree for Web a library, when it gets ready you will be able to load it inside any page, or Moodle, with just loading a javascript file, as everybody does with JQuery. This new version of Handsfree for Web won't require to be installed as a Chrome extension and it will support a subset of the commands that the current extension supports.

I think this is specifically what you need for your project, but sadly I cannot confirm that it will be ready for your deadline. What I can confirm is that the Voice Commands API will be pretty similar or equal, which means that if you define voice commands for the chrome extension they also work for the new library.

Thanks for reaching me out and play with the extension, I'll be here to help you if you decide to integrate Hadsfree for Web with your project.
Javier

Wolner

Hi Javi,

Sorry for the delay in responding. I have not received any email notifications of your reply.

Your suggestion is good. Even if it changes the course of the project, it is very interesting. I'll take it to my research advisor. Thank you.

As for your API, is there any way to help you with this project? Any PHP, JS or HTML files that need help? Would like to help!

Another thing: Before I got to know Handsfree for Web I was researching how to do voice recognition and came across the JS Web Speech API.
It may help you with something in the project!

If I do a voice recognition program from the start, my biggest challenge right now is to enumerate the HTML elements as you did. Can you help me in this part?

hugs

javi

Hi Wolner ,

As for your API, is there any way to help you with this project? Any PHP, JS or HTML files that need help? Would like to help!
Thanks for the offer, I think there are many ways you can help me to improve the extension/lib. At this moment I think you have touched an excellent point. Providing voice commands for Moodle, it is an excellent and valuable idea. Other kinds of improvements require I have to release the code as open-source, I've been doing this since a few months ago, and I plan to continue doing that, but it isn't everything public yet.

If you are interested in aspects of the current extension, just let me know, I'll see how I can release that piece of code earlier so you can access it and change it.

Another thing: Before I got to know Handsfree for Web I was researching how to do voice recognition and came across the JS Web Speech API.
It may help you with something in the project!
That's great! It's what Handsfree for Web uses.

If I do a voice recognition program from the start, my biggest challenge right now is to enumerate the HTML elements as you did. Can you help me in this part?
I agree it was a painful task. You can find that analysis in my thesis https://www.handsfreeforweb.com/img/tesis.pdf
Check the pages 93-138
It's written in Spanish, but I think you will be able to read it, Portuguese is a similar language.

It will be great if we can work together ?

Wolner

Hi Javi,

Sorry for the delay in responding. These days have been busy for me. But I'm closing a few cycles to dedicate myself more to new projects.

Yes, I am interested in helping you with this project. It would be great for my learning about voice command and HTML element manipulation. However, my knowledge of web programming is still under development. Still I want to collaborate !!

If you are interested in aspects of the current extension, just let me know, I'll see how I can release that piece of code earlier so you can access it and change it.

The current extension is excellent for my proposal with Moodle. If you can release parts of the code it will be extremely helpful for me as I want to make more progress in aplugin Moodle and currently, with the Web Speech API, I am only able to convert voice to string. It remains to enumerate the elements of the page and subsequently be able to perform actions by voice.

Thank you for your attention so far on my problem and also thank you for sharing your thesis. I will read very carefully. I also understand some Spanish so it won't be a problem.

It will be great if we can work together ?
I'm looking forward ?

Good night!

javi

Wonderful ?

Let's start with something "easy". I strongly recommend you read these two links

It should be enough to let you know how to add any kind of voice command to any website.

As a second step, I would recommend you to define the voice commands you want to support. e.g. "Go to my inbox", "Go to my profile", "Write a message", "Search course". And describe what it should happen after the voice command is executed. Sometimes the user needs to execute several voice commands to finish an action.

Have a great day!

Wolner

Great!

I will read all the material and absorb more knowledge so we can discuss more about it.

Thank you very much Javi.
? ? ? ?

Wolner

Hi Javi,

I learned a lot from the documentation and thesis content. Even chapter 5 really helped me a lot.

In my college project, I was able to develop experimental code to select page elements. I am integrating with the voice command "Click" and conducting experiments.

I would like to thank you again for the support you have given me. 😁

When it comes to Handsfree for web, I wish I could help select drag and drop elements, because I once tried this on a website and it was not possible. Making my test interaction limited. Could you show me the next step so I can help you?

javi

I'm very happy that you could learn from the docs and the thesis.

It would be great if you can implement the drag and drop feature.

Let's start for the voice commands part, what do you think the user will need to mention in order to execute a drag&drop? After defining the commands we can start working on the javascript functions.

Wolner

Right!

About drag and drop I would like to analyze two conditions that we have with the mouse: the click and the press and hold. We can see that these are two different contexts. Therefore, I suggest that the "name for the drag" be in the same context (root) as the "click".

The names I suggest to activate it is "Move, Drag, Catch or Take". In Portuguese I suggest "Mover or Arrastar".

That said, it would be interesting for the user to say "Move", so only drag elements are selected and enumerated. The user speaks "two" and element two undergoes the "press and hold action". Then the drop elements are automatically selected and enumerated. The user says "three" then the element is moved from position two to position three.

This is my perspective on the drag & drop situation 🙂

javi

So far so good, let me summarize what you have described

Drag&Drop voice command:

User says "Move"
HFW highlights draggable elements
User says "{Draggable element number}"
HFW highlights elements that are considered possible destinations
User says "{Destination element number}"
HFW moves the selected element to the selected destination

Contexts:

Root: Move
Pick Label: Element selections (draggable element and target element)
Drag and Drop: To be defined (We'll need it for sure)

Correct me if I'm wrong

The next step is to define the module 🙂

Wolner

You are right! 😃

I'm working on the module and as soon as I finish it I send here

Wolner

Hi Javi,

A help please!

Did you have trouble testing with the "Click" voice command? In the last test I noticed a certain SpeechRecoginition API deley to identify the "Click" command.
Do you know any way to "calibrate" the API for this and other words?

Any tips or suggestions are welcome.

javi

Hi Wolner!

Some times I have experimented these delays you are describing, most of the times it was because of network issues.
Could you review that first?

I'll push a new version of the extension that might improve the speed a bit, but I'm not sure if it will help you to detect "click" faster.

Thanks for reporting the issue

how the new module is going? I think I should be able to help you with the "select element" part. It's already implemented for the "click" and "select" commands.

Wolner

Hi Javi,

Excuse me. I must have expressed myself badly.
The deley I mentioned was for my college project. I was deploying the click code and noticed this problem.

I realized that in Handsfree for Web you used the understanding of "click" and "like" as "click". This helped me in my code too.

I am tweaking the new module as I progress in my college project. I currently have a "click" command that calls a function that selects all elements of the page.
My current challenge is to create the javascript context structure. I will not create any options for the user to add new modules, so I will not use the JSOn you used. I want to gradually evolve the application to better understand how it works.

I'll send you my speech recognition code and selection code (not so pretty because it's more for research, but once it's working, I'll improve the design).

I'm taking too long to reply to your messages, because this month I'm doing a code marathon ("IBM Behind the Code") to get the extra 60-hour college certificate. Until the 30th I will be in this delay. I'm sorry for this. But as soon as this date passes, I will pay special attention to our purpose.

javi

No worries about delays in posting here.

It would be great if you can share the code because you might be writing code for things that are already implemented and can be provided by the existent extension.

Good luck with the IBM project! hope you win 🙂

Wolner

As promised, here is the adapted English code:

https://github.com/Gwolner/colaboracaoPluginHandsfreeForWeb

When starting the index file, the default context is "root" and the voice object is created. Saying "click", the voice object is aborted and the contextClick () function is called.

I need to unify the selected elements of the selectAllPure () function with the number spoken in response to the contextClick () function. Even for that I need to turn both functions into one.
Here I would like your help with tips, suggestions, criticisms, etc.

To test the code, there is not much to say. Say a word without clicking to observe the behavior. Then say "click" and notice this behavior change.

Thanks for encouraging me in the marathon!

javi

Good morning @Wolner

I just took a look at the code and I saw a few things you can improve using existing libraries I already have released.

https://github.com/sljavi/web-speech-recognizer
Useful to connect with the Speech Recognition API, it simplifies the interaction

https://github.com/sljavi/text-sound-similarity
Useful to improve the words recognition (like you what did with "click" ₌ "like" but more complete)

https://github.com/sljavi/dom-element-types
Useful to identify elements that can be clicked. When you want to implement the drag&drop command I think we'll need to extend this module to recognize elements that can be dragged and where they can be dropped.

I like to see the progress you are making, keep me posted when you have more updates.

As a final word, you might be able to replace part of the current implementation as soon I can release the stand-alone version of the handsfree for web. (if you want to use it 🙂 )

Have a great day

Wolner

Hi Javi!

Thank you for your patience and attention. I will read the material and give feedback.

Thank you so much and a great day to you! 😁

Wolner

Hi Javi,

Good Morning!!

Questions so far:

Should these APIs be used only on the Linux server? I ask this because of the "npm" commands that do not run on Windows.

If I need to install the APIs on the server, are they available to any browser that receives javascript code from the server?
Or does the user also need to install it locally?

Sorry for the basic question, but my situation is a bit confusing because I develop on Windows and the Moodle server is on Linux. Would it be an occasion to develop directly on the server using these APIs?

I can develop on an Ubuntu VM, but can I run this code on Windows without running npm?