User Feedback and Offer to Contribute

freelyassociate

Thank you for creating Handsfree for web! There are many acute care patients at the Vancouver General Hospital in Canada without ability to use their hands that have enjoyed your project to browse Facebook, watch Netflix and play songs on Spotify. This has helped them pass time in the hospital instead of just sitting around and doing nothing so thank you very much for that! The following is a few bugs we have encountered for your feedback.

1) If a user says "OPEN website-that-does-not-exist.com" i.e., tries to open a website that's DNS does not resolve, the application currently crashes and they are unable to navigate away from the page using the plugin.

2) If a page has multiple scrollbars, there is currently no way to specify which area should be scrolled with the scroll command. An example of this is on Spotify, where you may have a scroll bar for your playlists on the left, as well as a scrollbar for the main page with the list of songs in that playlist. Note both the playlists section and songs section have to be big enough for the scrollbar to appear to reproduce this bug.

3) Would be convenient to stop listening for voice commands when a video or movie is playing, and only resume it after the video stops. This would avoid the problem of the voice recognition picking up a voice inside a video/movie. Currently, this can be worked around by starting a video and then putting the plugin to sleep and not waking it up until the video is done, but it would be nice if there was a simpler way to do this!

4) This is a feature request but many users have looked for an ability to create "custom commands". For example some users would like to map "Open Spotify.com" to a custom command like "Play my music".

Overall, users are very happy with your product and I can personally say it is one of the best voice control software that I have used. It has given patients hope and a feeling of being normal after major accidents and I cannot thank you all enough for your work.

In fact, I would love to help with the implementation of the above the features and bug fixes. I am a front end software developer with 5 years experience and will have the ability to spend around 9 hours a week on the project over the next 6 months to help improve the product and bring it to more users in the Vancouver community. I have seen the positive impact that this product can bring and if there is any way I can help contribute to the project please let me know.

javi

Thanks for posting such valuable feedback, this kind of message motivates me to continue supporting the project.

It would be great if we can work together in order to improve the tool, we can talk about it here or we can create a slack room for it. What do you think?

About the described issues:

We can create a command to support that behavior. At this moment you can open a website speaking two commands. i.e. "open" wait "facebook.com"
I think we can create a few alternatives commands for it. "scroll area" "{area_number}" "up | down | exit"
The problem with the proposed solution is that if you stop the recognition then you can't interact with the video until it ends. Voice commands like "pause", "mute", "rewind", "forward" will become useful and the user won't have a way to trigger them. I think the best solution here is to use a headset. What do you think?
I recommend you read https://www.handsfreeforweb.com/es/docs/ . It's not user-friendly but maybe we can think in a SDK where the user can create some basic modules with commands.

I'll more than happy to work with a FE developer, if you want we can talk via hangout in order to speed up the tool improvements.

freelyassociate

javi Thanks for the quick reply! I have created a slack room for us to use and I’ve sent you an invite at your gmail account. Alternatively, if you have a slack room already available feel free to send me an invite at klee2010@gmail.com.

I looked at the API docs for custom voice commands, and have started implementing a small custom command "scroll area" "{area_number}" "up | down | exit”. The boilerplate code is modelled after your zoom extension and is available here: https://github.com/leekevinyg/handsfree-for-web-scroll-area-module

A few questions:

I currently have a “scroll area” command in the root context, and when it is triggered I would like to trigger the element picker much like how the “click” command is currently functioning. Would you be able to help explain how the click command currently works or point me to the codebase where it’s implemented? I believe parts of the new command “scroll area” "{area_number}" "up | down | exit” will look very similar! (please correct me if I’m wrong).
I noticed that in the zoom module, you have i18n placeholders set up in src/index.js. Are you using a custom translation tool to substitute the translation text into the distributed files? If so, can you help me get ramped up on it? 🙂
Is it possible to access the chrome extension code code to run it in dev mode? If not, what would be the next best way to test new custom commands that require context to be passed to the “element picker” command?

I appreciate your time and taking the effort to get me ramped up. I am excited to get our collaboration process going and help get some new features and bug fixes out to the users!

javi The problem with the proposed solution is that if you stop the recognition then you can't interact with the video until it ends. Voice commands like "pause", "mute", "rewind", "forward" will become useful and the user won't have a way to trigger them. I think the best solution here is to use a headset. What do you think?

Good point! I will suggest the headset solution to the users for now. If I have any other ideas I will let you know!  

javi

Thanks for taking the lead and create the slack space. I just joined it.

Could you upload what you have to a github repository, It'd be easier for me to see what you have and suggest over the code the following steps required to get the desired voice commands.

About the questions
1 - You can call the label picker functionality returning the following object when calling to the voice command action.

{
      context: 'pickLabel',
      labels: 'div',
      selectedElementHandler: (el) => ({selectedElement: el})
 }

what you would get there is an action that asks the user to select which is the div that wants to scroll, and then, when a div is selected the element is stored as the selected one inside the state of the module. The next step is to create another context that is able to start working as soon as a div is selected. I think you have chosen the most complicated voice command to implement, but don't worry this is almost half of the work to do and I can show you the next steps.

Yes, there is a custom translation tool, it was implemented by myself so I can't share with you some docs yet. But as soon the voice command starts working I can show you how to use it if you don't have already discovered it.

In order to test the module, you can import it via the extension dashboard, let me know if you have some troubles.

Feel free to post here or reach me out via slack.

Thanks!

freelyassociate

javi Hi Javi, this is what I have so far https://github.com/leekevinyg/handsfree-for-web-scroll-area-module/blob/master/src/index.js

Currently, the module is responding to "scroll area" command from the root context, and then showing the element picker to the user. I have also created a "scroll-area" context that will be handling the "up" and "down" commands after the user has selected an area on the screen to scroll.

However, on user selection of an area, I am not sure how to how to pass the selected element to the scroll-area context, so that the up/down commands can be handled. Can you advise? Thank you!

javi

Glad to see you almost finish the module.

The missing part is something that isn't documented yet.

You just need to add the following prop to the 'scroll-area' context object. Basically it tells to the tool when to take control.

  switchOnSelectElement: (el) => el.tagName === 'DIV',

javi

I forgot to tell that as part of the action arguments you can find the selected element

i.e.

{
        name: 'down',
        action: ({selectedElement}) => {
          //scroll down the selectedElement
        },
        group: 'Scroll Direction',
        help: 'Scrolls a selected element down',
        switchToContext: 'root'
}

freelyassociate

javi Thank you! Just noticing one issue now when the label picker is highlighting all the divs on the webpage. As there can be quite a lot of divs on a webpage, this can make the number selection process quite messy. Furthermore, not everything that can be scrolled is a div (sometimes it's an unordered list for example).

Instead of making the label picker look for specific elements like a div or ul, is it possible to make the label picker only highlight scrollable elements on a page? That is, elements where the current scrollHeight is greater than the clientHeight? I think this will make the UI and interactions a lot cleaner, but I am not sure if it's possible. What do you think? Thanks again!

javi

I think it makes sense, and you are also correct, at this moment there is no way to select elements using such level of a custom selector it just accepts css queries. i.e. div.some-class

In the meantime, while I think how to support custom element selectors-

What do you think if before returning the object that asks for the pickLabel command, you walk over the elements and add a CSS class or another kind of attribute to those elements that can be scrolled? If you do that then you can change the current label from div to div.scrollable

freelyassociate

javi Good idea, the workaround works! This is what I have so far: https://github.com/leekevinyg/handsfree-for-web-scroll-area-module/blob/master/src/index.js

I am noticing a new issue where:

1) User says "scroll area" and scrollable areas on the page are highlighted
2) User selects a scrollable ul element to scroll using the element picker
3) Instead of the scroll-area context taking control, the text-actions context takes control.

This looks like a conflict in the switchOnSelectElement logic. Currently I have the following for the scroll-area context:

{
      context: 'scroll-area',
      switchOnSelectElement: (el) => el.classList.contains('hands-free-scrollable'),
}

hands-free-scrollable is the className that I am adding to scrollable elements on the page.

Can you advise on why the text-actions context is taking precedent over scroll-area? Thank you!

javi

You're almost there!

I just published a new version of the extension including the following changes:

If the selected element has the class name 'hands-free-scrollable' don't activate the text actions
Share a new parameter called tools when executing an action. The tool parameter is an object that shares relevant helpers or libraries such as: lodash, jquery, moment, domElementTypes and scroll.

I think you will find interesting two of these libs.

scroll: it implements four actions: up(el), down(el), top(el) and bottom(el). It uses jquery to animate the scrolling, there is also a second optional parameter you can use that is distance if you omit it a default distance is calculated based on the height of the element.
domElementTypes: You can see the docs here. https://github.com/sljavi/dom-element-types I think, you might want to use isVisible(el) and also it will be great if at some point we add a new method called isScrollable

Sometimes it takes time to Google Store to publish a new version of the extension, make sure of having the version 3.3.4 of the tool before looking for these changes

Have a great hands-free day 🙂

freelyassociate

javi Thanks Javi! The text actions are not activated now in the new version 3.3.4, and I am able to execute the up/down scroll actions. I am trying to integrate the scroll tool into the action at the moment and have tried passing both tool and tools as action parameter in the following way shown below, but it does not seem to be working. Can you give me an example of how the tools parameter is intended to be passed? Thank you!

{
        name: 'down',
        action: ({ selectedElement, tools }) => {
          tools.scroll.down(selectedElement);
          //selectedElement.scrollTop += 100
        },
        group: 'Scroll Direction',
        help: 'Scrolls a selected element down'
      }

Note the above snippet is from this line inside the scroll-area context: https://github.com/leekevinyg/handsfree-for-web-scroll-area-module/blob/cc54c15294fd87a2cbc62c1d5a9d245e96e3adf9/src/index.js#L48

javi

I just checked the latest version of the module and I can confirm that the scroll object is inside the tools parameter as your code shown.

I have found another issue that is preventing the scrolling. The selectedElement parameter has to be inside a jquery wrapper in order to play correctly with the scroll helpers. I'll change the implementation of it now to avoid requiring the wrapper and I'll release a new version of the extension but it will take a while until it gets published. In the meantime I recommend you change the action to

action: ({ selectedElement, tools }) => {
  tools.scroll.down(tools.jQuery(selectedElement));
},

I recommend you add a property called name to the object that defines the context 'scroll-area'. It will replace the Global label that is next to the mic picture after selecting an element that can be scrolled.

context: 'scroll-area',
name: 'scroll area'
switchOnSelectElement: el => el.classList.contains('hands-free-scrollable'),

Something to think about, what do you want to happen if there are no elements that can be scrolled? Maybe you can default to the entire body in that case?

I'll think about how you can ask the extension to render the allowed voice commands as a list at the bottom of the screen. I think the list would be [Up | Down | Exit]

freelyassociate

javi Thanks for the suggestions! I have added a name for the scroll-area context, as well as your fix for the scroll issue. The latest code is here: https://github.com/leekevinyg/handsfree-for-web-scroll-area-module/blob/master/src/index.js

As for the case where no elements can be scrolled, do you think it would make sense to just return the root context?

javi I'll think about how you can ask the extension to render the allowed voice commands as a list at the bottom of the screen. I think the list would be [Up | Down | Exit]

Ok sounds good. On a related note, would it be useful to show error messages in red on the bottom of the screen as well if possible? For example, another solution to the "no scrollable elements found" issue you mentioned above can be to show a red toast message on the bottom of the screen that says "no scrollable elements found", and then just return back to the root context. Just an idea! Let me know if you think it will be useful and I can help implement it with your guidance 🙂 Thanks again for you time.

javi

Glad to see the module is working 🙂

I think it'd be better to focus on the user's intention, for some reason he wants to scroll something, if we detect that there is no area to scroll, we can suppose that he wants to scroll the entire screen.

I just released a new version of the extension including the latest changes and the option to show the supported voice commands of the current context.
https://twitter.com/javipzv/status/1192092962362155009

You can tell the tool to show the supported commands returning the property showCommandList: true when executing an action (up and down), but if you do that you will see the command list only after executing some of these commands. What you really want is to see the list as soon as the context scroll-area is activated to do that you can use the setup hook.

i.e.

      context: 'scroll-area',
      switchOnSelectElement: el => el.classList.contains('hands-free-scrollable'),
      setup: () => ({ showCommandList: true })

This new feature will be supported by v3.4.0, hope Chrome Store will make it public in the following hours.

javi

Hi @freelyassociate, could you check the latest changes?

What do you think is the next step to implement? We can internationalize it and then include it within the supported modules so everybody can use the new commands.

Have a good day

freelyassociate

javi Hi Javi, sorry for the late reply! I have been busy with moving to a new place this last week. I will take your suggestion to scroll the main page when multiple elements are not present on the screen, as well as add in the showCommandList action. I will do this tonight and have it in by tomorrow morning 🙂

freelyassociate

Hi @javi, the latest changes are available at this link!

When no detectable scrollable elements are found we still go to the scroll-area context, and try to scroll the body. However in my testing this doesn't seem to be working on all pages, whereas the default "scroll down" command behaviour from the root context does work. Is it possible to trigger the default "scroll down" command behaviour from this line here?

Also, please advise on what I can do to help with internationalization to help get this module out to the public!

Additionally, I am not sure if this will be an issue when we release the module, but it seems that while the src/index.js file is able to be imported properly into the extension, trying to import the minified file under dist here results in Error when trying to import the module. Seems that module structure is invalid. Will this be an issue, and if so can you advise on how to fix it?

One last thing - I added an isScrollable helper to the dom-element-types tool and created a pull request here. Feel free to let me know if anything there needs to be fixed up!

Thank you 🙂

javi

I'm excited to see the progress here 🙂

I have posted a comment in your repository to fix the scroll issue of the body element and I reviewed the PR of isScrollable

I have published a repository including all the voice command modules that don't require access to the chrome extension permissions.
https://github.com/sljavi/handsfree-for-website-modules

At this moment I'm working on adding docs and tests to the repository, I hope the docs will be ready for Monday so you can read about the internationalization feature. In case you want to try to guess how does it work, as a short clue I can say that

module.name and module.description will be translated based on what module.i18n[selectedLang] defines.
module.context[index].actions[index2].name is translated based on what module.context[index].i18n[selectedLang] defines.
keep in mind that if a name must be translated it has to start with i18n-. i.e. 'i18n-scroll.command' otherwise, it isn't translated

My idea is to load the voice commands from two repositories "handsfree-for-website-modules" and "handsfree-for-web-modules". I'm also working on a new version of handsfree for web that can be loaded by any developer in a website, like jquery. As soon as it gets loaded allows the user to browse the page handsfree and also will allow the developer to add custom voice commands on the fly.

So in this new scenario,

Handsfree for web loads handsfree-for-website-modules and handsfree-for-web-modules
Handsfree for website loads handsfree-for-website-modules

I'll keep you posted,
Thanks!

freelyassociate

javi

javi I have posted a comment in your repository to fix the scroll issue of the body element and I reviewed the PR of isScrollable

Thank you, I updated the PR based on your comments! Just one more open question/issue here regarding the scroll issue with the body element.

I will start preparing a PR for the "scroll area" command in your new repository with the valid i18n strings. Is there anyway I will be able to test the translations before I make a PR? Thanks again.