Accessing Smartbox Grid 3 using Python and win32gui

Summary

Smartbox’s Grid 3 communication software creates two windows containing the words ‘Grid 3’ in their titles, even though you can only see one. If you are trying to interact with this software using your own program, you need to make sure to access the window that you intend to.

Problem

I wrote some Python code to detect the use of Grid 3 or Tobii’s Communicator software for this project, to visually show when somebody who uses eyegaze technology interacts with the software.

This post concentrates on the issue I had with finding the correct window that Grid 3 runs in. Grid 3 runs under Windows.

I use the pywin32 library to access the win32gui library. This library allows me to find which window is running the software that I want to monitor. However, after using this library to find the ‘grid 3’ window, my code kept on telling me that nothing was changing in the window, when I could clearly see something was. To make matters more confusing, the code seemed to run fine on one machine and not another.

Solution

Please find the the parts of the Python script needed to explain my solution below. All of the script is on my GitHub site.

import logging
import win32gui

logging.basicConfig(
    format='%(asctime)s.%(msecs)03d %(message)s',
    level=logging.INFO,
    datefmt='%H:%M:%S')

COM_SOFTWARE = ['grid', 'communicator']
IGNORE = ['grid 3.exe', 'users']

def find_window_handle(com_software=COM_SOFTWARE, ignore=IGNORE):
    ''' Find the window for communication software. '''
    toplist, winlist = [], []

    def _enum_cb(window_handle, results):
        winlist.append((window_handle, win32gui.GetWindowText(window_handle)))

    win32gui.EnumWindows(_enum_cb, toplist)
    for sware in com_software:
        # winlist is a list of tuples (window_id, window title)
        logging.debug('items in ignore: {}'.format([item.lower() for item in ignore]))
        for window_handle, title in winlist:
            #logging.debug('window_handle: {}, title: {}'.format(window_handle, title))
            if sware in title.lower() and not any (x in title.lower() for x in ignore):
                logging.info('found title: {}'.format(title))
                return window_handle
    logging.info('no communications software found for {}'.format(com_software))
    time.sleep(0.5)

The critical debugging line is the commented out line 24:

logging.debug('window_handle: {}, title: {}'.format(window_handle, title))

When uncommented, and running the logging in debug mode, this listed out two windows that contained ‘Grid 3’ as part of their title, even though only a single Grid 3 window was visible. Even with just the ‘Users’ screen up, before launching a grid communication window, the logging.debug line returned two windows containing the name ‘Grid 3’ in their title:

grid: [(66532, 'GDI+ Window (Grid 3.exe)'), (197532, 'Grid 3 - Users')]

When running one of the Grids (for testing I used the Super Core grid), the software still tells me there are two windows with ‘grid’ in the title:

grid: [(66532, 'GDI+ Window (Grid 3.exe)'), (263256, 'Grid 3 - Super Core - .CORE')]

For this example, I could take the second window found and be done. However, to be robust, I created an IGNORE list, containing strings that are in the window titles that I do not want to use.

In the code example above, line 25 looks for the correct string to be in the window title and also checks that none of the strings in the IGNORE list are in the title:

if sware in title.lower() and not any (x in title.lower() for x in ignore):

This only passes the title for the window that I am interested in – the one containing the communication grid.

Testing

I use a Windows 10 virtual machine running in VirtualBox, with Debian Linux as the host. I also test on a separate Windows 10 only PC. I use a virtual machine for Windows for development as I run Linux on my laptop. The virtual machine allows me to create a static and controlled testing environment with only the software that I am working on in it. I double test on a stand alone Windows 10 machine in case the virtual environment somehow effects the software.

In this case, my script seemed to run well on one system and not another. I now suspect that sometimes the window that I was interested in was the only one generated by Grid 3 and at other times, the extra spurious Grid 3 window was generated as well. This spurious window was then selected by the software.

Give Me a Minute idea in use in commercial AAC software

To understand this post, please read the first couple of paragraphs of my give-me-a-minute project page.

The idea presented in give-me-a-minute looks to be making it into commerical AAC products. I don’t pretend that my project influenced this development. Be nice to think it did though… I’ll come back to reality now.

One implementation is as ‘partner windows’ where the text being composed on an AAC display is mirrored to a second screen that faces away from the person composing the test. A pattern that indicates that text is being composed can be displayed instead. This can be easily seen by somebody facing the composer. The partner windows are incorporated in some of the I-series AAC devices made by Tobii Dynavox.

When I presented Give Me a Minue at Communication Matters in 2019, the father of somebody who uses AAC approached me with a concern over the idea. His son sometimes wants to compose private text and not to have anybody else know that he is doing this. Having the text replicated completely removes this privacy. Having a pattern indicating that the device is in use, but not giving away the content, is a step to preserving privacy. Give Me a Minute displays a pattern. The I-series has an option to display a pattern of bouncing dots instead of text. In both cases some privacy is maintained until the final text is delivered as speech. However, even having this pattern on the display gives away some privacy. The father of the AAC user who talked to me indicated that he would want to be able to turn the feature on and off easily.

As I have no access to the AAC software, I don’t have the ability to add this extra feature to Give Me a Minute. The instructions for the partner window software on the I-series is here. It looks like the only way to turn the indicator window on or off is to go through the settings menu.

The Twitter feed where I learned of these developments is here: https://twitter.com/TinaBoardmaker/status/1278723338907660291. My contributions are under the name ‘Oppy’.

Therapy Box plan to incorporate their idea in the next release of their predictable AAC software.

Using the microbit to control switch access software

What is switch access software

Many disabled people use specialist software to create speech or to interact with the environment (e.g. turn on lights). Some are unable to use keyboards or mice to operate this software, so use a variety of ‘switches’, such as push buttons. These buttons act like keys on a keyboard, or pretend to be a mouse click.

I took some hand held video of a switchable software package which enables speech to be created. The software is called Liberator. A big red button was configured as the switch controller. When the button is pressed, a row is highlighted. The highlighted row scans down. A second click selects a row. The software now scans across the single cells in the selected row. A third click now selects that cell and the text for that cell appears in the speech window. Sometimes a cell will lead to a new grid. Once the speech text is composed, a cell can be selected for the text to be sent to a speaker. I tried this out at the Communication Matters conference in Leeds.

Trying out the Linberator switch access software at the Communication Matters Conference.

Wikipedia has a page (of course) explaining what an assistive technology (AT) switch is here. Copying the one line summary at the start of that page: “A switch is an assistive technology device that replaces the need to use a computer keyboard or a mouse.”

The system we developed allows the BBC microbit to pretend to be an AT switch, so that movement sensor devices I make using the microbit can be used to control switch accessible communication software. For instance, Grid 3 by Smartbox.

The picture below shows a typical button switch and a Joybox switch to USB adapter. The adapter enables the button switch to be connected to a USB port. This allows the button to act as if a key is pressed on a keyboard. This simulated keyboard key press then controls software, to e.g. create speech. The standard connector for a switch is the venerable 3.5mm audio plug. The 3.5mm plug is on the end of the cable attached to the button switch. A 3.5mm socket is attached to the USB to switch converter.

My task was to enable a microbit board to connect with a 3.5mm plug and act as a switch, so that the signal would be recognised by the switch to USB adapter. How hard could this be?

Button switch and interface dongle.

How does the switch work? The 3.5mm connector has 2 contacts inside of it. When the switch is operated, these are connected together internally. So, the contacts are normally open and closed when the switch is pressed. How do I recreate this switch electronically?

I used a Grove M281 Optocoupler Relay.

Grove M281 optocoupler relay board. The control pins are on the left, the relay pins are on the right.

This acts as an electronically controlled switch. When the CTR pin on the board goes high, the two connectors with the screws on top are connected. When the CTR pin is low, they are disconnected. The CTR pin can be seen on the left of the photo. There are connections for ground (GND), power (VCC) as well. The NC pin is Not Connected.

I could maybe lash up something cheaper using a transistor or two, but for around £6 I had an off the shelf solution that I got tested and running within a day. The microbit connects to the pins on the left of the board in the photo. The 3.5mm plug connects to the screw top terminals on the right of the photo.

The advantage of using optocoupler is that the microbit is isolated from the communication device that the 3.5mm plug is connected to. My slight worry was creating a ground loop. If I didn’t have any isolation between the microbit and the 3.5mm plug, if the microbit is powered from a USB source – say another computer – and then the microbit is connected to a communication device that is also connected to the mains, we may create a ground loop. The optoisolator prevents this. I don’t think this is a likely scenario with the tiny currents involved, but I am working with a vulnerable user group, so am a little more cautious than usual.

The optocoupler relay board is specced at 3.3V, but worked with the 2xAAA battery pack powered microbit at 2.4V. Nobody is more surprised than I am when something I build works!

The photo below shows all of the parts of the system, apart from the switch to USB adapter, shown in the photo at the top of the page. The microbit board slides into a Kitronik edge connector and break out board:

https://www.kitronik.co.uk/5601b-edge-connector-breakout-board-for-bbc-microbit-pre-built.html

microbit board slid inside a Kitronix break out board connected to a solid state relay board connected to a 3.5mm plug. The plug is unscrewed so you can see the two connectors.

The Kitronix break out board allows all the signal pins on the microbit board to be accessed. I used digital pin 16 on the microbit board to connect to the CTR pin on the M281 board, as it allowed for the neatest wiring. The photo below shows the wiring on the right. Pin 16 connects to the yellow wire. Ground is the black wire and the 3V output is connected to the red wire. Ignore the connectors and resistor on the lower left of the photo – these are used for connecting a motion detecting sensor, which I will write up on a different post.

The two screw top terminal connectors on the M281 are connected to each of the two contacts in the 3.5mm plug. The wires connected to the plug by soldering and are connected to the M281 by screwing down the terminals.

I wrote a small program to toggle pin 16 on the microbit high to simulate the action of the sensor detecting a hand motion, which is the action we would like to use to trigger the switch.

Pin 16 is connected to the solid state relay board using the yellow wire. The red and black wires connect to +3V and ground pins.

The 3.5mm plug goes into the socket of the interface dongle shown in the photo at the top of the blog. The dongle plugs into your PC on a USB port. The dongle is recognised as a switch interface using the free software at:

https://thinksmartbox.com/product/switch-driver-6/#

I used Windows 10 to check that everything works as it should. Which it did. Screen shot of the software below. Hurrah.

The final code used for this project can be found on my github page in the hand_wave folder.

Smartbox Switch Driver software used to test the switch.

One final Top Tip is to replace the AAA battery pack that comes with the microbit with one that has a power switch. These are about £4 + £0.75 postage from eBay. The title for the switched battery box I bought is ‘Switched battery power box for BBC Micro:Bit 2 x AAA’.