Accessing Smartbox Grid 3 using Python and win32gui

Summary

Smartbox’s Grid 3 communication software creates two windows containing the words ‘Grid 3’ in their titles, even though you can only see one. If you are trying to interact with this software using your own program, you need to make sure to access the window that you intend to.

Problem

I wrote some Python code to detect the use of Grid 3 or Tobii’s Communicator software for this project, to visually show when somebody who uses eyegaze technology interacts with the software.

This post concentrates on the issue I had with finding the correct window that Grid 3 runs in. Grid 3 runs under Windows.

I use the pywin32 library to access the win32gui library. This library allows me to find which window is running the software that I want to monitor. However, after using this library to find the ‘grid 3’ window, my code kept on telling me that nothing was changing in the window, when I could clearly see something was. To make matters more confusing, the code seemed to run fine on one machine and not another.

Solution

Please find the the parts of the Python script needed to explain my solution below. All of the script is on my GitHub site.

import logging
import win32gui

logging.basicConfig(
    format='%(asctime)s.%(msecs)03d %(message)s',
    level=logging.INFO,
    datefmt='%H:%M:%S')

COM_SOFTWARE = ['grid', 'communicator']
IGNORE = ['grid 3.exe', 'users']

def find_window_handle(com_software=COM_SOFTWARE, ignore=IGNORE):
    ''' Find the window for communication software. '''
    toplist, winlist = [], []

    def _enum_cb(window_handle, results):
        winlist.append((window_handle, win32gui.GetWindowText(window_handle)))

    win32gui.EnumWindows(_enum_cb, toplist)
    for sware in com_software:
        # winlist is a list of tuples (window_id, window title)
        logging.debug('items in ignore: {}'.format([item.lower() for item in ignore]))
        for window_handle, title in winlist:
            #logging.debug('window_handle: {}, title: {}'.format(window_handle, title))
            if sware in title.lower() and not any (x in title.lower() for x in ignore):
                logging.info('found title: {}'.format(title))
                return window_handle
    logging.info('no communications software found for {}'.format(com_software))
    time.sleep(0.5)

The critical debugging line is the commented out line 24:

logging.debug('window_handle: {}, title: {}'.format(window_handle, title))

When uncommented, and running the logging in debug mode, this listed out two windows that contained ‘Grid 3’ as part of their title, even though only a single Grid 3 window was visible. Even with just the ‘Users’ screen up, before launching a grid communication window, the logging.debug line returned two windows containing the name ‘Grid 3’ in their title:

grid: [(66532, 'GDI+ Window (Grid 3.exe)'), (197532, 'Grid 3 - Users')]

When running one of the Grids (for testing I used the Super Core grid), the software still tells me there are two windows with ‘grid’ in the title:

grid: [(66532, 'GDI+ Window (Grid 3.exe)'), (263256, 'Grid 3 - Super Core - .CORE')]

For this example, I could take the second window found and be done. However, to be robust, I created an IGNORE list, containing strings that are in the window titles that I do not want to use.

In the code example above, line 25 looks for the correct string to be in the window title and also checks that none of the strings in the IGNORE list are in the title:

if sware in title.lower() and not any (x in title.lower() for x in ignore):

This only passes the title for the window that I am interested in – the one containing the communication grid.

Testing

I use a Windows 10 virtual machine running in VirtualBox, with Debian Linux as the host. I also test on a separate Windows 10 only PC. I use a virtual machine for Windows for development as I run Linux on my laptop. The virtual machine allows me to create a static and controlled testing environment with only the software that I am working on in it. I double test on a stand alone Windows 10 machine in case the virtual environment somehow effects the software.

In this case, my script seemed to run well on one system and not another. I now suspect that sometimes the window that I was interested in was the only one generated by Grid 3 and at other times, the extra spurious Grid 3 window was generated as well. This spurious window was then selected by the software.

Running handShake in administrator mode to operate Grid 3

Sensory Software’s Grid 3 is a popular communication software package, running in Windows. Naturally, I would like handShake to be able to operate this software through the software keystrokes that handShake generates.  To get Grid 3 to respond to a software keystroke, I have to ‘elevate’ the base.py script which runs on the communication device to run as an Administrator.

There is a second solution. I can use a Freetronics Leostick USB dongle as a pretend keyboard and have this generate keystrokes that appear as coming from a physical keyboard. I did this for a while, but this adds a layer of complexity and expense to the project. The simplest solution is to run handShake as an Administrator when using Grid 3, or other software that requires software keystrokes to come from an elevated source.

I tested out adding the functionality for handShake to detect when Grid3 was running, then automatically try to elevate the base.py script to run in Administrator mode. I got this running. Then removed the functionality. Why? Security. Software running as Administrator can damage your system if incorrectly or maliciously written.

Now handShake detects if Grid 3 is running and advises that this requires the software to be restarted as an Administrator, but does not try to automate this restart. The decision is left to the user.

The Grid 3 software is detected using the code shown below. The code looks through the titles of the windows for any that match the ones in the list called ADMIN_SOFTWARE. At the moment there is only one title to check for – ‘grid’. This is the title for a window running Grid3. As I find other packages that demand that my script runs as Administrator to have the ability to interact with it, then I will add their titles to the ADMIN_SOFTWARE list.

ADMIN_SOFTWARE = ['grid']

def target_admin_sware(software=ADMIN_SOFTWARE):
    ''' Check if target software requires this script to run as Administrator. '''
    toplist, winlist = [], []
    logging.info('Looking for software that requires elevation to Aministrator: {}'
        .format(ADMIN_SOFTWARE))
    def _enum_cb(hwnd, results):
        winlist.append((hwnd, win32gui.GetWindowText(hwnd)))
    win32gui.EnumWindows(_enum_cb, toplist)
    for sware in software:
        # winlist is a list of tuples (window_id, window title)
        for hwnd, title in winlist:
            if sware in title.lower():
                logging.info('found software requiring Administrator mode {}'
                    .format(title))
                return True
    return False

Running handShake as Administrator is a choice that the user makes and implements if he or she deems necessary.  As all of the code is on the project GitHub site, the code can be reviewed to check that it is safe to run as an Administrator.

One of the advantages of open source projects is that they are open to this kind of scrutiny to find security flaws.

For interest, I detail the code I added and then removed from the base.py script to enable it to detect if it is running as an administrator and if not, request to be restarted as an Administrator.

I added the option to run the script as an Administrator from the command line using the click library.

click.command()

@click.option('-a', '--admin', default=False, 
    help='Run as administrator. Required for Grid 3.')

@click.option('-k', '--keystroke', default='F1',
     help='Keystroke to send. Default is F1.')

def main(admin, keystroke):
    logging.info('software keystroke is {}'.format(keystroke))
    logging.debug('admin flag is {}'.format(admin))
    if is_admin():
        logging.info('running as administrator')
    else:
        logging.info('not running as administrator')
        if admin:
            logging.info("restarting as administrator")
            elevate()
    service_microbit(keystroke) 

To test if the script is running as an administrator I added this method:

def is_admin():
    ''' Is the script running as an Administrator? '''
    try:
        return windll.shell32.IsUserAnAdmin()
    except:
        return False

Using Python to detect activity in Sensory Software’s Grid 2

Update: March 2018. This work is being submitted to the Communications Matters conference.

Following on from the eyeBlink post, with the help of Fil at Beaumont, I modified the algorithm I’m using to detect when the Grid 2 or Grid 3 software is being used. The image below shows Sensory Software’s Grid 2 software being used to construct a sentence. The new text appears in the white area at the top of the window. Fil suggested that I change the Python script to just monitor this area at the top of the window. The script now looks for a change in the amount of black text in this area. After the usual software wrangling I think I got it working. The Python script looks at the top 20% of the window and counts the number of black pixels in this area. Every half second it recounts the number of black pixels. If there is a change in the number of black pixels above a threshold, then a trigger is sent to indicate that the Grid software is being actively used. I’m using a threshold of 20 pixels, so there needs to be an increase or decrease of 20 or more black pixels for a change to be detected. This allows you to move your mouse cursor around in the text area at the top of the Grid window without triggering that there has been a change. The activity detection script needs more testing, but preliminary results seem to show it works. Prior to this, I was monitoring the entire Grid window and looking for a change in the whole window above a threshold. This led to false triggers when cells were selected, but not activated. When a cell is selected, the colour of the cell changes, even when it is not activated to produce text. This change in colour was being detected.

Each time we test the script, we find new ways to break it, leading to some ‘try except’ exception handling clauses. The script is designed to run on Windows as Grid 2 and Grid 3 only work on this operating system. I use the win32gui library to interact with Windows and the python imaging library, PIL (known as pillow), to do the image processing.
 
Sensory Software’s Grid 2 Chatterbox grid being used to construct a sentence:
 

Grid 2 communications software by Sensory Software used to create speech.

The latest code and installation details on how to get this running using the BBC microbit to give a flash when the Grid software is being actively used can be found on my github site at:
 
If you have any questions, please ask.