One of the projects I’ve been working on for the last week or so has been particularly vexing. Learning to use Vim and switching to a tiling window manager (Xmonad) has left me thinking it would be great if I never had to touch a mouse (or trackpad, or nubbin) at all - think how much faster I might be if I didn’t have to move on and off the keyboard! But mice are a useful innovation as well. Perhaps they haven’t quite lived up to Doug Engelbert’s hopes as far as making us one with our computers, but they definitely make a lot of interactions more intuitive. And purely keyboard-based navigation is often frustrating for a number of reasons (as I have been repeatedly reminded during the last week of Vim-transition-pains). It seems like there must be a better way.
The really obvious ‘better way’ - at least to me - is this: Whenever I’m focused on a specific window, or attempting to click on something, I’m looking at that spot on the screen. This is more or less the point of windows and screens - if I don’t need to look, why have I not automated the task entirely? So rather than using a separate device (i.e. a mouse) to keep track of what I’m focused on, wouldn’t it be nice to just automatically focus on whatever window I’m looking at?
There have been a few attempts in this direction. In particular, Tom Ballinger (one of the Hacker School facilitators) has done a proof-of-concept pupil tracker that attempts to move the cursor to wherever the user is looking. Unfortunately I was unable to get it working, so I forked the repo to switch over to openCV 2 (and add linux pointer scripting with xdotool). That runs, but the pointer jitters considerably and has some other issues (eg strongly preferring positions along the diagonal from top left to bottom right)… After playing with that for a while, I decided to opt for a different approach.
Since most people have two eyes, and they are a constant distance apart for any particular individual, intuitively it seems as though it should be possible to draw a vector from the center of the eye to the pupil for each eye, then take those two vectors and the vector between the centers of the eyes and triangulate to a particular location on the screen (presuming the user’s face is at a more-or-less constant angle relative to the screen). I gave this a shot - in C++ rather than python this time - and had some success; I can find the face, find the eyes within the face, and find the pupils within each eye. But I can’t find a reliable difference between the center of the eye and center of the pupil. This has been vexing me for quite some time. Part of the issue is that Haar classifiers are essentially a black blox to me, and I don’t fully understand how to tune them for this application aside from using some of the training data distributed with OpenCV. Part of the issue may be with the webcam on my laptop, which was not exactly top-of-the-line five years ago when I got it. I’m doing pupil detection manually (by looking for circular regions of darkness in the eye region of interest), so that much I at least know is working(ish), but the exact mechanism by which the cascade classifier finds eyes in an image remains a question mark in my mind. My impulse is to keep pounding my head against the problem until it works, but (as the title of the post suggests), I don’t think that’s an effective use of my time. I may return to this project, or go back to working on my fork of Tom’s Python version and try to tune way the jitter, but for today I need a change of pace, so I’ve been working on two other things.
The first of those two other things is my attempt at implementing Veness et alia’s Monte Carlo approximation to AIXI in Clojure, which I started earlier in the batch, but have mostly not followed up on. This is an interesting project to be sure, but the going is slow there as well for a number of reasons involving the relative sizes of my eyes and stomach. But as Browning put it, “a man’s reach should exceed his grasp, or what’s a heaven for?”
In any case, after spending several hours being frustrated by the above, I decided that what I really need to do to get myself feeling useful again is to work on something that I can definitely finish in one day, or even one hour. That something is the subject of my next post, which is shortly to follow.