Modern Touchscreen Experiment

A Practical Experiment with Modern Touch Screen Technology

In recent years touch screen technology has advanced such that new human computer interaction is possible. This prototype experiment has been devised to gain a better understanding of how this technology could be used in the Transport Travel & Logistics industry. It was decided to investigate a touch screen monitor complete with 3D gestural software to see how it might be used on a Gantt based problem. A limited budget existed and readily available components were used.

The objectives of this experiment were:

• Investigate the capabilities of a touch screen monitor.
• Investigate the capabilities of a 3D gestural software package with a view to understanding what kind of skills are required of a developer to use it.
• Observe user interaction with a touch screen device configured to present a Gantt based data.

Hardware

A number of manufactures are now producing low cost touch screen monitors:

• HP L2105tm 21.5-inch
• Dell SX2210T 21.5-inch
• ViewSonic VX2258wm 22-inch
• Acer T230H 23-inch

After reading up about these devices on the Internet, it was concluded that at this low end of the touch screen market there wasn't very much difference between hardware manufactures. The major disappointment is that despite how these devices are marketed, all are dual-touch as opposed to multi-touch. It was only on reading up on their technical specs that this important point is revealed.

In the end it was decided to go with Acer T230H due to its slightly larger size, good reviews and price.

Acer T230H

A look around the back of the T230H is interesting. There are the usual analogue and digital video inputs along with an analogue stereo input jack for the built in speakers. However what is different is a USB port, which is used by the monitor to send the touch signal to the computer.

A closer look around the edge of the screen reveals a raised bezel that houses an infrared camera that is used to capture the on-screen finger movements. In the above picture a pen is placed on the screen in order to illustrate the raised height of the bezel.

Operating System

Windows 7 was chosen as the Operating System. It automatically recognized the Acer T230H and enabled additional touchscreen specific user interfaces such as an on-screen keyboard.

Software

Two software packages were considered for this experiment; Sparsh & MT4J.

Sparsh

Sparsh (Hindi for "touch") has been developed by Iowa State University's Virtual Reality Applications Center (VRAC). It is a multi-touch API for enabling users to create multi-touch applications on a variety of hardware platforms. The API supports custom hardware drivers and is platform independent. Sparsh is written in C++ with a Java wrapper.

Here is a link to the Sparsh project page: http://code.google.com/p/sparsh-ui/

Sparsh in action:

MT4J

Fraunhofer-Institute for Industrial Engineering, Germany, has developed MT4J as an open source Java framework, created for rapid development of visually rich applications. MT4j is designed to support different kinds of input devices with a special focus on multi-touch support.

Here is a link to the MT4J project page: http://www.mt4j.org

MT4J in action:

Choice of Software

For the experiment MT4J was chosen over Sparsh as it appeared to be easier to set up.

Custom Test Code

MT4J is a general purpose multi-touch library and as such it does not have any support for schedule or roster display therefore in order to test the capabilities and appropriateness of multi-touch technology for Constraint Technologies, it was required that some custom test code was written.

Above is an image of a simple Gantt pane, created using MT4J. The user is able to move, rotate and resize the Gantt as well as selecting individual roster allocations or lines. The Gantt's content is randomly generated and there are no legality checks. The black right-angled piece in the upper left-hand side is a "handle" that is meant for the user to hold on to whilst moving the Gantt pane.

MT4J relies heavily on OpenGL and after writing this custom test code it was clear that a developer with strong OpenGL skills is required to develop further code for MT4J.

Final Platform

In summary the following diagram illustrates the final platform used in this experiment.

Results

Below is presented a series of images that document the experiment.

The first finding was that when the Acer T230H was mounted at a right angles to the bench, wrist fatigue was a problem. In the above image the Acer T230H was mounted inside the polystyrene box it came in. With this setup, wrist fatigue was no longer an issue. The additional height provided by the polystyrene box made the device easier to interact with.

MT4J allows for multiple touch frames. In the above images we can see a single touch frame. In the first image the touch frame has just been opened and is centred in the middle of the screen. In the second image the same touch frame has been slightly rotated and enlarged by a user.

MT4J comes with only a few high level widgets:

• Text Box
• Slider
• Touch Frame
• Keyboard
• Button

The following basic widgets were missing:

• Radio Buttons
• Drop Down List
• Menus

The lack of these widgets would require a developer to implement them, hence increasing the cost of a project.

In the above pair of images we can see two keyboard widgets. In the first image only one user is interacting with the system; however in the second image a second user has joined in. Neither user is advantaged or disadvantaged by their position. Both users can also see what the other user is doing. This is a clear example of collaboration.

In the above image we see two users interacting with the system. However, unlike the previous image these users are both performing a different task as denoted by the separate touch frames. One user is performing a typing task whilst the other is viewing a roster Gantt. You should observe that with two users the device has become crowded. It was found that at the three-user level the device was saturated due to the lack of available on screen space. Clearly the T230H's 23.5 inch screen is too small for a collaborative device.

In this image we see the expanded touch frame containing the simple Gantts from the previous image. The touch frame now occupies the entire screen. This illustrates how views can be controlled in MT4J.

This set of images above illustrates the problem of finger size. In the first image the Gantt has been scaled to such a size that a finger can accurately select a roster allocation. This is not the case in the second image, where you will observe that the finger is obscuring a number of buttons whilst making a selection. Therefore average finger size is a limitation to the size of selectable items and therefore the amount of information that can be displayed. It is worth noting that haptic feedback could assist in this situation.

In the above image we can see an example of a user selecting, rotating and moving a Gantt pane to a position that is closer to him.

The above three images illustrate the pinch and stretch gestures. In the first image the user is about to pinch the screen. This results in the Gantts zooming out as illustrated in the second image. The final image is an example of a stretch gesture zooming in on the Gantt panes.

In these final images we can see an example of the two-fingered stroke gesture, which results in the workspace moving in the direction of the gesture.

It was found that MT4J had excellent basic gestural behavior. MT4J also supports customised gestures, which offer the ability to add specific gestures to any customised components created. Further to this MT4J supports nice transitions such as dissolve, sweep and flip. These are easy to code and the results look good.

Feedback from Users

The final part of the experiment was to invite colleagues to try the prototyped system. These are some of the observations from their interaction with the prototype:

• Initial "wow" factor, which lead them to wanting to use the system.
• Evidence of collaboration when it came to helping each other use the system.
• Intuitive navigation and exploration.
• Frustration when they realized that only two fingers could be used. This was particularly noticeable when two people used the system.
• Frustration when the hardware missed touches. I am not sure if this was caused by hardware or software problems. Wrist, palm and sleeve interference did explain some of these missed touches.
• Screen looked filthy when switched off but surprisingly clean when switched on.
• Hardware performed well. The experience was smooth and there was no lag in the 3D rendering.

Conclusion

This experiment taught Constraint Technologies a lot. It was surprising how quick it was to develop an application using this technology. MT4J was quite advanced in its 3D capabilities and gestural processing. The primary concerns with MT4J were its lack of high-level widgets and automated testing framework. The developer found that his biggest problem was his lack of OpenGL experience. He found it frustrating at times to implement his concepts. Once he had learnt the necessary OpenGL required to implement a concept, he was surprised how brief yet elegant the code segment was.

The Acer T230H was very limited in its input capabilities and was very prone to finger errors. For multiple users it is important that the device is large enough to fit a number of people around.

It has been proposed that a similar experiment should conducted with a tablet computer, such as an iPad, to better understand the capabilities and user experience of a device that whilst being touch enabled, is also small and meant for personal computing as opposed to collaborative computing. Colleagues that used the prototype were impressed with the user experience. In particular they liked the smooth use of 3D graphics. Therefore any project that was to develop a product using touch-screen technology should include a developer with OpenGL experience and an industrial designer with user interface experience.