A generic and visual interfacing framework for bridging the interface between application systems and recognizers

Abstract

Application systems that utilize recognition technologies such as speech, gesture, and color recognition provide human-machine interfacing to those users that are physically unable to interact with computers through traditional input devices such as mouse or keyboard. Current solutions to interface application systems with recognizers, however, use an ad hoc approach and lack of a generic and systematic way. The common approach used is to interface with recognizers through low-level programmed wrappers that are application dependent and require the details of system design and programming knowledge to perform the interfacing and to make any modifications to it. Thus, a generic and systematic approach to bridge the interface between recognizers and application systems must be quested. In this research work, we provide a generic and visual interfacing framework for bridging the interface between application systems and recognizers through the application system's front end, applying a visual level interfacing without requiring the detailed system design and programming knowledge, allowing for modifications to an interfacing environment to be made on the fly and more importantly allowing the interfacing with the P party applications without requiring access to the application's source code. Specifically, an interfacing script language for building the interfacing framework is designed and implemented. The interfacing framework uses a transparent grid layout mechanism to position the graphic user interface icons defined in the interfaced application system. The proposed interfacing framework is then used to bridge the visual interface commands defined in application systems to the voice commands trained in speech recognizers. The proposed system can be applied to GUI based commercial software without the need of accessing their internal code, and also allowing the composition of macros to reduce interaction overhead to users. Examples are applied using the proposed interfacing mechanisms to demonstrate the applicability and feasibility of the proposed visual interfacing approach.