Institute of Computer Science,
Foundation for Research and Technology-Hellas (FORTH),
Science and Technology Park of Crete,
P.O. Box 1385, GR 711 10, Heraklion, Crete, Greece,
Tel: +30-81-391741, Fax: +30-81-391740
Email: as@csi.forth.gr, cs@csi.forth.gr
It is therefore evident that there is a need for better quality User Interfaces accessible by blind users and for the corresponding development support. In this context, the Dual User Interface concept has been defined as an appropriate basis on which the previous problems can be efficiently addressed. A Dual User Interface is characterized by the following properties: (i) it is concurrently accessible by blind and sighted users, (ii) the visual and non-visual metaphors of interaction meet the specific needs of sighted and blind users respectively (iii) the visual and non-visual syntactic and lexical structure meet the specific needs of sighted and blind users respectively (iv) at any point in time, the same internal (semantic) functionality should be made accessible to both user groups through the visual and non-visual "faces" of the Dual User Interface (v) at any point in time, the same semantic information should be made accessible through the visual and non-visual "faces" of the Dual User Interface. The first requirement is important for enabling cooperation between a blind and a sighted user and thus avoiding further segregation of blind people in their working environment. The second requirement explicitly poses the need of a metaphor specifically designed for blind users in the non- visual environment. The third requirement expresses the need for separate non-visual interface design. Finally, the fourth and fifth requirements indicate that the underlying functionality and information structures should be made equivalently accessible, though via different interactive channels.
A tool has been developed, called HOMER, which facilitates the development of Dual User Interfaces. It falls in the domain of language-based User Interface Management Systems (UIMS) and, apart from Dual User Interfaces, it supports the development of separate visual and non-visual User Interfaces as well. The HOMER system can integrate visual and non-visual lexical technologies and it supports concurrently communication with a visual and non-visual lexical technology at run-time. The dialogue specification is facilitated through the HOMER language. This language enables abstraction of the physical constructs, which are introduced through lexical technology integration in the HOMER system, to metaphor independent interaction objects. Consequently, sub-dialogues which can be described with such abstract objects are specified with shared descriptions for both the visual and non-visual environments. The Athena and the OSF/Motif widget sets have been integrated as visual technologies, while a simple toolkit for non-visual interface development, called COMONKIT, has been developed and integrated in HOMER. The Dual User Interfaces generated by HOMER can function in two modes: (i) local collaboration, where the visual and non-visual versions run on the same host (support to interface designers for handling keyboard sharing for this case is provided), and (ii) remote collaboration, in which case the visual and non- visual versions run on remote hosts.
A virtual interaction object can be constructed through the genesis mechanism. Construction of objects at the virtual level does not imply construction at the physical level, since no facilities are provided for extending the set of interaction objects supplied by a target platform. The HOMER system provides methods for integrating and utilising the physical constructs which are available. Engagement in dialogue specification of a virtual interaction object is enabled only after a physical instantiation scheme has been defined for the target environment(-s); for instance, Dual objects require both a visual and a non-visual instantiation scheme to be specified. In Example 1, the genesis of a Selector object is demonstrated at the virtual level, supporting one method called Selected and two virtual attributes, totalOptions and userOption. Following the genesis of the virtual class Selector, a visual physical object class is introduced, named VIMenu; physical attributes like foreground colour, background colour and font are defined.
dual genesis Selector [
int totalOptions;
int userOption;
method Selected;
constructor (int N) [
totalOptions=N;
userOption=-1;
]
destructor []
]
visual object VIMenu [
int N, userOption;
string fg, bg, font;
string* optionsList;
method VISelected;
...
]
Example1: Genesis of virtual objects and specification
of physical objects.
visual instantiation Selector [
simpleMenu : VIMenu [
-> {me}.userOption:={me}visual.userOption;
method Selected : VISelected;
constructor [
{me}visual.N={me}.TotalOptions;
]
-> ]
default=simpleMenu
]
1: {x}.attr {x}visual.attr {x}nonvisual.attr
2: method {x} Selected [ stmts ]
3: method {x}visual VIselected [ stmts ]
4: method {x}nonvisual NVselected [ stmts ]
Example 2: Specification of instantiation, access
to virtual / physical attributes and virtual / physical
method specification for virtual interaction objects.
In Example 2, the specification of the visual instantiation for the virtual Selector class is demonstrated. Multiple physical instantiation schemes can be defined for each environment. The example scheme is named simpleMenu and the physical class in which the object will map is the VIMenu. The block indicated between arrows is the description of the instantiation scheme named simpleMenu. Attribute associations can be specified either through constraints or monitors. In this example, a constraint enforces the virtual variable userOption to always have the value of the physical attribute userOption; the qualifier visual is used to distinguish physical visual attributes. Following, is the correspondence of virtual to physical methods. The virtual method Selected is associated to the physical method VISelected; when physical VISelected is called, the virtual Selected will be called as well. The last four lines show the syntax for accessing virtual or physical attributes and methods. The notation {X} provides access to the virtual structure of object X while the notations {X}visual and {X}nonvisual provide access to the visual and non-visual physical structures respectively. Since for each virtual object class it is allowed to specify multiple physical instantiation schemes, it is possible to have objects of the same physical class which are instantiated differently at the physical level. This is the first level of polymorphism (i.e. from the virtual to the physical level) enabled in HOMER. Moreover, the physical level can be modelled in such a way that it forms a generalisation over platforms instead of exactly "visualizing" the conventions used in the particular lexical technology being integrated.
This would practically serve as a second level of polymorphism. Hence, virtual objects are potentially characterized by a meta-polymorphic lexical nature. In Figure 1, the intermediate levels for physically instantiating a Dual virtual object are indicated.
Figure 1:The various intermediate levels for physical realisation of virtual interaction objects in HOMER.
1: agent Dbox create if (precondition)
2: agent Dbox (call parameters)
destroy if (precondition) [
interaction objects, event
handlers, methods, constraints,
monitors, functions, data
structure definitions, variables,
agents
constructor [ stmts ]
destructor [ stmts ]
]
Example 3: Specification of dialogue agents enabled
by precondition or by parameter passing, and constructs
internally to agents.
The skeleton of an agent specification in HOMER is indicated in Example 3. The first line shows the header of an agent instantiated by precondition, while the second a parameterized agent. Agents may optionally have a destruction precondition. Apart from objects, event handlers and children agents, the specification of arbitrary method implementations, constraints, monitors, functions, data structures and variables may take place within the definition body of agents. Also, communication events related to the application interface (e.g. creation of shared objects, receipt of messages in a channel, etc) can be also used as instantiation / destruction preconditions for agents. One way of achieving syntactic differentiation is via dialogue agents and virtual objects. Assume that task-structure differentiation is required for a particular simple task of deleting a file as shown in Figure 2. This scenario is implemented in HOMER via a single agent which owns all the objects indicated in Figure 2. Objects with a prefix VI, represent visual objects and those with prefix NV represent non-visual objects; the rest are dual objects. The designer defines the hierarchical structure in the context of the owner agent for both environments - notice that dual objects have two parents - and subsequently assigns the desired behaviour for each environments via corresponding method implementation. Figure 2:A simple scenario for task structure differentiation.
visual eventhandler DrawArea [
(pointDragged==true) PointerMoved [
// arbitrary code here which
// accesses PointerMoved.x,y
]
(true) PointerMoved [
!! ButtonPressed (PointerMoved.x,
PointerMoved.y,
1);
]
]
Example 4: Specification of a simple event handler.
The specification of a simple visual event handler is illustrated in Example 4. Two event blocks are shown for the same event class, thus demonstrating the parallelism for event processing. Also, the first event block has a simple precondition while the second event block indicates the way in which unconditional handling of events can be specified (i.e. setting (true) as the precondition). The statement of the second event block (!! ButtonPressed) concerns the generation of artificial events (i.e. local event broadcasting).
For instance, the object oriented approach which is based on the information structures manipulated by the functional core is more naturally realised through the shared space approach, while the (older) procedural approach of treating the application as a semantic server is more easily specified by means of direct messages which carry commands and parameters. The coordination scheme is asynchronous, but synchronous behaviour can be modelled; for instance, the dialogue control can wait for a notification that an action has been completed by the functional core and vice versa. Conceptual abstraction or adaptation of the internal structures and services is facilitated. In Example 5, specification of a simple application interface structure is shown.
enum Commands=[Del,Copy,Ren,Open,Protect];
channel CommandsChannel of Commands;
structEmployee [
string Name, Surname, Address;
real Salary;
];
shared Employee;
Example 5: Specification of application interface
structure.
nonvisual inputevent gestureSelect [
objectid obj;
]
visual inputevent LocatorMoved [
int x,y;
]
nonvisual outputevent Say [
string speechText;
]
visual outputevent DrawLine [
objectid obj;
int theGC, x1,y1,x2,y2;
]
Example 6: Specification of non-visual and visual
input / output events for non-visual and visual
technology integration.
In Example 7, the prefix DL indicates Dual virtual objects. The method Activated is a virtual method of the DLCommand class, and is called whenever the corresponding visual or non-visual instance is activated (i.e. enabling shared behaviour specification for both environments). As it has been previously discussed, such correspondence between virtual methods and physical methods is defined as part of the visual and non-visual instantiation schemes for virtual interaction objects.
agent DualDbox (
string msg,
bool* okSelected,
bool* cancelSelected,
string* userText ) [
object VisualPopupContainer viwin();
object NonvisualPopupContainer nvwin();
#define MY_PARENTS \
: visual parent=viwin \
: nonvisual parent=nvwin
object DLMessage theMsg(msg) MY_PARENTS;
object DLCommand theOk() MY_PARENTS;
object DLCommand theCancel() MY_PARENTS;
object DLTextEntry theText("") MY_PARENTS;
method {theOk} Activated [
*okSelected=true;
destroy myagent;
]
method {theCancel} Activated [
*theCancel=true;
destroy myagent;
]
*userText := {theText}.Content;
constructor [...]
destructor [...]
] // agent DualDbox end
Example 7: Specification of a Dual dialogue
box.
bool delOk=false,delCancel=false;
string file;
create DualDbox (
"Specify the file to be deleted",
&delOk;,
&delCancel;,
&file;);
delOk : [ // called when Ok selected
// behavior here for Ok
]
delCancel : [ // called when Cancel selected
// behavior here for Cancel
]
Example 8: Specification of dialogue for handling
Dual dialogue boxes.
Figure 3:The Dual run-time model.
The Application Server, the Dialogue Manager, the application interface (API) and the technology servers are implemented as separate processes. The lexical technology server interfaces (VI-TSI and NV-TSI) are not implemented as separate processes (as it is the case with the API which handles the shared space and message channels). Instead, the communication interface between the Dialogue Manager and the technology servers is realized on the basis of a general and efficient protocol (the technology server interface protocol) which explicitly introduces entities such as interaction objects and input / output events. The protocol implementation is provided to technology server developers by means of a class library which hides lower level communication aspects.
The HOMER system supports an integrated and powerful method for handling constraints, preconditions and monitors. Arbitrarily complex expressions can be used and automatic update propagation is efficiently managed. Constraints are uni-directional, however, since cycle elimination has been implemented, re-calculations are completely excluded (a cycle is eliminated exactly when it is created) and consequently bi-directional constraints can be safely modelled. Cycle elimination has been also incorporated to monitors. Constrains, monitors and preconditions may engage all types of variables (including aggregate types and pointer variables as well). The programming kernel which is supported in the HOMER language combines a plethora of C and Pascal features, while a flexible and simple model for dynamic memory allocation is provided as well. Regarding inter-component communication, even though the various components of the run-time architecture are separate UNIX processes, the HOMER system supports transparently to the dialogue designer, transfer of dynamic (linear or non-linear) data-structures either via the shared space or through the message channels. The HOMER compiler will generate code to decode / encode dynamic data structures, which are automatically monitored by the run-time system, to / from linear buffers when communication events are exported / imported. This method adds more expressive power to the notational methods for describing communication with either the application component or the technology servers.
Figure 4:llustration of the designed "rooms" metaphor for navigation in the non-visual environment.
Using HOMER, many Dual and dedicated non-visual prototypes of interactive applications have been built, like a payroll management system, a personal organizer and a simple "story teller" which includes textual descriptions of pictures and realizes a specifically designed scenario for non-visual picture exploration. A Dual version of the UNIX chess game is also underway. The second version of the non-visual environment, which is currently under development, combines 3D acoustics, 3D pointing and also speech input (recognition of keywords for navigation commands) in addition to the conventional keyboard (see Figure 5). An acoustic sphere illusion is provided to the blind user for navigation purposes which enables a direct perception of the structural aspects of the User Interface in contrast to the highly modal and synchronous approach of the present implementation.
Figure 5: A 3D acoustic non-visual environment with 3D
pointing and keyword speech recognition.
The lack of a non-visual toolkit posed the necessity of
explicitly developing one for subsequent integration in
HOMER. The COMONKIT library was developed, on the
basis of the designed "rooms" metaphor, with particular
emphasis on providing efficient navigation facilities. A
primary requirement for the development of the HOMER
system was the efficient support for interface development
as such, in order to enable high quality Dual, visual and
non-visual User Interfaces to be assembled.
Porting of the HOMER system to MS-WINDOWSTM and
integration of the InterViews library has been planned. Also,
another group will be studying a more effective metaphor
for non-visual environment than "rooms", and will build a
COMONKIT server for handling multiple clients for the
non-visual environment; this will include the implementation
of new efficient non-visual navigator handling all different
clients (the analogy of a visual window manager). The
HOMER system provides an effective framework for Dual
User Interface construction, it fills the gap due to the lack of
tools for non-visual User Interfaces and provides a ground
for further developments and experimentations.
SUMMARY AND CONCLUSIONS
Existing methods for providing accessibility for blind users
to Graphical User Interfaces suffer from many theoretical
and practical problems. It is argued that the main cause of
the present situation was the lack of tools for supporting
development of User Interfaces meeting the specific blind
user needs. In this context, the concept of Dual User
Interfaces was defined as the most appropriate basis for
integrating blind and sighted user groups. A UIMS has been
developed, called HOMER, which supports the development
of Dual User Interfaces. The HOMER system is platform
independent and it provides methods for integrating visual
and non-visual lexical technologies.
ACKNOWLEDGEMENTS
This work has been partially funded by the TIDE
Programme of the Commission of European Union (DG
XIII), under the projects GUIB (TP 103) and GUIB-II (TP
215). The partners of the GUIB consortium are: IROE-CNR,
Italy; Institute of Computer Science-FORTH, Greece; Vrije
Universiteit Brussel, Belgium; Department of Computer
Science-FUB, Germany; Institute of Telecommunications-
TUB, Germany; IFI, University of Stuttgart, Germany; VTT,
Finland; RNIB, England; F. H. Papenmeier GmbH&Co;, KG,
Germany.
