Looking and Lingering as Conversational Cues in Video-Mediated Communication

GAZE AND VIDEO-MEDIATED COMMUNICATION

Another approach takes a psycholinguistic perspective on VMC, testing specific hypotheses regarding visual contributions to mediated conversations. For example, Isaacs and Tang (3, 8) have argued that adding video--even at fairly low frame rates--changes the nature of remote audio interactions by providing visual feedback, helping to manage pauses, supporting iconic gestures, and allowing gaze co-ordination. O'Conaill and colleagues (5) report related research in which they conclude that directional gaze information appears necessary to promote seamless conversational turn-taking.

In this paper, we focus on video gaze cues and discuss some experimental work-in-progress on the cognitive implications of observed gaze duration and timing. With Ishii and colleagues (4), we believe that the ability to easily track where one's partner is looking can strongly influence the course of collaborative interactions. Studies of gaze patterns in face-to-face conversations (6) are consistent with the view that gaze cues are used--at least in part--to track comprehension, and thus to assess confirmation of conversational contributions. Previous research by Clark and colleagues has demonstrated the necessity of such cognitive tracking processes (2). It is not enough for a speaker to merely utter a verbal message; he or she must also ensure that the listener has understood what was said as it was intended. While gaze cues alone can support this function with great efficiency and precision, non-visual media such as the telephone require more cumbersome conventions or explicit modes of speech whenever comprehension is in doubt (3, 5, 6). Despite such precautionary measures, the risk of major misunderstanding, especially of complex ideas, can often remain quite large.

The present study is part of a larger research endeavour to systematically investigate the role(s) temporal gaze cues play in VMC. We examine inferences observers make simply from the length of time someone spends looking at an unknown problem. We predict that--in lieu of other information--the total time spent looking at the problem will be used to infer the degree of difficulty that person is having in trying to solve it. Moreover, we expect that subtle timing cues, such as a tendency to let one's gaze linger on the problem even after being instructed to move on, may add to perceived difficulty. Such effects could have important implications for the design of VMC systems, especially in choosing video sampling and compression techniques to best support natural conversational processes.

EXPERIMENT

Subjects

Materials

Procedure

RESULTS

Figure 1. Effect of Gaze Duration on Rated Difficulty

Look Duration:

Linger Duration: Initial Findings

DISCUSSION

Conversation is a collaborative activity in which subtle visual timing and gestural cues must be interpreted in conjunction with speech acts; cues involving the face and eyes are especially salient (6). In designing VMC systems, decisions are frequently made to compress or otherwise manipulate the video stream, often to "keep up with the audio" (3, 5). Our findings suggest that these decisions may influence the nature and course of the mediated interaction, since subtle gaze cues that suggest cognitive difficulty may be introduced or eliminated unwittingly. Given limited resources, decisions regarding video lag, sampling frame rate and the choice of compression algorithm must be approached with caution to avoid introducing timing biases with cognitive and communicative implications. The leverage gained by choosing to accurately represent visual dynamics may well be worth some loss in resolution or completeness of the video image.