Handjabber

Handjabber is an interactive installation used to facilitate non-verbal gesture communication. The primary builders of Handjabber are myself, Stjepan Rajko, and Christopher Martinez. Handjabber is designed to help encourage communication amongst multiple participants with the potential for sustained creative collaboration. This system is an interactive sound installation and research environment that produces audio feedback. HandJabber utilizes the emergence of semantic meanings from gesture and interpersonal behavior as a single interface for real-time control of music. This approach offers a new way to think about media control in the context of collaboration. Our goal with Handjabber was to look at interface design that (a) responds to an individual participant’s hand and arm gestures, and (b) reacts to the non-verbal interpersonal behavior between two participants. We are exploring three major areas of non-verbal behavior, (a) metaphoric hand and arm gesture, (b) interpersonal space, and (c) body orientation.

Natural gestures and behavioral movements often aid and accompany verbal communication. Hand and arm gestures, for example, commonly accompany speech. Other, more full-bodied movements such as those associated with interpersonal behavior, may or may not accompany speech. Examples of this include the use of full-bodied behaviors to express agreement during verbal conversation, to navigate through crowded spaces, or to claim territory within a public space. While some research explores how we use non-verbal behavior outside of the context of verbal communication, the majority of this research focuses on gesture used with speech. The small amount of research that does exist, focuses on how people use non-verbal behaviors to avoid a verbal conversation. For example, an individual may communicate to others by using aversive or territorial behavior to send a non-verbal message that suggests he or she does not want to engage in a personal conversation.

There are some codified nonverbal languages, such as American Sign Language (ASL). These languages are used in personal conversation without speech. However, these nonverbal languages are constructed with specific gestures that hold specific meanings. While these forms of communication may feel natural to those who know the language, they are not natural forms of communication for those who do not. Participants in these languages must first learn the gestures and their associated meaning before they can engage in non-verbal communication through the language. While these forms of gesture are valid and important, we are interested in fundamental forms of non-verbal behavior that derive from meanings implicit at the subconscious level most people, as opposed to codified nonverbal languages used by a select group of people.

While participants in HandJabber are not discouraged from using verbal communication, we discovered that they rarely engage in verbal communication while moving in the mediated space. Participants engage in the natural gesture and behavioral movements associated with speech, but these behaviors are often abstracted and exaggerated forms of original behaviors. We are exploring three major areas of non-verbal behavior observed to be present in HandJabber: (a) metaphoric hand and arm gesture, (b) interpersonal space, and (c) body orientation.

 

 

Metaphoric Gesture:

Metaphoric gestures are not dictated by speech, cultural representations, or concrete objects. Usually depend on the abstract meaning and communicative intent of the speaker. When performed with speech, metaphoric gestures usually occur at roughly the same moment in time as the associated spoken words. Metaphoric gestures do not present the same information as the associated spoken works, rather they compliment or relate in some way to the words spoken. In other words, they're redundant, but related.

 

Interpersonal space:

Interpersonal Space is the amount of physical distance that lies between two people during conversation. The four sub-categories of interpersonal space are:

  1. Intimate distance: physical contact to 18 inches;
  2. Casual – personal: 1 ½ feet to 4 feet;
  3. Social – consultative: 4 to 12 feet;
  4. Public – 12 feet to physical hearing and seeing distances;
Interpersonal space can fluctuate depending on several factors such as the familiarity between two people, formality of a situation, individual emotional states, or current feelings toward the other person.

 

Body Orientation:

Body orientation is the frontal facing of one person in relationship to another person or group of people. People often directly orient their body toward the person of interest in what Scheflen defines as “quasi-courtship” behavior, or “flirtatious behavior used to show non-sexual affiliation with another person.” A person may turn away from another to express anger, frustration, disgust, or to simply signify that a conversation is over. If the interaction is between more than two people, members of the group will typically situate the front of their bodies so they are open to those in the group but closed off to those not associated with the group.