Carlos A. Gómez Gallo
Research
Current Projects:
• Continuous Understanding Project with James Allen, Mike Tanenhaus, Mary Swift, Greg Aist and Ellen Campana
• Optional Deletion of Double Clitics in South American Spanish with T. Florian Jaeger
• Ellipsis resolution in Picture NP Eye tracking experiments with Jeff Runner
Previous Projects:
• Language Tutor through Finite State Machine Agent: MARINA with Ron Zacharski
At the University of Rochester, I have been involved in joint projects with the Departments of Computer Science, Brain and Cognitive Science, and Linguistics. Here, I have collaborated with a number of professors, research associates, and graduates students. Among these are: James Allen, Mary Swift, Michael Tanenhaus, T. Florian Jaeger, Greg Carlson, and Jeff Runner, and others.
Corpus Development
Developing a Semantically Annotated Multimodal Corpus: We have developed a corpus that consists on a collection of videos, transcripts and semantic annotation of multi-modal task-oriented dialogues (Aist et al. 06, Gómez Gallo et al. 07a), done in collaboration with Prof. Mike Tanenhaus and Dr. Ellen Campana in the Department of Brain and Cognitive Science and Prof. James Allen and Dr. Greg Aist in the Department of Computer Science at the University of Rochester. The experiment extends a typical eye-tracking visual world experiment by interleaving language and action in a task-oriented dialogue. For instance, a typical interaction starts with the subject’s utterance "Move a banana to Central Park", and is followed by the confederate’s execution of a move action. The corpus has unique features including varying degrees of object complexity; eye tracking data; incremental annotation of verb thematic roles, referring expressions, and spatial relations; language models; entropy estimation, etc. Annotation of speaker's verbal requests and domain action executions use the annotation software called ANVIL developed by Michael Kipp. Details about the experimental design and multilevel annotation can be found in (Gómez Gallo et al. 07a, Gómez Gallo et al. 08a). We have used the Fruit Carts corpus as an empirical base for the development of computational and mathematical models to explore cognitive issues in language processing.
Language Interpretation
Developing Computational Models for Incremental Language Interpretation: We have built cognitively inspired computational algorithms that interpret language in an incremental fashion, in collaboration with Prof. James Allen, Dr. Greg Aist, and Dr. Mary Swift (Computer Science). Traditional dialogue systems have used a serial architecture which analyzes speech in a modularized manner. A speech recognition module passes its final analysis to a syntactic module which in turn feeds the next module with a final parse. Departing from this paradigm, we use incremental understanding to use semantic information to improve the parser’s efficiency (Aist et al. 06). We used semantic frequency effects of verb argument structure in the Fruit Carts domain and are currently extending this approach to use a discourse data structure, and domain semantic and pragmatic expectations to modify the constituents’ probabilities in a chart-based bottom-up parser. We demonstrate some advantages of incremental, non-modular processing in syntactic analysis (Gómez Gallo 07b), and user-system interaction (Aist et al. 07b). This interpretation system treats speech as a continuous stream and handles sentential and non-sentential utterances equally, aiming to overcome speakers’ disfluencies, speech recognition errors, or parsers’ incomplete analysis. This is being done within the continuous understanding framework developed at CISD (.pdf)
Language Production
Developing Mathematical Models for Incremental Language Production: We also used the Fruit Carts corpus to analyze cognitive issues in language production, in collaboration with T. Florian Jaeger (Brain and Cognitive Science, University of Rochester) and Ron Smyth (Linguistics and Psychology, University of Toronto). For instance, why people choose “Move an apple to Central Park” (mono-clausal plan I; implicit selection of the theme, i.e. the apple) over “Take an apple, move it to Central Park” (bi-clausal plan II; explicit selection of the theme), when both convey the same message. My work demonstrates that speakers a) prefer to distribute complex, high information theme expressions into multiple clauses, b) have an upper bound in the complexity and amount of information per clause, and c) have an early estimation of complexity and information density of referring expressions, crucially before onset of the verb (Gómez Gallo et al. 08b). This work draws connections between resource limitations and Uniform Information Density (UID). UID uses principles of communication in a noisy channel to explain mathematically why speakers prefer to expand information dense phrases. Our work is the first evidence for UID effects beyond the clause level, and goes beyond availability-based production effects. (CogSci-08.pdf)
Previous Projects
Other past projects include FASIL , a european funded consortium, for which I developed an email summarization tool that exploited replied text or ‘thread.’ These threaded emails were used to assign a topic and be able to cluster emails in a dynamic manner.
