Institute for Creative Technologies

Download a one sheet PDF overview of this project.

During face-to-face conversation, eye contact, and gaze direction provide important visual cues to express emotion, attention, and interest. Unfortunately, 2D video telecoferencing does not recreate accurate eye contact breaking the emotion connecting between participants. When a remote participant looks directly into the camera, everyone watching the video stream sees the same image of the participant looking towards them; if the participant looks away from the camera, no one receives the eye gaze. Our one-to-many teleconferencing system uses a novel arrangement of 3D acquisition, transmission, and display technologies to achieve accurate reproduction of gaze direction and eye contact.

Watch the LabTV video about 3D Teleconferencing.

The 3D display works by projecting high-speed video onto a rapidly spinning mirror made from brushed aluminum. As the mirror turns, it reflects a different and accurate image to each potental viewer. The size, geometry, and material of the spinning surface have been optimized for the display of a life-sized human face. Its two-sided shape provides two passes of a display surface to each viewer per full rotation, achieving a 30Hz visual update at 900 rpm. A pair of high speed DLP projectors project 8,640 1-bit (black or white) frames per second using a specially coded DVI video signal. Instead of rendering a color image,each projector takes a 24-bit color frame of video and displays each bit sequentially as separate frames. Effectively, the mirror reflects 144 unique views of the scene across a 180-degree field of view with an angular view separation of 1.25 degrees.

A 90 degree field of view 2D video feed allows the remote participant to view the audience interacting with their three-dimensional image on the 3D display. A polarized beam-splitter is used to virtually place the camera close to the position of the eyes of the three-dimensional head. The video from the aligned 3D display camera is transmitted to the remote participant where it is shown on a geometrically calibrated LCD screen.

To correct the vertical perspective on the 3D display, we use marker-less face detection from OpenCV to track viewers based on the 2D video feed. In this way, the display's horizontal parallax provides binocular stereo with no lag as the viewers move their heads horizontally, while vertical parallax is achieved through tracking.

Watch the original LabTV video at http://www.ndep.us/LabTV2.aspx?id=38&t=3-D%20Teleconference.

Tags: 3d, display, human, teleconferencing, video

View All Projects »