NYU Holodeck: A simulated Reality Environment

NSF MRI Award #1626098

NYU Holodeck graphic

We are building a Holodeck.

The NYU Holodeck is a National Science Foundation MRI Track 2 Development project (NSF award # 1626098) that will create an immersive, collaborative, virtual/ physical research environment providing unparalleled tools for research collaborations, intellectual exploration, and creative output.

To prototype the transition from projection to functional AR, MR and VR simulations, our interdisciplinary team will develop an instrument and educational environment that facilitates enticing research and creates an experiential supercomputing infrastructure that empowers students, faculty, and researchers to develop new knowledge and disseminate powerful new integrated, multimodal tools and techniques.

The NYU Holodeck will be a well-integrated software/hardware instrument that fosters creative capacity by incorporating:

  • Visual, audio, and physical components;
  • Novel technologies to enhance human-human, human-agent, and human-robot social interactions;
  • Rapid prototyping and fabrication tools,
  • Tightly coupled interactive visual, audio, and physical experiences.

The NYU Holodeck instrument will:

  • Provide a flexible, modular, reconfigurable infrastructure with comprehensive capacity to capture behavioral, physiological, affective, and cognitive data, and provide real time data visualization and analysis.
  • Enable researchers to explore questions involving virtual environments, telepresence, collaborative engagement, and remote interaction.
  • Advance next-generation experiences in human interaction
  • Provide deep integration of virtual and physical settings.
  • Potentially transform fundamental understanding of collaboration, learning, creativity, discovery and innovation.

Our interdisciplinary project team includes expertise in computer science, health-informatics, gaming, digital media, design engineering, and educational technology.

Initial NYU Holodeck research collaborations leverage current NSF funding in disciplines including: cyberlearning; healthcare; acoustics, applied mathematics; urban planning; and gaming.

The NYU Holodeck will be an open source networked instrument linking facilities at the Future Reality Lab (FRL) in Washington Square, and Media and Games Network (MAGNET) in Brooklyn to create a networked collaboration that can be readily and broadly extended and disseminated to national and international research and educational communities.

Our goal is to create and generate rich actualizing environments to support research and discover new paradigms.

Research Areas

  • Visual
  • Auditory/Acoustic
  • Physical Robotic
  • Human Dynamics
  • Distributed Collaboration
  • Simulation

Applications

  • Modeling, Visualization & Verisimilitude
  • Urban and Planetary Environments
  • Applied Mathematics
  • Physical Acoustics and Collaboration
  • Studies of Teamwork, Design, and Outcomes
  • Cyberlearning (Learning Sciences and Games for Learning):

NYU Collaborators

Dr. Ken Perlin

Professor in the Department of Computer Science at NYU, founding director of the Future Reality Lab at NYU, the Director of the Games for Learning Institute, and a member of MAGNET and founding director of the Media Research Laboratory and director of the NYU Center for Advanced Technology.

Technical Expertise: graphics, animation, multimedia, and science education. He will provide integration expertise in scientific visualization and interface development, motion capture, human-computer interaction, and applied mathematics.
Future Reality Lab

Dr. Agnieszka Roginska

Associate Professor, Music Technology at NYU.

Technical Expertise: accurate simulations of acoustic spaces and sound reproduction for interactive environments. She will provide integration expertise in sound and acoustic modeling, and human-computer interaction. http://steinhardt.nyu.edu/faculty/Agnieszka_Roginska

Dr. Jan Plass

Paulette Goddard chair in Digital Media and Learning Sciences in the NYU Steinhardt School of Culture, Education, and Human Development, director Educational Communication and Technology programs, co-director Games for Learning Institute and founding director of CREATE Consortium for Research and Evaluation of Advanced Technology in Education.

Technical Expertise: cognitive and emotional aspects of information design, and interaction design of simulations and educational games for science education and second language acquisition. He will provide expertise in educational technology and learning science, team interaction and integration.
http://steinhardt.nyu.edu/faculty/Jan_Plass

Dr. Luke DuBois

Co-Director/Associate Professor of Integrated Digital Media (SoE), Music Technology (Steinhardt), and Interactive Telecommunications (Tisch) at NYU.

Technical Expertise: interactive sound, video and electronic performance, software engineering for creative practice. He will provide expertise in linkages of audio and visual stimuli. http://engineering.nyu.edu/user/1797

Dr. Claudio Silva

Professor of Computer Science and Engineering at NYU, Head of Disciplines, Center for Urban Science + Progress (CUSP). His degrees are in mathematics and computer science.

Technical Expertise: intersection of visualization and geometric computing, and analysis of urban data. He will provide development support for multimodality visualization and analytic tools.
http://cusp.nyu.edu/people/claudio-t-silva/

Dr. Jeremy Rowe

Senior Research Scientist, Courant , NYU. Emeritus Executive Associate Director School of Computing and Informatics, Co-Director Decision Theater and Partnership for Research in Spatial Modeling at Arizona State University.

Technical Expertise: 3D modeling and scientific visualization. He will provide expertise in design, development and integration experience from ASU Decision Theater, 3D modeling and visualization and educational technology.

Dr. Rob Pahle

Senior Research Scientist, NYU IT Research Technology

Technical Expertise: 3D modeling and scientific visualization. He will provide expertise in design, development and integration experience from ASU Decision Theater, 3D modeling and visualization and educational technology.

University of Arizona NYU Holodeck Node

Dr. Win Burleson

Associate Professor Technical Expertise: human computer interaction, context aware computing, cyber learning, individual and team creativity research, affective computing, intelligent environments, and media arts. He will provide expertise in human-computer interaction, design and fabrication, biosensors, and prior research experience at ASU Decision Theater and other CAVE and immersive visualization environments. NEED WEBSITE

NYU Holodeck Advisory Board

Our Advisory Board will guide and assist in technology selection and research collaborations, assess progress, and interact with project teams.

Steven Feiner

Prof. Computer Science; Dir., Computer Graphics Lab and User Interface Laboratory, Columbia University.

Henry Fuchs

Prof. Computer Science; Founding Dir., Being There Center, University of North Carolina; Fellow of (AAAS) and (ACM). Ken Goldberg Prof. Ind. Eng. and Oper. Research (IEOR), College of Eng.; Dir., Automation Sciences Research Lab; Co-Dir., Center for Automation and Learning for Medical Robotics; Co-Founder Berkeley’s Center for New Media.

Ken Goldberg

Craigslist Distinguished Professor of New Media and Professor of Industrial Engineering and Operations Research (IEOR), with joint appointments in Electrical Engineering and Computer Science (EECS), Art Practice, and the School of Information at the University of California Berkeley

Hiroshi Ishii

Assoc. Dir., MIT Media Lab; Jerome B. Wiesner Prof. Media Arts and Sciences; Co-Dir., Things That Think; Head, Tangible Media Group. Robert Jacob Prof. Computer Science at Tufts University, ACM CHI Academy member.

Rob Jacob

Professor of Computer Science at Tufts University.

Ari Kaufman

Distinguished Prof. and Chair of the Department of Computer Science at Stony Brook University and Dir. Reality Deck and Center for Visual Computing.

Alan Kay

President, Viewpoints Research If Nursingnstitute; Adj. Prof., Computer Science, UCLA; Fellow: AAAS; NAE.

Mary Lou Maher

Prof. and Chair, Department of Software and Information Systems; Dir. Center for Education Innovation, UNC-Charlotte.

Collaborating NYU Facilities and Resources

The Applied Mathematics Laboratory at the Courant Institute http://math.nyu.edu/aml/

Center for Urban Science + Progress (CUSP) http://cusp.nyu.edu/

Consortium for Research and Evaluation of Advanced Technologies in Education (CREATE) http://create.nyu.edu/

Game Innovation Lab (NYU Polytechnic) http://gil.poly.edu/

NYU Media and Games Network (MAGNET) http://magnet.nyu.edu/

NYU Music Technology Recording and Computer Music Studios http://steinhardt.nyu.edu/music/technology/facilities/

NYU Wireless http://wireless.engineering.nyu.edu

External Collaborators

Academic

Aalborg University Copenhagen

Arizona State University

Carnegie Mellon University

Columbia University

Drexel University

Georgia Institute of Technology

Harvard Graduate School of Education

Massachusetts Institute of Technology

North Carolina State University

Northwestern University

Queens University

RMIT University

Stony Brook University

The University of Sydney

Tufts University

University of California Berkeley

University of California Irvine

University of California Santa Barbara

University of California Santa Cruz

University of Denver

University of North Carolina, Chapel Hill

University of North Carolina, Charlotte

University of Southern California

Vanderbilt

Industry

Alelo

Auro

IBM

Lego Foundation

NASA Ames Research Center

Sennheiser

Tactonic Technologies

Viewpoints Research Institute

XUUK Holdings