CS 377M - HCI Issues in Mixed and Augmented Reality

Course Schedule

Date	Class	Readings	Homework
March 28	Introduction. General class intro: goals, format, assignment overview Introduction to mixed and augmented reality. History of MR technology. VR & AR, Hard AR vs soft AR. Overview of HCI issues and class. Discussion around what is Mixed Reality? What is Augmented Reality? Where is it appropriate? AR in film and popular culture Guest Speaker: Prof. Mark Billinghurst	Readings for next week: Required: HoloDesk: direct 3d interactions with a situated see-through display Suggested: Augmented reality: a class of displays on the reality-virtuality continuum A review of head-mounted displays (HMD) technologies and applications for consumer electronics	Assignment 1 (due 4/4): Find an existing augmented reality system (check out at least 5-10 before picking) Write 1-2 pages that: Provide a link to the video, app, page, system, etc... Describe the system & the key interaction techniques Critique the video Is it a real application or concept video? Does it need to be augmented reality? does augmented reality improve the experience? What works & doesn't work? How would you improve the interaction?
April 4	Displays. Types of displays. Hand-held AR. Spatial AR. Optical and video see-through glasses. See-through displays. Discuss issues specific to each of these modalities. Non-visual modalities Input and Output hardware.	Readings for next week: Required: How to read a paper A Review of 3D Gesture Interaction for Handheld Augmented Reality Suggested: Marker tracking and hmd calibration for a video-based augmented reality conferencing system Parallel tracking and mapping for small AR workspaces KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera	Need finding. What is the interesting opportunity? Explore the set of user, context, and task/opportunity that you want to address and be able to articulate that. We're interested in the why as much as the who/what. Initial project ideas. We don't expect you to have fully baked solutions but you should be brainstorming generally how you might address the opportunity and exploring the possibilities. Have at least three potential directions. Deliverables - 1-2 pages describing the above, create a slack channel, be able to discuss/present in class
April 11	Tracking and Registration. How do we understand the world and provide that information to the person and system. Introduction to existing tools like ARToolkit, Vuforia, Wikitude, and more advanced work like PTAM. Model on a Marker. Implications for scene understanding in AR. Project teams due	Readings for next week: Required: REVEL: tactile feedback technology for augmented reality Twiddler typing: one-handed chording text entry for mobile phones Suggested: Haptic Ring Interface Enabling Air-Writing in Virtual Reality Environment GyroWand: IMU-based Raycasting for Augmented Reality Head-Mounted Displays	Model on a marker For the group assignment, you need to write up a a proposal for the project you're selecting and be prepared to discuss: Project title Group members and roles (e.g. Will has experience with Arduinos and will be making our magic wand, Sean built image processing systems and will be making a face recognizer) Project idea and motivation (opportunity/problem, users, context, task) Project plan of attack with weekly milestones (e.g., a rough plan of how you will accomplish the work) Initial literature review (Has this been done before? What prior work is relevant? If your project is different, how is it different?) If you're creating a new contribution, like many of the papers, what might you be contributing (e.g. a study, a new concept, old concept in a new application) What hardware, software, or other resources you might need, and how you plan to get them? Do you need glasses (HoloLens, ODG, Meta), handheld displays, Tango or occipital depth cameras? Will you use Wikitude, Vuforia, or some other tracking system? Success criteria (How will you know if you succeed? I will grade you partially on whether you met your own criteria.) Note that along with a presentation and video of the project, one of your final deliverables will be writing up your work in the ACM SIGCHI Extended Abstract format. The official final for this class is June 7th from 7-10pm, when you'll present your projects to each other and guests.
April 18	Rendering & Authoring, Shadows, Relighting, Occlusions. How are virtual objects and affordances perceived as physical? Should they be? Discuss draft project proposals Guest Speaker: Dr. Kent Lyons	Readings for next week: Required: Perceptual Issues in Augmented Reality Revisited	Start on design sketches and exploration
April 25	Perceptual & Cognitive Issues. Visibility, object relationships, registration errors, layer interference, individual differences, depth perception cues. Design implications and strategies Final project proposals due Guest Speaker: Dr. Oliver Aalami, M.D.	Readings for next week: Required: An introduction to 3-D user interface design Optional: New Directions in 3D User Interfaces	Project Design
May 2	Interaction Techniques. 3DUI, Selection & manipulation, Tangible AR, Wayfinding, System Control, Evaluation techniques for MR systems. Guest Speaker: Dr. Ron Azuma	Readings for next week: Required: DART: a toolkit for rapid design exploration of augmented reality experiences. Authoring Tools in Augmented Reality: An Analysis and Classification of Content Design Tools Optional: Authoring 3d hypermedia for wearable augmented and virtual reality Tiles: A mixed reality authoring interface Immersive authoring of tangible augmented reality applications Kharma: An open kml/html architecture for mobile augmented reality applications	Project Design & Prototype
May 9	Authoring. Creating AR systems, content in AR systems, out of system and in-situ authoring. AR-specific infrastructure. Unity. Guest Speaker: Charlie Sutton	Readings for next week: Required: View management for virtual and augmented reality An augmented reality x-ray system based on visual saliency Optional: The Personal Interaction Panel - a Two-Handed Interface for Augmented Reality Smart Vidente: advances in mobile augmented reality for interactive visualization of underground infrastructure Image-driven view management for augmented reality browsers	Project Checkpoint
May 16	Visualization: Types of representation, situated visualization, situated learning, situated analytics. Spatial data. ZUIs in AR. Project updates in class Guest Speaker: Brian Mullins	Readings for next week: Required: Of vampire mirrors and privacy lamps: Privacy management in multi-user augmented environments Security and privacy for augmented reality systems	Project Checkpoint II
May 23	Applications: Enterprise, Consumer, Medicine, Manufacturing, Education, Games, Art, Media, & Humanities Ethical issues Future of Mixed and Augmented Reality	No Readings	Work on presentation and documentation
May 30	No Class (Memorial Day)	No Readings	Keep working on the project and presentation
June 7	Final Project Presentations	No Readings

Having problems? Let us know!