Desk Console.
Augmenting 3D Virtual Controls on Physical Desks for Immersive AuthoringMaster’s Thesis Research
Defense completed at KAIST, 2023
THESIS︎︎︎ SLIDES︎︎︎
Timeline
Jan 2023 - June 2023 Master’s Thesis
- Personal Project as Master’s Thesis
- Advisor: Prof. Woohun Lee (WonderLab, KAIST)
-
Thesis Committee:
Prof. Andrea Bianchi / Prof. Seok-Hyung Bae
Keywords
- Virtual Reality Interface Design
- Immersive Authoring Tool
- Creativity Support
Tools
Unity VR(C#). Oculus Quest 2.Oculus interaction SDK. Figma.
User Study: Contextual Inquiry. Thematic analysis. Quantitative analysis (SPSS). Data visualization ( Python).
What if more expressive and stable immersive creation is possible with bare hands in virtual reality?
Complex control panel screens in creative work pose challenges for natural bare-handed interaction in VR, limiting exploration.
I present Desk Console, an immersive authoring interface that represents 2D control panels as 3D virtual controls on physical desk.
This contributes to the future of work in augmented and virtual environments by enhancing immersive authoring and promoting creativity.
I present Desk Console, an immersive authoring interface that represents 2D control panels as 3D virtual controls on physical desk.
This contributes to the future of work in augmented and virtual environments by enhancing immersive authoring and promoting creativity.
My Role Master’s Thesis Author
I led this project
as part of
my master's thesis at the Korea Advanced Institute of Science and Technology (KAIST).
Throughout this journey, I wore multiple hats as an interaction designer, VR developer, and UX & HCI researcher.
I received valuable advice from my advisor, committee members, and WonderLab members.
Throughout this journey, I wore multiple hats as an interaction designer, VR developer, and UX & HCI researcher.
I received valuable advice from my advisor, committee members, and WonderLab members.
Motivation
Bare hands, while natural and expressive, can limit creativity in VR due to unstable + awkward interaction.
As we can see from Apple Vision Pro and Meta Quest 3, XR interactions have evolved to use our hands naturally.
Imagine the future of work. We can work anywhere with the HMD device. When interacting with the virtual screen by hand, we often use raycasting, which works well for basic tasks. However, when we consider who works in VR, it is almost creators who have to check their work in real time. Their tools include these complex control panels. That's where raycasting becomes annoying, limiting creativity with low stability and awkward interactions.
How can we design an immersive authoring interface
enabling intuitive screen UI interaction to capture both stability and naturality in operation?
- RQ1a. What are current user experiences of creative work with virtual screens?
-
RQ1b. How can immersive authoring interface be designed to enable intuitive and natural interaction, while ensuring the stability of bare-handed operation with a virtual control panel?
- RQ2. How does the novel interface impact users’ authoring experience, and to what extent does it enhance various aspects of creativity support?
Formative Study
Contextual Inquiry.
Study Goal.
To clearly understand the user's
experience of creative working with virtual screens (RQ1a)- Identify specific work breakpoints during the process
- Gain insights into the limitations of the current interface
Setup.
Result - Key findings.
Result - Sequence Model.
︎︎︎
Initial Concept Sketch
Based on the findings from related works research and formative study, I explored leveraging the tactile sensation of the desk surface to facilitate intuitive interactions.
Throughout this process, I recognized the need for a more systematic approach to interaction design to achieve scalability and flexibility.
︎︎︎
Desk Console Interface Design
1. Define Interaction Design Consideration
2. Categorize UI Elements of Creative Tools
For adaptability, I examined the interfaces of design tools like Unity to identify the main UI elements. This is the Categorized UI elements according to input method.
3. Design
interaction for each UI
type
The UI elements were transformed into 3D representations based on the input methods.
Binary Input was represented as Buttons, Value Input as Sliders, and Navigation Input as Frame with Pin.
4. Apply to example panel
Implementation
Desk Console interface was implemented in the VR to control the environment and variables systematically. I used Oculus Quest 2 head-mounted display (HMD) with real-time hand tracking via its built-in camera.
The software was developed using Unity Game Engine (version 2020.3.33f1),
utilizing the Oculus Interaction SDK (version 49.0).
Evalutation - User Study.
Procedure:
To better understand user experience with the designed interface, I used Raycasting, a widely adopted method for interacting with virtual screens, as the baseline condition for the study.
The study was conducted as within a group, considering the order effect.
Results
︎ Results of content creations by 14 participants using each interface
︎ Results of content creations by 14 participants using each interface
︎ Performance Indicators Exploration
Users
were
able
to explore more various controls within
the
same
amount
of time
using
Desk
Console.
The vertical slider and color picker were adjusted significantly faster on the Desk Console.
"I felt more comfortable with this(Desk Console), so I tried different controls earlier and experimented with more various options, whereas in Raycasting, I focused on performing one task well rather than exploring multiple options.＂ (P9)The vertical slider and color picker were adjusted significantly faster on the Desk Console.
︎ Workload Evalutation + Interface Impact on Creative Workflow
Reduced Workload:
As the interaction in work becomes more realistic, users can focus more on the task, reducing mental demand. Providing suitable tactile feedback can affect not only physical comfort but also cognitive processes.
As the interaction in work becomes more realistic, users can focus more on the task, reducing mental demand. Providing suitable tactile feedback can affect not only physical comfort but also cognitive processes.
"When working on a desk, it's almost instinctively connected to physically moving objects, so I feel comfortable to do it like I usually would.” (P14)
“With second one(Desk Console), it's like just pressing buttons right away, so thinking process was a bit shorter, as I feel. Since I immediately get feedback right in my fingure, it feels like I was able to do things right away without hesitation.” (P9)
Expanded Exploration and Expressiveness:
Facilitating users' exploration is enhanced by controllability with easily reversible previous states. Effective expression is facilitated by passive haptic providing a sense of scale and enhancing spatial awareness.
Facilitating users' exploration is enhanced by controllability with easily reversible previous states. Effective expression is facilitated by passive haptic providing a sense of scale and enhancing spatial awareness.
"It was easier to go back to previous state compared to raycasting, so I could discover something good while exploring…” (P12)
“When I used it (on the desk), there was a sense of scale that allowed me to control more accurately and precisely because I could see the difference of double the size between this and that.“ (P10)
... And “it was Fun to Use!”
Participants frequently likened Desk Console to real-world objects during the task.
The alignment with familiar objects
and associated tactile experience enhanced the overall fun factor
mouse,
keyboard (P5, P9, P14) / airplane or helicopter cockpit (P13) / game
console pad (P3) / augmented reality (P9)︎ Qualitative Observations
Potential for Bimanual Manipulation
"Buttons or something similar in a cognitive sense, they make it so much easier, I'd say. I mean, I'm not quite used to it yet, so I couldn't use both hands this time, but I feel like I could next time. It's just that using both hands really lightens the load." (P11)
Gaze Dispersion Extent
"I get the sense that moving up and down is way less distracting for the eyes. With this Desk Console, it feels secure in my hand, so I approached it with a similar mindset, much like how I'd work in Photoshop with a mouse, where I don't really look at the controls and just focus on the task at hand." (P6)
Movement Traces for Color Exploration in Color Picker in Each Interface
In the interviews, participants expressed notable differences in their experience with the Color Picker across the two interfaces. To gain deeper insights into this observation, Log Data, which captured the pointer (or pin) movement traces on the color picker for color selection, was visualized.
This implies more frequent operation adjustments in the Raycasting, and more detailed adjacent color exploration in the Desk Console.
Discussion.
Making & Presentation
