Jack Yang
Building Real-Time Multimodal Human-AI Interaction
I’m a software engineer building real-time, multimodal AI in production. At Microsoft Teams AI, I work on Facilitator, an agent that participates in live, multi-person meetings. Seven years shipping real-time, multimodal human-computer interaction at scale — expressive avatars, embodiment, and XR at Microsoft Mesh — is exactly what AI needs as it moves beyond the chat box.
- Location
- Seattle, United States
- jackyangzzh@gmail.com
- GitHub
- jackyangzzh
- Medium
- jackyangzzh
- jackyangzzh
Experience
– present
Software Engineer II, Teams AI at Microsoft
Highlights
- Build Facilitator, Microsoft's conversational AI agent that participates in live, multi-party Teams meetings — real-time notes, agenda tracking, decision capture, and in-conversation Q&A — shipping to Microsoft 365 Copilot customers.
- Design the turn-taking layer of a proactive agent — when and how AI speaks into a live group conversation without interrupting — alongside early-stage multimodal LLM initiatives.
–
Software Engineer II, Mixed Reality (Mesh) at Microsoft
Highlights
- Engineered the shared UI framework powering Microsoft Mesh and Avatars in Teams for millions of users — reusable interaction patterns, input abstractions, accessibility, and spatial UI components adopted across Mesh product teams.
- Owned the real-time avatar rendering and performance pipeline across HoloLens, Quest, web, and desktop — profiling CPU/GPU frame budgets and applying LOD switching, GPU instancing, and other optimizations to hold target framerates on constrained hardware.
- Shipped end-to-end avatar customization and character-building systems with data-driven content pipelines, a weekly release cadence, and 20+ language localization.
–
AR/VR Interaction Engineer at Holos
Highlights
- Developed AR/VR training platform featuring gesture recognition, hand tracking, and contextual guidance.
- Implemented multiplayer networking, automated 3D model ingestion, and advanced interaction mechanics.
- Partnered with leadership to secure $750K U.S. Air Force contract and acceptance into Techstars accelerator.
Research
–
XR Researcher at Kats Research Group
Highlights
- Built an AI model that mimics human color vision to simulate color vision deficiency on hyperspectral imagery with 90% chromatic-adjustment accuracy, deployed in a color-calibrated Oculus Rift demonstration.
–
Graphics Researcher at UW Graphics Group
Highlights
- Built a real-time VR control loop mapping hand gestures to remote robotic-arm manipulation, plus a Unity UI for replaying arm motion from 15+ lab-experiment datasets.
–
AR/VR Researcher at UW HCI Group, People and Robots Lab
Highlights
- Added speech recognition and NLP-driven interaction to children's reading-companion robots; validated in an eight-family trial.
Education
–
M.S. Computer Science (HCI) from Georgia Institute of Technology
Courses
- Human-Computer Interaction specialization. Expected Spring 2027.
–
Bachelor of Science from University of Wisconsin–Madison
Courses
- Double-majored in Computer Science and Communication Arts.
- Served as a teaching assistant for CS 559 (Computer Graphics) for 2 semesters.
Publications
Passive frequency conversion of ultraviolet images into the visible using perovskite nanocrystals
Co-authored research on hyperspectral imaging and color perception.
Effective Leaders: Four Attributes That Underpin The Core Characteristics of Effective Leadership
Featured article ‘Critical Conversation’ published in the book.
Skills
- AI / Agents
- Level: 3/3Keywords:
- Real-Time 3D / XR
- Level: 3/3Keywords:
- UI / UX
- Level: 2/3Keywords: