Software engineerSeattle
Jack Yang
Building Real-Time Multimodal Agents for Live Human-AI Interaction
I build real-time AI agents at Microsoft Teams AI, working on Facilitator in Microsoft 365 Copilot. My focus is conversation orchestration, tool execution, context management, and multimodal meeting features that hold up in live conversation.
Previously, I built AR/VR interaction and multiplayer platforms at Holos, then was a key contributor to avatar rendering pipelines and shared UI at Microsoft Mesh. My background spans early-stage startups and large-scale interactive systems.
Outside work, I created FakeHaul, an AI-generated marketplace covered by The New York Times and Fast Company. I enjoy connecting low-latency systems with generative models, with as much attention to the interaction as the engineering.
- Location
- Seattle, United States
- jackyangzzh@gmail.com
- GitHub
- jackyangzzh
- Medium
- jackyangzzh
- jackyangzzh
Experience
– present
Software Engineer II, Teams AI at Microsoft
Highlights
- Build Facilitator, a meeting agent for Microsoft 365 Copilot that takes live notes, tracks agendas and decisions, and answers questions in Teams.
- Contribute to conversation orchestration, tool calling, context management, observability, and multimodal LLM features.
–
Software Engineer II, Mixed Reality (Mesh) at Microsoft
Highlights
- Built avatar customization systems and shared UI components (input handling, accessibility) behind Microsoft Mesh and Avatars in Teams, supporting weekly releases and 20+ languages for millions of users.
- Owned the avatar rendering and performance pipeline across HoloLens, Quest, web, and desktop. Profiled CPU and GPU frame budgets, using LOD switching and GPU instancing to maintain target framerates on constrained hardware.
–
AR/VR Interaction Engineer at Holos
Highlights
- Built core AR and VR training platform capabilities, including hand tracking, gesture recognition, multiplayer networking, and 3D model ingestion workflows.
- Worked with leadership on product and engineering execution, helping Holos secure a $750K U.S. Air Force contract and a place in Techstars.
Research
–
XR Researcher (part-time) at Kats Research Group
Highlights
- Developed a computer vision model that simulates color vision deficiency in hyperspectral images. It reached about 90% agreement in pilot color tests and ran in a color-calibrated Oculus Rift demo.
–
Graphics Researcher (part-time) at UW Graphics Group
Highlights
- Built a real-time VR control loop mapping hand gestures to remote robotic-arm manipulation, plus a Unity UI for replaying arm motion from 15+ lab-experiment datasets.
–
HCI Researcher (part-time) at UW HCI Group, People and Robots Lab
Highlights
- Added speech recognition and NLP-driven interaction to children's reading-companion robots; validated in an eight-family trial.
Education
–
M.S. in Computer Science (HCI) from Georgia Institute of Technology
Courses
- Human-Computer Interaction specialization; in progress, expected May 2027.
–
Bachelor of Science from University of Wisconsin–Madison
Courses
- Double-majored in Computer Science and Communication.
- TA for CS 559 (Computer Graphics) for 2 semesters.
Certifications
- Machine Learning Specialization Stanford University
- Human-Computer Interaction Professional Certificate Georgia Tech
- Computer Vision Nanodegree Udacity
- VR Development Professional Certificate UC San Diego
- Virtual Reality Specialization University of London
- Interaction Design Specialization Certificate UC San Diego
Skills
- AI & Agent Systems
- Keywords:
- Software & Cloud
- Keywords:
- Real-Time & Spatial Computing
- Keywords:
Publications
Too Late to Rely: A Self-Concealing Failure in State-Only Transcript Grounding Trackers
Under review for CHI 2027. Used a 264-dialogue benchmark to study how transcript-based grounding trackers can lose action-time warnings when later evidence updates the recorded state.
Passive frequency conversion of ultraviolet images into the visible using perovskite nanocrystals
Co-authored a peer-reviewed paper on a passive system that converts ultraviolet images into visible light using perovskite nanocrystals.
Breaking Binocular Redundancy Through Virtual Reality
Wrote a technical report and released an open-source VR demo that simulates static and dynamic binocular rivalry.
Effective Leaders: Four Attributes That Underpin The Core Characteristics of Effective Leadership
Contributed the article “Critical Conversation” to Mike Cameron’s collected volume.