Software engineerSeattle
Jack Yang
Building real-time AI agents for live conversation
I build real-time AI agents at Microsoft Teams AI, where I work on Facilitator, the Microsoft 365 Copilot agent that runs inside live Teams meetings. My focus is the orchestration, tool execution, context management, and multimodal features that let an agent keep up with a conversation as it happens.
Before Teams AI, I was a key contributor to the avatar rendering pipelines and shared UI at Microsoft Mesh, which shipped weekly in 20+ languages to millions of users. Earlier, I built AR/VR interaction and multiplayer systems at Holos, a Techstars startup, and helped it secure a $750K U.S. Air Force contract.
Outside work, I created FakeHaul, an AI-generated marketplace where every checkout is $0.00, covered by The New York Times and Fast Company. I’m also completing an M.S. in HCI at Georgia Tech, with a paper under review at CHI 2027 on how AI systems that track decisions from transcripts can hide their own mistakes. I enjoy connecting low-latency systems with generative models, and I care as much about how the interaction feels as how the system is built.
- Location
- Seattle, United States
- jackyangzzh@gmail.com
- GitHub
- jackyangzzh
- Medium
- jackyangzzh
- jackyangzzh
Experience
– present
Software Engineer II, Teams AI at Microsoft
Highlights
- Build Facilitator, a meeting agent for Microsoft 365 Copilot that takes live notes, tracks agendas and decisions, and answers questions in Teams.
- Design the turn-taking layer of a proactive agent, deciding when and how the AI speaks into a live group conversation without interrupting.
- Contribute to conversation orchestration, tool calling, context management, observability, and multimodal LLM features.
–
Software Engineer II, Mixed Reality (Mesh) at Microsoft
Highlights
- Built avatar customization systems and shared UI components (input handling, accessibility) behind Microsoft Mesh and Avatars in Teams, supporting weekly releases and 20+ languages for millions of users. Satya Nadella showed the outfit-selection flow I built at Microsoft Ignite.
- Key contributor to the avatar rendering and performance pipeline across HoloLens, Quest, web, and desktop. Profiled CPU and GPU frame budgets, using LOD switching and GPU instancing to maintain target framerates on constrained hardware.
–
AR/VR Interaction Engineer at Holos
Highlights
- Built core AR and VR training platform capabilities, including hand tracking, gesture recognition, multiplayer networking, and 3D model ingestion workflows.
- Worked with leadership on product and engineering execution, helping Holos secure a $750K U.S. Air Force contract and a place in Techstars.
Research
–
XR Researcher at Kats Research Group
Highlights
- Developed a computer vision model that simulates color vision deficiency in hyperspectral images with 90% accuracy, and ran it in a color-calibrated Oculus Rift demo. Co-authored an arXiv publication.
–
Graphics Researcher at UW Graphics Group
Highlights
- Built a real-time VR control loop mapping hand gestures to remote robotic-arm manipulation, plus a Unity UI for replaying arm motion from 15+ lab-experiment datasets.
–
HCI Researcher at UW Human-Computer Interaction Group
Highlights
- Added speech recognition and NLP-driven interaction to children's reading-companion robots and validated the interaction in an eight-family trial.
Education
–
M.S. in Computer Science (HCI) from Georgia Institute of Technology
Courses
- Human-Computer Interaction specialization; in progress, expected May 2027.
–
Bachelor of Science from University of Wisconsin–Madison
Courses
- Double-majored in Computer Science and Communication.
- TA for CS 559 (Computer Graphics) for 2 semesters.
Certifications
- Machine Learning Specialization Stanford University
- Human-Computer Interaction Professional Certificate Georgia Tech
- Computer Vision Nanodegree Udacity
- VR Development Professional Certificate UC San Diego
- Virtual Reality Specialization University of London
- Interaction Design Specialization Certificate UC San Diego
Skills
- AI & Agent Systems
- Keywords:
- Software & Cloud
- Keywords:
- Real-Time & Spatial Computing
- Keywords:
Publications
Too Late to Rely: A Self-Concealing Failure in State-Only Transcript Grounding Trackers
Under review for CHI 2027. Transcript-based trackers that keep only the latest state can hide their own failures: if someone acts on a decision before the conversation supports it, later agreement overwrites the warning that they acted too early. Used a 264-dialogue benchmark to show the failure and test an event-sourced record that preserves those warnings.
Passive frequency conversion of ultraviolet images into the visible using perovskite nanocrystals
Co-authored a peer-reviewed paper on a passive system that converts ultraviolet images into visible light using perovskite nanocrystals.
Breaking Binocular Redundancy Through Virtual Reality
Wrote a technical report and released an open-source VR demo that simulates static and dynamic binocular rivalry.
Effective Leaders: Four Attributes That Underpin The Core Characteristics of Effective Leadership
Contributed the article “Critical Conversation” to Mike Cameron’s collected volume.