Jack Yang

Jack Yang: Real-time AI for real conversations.

I’m Jack Yang, a software engineer at Microsoft Teams AI
building real-time multimodal agents for Microsoft 365 Copilot.
Before that, I built avatar and XR systems for Microsoft Mesh.

Open résumé Connect on LinkedIn
or email jackyangzzh@gmail.com

 About

Jack Yang, Software Engineer at Microsoft Teams AI
Now
Software Engineer, Microsoft Teams AI
Before
Microsoft Mesh, Holos
Studying
M.S. in Computer Science (HCI), Georgia Tech
Based in
Seattle

I build real-time AI agents at Microsoft Teams AI, where I work on Microsoft 365 Copilot in live Teams meetings. My focus is the orchestration, tool execution, context management, and multimodal features that let an agent keep up with a conversation as it happens.

Before Teams AI, I engineered the shared UI framework adopted by every product team at Microsoft Mesh and was a key contributor to its avatar rendering pipeline, shipping weekly in 20+ languages to millions of users. Earlier, I built AR/VR interaction and multiplayer systems at Holos, a Techstars startup, and helped it secure a $750K U.S. Air Force contract.

Outside work, I created FakeHaul, an AI-generated marketplace where every checkout is $0.00, covered by The New York Times and Fast Company. I’m also completing an M.S. in Computer Science (HCI) at Georgia Tech, where I research how AI systems interpret evidence in conversations. My research project, Ground Tracker, examines how later confirmation can hide a lack of support for an earlier action; the manuscript is under consideration at Dialogue & Discourse. I enjoy connecting low-latency systems with generative models, and I care as much about how the interaction feels as how the system is built.

 Experience

Résumé (PDF)
Work
Research
Study
Current role Expected Select a bar to jump to the entry

Industry

  1. to PresentNow

    Microsoft Teams AI

    Software Engineer II
    • Build Facilitator, a meeting agent for Microsoft 365 Copilot that takes live notes, tracks agendas and decisions, and answers questions in Teams.
    • Design the turn-taking layer of a proactive agent, deciding when and how the AI speaks into a live group conversation without interrupting.
    • Contribute to conversation orchestration, tool calling, context management, observability, and multimodal LLM features.
  2. to

    Microsoft Mesh

    Software Engineer II, Mixed Reality
    Case study
    • Engineered the shared UI framework behind Microsoft Mesh for millions of users, defining reusable interaction patterns, input abstractions, accessibility integration, and spatial UI components adopted by every Mesh product team.
    • Built avatar customization systems with weekly releases in 20+ languages. Satya Nadella showed the outfit-selection flow I built at Microsoft Ignite.
    • Key contributor to the avatar rendering and performance pipeline across HoloLens, Quest, web, and desktop. Profiled CPU and GPU frame budgets, using LOD switching and GPU instancing to maintain target framerates on constrained hardware.
  3. to

    Holos

    AR/VR Interaction Engineer
    Case study
    • Built core AR and VR training platform capabilities, including hand tracking, gesture recognition, multiplayer networking, and 3D model ingestion workflows.
    • Worked with leadership on product and engineering execution, helping Holos secure a $750K U.S. Air Force contract and a place in Techstars.

Research

  1. to PresentNow

    Georgia Institute of Technology

    HCI Researcher
    • Built Ground Tracker, which separates language-model perception from deterministic grounding rules, and a 264-dialogue benchmark showing how later confirmation can hide missing evidence at the time of an earlier action.
    • Authored Preserving Action-Time Grounding Judgments in Transcript Trackers, under consideration at Dialogue & Discourse.
  2. to

    Kats Research Group

    XR Researcher
    Case study
    • Developed a computer vision model that simulates color vision deficiency in hyperspectral images with 90% accuracy, and ran it in a color-calibrated Oculus Rift demo. Co-authored an arXiv publication.
  3. to

    UW Graphics Group

    Graphics Researcher
    Case study
    • Built a real-time VR control loop mapping hand gestures to remote robotic-arm manipulation, plus a Unity UI for replaying arm motion from 15+ lab-experiment datasets.
  4. to

    UW Human-Computer Interaction Group

    HCI Researcher
    • Added speech recognition and NLP-driven interaction to children’s reading-companion robots.
    • Designed and built a collaborative-filtering recommender system, successfully trialed with eight families.

 Education

Degrees

  1. to Expected

    Georgia Institute of Technology

    M.S. in Computer Science (HCI)
    • Human-Computer Interaction specialization.
  2. to

    University of Wisconsin–Madison

    Bachelor of Science
    • Double-majored in Computer Science and Communication.
    • TA for CS 559 (Computer Graphics) for 2 semesters.

Certifications

 Selected work

All 6 projects
Showreel My work in 20 seconds.
  1. Microsoft Mesh immersive event with avatars gathered around a shared presentation

    Microsoft Mesh

    01Professional, 2022–2024

    The shared UI framework every Mesh product team adopted, plus avatar editing and rendering across mobile, desktop, and AR/VR.
  2. FakeHaul checkout graphic showing a $1,248.50 cart reduced to $0.00

    FakeHaul

    02Personal, 2026

    A fake store where every checkout ends at $0.00, built with coding agents, an offline software factory, and my own product taste.
  3. Holos immersive training scene with an aircraft cockpit model and interactive instruction panels

    Holos

    03Professional, 2019–2022

    Hand interaction, multiplayer networking, and 3D content workflows I built for Holos’s AR and VR training platform.

 Publications

  1. Under review Dialogue & Discourse

    Preserving Action-Time Grounding Judgments in Transcript Trackers

    Sole-authored study of how transcript assistants that judge an earlier action by the conversation’s final state lose the evidence available when it happened, tested on a 264-dialogue benchmark built with Ground Tracker.

  2. Journal of Optics (IOP Publishing)

    Passive frequency conversion of ultraviolet images into the visible using perovskite nanocrystals

    Co-authored a peer-reviewed paper on a passive system that converts ultraviolet images into visible light using perovskite nanocrystals.

  3. UW–Madison technical report

    Breaking Binocular Redundancy Through Virtual Reality

    Wrote a technical report and released an open-source VR demo that simulates static and dynamic binocular rivalry.

  4. Strategically Yours

    Effective Leaders: Four Attributes That Underpin The Core Characteristics of Effective Leadership

    Contributed the article “Critical Conversation” to Mike Cameron’s collected volume.

 Writing

All writing
  1. How I built FakeHaul (opens on Medium)
  2. I leave AI models constantly. I have never once left the platform. (opens on Medium)
  3. I built a fake shopping app. Nothing ships. It might be the most useful thing I’ve made. (opens on Medium)
  4. My Top 5 Books of 2025 (opens on Medium)
  5. Making a Comeback Antaeus AR (opens on Medium)