Jack Yang
Software Engineer II · Microsoft Teams AI Based in Seattle

Jack Yang: Real-time AI for real conversations.

I’m Jack Yang, a software engineer at Microsoft Teams AI building real-time multimodal agents for Microsoft 365 Copilot. Before that, I built avatar and XR systems for Microsoft Mesh.

Book time with me Résumé LinkedIn
or email jackyangzzh@gmail.com

About

I build real-time AI agents at Microsoft Teams AI, where I work on Facilitator, the Microsoft 365 Copilot agent that runs inside live Teams meetings. My focus is the orchestration, tool execution, context management, and multimodal features that let an agent keep up with a conversation as it happens.

Before Teams AI, I was a key contributor to the avatar rendering pipelines and shared UI at Microsoft Mesh, which shipped weekly in 20+ languages to millions of users. Earlier, I built AR/VR interaction and multiplayer systems at Holos, a Techstars startup, and helped it secure a $750K U.S. Air Force contract.

Outside work, I created FakeHaul, an AI-generated marketplace where every checkout is $0.00, covered by The New York Times and Fast Company. I’m also completing an M.S. in HCI at Georgia Tech, with a paper under review at CHI 2027 on how AI systems that track decisions from transcripts can hide their own mistakes. I enjoy connecting low-latency systems with generative models, and I care as much about how the interaction feels as how the system is built.

Selected work

All 7 projects
  1. FakeHaul checkout graphic showing a $1,248.50 cart reduced to $0.00
    01Personal2026

    FakeHaul

    A pretend shopping marketplace where every checkout is $0.00 and nothing ships. The ritual of online shopping, without spending money.
  2. Microsoft Mesh immersive event with avatars gathered around a shared presentation
    02Professional2022–2024

    Microsoft Mesh & Avatars for Teams

    The shared UI, avatar editor, and rendering systems I worked on for Microsoft Mesh and Avatars in Teams, shipped weekly in 20+ languages to millions of users.
  3. Holos immersive training scene with an aircraft cockpit model and interactive instruction panels
    03Professional2019–2022

    Holos

    Hand interaction, multiplayer networking, and 3D content workflows I built for Holos’s AR and VR training platform.

Experience

Résumé (PDF)
Work
Research
Study
Current role Expected Select a bar to jump to the entry

Industry

  1. to PresentNow

    Microsoft Teams AI

    Software Engineer II
    • Build Facilitator, a meeting agent for Microsoft 365 Copilot that takes live notes, tracks agendas and decisions, and answers questions in Teams.
    • Design the turn-taking layer of a proactive agent, deciding when and how the AI speaks into a live group conversation without interrupting.
    • Contribute to conversation orchestration, tool calling, context management, observability, and multimodal LLM features.
  2. to
    Case study

    Microsoft Mesh

    Software Engineer II, Mixed Reality
    • Built avatar customization systems and shared UI components (input handling, accessibility) behind Microsoft Mesh and Avatars in Teams, supporting weekly releases and 20+ languages for millions of users. Satya Nadella showed the outfit-selection flow I built at Microsoft Ignite.
    • Key contributor to the avatar rendering and performance pipeline across HoloLens, Quest, web, and desktop. Profiled CPU and GPU frame budgets, using LOD switching and GPU instancing to maintain target framerates on constrained hardware.
  3. to
    Case study

    Holos

    AR/VR Interaction Engineer
    • Built core AR and VR training platform capabilities, including hand tracking, gesture recognition, multiplayer networking, and 3D model ingestion workflows.
    • Worked with leadership on product and engineering execution, helping Holos secure a $750K U.S. Air Force contract and a place in Techstars.

Research

  1. to
    Case study

    Kats Research Group

    XR Researcher
    • Developed a computer vision model that simulates color vision deficiency in hyperspectral images with 90% accuracy, and ran it in a color-calibrated Oculus Rift demo. Co-authored an arXiv publication.
  2. to
    Case study

    UW Graphics Group

    Graphics Researcher
    • Built a real-time VR control loop mapping hand gestures to remote robotic-arm manipulation, plus a Unity UI for replaying arm motion from 15+ lab-experiment datasets.
  3. to

    UW Human-Computer Interaction Group

    HCI Researcher
    • Added speech recognition and NLP-driven interaction to children’s reading-companion robots and validated the interaction in an eight-family trial.

Education

Degrees

  1. to Expected

    Georgia Institute of Technology

    M.S. in Computer Science (HCI)
    • Human-Computer Interaction specialization; in progress, expected May 2027.
  2. to

    University of Wisconsin–Madison

    Bachelor of Science
    • Double-majored in Computer Science and Communication.
    • TA for CS 559 (Computer Graphics) for 2 semesters.

Certifications

Publications

  1. Under review CHI 2027

    Too Late to Rely: A Self-Concealing Failure in State-Only Transcript Grounding Trackers

    Transcript-based trackers that keep only the latest state can hide their own failures: if someone acts on a decision before the conversation supports it, later agreement overwrites the warning that they acted too early. Used a 264-dialogue benchmark to show the failure and test an event-sourced record that preserves those warnings.

  2. Journal of Optics (IOP Publishing)

    Passive frequency conversion of ultraviolet images into the visible using perovskite nanocrystals

    Co-authored a peer-reviewed paper on a passive system that converts ultraviolet images into visible light using perovskite nanocrystals.

  3. UW–Madison technical report

    Breaking Binocular Redundancy Through Virtual Reality

    Wrote a technical report and released an open-source VR demo that simulates static and dynamic binocular rivalry.

  4. Strategically Yours

    Effective Leaders: Four Attributes That Underpin The Core Characteristics of Effective Leadership

    Contributed the article “Critical Conversation” to Mike Cameron’s collected volume.

Writing

All writing
  1. I leave AI models constantly. I have never once left the platform. Medium (opens on Medium)
  2. I built a fake shopping app. Nothing ships. It might be the most useful thing I’ve made. Medium (opens on Medium)
  3. My Top 5 Books of 2025 Medium (opens on Medium)
  4. Making a Comeback Antaeus AR (opens on Medium)