Software engineerSeattle

Jack Yang

Building Real-Time Multimodal Agents for Live Human-AI Interaction

Jack Yang
View projects Email Me Resume PDF (opens in a new tab) Schedule a Call (opens in a new tab)

I build real-time AI agents and interactive systems that have to hold up inside the latency, noise, and unpredictable turn-taking of live human conversation.

At Microsoft Teams AI, I work on Facilitator in Microsoft 365 Copilot, focusing on conversation orchestration, tool execution, context management, and multimodal LLM features for meetings across millions of enterprise users.

My background spans 0-to-1 startups and deep systems engineering: I built AR/VR interaction and multiplayer platforms at Holos (a fast-moving Techstars startup), then led avatar rendering pipelines and shared UI at Microsoft Mesh.

Outside of work, I created FakeHaul, an AI-generated marketplace covered by The New York Times and Fast Company. Bridging low-latency client runtimes with generative models is where I spend my time: building software where deep engineering rigor and polished interaction design cannot be separated.

Location
Seattle, United States
Email
GitHub
jackyangzzh
Medium
jackyangzzh
LinkedIn
jackyangzzh

Experience

– present

Software Engineer II, Teams AI at Microsoft

Highlights

  • Build Facilitator, a meeting agent for Microsoft 365 Copilot that takes live notes, tracks agendas and decisions, and answers questions in Teams.
  • Contribute to conversation orchestration, tool calling, context management, observability, and multimodal LLM features.

–

Software Engineer II, Mixed Reality (Mesh) at Microsoft

Highlights

  • Built avatar customization systems and shared UI components (input handling, accessibility) behind Microsoft Mesh and Avatars in Teams, supporting weekly releases and 20+ languages for millions of users.
  • Owned the avatar rendering and performance pipeline across HoloLens, Quest, web, and desktop. Profiled CPU and GPU frame budgets, using LOD switching and GPU instancing to maintain target framerates on constrained hardware.

–

AR/VR Interaction Engineer at Holos

Highlights

  • Built core AR and VR training platform capabilities, including hand tracking, gesture recognition, multiplayer networking, and 3D model ingestion workflows.
  • Worked with leadership on product and engineering execution, helping Holos secure a $750K U.S. Air Force contract and a place in Techstars.

Research

–

XR Researcher (part-time) at Kats Research Group

Highlights

  • Developed a computer vision model that simulates color vision deficiency in hyperspectral images. It reached about 90% agreement in pilot color tests and ran in a color-calibrated Oculus Rift demo.

–

Graphics Researcher (part-time) at UW Graphics Group

Highlights

  • Built a real-time VR control loop mapping hand gestures to remote robotic-arm manipulation, plus a Unity UI for replaying arm motion from 15+ lab-experiment datasets.

–

HCI Researcher (part-time) at UW HCI Group, People and Robots Lab

Highlights

  • Added speech recognition and NLP-driven interaction to children's reading-companion robots; validated in an eight-family trial.

Education

–

M.S. in Computer Science (HCI) from Georgia Institute of Technology

Courses

  • Human-Computer Interaction specialization; in progress, expected May 2027.

–

Bachelor of Science from University of Wisconsin–Madison

Courses

  • Double-majored in Computer Science and Communication.
  • TA for CS 559 (Computer Graphics) for 2 semesters.

Certifications

Skills

AI & Agent Systems
Keywords:
  • LLM Agents & Orchestration
  • Tool Calling & Context Management
  • Retrieval-Augmented Generation (RAG)
  • Multimodal AI
  • Agent Interoperability (MCP / A2A)
  • Python / PyTorch
Software & Cloud
Keywords:
  • Python
  • C# / C++
  • TypeScript / React
  • Azure / Cloud Services
  • Observability & Performance
  • Accessible UI Engineering
Real-Time & Spatial Computing
Keywords:
  • Unity / Unreal Engine
  • Computer Vision / OpenCV
  • WebXR / Three.js
  • visionOS / Swift
  • GPU Rendering / Vulkan / OpenGL

Publications

Passive frequency conversion of ultraviolet images into the visible using perovskite nanocrystals

Co-authored a peer-reviewed paper on a passive system that converts ultraviolet images into visible light using perovskite nanocrystals.

Breaking Binocular Redundancy Through Virtual Reality

Wrote a technical report and released an open-source VR demo that simulates static and dynamic binocular rivalry.

Effective Leaders: Four Attributes That Underpin The Core Characteristics of Effective Leadership

Contributed the article “Critical Conversation” to Mike Cameron’s collected volume.

© Jack Yang