215 jobs for "Software Engineering Researcher"

Applied Researcher: On-Device Multimodal Reasoning

At Apple - Sunnyvale

Posted on Oct 19

View job Apply on company site

2027 Summer Intern - Research and Development - Defense Lab Support

At General Motors - Warren

Posted on Oct 12

View job Apply on company site

Applied Researcher 4 (AI Safety & Multimodal)

At Capital One - New York · McLean

Posted on Nov 6

View job Apply on company site

Software Engineer - Face ID

At Apple - Sunnyvale

Posted on Oct 19

View job Apply on company site

2027 Summer Intern - Research and Development - Defense Lab Modeling and Simulation

At General Motors - Warren

Posted on Oct 12

View job Apply on company site

Entry-Level - Defense Lab Modeling and Simulation Researcher

At General Motors - Warren

Posted on Oct 12

View job Apply on company site

Applied Researcher 5 (AI Foundations)

At Capital One - New York · McLean

Posted on Nov 5

View job Apply on company site

Machine Learning Engineer, Apple Services Engineering

At Apple - Seattle

Posted on Oct 19

View job Apply on company site

Applied Researcher 4 (AI Foundations, VLM Customization)

At Capital One - San Jose · McLean

Posted on Nov 1

View job Apply on company site

Design Engineer: Optical Sensor System - Integrated Optics

At ASML - Wilton

Posted on Sep 18

View job Apply on company site

Backend Software Engineer, TikTok Live Recommendation Infrastructure

At TikTok - San Jose

Posted on Apr 1

View job Apply on company site

Applied AI Engineering Lead - VP, Markets Operations

At JPMorgan Chase - London

Posted on Oct 5

View job Apply on company site

Senior Software Engineer

At Labcorp - Bloomfield

Posted on Mar 30

View job Apply on company site

Offensive Security Researcher

At Salesforce - Tel Aviv

Posted on Oct 7

View job Apply on company site

Lead Data Scientist - Nike Sports Research Lab - Nike Innovation

At NIKE, Inc. - Beaverton

Posted on Oct 3

View job Apply on company site

Embedded Systems Design Engineer

At Eaton - Dublin

Posted on Jul 9

View job Apply on company site

Senior Principal Software Engineer (C++-low level-Prisma Browser)

At Palo Alto Networks - Santa Clara

Posted on Sep 24

View job Apply on company site

Principal Security Researcher (AI-Assisted Vulnerability Research)

At Palo Alto Networks - Santa Clara

Posted on Jun 6

View job Apply on company site

Software Engineer - Model Infrastructure, TikTok Feeds

At TikTok - San Jose

Posted on Jun 12

View job Apply on company site

Lead Software Engineer (AI Platform) - Vice President

At JPMorgan Chase - New York

Posted on Oct 1

View job Apply on company site

Apple

Applied Researcher: On-Device Multimodal Reasoning

Sunnyvale, CA$150,400 – $277,600Posted 20 days ago

Apply on company site View full job

The Video Computer Vision (VCV) organization is an applied research and engineering team developing real-time, on-device Computer Vision and Machine Perception technologies across Apple products. Within VCV, our team builds next-generation multimodal AI systems that combine on-device multimodal encoders, large language models, and foundation models to create intelligent systems capable of understanding, reasoning, and acting across language, vision, audio, and tools. Our work is deeply integrated into the Apple ecosystem, partnering across hardware, software, and ML teams to deliver real-time, scalable, and privacy-preserving experiences reaching millions of users.

Description

We are seeking an Applied Researcher with deep expertise in multimodal reasoning at small model scale - making vision-language models in the smaller regime (under ~10B parameters, down to sub-1B) think, plan, and act reliably under strict compute, memory, and latency constraints. In this role you will own the reasoning side of the on-device multimodal stack: designing compact VLMs that reason over images, video, and 3D scene content; compressing and distilling the reasoning process itself; and engineering the decoding and inference path that makes multi-step reasoning affordable on an Apple device. This role offers the unique opportunity to define what on-device intelligence looks like for hundreds of millions of users. You'll push the boundaries of what small models can achieve - enabling real-time multimodal understanding and multi-step reasoning without reliance on cloud connectivity. You'll collaborate with hardware teams, compiler engineers, and ML researchers to unlock capabilities that few organizations can deliver at Apple's scale and quality bar.

This role spans multiple dimensions of efficient on-device reasoning - including VLM architecture and connector design, reasoning post-training (SFT/RL), chain-of-thought compression, speculative and structured decoding, visual token reduction, quantization and distillation, and hardware-aware inference optimization.

A core focus of this role is efficient reasoning: compressed and latent chain-of-thought, reasoning distillation from frontier teachers, adaptive test-time compute (knowing when - and how long - to think), speculative and structured decoding, KV-cache compression, and visual-token efficiency. A second focus is reasoning over real-time visual perception experts. Rather than consuming pixels alone, the VLM should be able to invoke and reason over the outputs of specialist on-device vision models - feed-forward 3D scene and geometry estimators (VGGT-style reconstruction, depth, camera pose), human body and hand mesh/pose recovery, object detectors, localizers, and trackers - and fuse those structured, metric outputs into its reasoning about the scene. This raises real research questions: how to represent geometry, body parameters, and detections compactly in a token-budgeted context; how to schedule which experts run at which frame rate within a real-time budget; and how to train a small model to invoke, trust, and cross-check them. Efficiency is treated as a first-class metric here: reasoning quality is measured at a fixed latency, memory, and power budget.","responsibilities":"Design, train, and post-train compact vision-language models (under ~10B, including sub-1B) that perform multi-step visual reasoning, grounded visual understanding, and language generation within on-device resource budgets

Research and implement efficient reasoning techniques - compressed and latent chain-of-thought, reasoning-trace distillation, early-exit and budget-aware reasoning, adaptive compute allocation, and test-time scaling that maximizes reasoning quality per FLOP

Own the decoding stack for on-device inference: speculative and self-speculative decoding, draft models and multi-token prediction, structured/constrained generation, KV-cache compression and quantization, prefill/decode scheduling, and streaming latency (TTFT, tokens/sec)

Build reasoning over real-time perception experts: enable a small VLM to invoke and reason over on-device 3D scene reconstruction and geometry (VGGT-style feed-forward reconstruction, depth, camera pose), human body/hand pose and mesh recovery, object detection, localization, and tracking - designing the representations, interfaces, and training signals that make those outputs usable inside a limited context

Develop structured-output fusion and expert scheduling: compact tokenizations for geometry, body parameters, and detections; policies for which perception models run at which resolution and frame rate; and mechanisms for the reasoner to resolve conflicts between experts and its own visual features

Apply reasoning-focused post-training: supervised distillation from frontier teachers, preference and RL methods (GRPO, RLVR, STaR, rejection sampling), verifier- and reward-guided decoding, and process supervision for multimodal and spatially grounded chains

Drive visual token efficiency and feature compression: token pruning, merging, and resampling, adaptive resolution and frame-rate policies, and learned connectors that preserve reasoning accuracy at a fraction of the visual token budget

Develop distillation, pruning, and quantization strategies that preserve multimodal reasoning fidelity at reduced model sizes - including QAT and mixed-precision inference across the language decoder, vision encoder, and perception experts

Explore hybrid architectures combining stateful components (SSM/Mamba, linear attention) with attention-based components to balance long-context visual reasoning - long video, multi-image, extended dialogue - against inference efficiency and bounded memory

Optimize reasoning and encoder architectures for Apple silicon, including Neural Engine, GPU, and ANE-aware design patterns; profile and iterate on latency, memory footprint, thermal behavior, and power consumption across Apple's device portfolio

Partner with our visual representation and video-encoder efforts (self-supervised and joint-embedding pretraining, streaming/stateful encoders, world models) to make representations reasoning-ready, and co-design the encoder-LLM interface

Build evaluation frameworks that measure reasoning fidelity, visual and spatial grounding, hallucination, and robustness under device budgets, including accuracy-vs-latency and accuracy-vs-power trade-off curves

Collaborate with hardware, compiler, and platform teams to co-design model architectures that exploit device-specific acceleration capabilities

Preferred Qualifications

PhD with research in efficient multimodal reasoning, LLM reasoning, model compression, efficient inference/decoding, or lightweight VLM architectures

Experience training or post-training vision-language models end-to-end - connector/projector design, visual instruction tuning, resolution and token-budget trade-offs, small-model recipes

Hands-on experience with reasoning techniques: chain-of-thought distillation and compression, latent/implicit reasoning, reward-guided decoding, RL for reasoning (GRPO, RLVR, STaR), or test-time compute allocation

Expertise in decoding and serving optimizations: speculative decoding, structured/grammar-constrained generation, KV-cache quantization and eviction, continuous batching, long-context inference

Experience combining LLMs with real-time perception models - 3D reconstruction and geometry (VGGT, DUSt3R/MASt3R-style, SLAM, monocular depth), human pose and body/hand mesh recovery (SMPL-family), detection, segmentation, or tracking - and with spatial or 3D-grounded reasoning and embodied/spatial VQA

Experience deploying LLM or multimodal models on mobile or edge hardware (CoreML, MLX, TensorRT-LLM, or equivalent), with attention to ANE/GPU kernel and memory constraints

Experience with quantization-aware training, mixed-precision inference, and knowledge distillation for vision-language models

Familiarity with efficient vision encoders and self-supervised/joint-embedding pretraining (V-JEPA, I-JEPA, MAE, DINO/DINOv2, SigLIP, CLIP), Mamba/SSM vision backbones, or streaming architectures for real-time video with fixed memory budgets

Interest in Video-LLMs, long-video reasoning, and world models for prediction and planning

Publication record in top-tier venues is a plus (NeurIPS, ICML, ICLR, CVPR, ECCV, ACL, MLSys, ICRA, etc.)

Minimum Qualifications

MS in Computer Science, Machine Learning, AI, Computer Vision, or a related field (or equivalent practical experience)

Strong foundation in deep learning, with specific experience in LLM or VLM training, post-training, or inference optimization

Demonstrated experience working with multimodal models (vision-language models, multimodal LLMs) in resource-constrained environments, including hands-on work with reasoning quality, decoding, or model compression

Proficiency in Python and modern deep learning frameworks (PyTorch preferred), with familiarity with inference and optimization toolchains (quantization, distillation, pruning - e.g., vLLM/SGLang, llama.cpp, MLX, CoreML)

Pay & Benefits

At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $150,400 and $277,600, and your base pay will depend on your skills, qualifications, experience, and location.

Apple employees also have the opportunity to become an Apple shareholder through participation in Apple's discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple's Employee Stock Purchase Plan. You'll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses - including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits

Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.

About Apple

Working at Apple means doing more than you ever thought possible and having more impact than you ever imagined.

Size: 10,000+ employees
Industry: Technology

View company profile