Vision-Language Models (VLMs) are a foundational pillar of our Autonomy stack. In this Staff
Research Engineer role, you will play a key role in delivering the overarching VLM strategy,
especially training, shipping, optimizing the VLM models, as well as extending to
multi-modalities and enabling new use cases, among others. In this role, you will also be
responsible to define and deliver VLM-driven solutions to solve some of autonomy's hardest
challenges, including automated data mining, handling long-tail distributions, rare edge-case
detection, and scene anomaly reasoning. As part of the model delivery, you will also own the
whole end-to-end lifecycle of VLM model delivery: data acquisition, metrics definition,
benchmarking, model performance optimization, deployment, feedback loop.
Drive and deliver the VLM model strategy: Define, drive and execute the roadmap of
VLM model delivery, including training and delivering VLM models, optimization,
deployment, as well as the extension to other multi-modalities.
- Accelerate data mining: Design and deliver VLM/LLM related models and strategies
that power automated data mining, long-tail distributions, rare/edge case detection, and
anomaly detection at scale, across multiple modalities (vision, lidar, text, etc).
- Iterate and optimize performance: Establish rigorous evaluation and monitoring
benchmarks. Identify and root-cause top-tier system anomalies, prioritizing high-impact
optimizations to continuously push the needle on performance.
- Cross-functional collaboration: Partner closely with core Autonomy teams
(Perception, Planning, Calibration, Systems, etc) to translate vehicle feature
requirements into concrete ML deliverables.
- Influence trade-offs & requirements: Define system requirements and guide
cross-functional efforts through technical trade-off decisions.
Education: BS, MS, or PhD in Computer Science, Robotics, Electrical Engineering, or a
highly related quantitative field.
- Experience: 5+ years of professional experience scaling ML solutions, with a strong
focus on the following:
- VLM model training: Hands-on experience training or fine-tuning VLMs using
modern parameter-efficient techniques (LoRA, QLoRA) and RL alignment.
- Large-scale data mining: Proven track record developing VLM/LLM-related
techniques for data mining, long-tail distributions, rare cases, safety-critical
events.
- Zero/few-shot capabilities: Experience with open-vocabulary, zero-shot, or
few-shot classification models, particularly in long-tail scenarios.
- System engineering: Strong proficiency in Python alongside a solid
understanding of modern Perception pipelines, benchmarking tools, and
infrastructure.
- Execution: Demonstrated ability to root-cause complex issues across a
distributed, cross-functional stack in a fast-paced environment.
Preferred Qualifications
- Experience applying VLMs within the Autonomous Vehicle domain.
- Experience with Auto Prompt Optimization (APO) and automated prompt engineering
techniques.
- Experience with spatial grounding in 2D and/or 3D.
- Experience extending foundational models to extra modalities (e.g., LiDAR, Radar, IMU,
ego-motion).
- Experience utilizing VLMs or Foundation Models for complex behavior reasoning and
planning.
- Experience with onboard edge deployment, cloud inference architectures, and balancing
compute/efficiency trade-offs.
- Experience with quantization techniques (PTQ, QAT) and high-performance inference
engines like TensorRT.
