自动驾驶基础模型（英）

Foundation modelsFor autonomous drivingVincent VanhouckeDistinguished EngineerWaymop. 2Confidential and proprietaryOUR MISSION —Be the world’s most trusted driver.p. 2p. 3p. 4Atlantap. 4Up nextMiamiD.C.p. 4Seattle, WALas Vegas, NVNew York, NYLas Vegas, NVSan, Diego, CANew Orleans, LANashville, TNComing soonComing soonComing soonComing soonComing soonComing soon2025 Road trip citiesSan Diego, CALos Angeles, CASan Francisco Bay Area, CATruckee, CADeath Valley, NVAustin, TXMiami, FLAtlanta, GAWashington DCUpstate NYBuffalo, NYUpper Peninsula, MIMetropolitanDetroit Area, MITokyo, JapanWaymo tested citiesOperation coming soonWaymo operating citiesPhoenix, AZNew Orleans, LANashville, TNSTEP 4What should I do?p. 8STEP 1Where am I?STEP 2What’s around me?STEP 3What will happen next?p. 9p. 10Confidential and proprietaryUnusual BehaviorsExtreme WeatherUnique InteractionsStray CyclistsToddlers on the LooseFalling SkateboardersFalling TreesFloodingRoad BlockagesLast Minute Lane ChangesForeign Objects on the RoadLong tail of driving scenarios presents significant challengesA BBQ GrillFalling Off TruckMotion token sequence:t=1t=2t=3t=4MotionLM: Multi-Agent Motion Forecasting as Language ModelingAri Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S. Refaat, Rami Al-Rfou, Benjamin SappICCV 2023Driving as a ConversationTrajectories as sentences in a new languageVocabulary consists of state / motion words (vectors)Like language, trajectories have local continuity and global contextModel architecture is similar to that of a LLMp. 12ExamplesMARGINAL PREDICTIONCAUSAL JOINT “DIALOGUE”p. 13Scaling LawsOnce you have a good architecture, performance scales with model size and datap. 14Inference Scaling LawsPerformance scales with the amount of compute used at runtimeFor more on motion scaling laws, don’t miss Ben Sapp’s talk tomorrow at the Workshop on Distillation of Foundation Models for Autonomous Driving!(NEW!) Now on ArXivWaymo | Confidential & ProprietaryPost-training Preference Alignment Large scale driving demonstration dataLet’s remember what the expert did and copy them! Direct Post-Training Preference Alignment for Multi-Agent Motion Generation ModelUsing Implicit Feedback from Pre-training DemonstrationsThomas (Ran) Tian, Kratarth GoelICLR 2025, SpotlightMisalignment: by optimizing an incomplete or mis-specified objective,these models lead to undesirable behaviors at best and safety hazards at worst!Waymo | Confidential & ProprietaryReconcile the disparity between the next-token prediction objective and human preferences.User ranks MotionLM responses Pre-trained MotionLM Post-training Preference Alignment Direct Post-Training Preference Alignment for Multi-Agent Motion Generation ModelUsing Implicit Feedback from Pre-training DemonstrationsThomas (Ran) Tian, Kratarth GoelICLR 2025, SpotlightWaymo | Confidential & ProprietaryPre-trained traffic simulation modelAfter post-training alignmentyield to pedestriantoo close to pedestr