{"id":57831,"date":"2025-12-21T17:00:00","date_gmt":"2025-12-21T11:30:00","guid":{"rendered":"https:\/\/officechai.com\/?p=57831"},"modified":"2025-12-10T22:29:58","modified_gmt":"2025-12-10T16:59:58","slug":"googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models","status":"publish","type":"post","link":"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/","title":{"rendered":"Google&#8217;s Logan Kilpatrick Explains Why Even Small Pre-Training Gains Lead To Big Improvements In Models"},"content":{"rendered":"\n<p>Google DeepMind had indicated that it was <a href=\"https:\/\/officechai.com\/ai\/improvements-in-both-pretraining-and-post-training-led-to-gemini-3s-impressive-performance-google-vp\/\">still seeing gains<\/a> in pre-training when it had released the Gemini 3 Pro model, and it turns out that even small gains in pre-training can have a big impact on the overall quality of the model.<\/p>\n\n\n\n<p>Logan Kilpatrick, Group Product Manager at Google DeepMind, recently addressed mounting <a href=\"https:\/\/officechai.com\/ai\/weve-already-run-out-of-human-data-to-train-ai-models-elon-musk\/\">speculation <\/a>about whether pre-training\u2014the foundational phase of AI model development\u2014has hit a wall. His response cuts through the noise with a clear message: the gains are real, and more importantly, they compound in unexpected ways throughout the model development pipeline.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img data-recalc-dims=\"1\" fetchpriority=\"high\" decoding=\"async\" width=\"640\" height=\"358\" src=\"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?resize=640%2C358\" alt=\"\" class=\"wp-image-52886\" srcset=\"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?resize=1024%2C572&amp;ssl=1 1024w, https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?resize=300%2C168&amp;ssl=1 300w, https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?resize=768%2C429&amp;ssl=1 768w, https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?resize=1536%2C858&amp;ssl=1 1536w, https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?w=1600&amp;ssl=1 1600w, https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?w=1280 1280w\" sizes=\"(max-width: 640px) 100vw, 640px\" \/><\/figure>\n\n\n\n<p>When <a href=\"https:\/\/x.com\/ForwardFuture\/status\/1998431449163210984?s=20\">asked <\/a>directly about whether pre-training scaling has plateaued, Kilpatrick framed the issue not as a matter of belief but of empirical fact. &#8220;It&#8217;s not whether or not you believe it, it&#8217;s whether or not it&#8217;s true,&#8221; he said. &#8220;And I think the truth is this: pre-training continues to deliver real gains.&#8221;<\/p>\n\n\n\n<p>But Kilpatrick&#8217;s most compelling insight came when he explained the mechanics of how these gains translate into final model performance. &#8220;The most important part is that the gains that you see in pre-training, even if the exponential is slightly shallower than it was before where we were in the early days of scaling up, principally every other step in the model development process is an amplification of the underlying pre-trained model,&#8221; he explained.<\/p>\n\n\n\n<p>He used a vivid metaphor to drive the point home: &#8220;So if you have a beast of a pre-trained model, and even if you&#8217;re only eking out a couple of percentage points increases in overall capability during pre-training, the other steps of the process amplify and are exponential on top of that.&#8221;<\/p>\n\n\n\n<p>The implications of Kilpatrick&#8217;s comments are significant for the AI industry. His explanation suggests that even modest improvements in pre-training\u2014say, a 2-3% gain in base capabilities\u2014can translate into substantially larger improvements in the final model after post-training techniques like reinforcement learning from human feedback, fine-tuning, and instruction tuning are applied. This multiplicative effect means that the returns on pre-training investments may be far greater than raw benchmark numbers suggest, challenging the narrative that scaling laws have broken down. The perspective also helps explain why major AI labs continue to invest heavily in larger training runs despite seemingly diminishing returns\u2014the downstream amplification makes even incremental pre-training gains worthwhile. And while there were concerns that companies had <a href=\"https:\/\/officechai.com\/stories\/data-is-the-fossil-fuel-of-ai-it-will-get-exhausted-ilya-sutskever\/\">already used up all the data<\/a> that was available to pre-train models, it appears that even small gains at this step could lead to significant improvements in model quality.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Google DeepMind had indicated that it was still seeing gains in pre-training when it had released the Gemini 3 Pro model, and it&#8230;<\/p>\n","protected":false},"author":1,"featured_media":52886,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1029],"tags":[],"class_list":["post-57831","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Google&#039;s Logan Kilpatrick Explains Why Even Small Pre-Training Gains Lead To Big Improvements In Models<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Google&#039;s Logan Kilpatrick Explains Why Even Small Pre-Training Gains Lead To Big Improvements In Models\" \/>\n<meta property=\"og:description\" content=\"Google DeepMind had indicated that it was still seeing gains in pre-training when it had released the Gemini 3 Pro model, and it...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/\" \/>\n<meta property=\"og:site_name\" content=\"OfficeChai\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/OfficeChai\/\" \/>\n<meta property=\"article:published_time\" content=\"2025-12-21T11:30:00+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?fit=1600%2C894&ssl=1\" \/>\n\t<meta property=\"og:image:width\" content=\"1600\" \/>\n\t<meta property=\"og:image:height\" content=\"894\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"OfficeChai Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@OfficeChai\" \/>\n<meta name=\"twitter:site\" content=\"@OfficeChai\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"OfficeChai Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/\",\"url\":\"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/\",\"name\":\"Google's Logan Kilpatrick Explains Why Even Small Pre-Training Gains Lead To Big Improvements In Models\",\"isPartOf\":{\"@id\":\"https:\/\/officechai.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?fit=1600%2C894&ssl=1\",\"datePublished\":\"2025-12-21T11:30:00+00:00\",\"author\":{\"@id\":\"https:\/\/officechai.com\/#\/schema\/person\/5861f1134993293cc28905de7624d6b2\"},\"breadcrumb\":{\"@id\":\"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/#primaryimage\",\"url\":\"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?fit=1600%2C894&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?fit=1600%2C894&ssl=1\",\"width\":1600,\"height\":894},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/officechai.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Google&#8217;s Logan Kilpatrick Explains Why Even Small Pre-Training Gains Lead To Big Improvements In Models\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/officechai.com\/#website\",\"url\":\"https:\/\/officechai.com\/\",\"name\":\"OfficeChai\",\"description\":\"Startups, Businesses And Careers\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/officechai.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/officechai.com\/#\/schema\/person\/5861f1134993293cc28905de7624d6b2\",\"name\":\"OfficeChai Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/officechai.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/61d744733248dc647d505d0676bb425323413132ee5447e86aa8eecbbb7b27d5?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/61d744733248dc647d505d0676bb425323413132ee5447e86aa8eecbbb7b27d5?s=96&d=mm&r=g\",\"caption\":\"OfficeChai Team\"},\"description\":\"Dotting the i's, crossing the t's.\",\"url\":\"https:\/\/officechai.com\/author\/admin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Google's Logan Kilpatrick Explains Why Even Small Pre-Training Gains Lead To Big Improvements In Models","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/","og_locale":"en_US","og_type":"article","og_title":"Google's Logan Kilpatrick Explains Why Even Small Pre-Training Gains Lead To Big Improvements In Models","og_description":"Google DeepMind had indicated that it was still seeing gains in pre-training when it had released the Gemini 3 Pro model, and it...","og_url":"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/","og_site_name":"OfficeChai","article_publisher":"https:\/\/www.facebook.com\/OfficeChai\/","article_published_time":"2025-12-21T11:30:00+00:00","og_image":[{"width":1600,"height":894,"url":"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?fit=1600%2C894&ssl=1","type":"image\/png"}],"author":"OfficeChai Team","twitter_card":"summary_large_image","twitter_creator":"@OfficeChai","twitter_site":"@OfficeChai","twitter_misc":{"Written by":"OfficeChai Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/","url":"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/","name":"Google's Logan Kilpatrick Explains Why Even Small Pre-Training Gains Lead To Big Improvements In Models","isPartOf":{"@id":"https:\/\/officechai.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/#primaryimage"},"image":{"@id":"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?fit=1600%2C894&ssl=1","datePublished":"2025-12-21T11:30:00+00:00","author":{"@id":"https:\/\/officechai.com\/#\/schema\/person\/5861f1134993293cc28905de7624d6b2"},"breadcrumb":{"@id":"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/#primaryimage","url":"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?fit=1600%2C894&ssl=1","contentUrl":"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?fit=1600%2C894&ssl=1","width":1600,"height":894},{"@type":"BreadcrumbList","@id":"https:\/\/officechai.com\/ai\/googles-logan-kilpatrick-explains-why-even-small-pre-training-gains-lead-to-big-improvements-in-models\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/officechai.com\/"},{"@type":"ListItem","position":2,"name":"Google&#8217;s Logan Kilpatrick Explains Why Even Small Pre-Training Gains Lead To Big Improvements In Models"}]},{"@type":"WebSite","@id":"https:\/\/officechai.com\/#website","url":"https:\/\/officechai.com\/","name":"OfficeChai","description":"Startups, Businesses And Careers","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/officechai.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/officechai.com\/#\/schema\/person\/5861f1134993293cc28905de7624d6b2","name":"OfficeChai Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/officechai.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/61d744733248dc647d505d0676bb425323413132ee5447e86aa8eecbbb7b27d5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/61d744733248dc647d505d0676bb425323413132ee5447e86aa8eecbbb7b27d5?s=96&d=mm&r=g","caption":"OfficeChai Team"},"description":"Dotting the i's, crossing the t's.","url":"https:\/\/officechai.com\/author\/admin\/"}]}},"jetpack_featured_media_url":"https:\/\/i0.wp.com\/officechai.com\/wp-content\/uploads\/2025\/06\/unnamed-2.png?fit=1600%2C894&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p685C6-f2L","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/posts\/57831","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/comments?post=57831"}],"version-history":[{"count":1,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/posts\/57831\/revisions"}],"predecessor-version":[{"id":57832,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/posts\/57831\/revisions\/57832"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/media\/52886"}],"wp:attachment":[{"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/media?parent=57831"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/categories?post=57831"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/officechai.com\/wp-json\/wp\/v2\/tags?post=57831"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}