Articles

Large Language Model Article

<a target="_blank" rel="noopener noreferrer" href="https://cmscritic.com/critics/scottliewehr">Scott Liewehr</a><a target="_blank" rel="noopener noreferrer" href="https://cmscritic.com/critics/dan-drapeau"> </a>is Global Vice President of Market Strategy &amp; Growth at <a target="_blank" rel="noopener noreferrer" href="https://cmscritic.com/products/sitecore">Sitecore</a> and a CMS Critic contributor. <hr /> In my <a target="_blank" rel="noopener noreferrer" href="https://cmscritic.com/youre-writing-for-the-wrong-audience">last piece for CMS Critic</a>, I introduced the H2A→A2H framework and argued that the audience for your content has fundamentally changed now that LLMs are in the mix. AI now sits on both sides of the equation: one AI helps you create, and a second AI decides whether your brand shows up at all. If you haven’t read it, the short version is this: you’re writing for the wrong audience, and the invisible editor (the LLM) that mediates between your brand and your buyer rewards substance, structure, authority, and specificity, and ignores everything else.This piece is about what happens next. Specifically, it’s about one of the most sacred practices in digital marketing and why I think it’s about to collapse under the weight of its own logic.I’m talking about A/B testing.Here’s the argument: if the primary consumer of your content is now an LLM, why are you still testing it on humans?<h3 id="you’re-optimizing-for-the-wrong-audience">You’re optimizing for the wrong audience</h3>The entire experimentation infrastructure that marketing has built over the past 15 years assumes that human behavior is the metric that matters. You publish multiple versions of a page, split your traffic, measure which one earns more clicks, more time on page, more conversions. The humans tell you which version wins.That makes perfect sense in a human-to-human world, when a human creates content and another human consumes it. It even makes sense in an H2A2H world, where AI helps the marketer create or scale the content, but the audience is still human. But in an H2A→A2H world, where an LLM is deciding whether your content even reaches the human on the other end, you’ve got a measurement problem. You’re measuring human reactions to determine what works, but the LLM is the one separating signal from noise before any human sees it.Think about it this way. When a buyer asks an LLM to recommend a product, the model assembles an answer by evaluating sources across a range of factors, including the substance, structure, authority, and specificity of your brand content. It decides which brands to cite and which to ignore. If your brand doesn’t pass that filter, no human ever sees it. It doesn’t matter which headline variant tested better with your website visitors if the invisible editor never selected your brand in the first place.The experimentation question has to change. The old question was: which version does my human audience prefer? The new question is: which version does the invisible editor cite, and does it carry my brand position faithfully when synthesized into an answer? Those are fundamentally different questions, and traditional A/B testing was never designed to ask the second one.I said in my last article that in an H2A→A2H reality, the list doesn’t exist. There is only one single, synthesized answer. If you’re still optimizing for a list by testing on the humans reading it, you’re solving last decade’s problem.<h3 id="the-opportunity-cost-nobody-talks-about">The opportunity cost nobody talks about</h3>There’s a secondary argument here that I think is worth sitting with.When you build three versions of a landing page for multivariate testing, you believe in all three. The test is how you discover which performs best. But the discovery comes at a cost: two-thirds of your audience bounces, fails to convert, or disengages entirely. Those are missed opportunities; real prospects who encountered the weaker versions and moved on. The learning is valuable, but the tuition of missed pipeline is expensive.That cost would be easier to absorb if the learning were quick. A/B and multivariate testing demand statistical significance, which demands large sample sizes. For brands with massive traffic, this can happen reasonably quickly. But for the mid-market company with 15,000 monthly visitors? Running a rigorous test is a struggle. The sample sizes are too small, the test duration stretches into weeks or months, and by the time you have a statistically valid result, the market context may have shifted. For a meaningful number of brands, traditional experimentation has always been more aspirational than practical, and those brands that can’t reach significance are left guessing. The opportunity cost compounds on both ends. Large-audience brands pay a “learning tax” across a lot of visitors. Small-audience brands can’t afford to run the test at all. Neither situation is great, and both are artifacts of a model that requires real people as its crash-test dummies.In an AI-native model, both problems disappear. You generate hundreds of content variations using AI, then test every one against thousands of simulated buyer personas – AI agents that behave like real buyers with real intent, real objections, and real decision criteria – before a single real person ever sees any of it. The AI generates. The AI evaluates. The AI selects. And your entire audience gets the optimal version from the first impression.No humans interacted with the underperforming version. The mid-market brand with limited traffic runs the same caliber of test as the enterprise with millions of visitors. The learning happened upstream, in simulation, at machine speed, and prior to publishing.The experimentation paradigm inverts: you’re testing for your audience, not on them.<h3 id="the-assumption-that-needs-rethinking">The assumption that needs rethinking</h3>At this point, a reasonable marketer might think: “Okay, but don’t I still need experimentation infrastructure to evaluate all those variations? You’ve moved the testing upstream, but someone still has to figure out which version wins.”It’s a fair instinct, and it’s rooted in how we’ve thought about experimentation for years. But I think it reflects an assumption that doesn’t hold in an AI-native workflow.When the AI system that generated 200 variations can also evaluate them against simulated buyer personas, score them for likely performance, and select the winner, the “experimentation” has collapsed into the content intelligence layer itself. It’s a function of the orchestration system. The evaluation capability has been absorbed into an intelligent platform that does more. You don’t need a standalone experimentation product sitting between your CMS and your audience when the intelligence that created the content can also judge it.There’s also a scope issue. Traditional experimentation asks a narrow question: Which version gets more clicks, more conversions, more engagement? Those are optimization questions within a fixed frame. The AI-native model lets you question the frame itself. You can ask: Does this content earn citations from the invisible editor? Does it hold up when a buyer asks a follow-up question and the AI goes back to the well? Does it carry my brand position faithfully across different query contexts?That’s a different category of evaluation entirely, and it requires content intelligence, not experimentation tooling.<h3 id="what-this-means-practically">What this means practically</h3>If you’re running a traditional experimentation program today, I’m not suggesting you shut it off tomorrow. There’s still value in testing live experiences for behavioral signals that simulated personas can’t fully replicate, at least for now. Human beings are unpredictable (I have teenagers; I can confirm this), and the gap between simulated behavior and real behavior is still real.But the trajectory is clear, and the planning window is shorter than you might think. Here’s where I’d focus:<ul><li>Start running parallel tracks. Keep your live experimentation program, but begin building the capability to pre-test content against AI personas before publication. Measure the delta between what your simulated testing predicts and what your live tests confirm. As that delta shrinks, and it will, you’ll have the evidence to shift investment.</li><li>Start evaluating your content against AI consumption metrics alongside human engagement metrics. Your experimentation program should be measuring whether your content earns citations from LLMs, whether it surfaces in synthesized answers, whether your brand position survives the compression of the invisible editor. Traditional A/B testing was never designed to measure any of that.</li><li>And if you’re one of the many brands that never had enough traffic to run meaningful multivariate tests in the first place, recognize that the constraint just evaporated. AI-native experimentation doesn’t require a single visitor to your site. The barrier to entry that kept rigorous testing out of reach for mid-market and niche brands is gone.</li></ul><h3 id="the-bigger-picture">The bigger picture</h3>This connects back to the H2A→A2H framework in a way that I think matters. The old experimentation model was built for an H2H world; a world where you published content for humans, tested it on humans, and optimized it for humans. The H2A→A2H world adds a new constituency that sits upstream of your human audience and makes editorial decisions about whether they see your content at all. Experimentation has to evolve to serve both.The brands that figure this out won’t just have better conversion rates. They’ll stop paying the learning tax on their audience, they’ll unlock experimentation capabilities that were previously gated by traffic volume, and they’ll start optimizing for the audience that actually determines their visibility.The crash test dummies are about to be retired. And honestly, it’s about time. <hr /><h3 id="upcoming-events">Upcoming Events</h3>  <a target="_blank" rel="noopener noreferrer" href="https://kontent.ai/events-and-webinars/raise-amsterdam/?utm_campaign=awareness-fy26&amp;utm_source=cms-critic&amp;utm_medium=referral&amp;pi_campaign_id=117478"><img style="width:23.45%" src="https://cmscritic.com/ms-content/uploads/2026/04/raise-logo-kontent.jpg" alt="" /></a><h4 id="raise-amsterdam"><a target="_blank" rel="noopener noreferrer" href="https://kontent.ai/events-and-webinars/raise-amsterdam/?utm_campaign=awareness-fy26&amp;utm_source=cms-critic&amp;utm_medium=referral&amp;pi_campaign_id=117478">RAISE Amsterdam</a></h4>May 20, 2026 – Amsterdam, NetherlandsSponsored by Kontent.ai, RAISE Amsterdam brings together bold marketing and technology leaders to explore how agentic AI is reshaping the way ambitious brands create and deliver content and what it takes to use it safely, responsibly, and with proper governance. Experience the ideas that are shaping the future and dive into big thinking, innovative strategies, and expert insights. Held at the Klein Canvas, Volkshotel, you'll hear from world-class speakers in dynamic sessions that will reveal how AI-powered content operations are accelerating production, improving governance, and driving creative impact. <a target="_blank" rel="noopener noreferrer" href="https://kontent.ai/events-and-webinars/raise-amsterdam/?utm_campaign=awareness-fy26&amp;utm_source=cms-critic&amp;utm_medium=referral&amp;pi_campaign_id=117478">Limited spots are available, so book yours today.</a> <img style="width:30.13%" src="https://cmscritic.com/ms-content/uploads/2025/12/image-1765546475634.png" /><h4 id="umbraco-codegarden-2026"><a target="_blank" rel="noopener noreferrer" href="https://codegarden.umbraco.com/">Umbraco Codegarden 2026</a></h4>June 10–11, 2026 – Copenhagen, DKJoin us in Copenhagen (or online) for the biggest Umbraco conference in the world – two full days of learning, genuine conversations, and the kind of inspiration that brings business leaders, developers, and digital creators together. Codegarden 2026 is packed with both business and tech content, from deep-dive workshops and advanced sessions to real-world case studies and strategy talks. You’ll leave with ideas, strategies, and knowledge you can put into practice immediately. <a target="_blank" rel="noopener noreferrer" href="https://codegarden.umbraco.com/">Book your tickets today.</a> <a target="_blank" rel="noopener noreferrer" href="https://www.boye-co.com/conferences/cmsconnect26"><img style="width:41.96%" src="https://cmscritic.com/ms-content/uploads/2026/05/cms-connect-logo-26.jpg" /></a><h4 id="cms-connect-26"><a target="_blank" rel="noopener noreferrer" href="https://www.boye-co.com/conferences/cmsconnect26">CMS Connect 26</a></h4>August 5-6, 2026 – Montreal, CanadaThe best conferences create space for honest, experience-based conversations. Not sales pitches. Not hype. Just thoughtful exchanges between people who spend their days designing, building, running, and evolving digital experiences. CMS Connect brings together people who share real stories from their work and platforms and who are interested in learning from each other on how to make things better. Over two days in Montreal, you can expect practitioner-led talks grounded in experience, conversations about trade-offs, constraints, and decisions, and time to compare notes with peers facing similar challenges. Space is limited for this exclusive event, <a target="_blank" rel="noopener noreferrer" href="https://www.boye-co.com/conferences/cmsconnect26">so book your seats today.</a> <img style="width:36.69%" src="https://cmscritic.com/ms-content/uploads/2026/01/open-source-cms-26.jpg" /><h4 id="open-source-cms-26"><a target="_blank" rel="noopener noreferrer" href="https://www.boye-co.com/conferences/opensourcecms26">Open Source CMS 26</a></h4>October 20–21, 2026 – Utrecht, NetherlandsJoin us for the first annual edition of our prestigious international conference dedicated to making open source CMS better. This event is already being called the “missing gathering place” for the open source CMS community – an international conference with confirmed participants from Europe and North America. Be part of a friendly mix of digital leaders from notable open source CMS projects, agencies, even a few industry analysts who get together to learn, network, and talk about what really matters when it comes to creating better open source CMS projects right now and for the foreseeable future. <a target="_blank" rel="noopener noreferrer" href="https://www.boye-co.com/conferences/opensourcecms26/tickets">Book your tickets today.</a> <img style="width:35.73%" src="https://cmscritic.com/ms-content/uploads/2026/05/image-1778200565145.png" /><h4 id="contentstack-contentcon-2026">Contentstack <a target="_blank" rel="noopener noreferrer" href="https://www.contentstack.com/contentcon">ContentCon 2026</a></h4>September 30 - October 1, Amsterdam  /  October 27-28, 2026 – ChicagoContentstack’s annual customer conference is the premier event for executives, marketing leaders, and developers to redefine their digital experience strategy. This is your opportunity to step out of the "status quo" and into "elite" status, learning exactly how the world’s most successful brands are using the technology you already own to do the impossible. Enjoy a full day of interactive workshops, certifications, and inspirational on-stage sessions designed to help you become an expert on cutting-edge digital strategies and how to turn Contentstack's CMS and adaptive personalization tools into your greatest competitive advantage.<a target="_blank" rel="noopener noreferrer" href="https://www.jdayusa.com/"> </a><a target="_blank" rel="noopener noreferrer" href="https://www.contentstack.com/contentcon">Book your seats today.</a>

Stop Testing on Your Audience

scottliewehr

<hr /> If you’re as old as I am, you might remember that the biggest hit on <a target="_blank" rel="noopener noreferrer" href="https://en.wikipedia.org/wiki/Use_Your_Illusion_I">Guns N’ Roses’ Use Your Illusion</a> was the iconic rock ballad, “November Rain.” It was cold. It was sweet. And that Slash solo? Fire. It was the soundtrack of 1991.What you might not recall (or choose to forget) was that the same album offered a cover of the McCartney/Wings ode to James Bond, <a target="_blank" rel="noopener noreferrer" href="https://en.wikipedia.org/wiki/Live_and_Let_Die_(song)#Guns_N'_Roses_version">Live and Let Die.</a> It turned out to be a sleeper hit, ranking near the top of a double album that went platinum seven times. I bring this up because, sometimes, there’s no logic to predicting human behavior. It’s an illusion. You can try to explain why a cover of a weird post-Beatles tune – soaked with an appropriate layer of pre-Grunge acid – appealed to the kids of my generation. But it ended up being one of the most popular songs on an album that sold 39 million copies. <a target="_blank" rel="noopener noreferrer" href="https://www.apple.com/">Apple</a> also knows a lot about music. And phones. And movies. Heck, they seem to have all media caught in their intrepid net of tech domination. Case in point: At their annual <a target="_blank" rel="noopener noreferrer" href="https://developer.apple.com/wwdc25/">Worldwide Developers Conference (WWDC)</a> – which kicked off today and runs through June 13th – they introduced some stunning enhancements across nearly every one of their products.I watched most of the opening, and I could spend a lot of time conveying how breathtaking the new Vision Pro UI is, how it brings flat 2D images to life with dramatic depth. I could wax philosophically about the emergence of spatial content, and how Apple is altering the fabric of reality with drag-to-life capabilities for retail products (try adding a couch to your living room from a web page and seeing it appear IRL).Instead, I’m going to zero in on everyone’s favorite subject: AI. Because there’s not enough fatigue around a subject that everyone wants to bank their future roadmaps on. If I sound cynical, it’s because I’m subjected to a lot of demos in our CMS and DXP corner of the software world, and I’m always trying to splice the hype from the hope. So here's the deal: Coinciding with WWDC, <a target="_blank" rel="noopener noreferrer" href="https://machinelearning.apple.com/research/illusion-of-thinking">Apple just released some new research</a> that casts doubt on advanced AI models. The study sharply reveals a truth that many have already suggested: AI isn’t that smart, and it’s a long way from pulling any sentient, Skynet-level shenanigans.Good news? Maybe for those who were concerned about its propensity to go all self-aware Terminator on us. But for hype machines promoting AI’s grand promise of higher-order thinking and logic, this might sour the apples, so to speak. The research paper, <a target="_blank" rel="noopener noreferrer" href="https://machinelearning.apple.com/research/illusion-of-thinking">The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity</a>, is a comparative study of <a target="_blank" rel="noopener noreferrer" href="https://medium.com/@fahey_james/large-reasoning-models-lrms-an-overview-19837b72540f">Large Reasoning Models (LRMs).</a> It validates that they show promise on a variety of reasoning benchmarks, and there's clear evidence of that. But when the rubber meets the road, their limitations are, as Apple surmises, “insufficiently understood.”Said another way: Not good. In the open salvo of the research, the authors explain how they have systematically investigated the gaps that exist around mathematical and coding constraints, which impact the accuracy of a model’s output as the challenges become more difficult. Through some extensive experimentation, they arrived at a core problem:LRMs face a complete accuracy collapse beyond certain complexities. Wow. And a deeper dive into the paper's content – a 30-page railgun of diagrams comparing failure tracks along a horizon of problem complexity – fires a cannon of chaos, illuminating just how real these limitations might be.<h3 id="bad-apples">Bad Apples</h3>BTW, that’s actually a song title on Use Your Illusion I. So, what’s all the fuss about this research? Well, if you believe the doomsayers, it describes a “total collapse” scenario, suggesting that leading frontier LRMs from <a target="_blank" rel="noopener noreferrer" href="https://openai.com/">OpenAI</a>, <a target="_blank" rel="noopener noreferrer" href="https://www.anthropic.com/">Anthropic</a>, and DeepSeek (there's a <a target="_blank" rel="noopener noreferrer" href="https://cmscritic.com/trust-and-verify-when-internet-drama-meets-ai-reality">great article on that one here</a>) fall apart when trying to solve really complex problems – and they don’t yield any remarkable benefits over LLMs that operate downstream.This scenario is based on the paper’s conclusion that these models aren’t, in fact, able to “think” or reason for themselves. They’re simply mimicking whatever they’ve been trained on. When given new and highly complex problems, even the most advanced LRMs perform less reliably than the standard LLMs on basic tasks. This is a “big pause” moment given the untamed investment in LRMs, which claim superior logic and reasoning for solving mathematical proofs, making decisions, and processing diverse data types with multi-modal capabilities. We’ve known about their heftier computational requirements – more tokens, slower response times – but those limitations were poopooed in light of their potential to handle higher-order problems.  To conduct its tests, Apple employed a series of “puzzles” that allowed for greater control over the granularity of complexity dimensions (they also designed them to minimize contamination that’s common in other relative benchmarks). It then compared reasoning-level responses from LRMs (such as o3-mini, DeepSeek-R1, Claude-3.7-Sonnet-Thinking) against a non-reasoning LLM. The paper provides more specifics, but what they discovered is that frontier LRMs experience a “complete accuracy collapse,” which results in a kind of logic bedlam. As observed, there were surprising limitations in the LRMs’ ability to perform exact computations, including a failure to benefit from explicit algorithms and their inconsistent reasoning across the different puzzle types.There’s a weird phenomenon at play here. Given the increased complexity of the puzzles, the models should have engaged in more detailed chain of thought computations, thus consuming more resources. But instead, they consumed less, effectively giving up. Throwing in the towel, so to speak.Not knowing the “why” in this case is particularly harrowing. As Apple states in the paper, they believe the lack of systematic analyses investigating these questions is due to “limitations in current evaluation paradigms.” In other words, we don’t even know how to test for the problem. And that’s a problem.<h3 id="knockin’-on-heaven’s-door?">Knockin’ on Heaven’s Door?</h3>This all might seem a little morose, but no one’s saying the future of AI is DOA. Well, not me, anyway. Still, the plague of inaccuracies has been scaling as of late, and Apple has been at the center of it. In January, shortly after launching its shiny new <a target="_blank" rel="noopener noreferrer" href="https://www.apple.com/apple-intelligence/?cid=wwa-us-kwgo-features-AIBrand-AppleIntellligence--AppleIntelligence-Brand-AppleIntelligence-apple%20intelligence&amp;aosid=p240&amp;ken_pid=go~cmp-21372186377~adg-168674639212~ad-757331429164_kwd-833708590225~dev-c~ext-~prd-~mca-~nt-search&amp;token=f8ed3188-10e0-4de1-9527-7c1db3c81d53">Apple Intelligence</a> feature on late-model iPhones, the company temporarily pulled the feature after the technology <a target="_blank" rel="noopener noreferrer" href="https://www.cnn.com/2024/12/19/media/apple-intelligence-news-bbc-headline/index.html">produced fake and misleading</a> summaries of news headlines that appeared almost identical to regular push notifications. The reversal was jarring, given the heavy holiday marketing they had done to promote their new AI capabilities. But this isn’t just an Apple problem. AI hallucinations are as prevalent as ever. A <a target="_blank" rel="noopener noreferrer" href="https://www.arxiv.org/abs/2407.17468">July 2024</a> study from Cornell, the University of Washington, and the University of Waterloo revealed that top AI models still can’t be fully trusted, given their propensity for making stuff up.No doubt, the timing of this research release was carefully orchestrated. With so many new product upgrades and announcements – and a gnarly video of <a target="_blank" rel="noopener noreferrer" href="https://www.apple.com/leadership/craig-federighi/">Craig Federighi</a>, Apple’s SVP of Software Engineering, circling the roof of the Cupertino saucer in a Formula 1 racer – there was ample cover. This was comms strategy at its finest, stimulating just the right amount of public discourse.There are, of course, those who might suggest Apple’s delayed arrival in the “Big AI Game” might be a key motivator. While Apple has forged a reputation for innovation, they have also been known to lag on trends like TV, autonomous vehicles, and other initiatives that fizzled. Given every player’s struggles with AI accuracy, a vilification of LRMs might provide cannon fodder to blow up the competitive matrix in an already tight race.So was it an illusion? Was this the LRM hype machine in overdrive, with Slash shredding outside the chapel on a dusty plain? I think we can safely say that some of it was. And herein lies the problem: With AI, we’re moving to product so quickly, we don’t always have the answers. We’re often not even asking the right questions. This research might be the best example of that.For the CMS and DXP industries, Apple is blazing some interesting trails with its portal to spatial content. There's a lot to unpack regarding the storage and structure of that content and expressing it in three dimensions, and that's exciting stuff that companies like <a target="_blank" rel="noopener noreferrer" href="https://cmscritic.com/first-look-cloudinarys-new-ai-vision-brings-brand-safe-gen-ai-capabilities-to-its-dam">Cloudinary</a> are already preparing for. At the same time, this AI research is offering us a moment of clarity that can't be overstated. As I said earlier this year in my <a target="_blank" rel="noopener noreferrer" href="https://cmscritic.com/in-a-fast-and-furious-tech-world-cms-kickoff-25-was-a-love-letter-to-slow-down">CMS Kickoff 25 wrap,</a> we need to slow down and be more thoughtful in this race to harness AI. The risks and consequences are simply too great, and the accuracy of content is in the crosshairs. Maybe we can use the illusion as a signal – and find better pathways to test, analyze, and validate what we’re building.Slowing down is hard advice to follow, especially in this hyperactive digital marketplace. But it might be the best prescription for writing the next hit. That – and making sure Slash gets his solo.  <hr /><h3 id="upcoming-events">Upcoming Events</h3> <img style="width:31.83%" src="https://cmscritic.com/ms-content/uploads/2025/01/cms-connect-2025-logo.png" alt="" /><h4 id="cms-connect-25"><a target="_blank" rel="noopener noreferrer" href="https://www.boye-co.com/conferences/cmsconnect25">CMS Connect 25</a></h4>August 5-6, 2025 – Montreal, CanadaWe are delighted to present the second annual summer edition of our signature global conference dedicated to the content management community! CMS Connect will be held again in beautiful Montreal, Canada, and feature a unique blend of masterclasses, insightful talks, interactive discussions, impactful learning sessions, and authentic networking opportunities. Join vendors, agencies, and customers from across our industry as we engage and collaborate around the future of content management – and hear from the top thought leaders at the only vendor-neutral, in-person conference exclusively focused on CMS. Space is limited for this event, <a target="_blank" rel="noopener noreferrer" href="https://www.boye-co.com/conferences/cmsconnect25">so book your seats today.</a>

Lose Your Illusion: Apple Sings a Sad Ballad About the Limitations of Reasoning Models

matt-garrepy

Explore a wide range of informative articles on Large Language Model, insights, and industry updates. Stay informed and ahead of the curve with our diverse collection of news and posts covering the latest trends and developments in the industry

News

Stop Testing on Your Audience

Lose Your Illusion: Apple Sings a Sad Ballad About the Limitations of Reasoning Models