Articles

Experimentation Article

<hr /> I’ve been following <a target="_blank" rel="noopener noreferrer" href="https://cmscritic.com/products/amplitude">Amplitude</a> for several years, specifically throughout this promising (if not tumultuous) period of AI disruption. The company has continuously pedaled the innovation wheels, exploring new ways to improve insights for marketers and brands as the digital world furiously changes around us. For example, we covered the introduction of its <a target="_blank" rel="noopener noreferrer" href="https://cmscritic.com/replay-it-again-sam-amplitudes-session-replay-everywhere-rocks-with-analytics-visualizations">Session Replay Everywhere</a> earlier this year, which expands on the company’s initial launch <a target="_blank" rel="noopener noreferrer" href="https://cmscritic.com/amplitude-expands-platform-capabilities-with-session-replay-and-simplified-pricing">in 2023</a>. This new approach incorporates the replay feature into every workflow to capture broader user engagement, minus the labor-intensive setup and rigid event taxonomies.Bridging this gap between qualitative and quantitative understanding is a powerful salve to what I call The Analytics Blues. It enables marketers to diagnose product issues, improve conversions, identify UX behaviors, innovate faster, and generally build better digital experiences.Experimentation is critical for marketers, and time is always the enemy. They simply can’t afford to wait weeks (or longer) to test an idea. Experimentation is the lifeblood of optimization, and when you run fewer tests, you get fewer learnings – and, ultimately, fewer conversions.In most cases, the holdup rests with the tooling. Many legacy experimentation platforms struggle with visual editors that break when used on modern sites, and marketers can’t address the fixes themselves. That means filing a ticket, waiting for engineering, and praying that the page doesn’t break once their changes go live. All these delays can impact a campaign’s effectiveness. And even when a test is finally running properly, the resulting metrics aren’t always useful.Tests need to work. And they need to be simple. Last year, Amplitude endeavored to solve these problems with its novel <a target="_blank" rel="noopener noreferrer" href="https://amplitude.com/web-experimentation">Web Experimentation</a> solution, and the company says that marketing and growth teams using the service have realized strong gains in testing velocity and conversion optimization. I’ve seen it in action, and there’s real value for marketers.That said, Amplitude has been listening to its customers and evolving the platform’s capabilities. This week, they announced a major update of Web Experimentation, including four new features designed to help marketers and growth teams do more. These features – drag-and-drop visual editing, control variant editing, group cohort targeting, and out-of-the-box widgets – show great promise. And according to Amplitude, users could test three times faster and build with greater autonomy via these enhancements.As part of this announcement, Amplitude is also teasing the launch of its own <a target="_blank" rel="noopener noreferrer" href="https://amplitude.com/ai">AI Agents,</a> which will give marketers and web PMs some slick capabilities like auto-generating experiment ideas based on conversion trends or drop-off points, or draft variant copy and layouts based on behavioral data.  <figure class="image image-style-align-center image_resized" style="width:30.19%"><img src="https://cmscritic.com/ms-content/uploads/2025/08/image-1756482273106.png" alt="A person smiling at the camera

AI-generated content may be incorrect." /></figure>Nithya Rajendran, Principal Product Manager, Amplitude I connected with Amplitude’s <a target="_blank" rel="noopener noreferrer" href="https://www.linkedin.com/in/nithya-r-2a72794/">Nithya Rajendran,</a> a project manager working on the Web Experimentation and AI initiatives, about the frontier of Amplitude’s agentic investment. She provided some insight on what kinds of functionality we can expect and the benefits. “Our Agents work in parallel with teams to tackle essential but tedious activities like monitoring data, spotting patterns and changes, watching user sessions, forming hypotheses, running experiments, shipping changes, and monitoring impact,” Rajendran said. “Our AI Agents are built directly into the platform to help automate these tasks, meaning work that used to take weeks can now take minutes."<h3 id="giving-marketers-more-control-for-expanded-use-cases-">Giving marketers more control for expanded use cases </h3>Marketers are dealing with a lot these days. With dozens of tools to balance in their expanding toolkit – not to mention the diminishing realities of search traffic and the advent of GEO – they need enhanced observability and better analytics to manage the chaos. And when it comes to testing, they need to reduce the friction to make the juice worth the squeeze.Amplitude Web Experimentation has always been geared towards helping marketers run faster, easier, and more impactful tests, so they can optimize and improve their outcomes. With this major update and new features, it promises to deliver marketing and growth teams with more power for building and running experiments in a truly self-service motion – giving them the control they need to be successful. As the company stated in its press release, there are four new features in Web Experimentation. The first is the ability to rearrange elements with true drag-and-drop editing, where you can shift a headline from below a form to above it in seconds. We take these things for granted in CMS and DXP ecosystems, but having an easy-to-use visual editor at the testing level has been elusive. Now, with Web Experimentation, you can move page components around visually instead of relying on an engineering team to do the heavy lifting. With its new out-of-the-box widgets, Web Experimentation now supports more test types. This means simplifying the time and complexity involved with adding banners, CTAs, or pop-ups without custom code.With freshly added group cohorting capabilities, you can target by account, allowing B2B teams to personalize experiments for specific companies or account lists. You can also edit control variants directly, allowing you to update your “default” experience without waiting for developers to push changes.<h3 id="a-seamless-add-on-for-cms-and-dxp-platforms">A seamless add-on for CMS and DXP platforms</h3>Web Experimentation’s new features are designed to give marketing and web PMs a much faster and streamlined pathway to testing. It’s also CMS agnostic, making it much more extensible and easier for organizations to onboard. “It works anywhere you can add a simple script tag, whether that is a traditional CMS or a headless CMS,” Rajendran said. “Once the script is installed, marketers can use a no-code, drag-and-drop Visual Editor to launch experiments directly on their site without developer support.”Amplitude also offers official plugins for <a target="_blank" rel="noopener noreferrer" href="https://c67dcd9a.streaklinks.com/CjlzcGqbWy6jkUcbTQdlL3tq/https%3A%2F%2Famplitude.com%2Fdocs%2Fdata%2Famplitude-wordpress-plugin">WordPress</a><a target="_blank" rel="noopener noreferrer" href="https://c67dcd9a.streaklinks.com/CjlzcGuKCuDX0DOR4QDezJ9N/https%3A%2F%2Famplitude.com%2Fdocs%2Fdata%2Famplitude-shopify-plugin"> and Shopify</a> that make it easy to install Web Experimentation – along with other solutions like Analytics and Session Replay – with minimal setup.Amplitude has also cozied up with the customer data layers across stacks, all the way down to the warehouse layer, where it offers self-service analytics capabilities. As Nithya explained, this connection to CDPs also enhances the overall flow. “Because Amplitude also integrates with CDPs like <a target="_blank" rel="noopener noreferrer" href="https://cmscritic.com/twilio-acquires-segment-for-3-2b-adds-market-leading-cdp-for-building-data-driven-apps">Segment</a> and <a target="_blank" rel="noopener noreferrer" href="https://tealium.com/">Tealium</a>, experiments are tied to consistent customer data that already flows through the rest of the stack,” she said.From a feedback perspective, Amplitude is touting its success with <a target="_blank" rel="noopener noreferrer" href="https://crometrics.com/">Cro Metrics</a>, where its drag-and-drop editor is having a notable impact. According to Co-CEO <a target="_blank" rel="noopener noreferrer" href="https://www.linkedin.com/in/gwen-hammes/">Gwen Hammes</a>, it’s giving Cro Metrics a new level of agility for its clients, allowing them to spin up high-impact experiments like banners and modals within their marketing and product teams – so they can move from concept to customer faster than ever.“That’s the sweet spot,” Hammes said. “Empowering our clients to iterate quickly to see measurable lift, while freeing up their valuable time for us to collaborate on larger strategic initiatives and integrate test learnings cross-functionally for even greater impact.”<h3 id="amplitude-ai-agents-are-entering-the-stack-">Amplitude AI Agents are entering the stack </h3>As I mentioned, this update also framed Amplitude’s foray onto the agentic AI scene, where we’re likely to see a great deal of automation surrounding various use cases. Analytics is one area where AI is already demonstrating its immense value to glean meaningful insights from large data sets and provide more actionable intelligence. I suspect we’ll see that value translated here. I’ve been watching a lot of demos from vendors in different categories, and the use cases for agentic AI and the harnessing of MCP are yielding some spectacular results. According to Amplitude, we’ll see their applications come to life this fall, with the foundation being laid through these Web Experimentation updates. When put to work, <a target="_blank" rel="noopener noreferrer" href="https://amplitude.com/blog/ai-agents?siteLocation=ai-agents-spotlight-cards">Amplitude’s AI Agents </a>will take testing velocity to new levels, simplifying tasks and reducing workloads that used to take weeks into hours. While it’s still early days, Amplitude said its agents will have a huge impact on a myriad of functions, like autonomously prioritizing tests that are most likely to drive lift or interpreting results and automatically recommending actions. AI agents can also assist with some of the more “nefarious” challenges that exist in a flow, like creating better checkout experiences to reduce abandoned carts or improving products by driving users toward their preferred behaviors (think signups or upsells). <figure class="image image-style-align-center image_resized" style="width:85.66%"><img src="https://cmscritic.com/ms-content/uploads/2025/08/image-1756482272869.png" alt="A screenshot of a computer

AI-generated content may be incorrect." /></figure>Source: Amplitude website Here’s where the connection to Web Experimentation comes in: Because these AI agents are built into the broader Amplitude platform, AI Agents won’t just analyze experiments in isolation – they’ll learn from the same analytics, session replays, and surveys you do, and act on them instantly. And the new drag-and-drop editor, widgets, and control features represent three critical channels that Amplitude's AI Agents will use on your behalf.Amplitude has also worked to make these agents as accessible as possible to marketing teams across all skill levels by harnessing a natural language interface. “While marketers will benefit from huge time-savings, they still stay in control,” Rajendran said. “They decide the level of autonomy their Agents have, and as they build trust, they can adjust these guardrails. So that means you can rely on an Agent to recommend content variants, but a marketer approves them before an experiment is rolled out. After the experiment runs, an Agent will analyze the results and recommend next steps, but a marketer makes the final call."<h3 id="pricing">Pricing</h3>Amplitude <a target="_blank" rel="noopener noreferrer" href="https://cmscritic.com/amplitude-expands-platform-capabilities-with-session-replay-and-simplified-pricing">introduced simplified pricing</a> early last year as part of its Sesson Replay launch, and there’s some decent guidance on their website. I like the inclusion of a free tier as a “jumping off” point, and the Plus plan starts at $49/month for 300K MTUs.Those dreaded MTUs. This is where the challenges exist, and hence why Amplitude offers a Growth tier with custom volumes. But all of its plans feature Web Experimentation, and hence, a gateway for experiencing these new features at any level. You can explore the full breadth of the packages on their <a target="_blank" rel="noopener noreferrer" href="https://amplitude.com/pricing">pricing page. </a>According to Rajendran, there aren’t any pricing details for the Amplitude AI Agents as of yet. In her words, the company is focused on enhancing the overall experience for customers and not monetizing the capabilities.<h3 id="the-verdict">The verdict</h3>Amplitude continues to establish its leadership through innovation, and these new features are aimed at giving marketers even greater control over their experimentation and testing, and that’s a win. Doing it faster – and with stable tools – is the name of the game, and Amplitude is delivering on that front with its easy drag-and-drop capabilities and out-of-the-box widgets.Based on preliminary positioning, the roadmap for Amplitude’s AI Agents looks exciting. Data and analytics are prime real estate for converting value around agentic investments, and the gains in automation could be significant. We’ll have to monitor feedback once we see them in play and assess how these capabilities measure up to other competitive AI enhancements across the stack. There are already AI agents farming analytics from platforms like <a target="_blank" rel="noopener noreferrer" href="https://beam.ai/integrations/mixpanel">Mixpanel</a> to enhance operations efficiency and strategic growth. It’s worth a few demos and some homework on the AI roadmaps as you evaluate options, but Amplitude has a clear vision – and they’re executing for success.  <hr /><h3 id="upcoming-events">Upcoming Events</h3> <img style="width:31.61%" src="https://cmscritic.com/ms-content/uploads/2025/08/storyblok-joyconf-2025-logo.jpg" /><h4 id="storyblok-joyconf-2025"><a target="_blank" rel="noopener noreferrer" href="https://joyconf.storyblok.com/">Storyblok JoyConf 2025</a></h4>October 14-15, 2025 – Amsterdam, NetherlandsJoin us at JoyConf 2025, where the future of content meets the people shaping it. Built for developers, marketers, and digital innovators, this free conference goes way beyond product updates or yet another strategy playbook. It’s where you’ll get inspired, get real, and get connected – to ideas, to the community, and to what’s possible when you build joyfully. Whether you're building lightning-fast frontends or launching cross-channel campaigns, this is where content becomes connection and collaboration becomes second nature. Come for the talks. Stay for the community. Leave with joy. <a target="_blank" rel="noopener noreferrer" href="https://joyconf.storyblok.com/">Save your spot.</a> <img style="width:28.9%" src="https://cmscritic.com/ms-content/uploads/2025/08/sitecore-symposium-2025.jpg" /><h4 id="sitecore-symposium-2025"><a target="_blank" rel="noopener noreferrer" href="https://symposium.sitecore.com/">Sitecore Symposium 2025</a></h4>November 3-5, 2025 – Orlando, FloridaAt Sitecore Symposium, you’ll learn how to meet your customers where they are – across channels, moments, and touchpoints that didn’t exist a year ago. You’ll hear how the boldest brands in the world are transforming content into outcomes and creating experiences that are fast, relevant, and human. Join 1,500 marketers, digital leaders, and technologists in Orlando to see what’s possible when personalization scales, AI accelerates creativity, and trust becomes your competitive edge. This is the Third Wave of the Internet. And this is your moment to lead in it. <a target="_blank" rel="noopener noreferrer" href="https://symposium.sitecore.com/">Get your tickets today.</a> <img style="width:31.44%" src="https://cmscritic.com/ms-content/uploads/2025/08/cms-kickoff-2026-logo.png" alt="" /><h4 id="cms-kickoff-2026"><a target="_blank" rel="noopener noreferrer" href="https://www.boye-co.com/conferences/cmskickoff26">CMS Kickoff 2026</a></h4>January 13-14, 2026 – St. Petersburg, FloridaMeet industry leaders at our fourth annual CMS Kickoff – the industry's premier global CMS event. Similar to a traditional kickoff, we reflect on recent trends and share stories from the frontlines. Additionally, we will delve into the current happenings and shed light on the future. Prepare for an unparalleled in-person CMS conference experience that will equip you to move things forward. This is an exclusive event – space is limited, <a target="_blank" rel="noopener noreferrer" href="https://www.boye-co.com/conferences/cmskickoff26">so secure your tickets today.</a>

Amplitude’s New Web Experimentation Upgrade Gives Marketers 3x Faster Testing and More Control

matt-garrepy

Explore a wide range of informative articles on Experimentation, insights, and industry updates. Stay informed and ahead of the curve with our diverse collection of news and posts covering the latest trends and developments in the industry

News

Amplitude’s New Web Experimentation Upgrade Gives Marketers 3x Faster Testing and More Control