I85PREP
⌘ K
OMCP study materials
OMCP study resources

OMCP practice questions

One question at a time. Build your OMCP understanding.

First 50 of 200 questions. Each one lists the choices, the correct answer, and an explanation. The checker at the top of this page saves progress in this browser when it is available.

Continue in the practice tool above to load more questions, 50 at a time.

Question 1Content Marketing

A content team defines its primary audience as 'operations managers at mid-sized logistics firms who evaluate route-planning software.' Which element of a buyer persona does this statement most directly supply?

  1. The persona's stage in the buying journey
  2. The persona's preferred content channels
  3. The persona's budget authority threshold
  4. The persona's role and organizational context Correct answer.

Explanation

Personas describe who the audience is: job role, company type, and the problem context they work in. This statement identifies role and firm profile, which are core persona attributes. Channel preference, funnel stage, and spending authority are separate dimensions that the statement does not address.

Question 2Content Marketing

A marketing team wants to measure whether blog content influences pipeline rather than just traffic. Which metric set best supports that goal?

  1. Keyword rankings, domain authority, and backlink count
  2. Assisted conversions, influenced pipeline value, and content-sourced opportunities Correct answer.
  3. Email open rate, unsubscribe rate, and list growth
  4. Pageviews, bounce rate, and average session duration

Explanation

To connect content to revenue, teams track how content touches deals: assisted conversions, pipeline influenced, and opportunities where content was a source. Traffic and engagement metrics describe consumption but not commercial impact. SEO and email metrics are channel health indicators, not pipeline evidence.

Question 3Content Marketing

A B2B software company publishes a detailed technical comparison of its product against two competitors. Which stage of the buyer's journey does this asset primarily serve?

  1. Awareness
  2. Consideration Correct answer.
  3. Retention
  4. Advocacy

Explanation

Comparison content helps buyers who already know their problem and are evaluating solution options, which is the consideration stage. Awareness content introduces the problem itself, while retention and advocacy content serve existing customers. Because the asset assumes a shortlist of vendors, it fits consideration.

Question 4Content Marketing

A content marketer is planning a pillar-and-cluster model. Which action best reflects correct implementation of that model?

  1. Publishing ten short posts on unrelated trending topics each week
  2. Restricting all internal links to the homepage to concentrate authority
  3. Replacing the pillar page every quarter with a new topic
  4. Creating one comprehensive pillar page and linking it to focused cluster pages on subtopics Correct answer.

Explanation

In a pillar-and-cluster model, a broad pillar page covers a core topic and links to narrower cluster pages that explore subtopics in depth, with links flowing both ways. This structure signals topical coverage and helps both readers and search engines. Unrelated posts, homepage-only linking, and constant replacement break the model.

Question 5Content Marketing

Which statement best describes the purpose of a content brief before a writer begins drafting?

  1. It guarantees the piece will rank on page one
  2. It locks the final word count so editors cannot request changes
  3. It aligns the writer on audience, angle, keywords, and success criteria Correct answer.
  4. It replaces the editorial calendar for the quarter

Explanation

A content brief is a planning document that gives the writer direction on target audience, angle, required topics, keywords, format, and how success will be judged. It reduces revision cycles by aligning stakeholders early. It does not fix word counts permanently, replace calendars, or guarantee rankings.

Question 6Content Marketing

A brand's editorial mission statement reads: 'We help first-time homebuyers make confident financial decisions.' Which content idea is least aligned with that mission?

  1. A checklist for saving a down payment
  2. A guide to understanding mortgage pre-approval
  3. An explainer on closing costs and who pays them
  4. A roundup of luxury yacht charter destinations Correct answer.

Explanation

An editorial mission defines the audience and the value the content delivers, and every idea should serve that promise. Three options address first-time homebuyer finances directly. A luxury yacht charter roundup targets a different audience and need, so it falls outside the mission.

Question 7Content Marketing

A team wants to repurpose a 45-minute webinar into multiple assets. Which approach best preserves value while fitting each channel?

  1. Cutting the webinar into 40 unedited two-minute segments
  2. Uploading the raw recording unchanged to every channel
  3. Extracting a transcript-based article, short video clips, and a quote graphic set Correct answer.
  4. Posting only the full recording and skipping other formats

Explanation

Effective repurposing adapts core ideas into formats native to each channel rather than duplicating one file everywhere. A transcript article, short clips, and quote graphics each suit different consumption habits. Unedited or excessive fragments dilute quality and fail to match channel norms.

Question 8Content Marketing

Which practice most directly improves the accessibility of a published blog article?

  1. Shortening the article to under 300 words
  2. Removing all subheadings to reduce clutter
  3. Increasing keyword density in the opening paragraph
  4. Adding descriptive alt text to meaningful images Correct answer.

Explanation

Alt text lets screen-reader users understand images, making content accessible to people with visual impairments. Accessibility also depends on clear structure, so removing subheadings harms it. Keyword density and arbitrary length limits relate to SEO and readability, not accessibility.

Question 9Content Marketing

A content team is choosing between gated and ungated versions of a research report. Which consideration most strongly favors gating?

  1. The team wants maximum organic reach and backlinks
  2. The report contains only evergreen educational material
  3. The team needs to capture leads for a sales follow-up motion Correct answer.
  4. The team has no email marketing system

Explanation

Gating trades reach for lead capture: visitors exchange contact details for the asset, enabling nurture and sales follow-up. Ungating favors reach, links, and sharing. Without an email system, captured leads cannot be nurtured, and purely educational evergreen content often performs better ungated.

Question 10Content Marketing

A company's style guide specifies voice as 'knowledgeable but approachable.' Which draft best follows that voice?

  1. Pursuant to the aforementioned provisions, users shall effectuate configuration changes.
  2. Configuration must be effectuated prior to utilization.
  3. Our platform is the unrivaled apex of technological supremacy.
  4. We'll walk you through setup step by step, and we'll flag anything that trips people up. Correct answer.

Explanation

Voice guidance translates into concrete choices in diction, sentence length, and tone. 'Knowledgeable but approachable' calls for plain, helpful language that still conveys expertise. The other options are either stiffly bureaucratic or inflated with hype, both of which conflict with an approachable expert voice.

Question 11Content Marketing

An editor is building a quarterly editorial calendar. Which entry is most useful for coordinating production across writers and designers?

  1. A note that says 'publish good content regularly'
  2. A list of competitor blog URLs
  3. A row listing topic, owner, format, target publish date, and dependencies Correct answer.
  4. A folder of unused stock images

Explanation

An editorial calendar coordinates work by assigning each piece a topic, owner, format, deadline, and dependencies such as design or review. These fields let teams sequence work and spot bottlenecks. Vague intentions, competitor lists, and image folders do not schedule or assign production.

Question 12Content Marketing

A content marketer wants to build topical authority for a niche. Which tactic is most consistent with that goal?

  1. Developing deep coverage of related subtopics with interlinked pages Correct answer.
  2. Publishing one post per topic and moving on quickly
  3. Buying guest posts on unrelated high-traffic sites
  4. Targeting only high-volume keywords outside the niche

Explanation

Topical authority grows when a site covers a subject comprehensively and connects related pages so search engines see depth and relevance. Deep, interlinked coverage signals expertise. One-off posts, off-topic links, and unrelated high-volume keywords dilute focus instead of building authority.

Question 13Content Marketing

Which element is essential to a well-formed call to action in a blog post?

  1. A request to share the post before reading it
  2. At least three competing offers in the same paragraph
  3. A guarantee of specific financial results
  4. A clear next step matched to the reader's stage Correct answer.

Explanation

A strong call to action tells the reader exactly what to do next and matches that ask to where they are in the journey. Competing offers split attention, premature share requests interrupt reading, and results guarantees are both ineffective and often noncompliant.

Question 14Content Marketing

A brand wants user-generated content for a campaign. Which practice best manages legal and brand risk?

  1. Editing creators' content heavily without disclosure
  2. Obtaining documented permission and giving credit per the agreed terms Correct answer.
  3. Using any tagged photo without contacting the creator
  4. Assuming public posts are free to use commercially

Explanation

User-generated content still belongs to its creator, so commercial use requires documented permission that defines scope, credit, and edits. Public visibility does not equal a license for commercial reuse. Heavy undisclosed edits can also breach the agreement and misrepresent the creator.

Question 15Content Marketing

A team notices a long-form guide ranks well but converts few readers. Which diagnostic step is most appropriate first?

  1. Increasing the word count by 2,000 words
  2. Reviewing search intent match, CTA placement, and offer relevance Correct answer.
  3. Adding more keywords throughout the body
  4. Immediately deleting the page and starting over

Explanation

Strong rankings with weak conversion usually signal a mismatch between what the page promises and what it offers next. Diagnosing intent alignment, CTA placement, and offer relevance addresses the likely cause. Deleting the page discards ranking equity, while adding keywords or length does not fix conversion friction.

Question 16Content Marketing

Which statement correctly distinguishes owned, earned, and paid media in a content strategy?

  1. Paid media is third-party coverage, earned media is purchased placement, and owned media is word of mouth
  2. Owned media is purchased placement, earned media is the brand blog, and paid media is press coverage
  3. Owned media is a channel the brand controls, earned media is third-party coverage, and paid media is purchased placement Correct answer.
  4. Earned media is the brand's blog, owned media is press coverage, and paid media is social sharing

Explanation

The three media types are defined by control and origin. Owned media such as a blog or email list is controlled by the brand, earned media is coverage or mentions generated by others, and paid media is placement the brand buys. Confusing these categories leads to misallocated budget and effort.

Question 17Content Marketing

A content leader is setting SMART objectives for a campaign. Which objective best meets that standard?

  1. Grow brand awareness significantly this year
  2. Increase newsletter subscribers by 2,000 within Q3 through the guide landing page Correct answer.
  3. Improve content quality across all channels
  4. Produce more engaging content than competitors

Explanation

SMART objectives are specific, measurable, achievable, relevant, and time-bound. The subscriber goal names a metric, a target, a deadline, and a mechanism. The other options lack measurable targets and time frames, making them aspirations rather than objectives.

Question 18Content Marketing

A brand discovers that a partner's article misrepresents its product's capabilities. Which response best protects credibility?

  1. Ignore it because third-party content is outside the brand's control
  2. Request a correction and publish a clarifying note where the claim is likely seen Correct answer.
  3. Threaten legal action without first contacting the partner
  4. Quietly remove all mentions of the partner from the brand's site

Explanation

Credibility is protected by correcting inaccurate claims quickly and transparently. Requesting a correction addresses the source, and a clarifying note reaches audiences who saw the error. Ignoring misinformation lets it spread, escalation without contact is disproportionate, and silent removal does not correct the record.

Question 19Content Marketing

Which approach best describes an effective content distribution plan?

  1. Publishing content and waiting for organic discovery alone
  2. Promoting only the newest asset and retiring all earlier ones
  3. Mapping each asset to specific channels and audiences with scheduled promotion Correct answer.
  4. Sending every asset to every channel simultaneously

Explanation

Distribution planning matches each asset to the channels and audiences where it fits, with scheduled promotion over time. Relying on organic discovery alone limits reach, blasting every asset everywhere ignores channel fit, and retiring earlier assets wastes their ongoing value.

Question 20Content Marketing

A content team wants to interview subject-matter experts efficiently. Which method yields the most usable raw material for writers?

  1. Sending a focused question list in advance and recording the session Correct answer.
  2. Requesting a written summary weeks after the interview
  3. Asking broad open questions with no preparation
  4. Recording only the expert's final conclusions

Explanation

A focused question list sent in advance lets experts prepare specifics, and recording preserves quotable detail and nuance. Unprepared broad questions produce vague answers, delayed summaries lose detail, and capturing only conclusions discards the examples and explanations writers need.

Question 21Content Marketing

A brand's content is being criticized for overpromising results. Which editorial governance practice most directly prevents this recurring issue?

  1. Adding more disclaimers to every page
  2. Requiring a documented claims review with evidence before publication Correct answer.
  3. Publishing more frequently to outpace critics
  4. Removing all specific data from content

Explanation

Overpromising is prevented at the source by requiring claims to be supported with evidence and reviewed before publication. Disclaimers do not fix unsupported claims, publishing faster does not improve accuracy, and stripping all data reduces credibility rather than ensuring honesty.

Question 22Content Marketing

A brand wants to build a community around its content. Which tactic most directly encourages sustained participation?

  1. Publishing announcements only, with comments disabled
  2. Asking members questions and highlighting their contributions Correct answer.
  3. Restricting all content to gated downloads
  4. Sending promotional offers daily without discussion

Explanation

Communities grow when members feel heard and recognized. Asking questions invites participation, and highlighting member contributions reinforces that their input matters. One-way announcements, relentless promotion, and gating everything suppress the interaction communities depend on.

Question 23Content Marketing

A team is deciding whether to refresh or consolidate two overlapping articles that compete for the same keyword. Which action best serves search performance?

  1. Merge the stronger content into one comprehensive page and redirect the other Correct answer.
  2. Add the same keyword to both and publish weekly
  3. Leave both unchanged to double the chances of ranking
  4. Delete both and write a brand-new short post

Explanation

Overlapping pages compete with each other and split signals. Consolidating the best material into one stronger page and redirecting the weaker one concentrates relevance and authority. Leaving duplicates, deleting both, or repeating keywords perpetuates the conflict.

Question 24Content Marketing

Which metric best indicates whether a piece of content is being read rather than merely opened?

  1. Impression count
  2. Domain authority of the hosting site
  3. Number of social shares
  4. Scroll depth and average engaged reading time Correct answer.

Explanation

Reading behavior is best captured by scroll depth and engaged reading time, which show how far and how long people actually consume the content. Impressions only show that content was displayed, shares reflect distribution, and domain authority measures site strength, not reader engagement.

Question 25Content Marketing

A global brand localizes content for several markets. Which practice best preserves message consistency while fitting local audiences?

  1. Removing the brand's core message in every market
  2. Using one market's content unchanged everywhere
  3. Adapting core messaging while adjusting examples, idioms, and formats for each market Correct answer.
  4. Translating word for word and keeping all idioms intact

Explanation

Localization keeps the strategic core message while adapting execution details such as idioms, examples, imagery, and format to each culture. Literal translation can produce awkward or offensive results, unchanged content ignores local context, and dropping the core message sacrifices brand consistency.

Question 26Conversion Rate Optimization

In a conversion rate optimization program, which metric is defined as the percentage of all sessions in which a visitor completes a defined target action, such as a purchase or lead form submission?

  1. Average order value
  2. Conversion rate Correct answer.
  3. Bounce rate
  4. Click-through rate

Explanation

Conversion rate is the ratio of sessions (or visitors) that complete the defined goal to the total number of sessions, expressed as a percentage. It is the core outcome metric that CRO programs seek to improve, and every test should tie back to a clearly defined conversion action.

Question 27Conversion Rate Optimization

A landing page receives 40,000 sessions per month and records 1,200 form submissions. What is the page's conversion rate?

  1. 33.3%
  2. 3.0% Correct answer.
  3. 1.2%
  4. 0.3%

Explanation

Conversion rate equals conversions divided by sessions, so 1,200 / 40,000 = 0.03, or 3.0%. Presenting the result as a percentage requires multiplying the decimal by 100. This baseline figure is what a subsequent A/B test would attempt to lift.

Question 28Conversion Rate Optimization

Which statement best describes the purpose of building a prioritized CRO hypothesis backlog?

  1. It stores the final statistical results of completed tests for regulatory recordkeeping
  2. It documents testable assumptions about why conversion is lost, ranked so the team works on the highest-expected-value ideas first Correct answer.
  3. It lists competitor landing pages so that the team can replicate their layouts exactly
  4. It records every design change shipped to production so that the analytics team can audit traffic sources

Explanation

A hypothesis backlog captures explicit, testable statements linking an observed problem to a proposed change and an expected effect. Prioritization frameworks such as PIE or ICE rank those hypotheses so limited testing capacity targets the highest expected impact.

Question 29Conversion Rate Optimization

A team wants to know where visitors abandon a multi-step checkout. Which analysis technique is most directly suited to that question?

  1. A cohort retention chart tracking repeat purchases over months
  2. A funnel report showing drop-off between each defined checkout step Correct answer.
  3. A keyword ranking report for organic search
  4. A heatmap of the home page hero section

Explanation

Funnel analysis measures how many users progress from one defined step to the next, exposing the exact stage where abandonment concentrates. That localization is what makes it the right diagnostic before forming a checkout hypothesis.

Question 30Conversion Rate Optimization

In an A/B test, what does statistical significance indicate?

  1. The test ran for the minimum recommended duration of two business cycles
  2. The sample size was large enough to detect any effect size of practical interest
  3. The observed difference is unlikely to have arisen from random variation alone under the assumed null hypothesis Correct answer.
  4. The winning variant will certainly produce the same lift when deployed to all traffic

Explanation

Significance is a statement about the probability of the observed result under the null hypothesis, typically judged against a chosen alpha such as 0.05. It does not guarantee replication, nor does it describe power or test duration.

Question 31Conversion Rate Optimization

Which element of a test hypothesis most directly guards against a vague, unfalsifiable statement?

  1. A description of the design agency that produced the mockups
  2. A note about which browser the tester used
  3. A stated predicted outcome and the metric by which it will be judged Correct answer.
  4. A list of the team members who approved the change

Explanation

A falsifiable hypothesis names the change, the expected effect direction, and the metric that will confirm or refute it. Without a predicted outcome and measurement definition, any result can be rationalized after the fact.

Question 32Conversion Rate Optimization

What is the primary purpose of an A/A test before running a series of A/B tests?

  1. To prove that the current page converts better than the proposed variant
  2. To estimate the average order value of the traffic segment
  3. To validate that the experimentation tooling and analysis produce a false-positive rate near the expected alpha Correct answer.
  4. To determine which of several variants has the highest statistical power

Explanation

An A/A test splits traffic between identical experiences, so any detected difference is a false positive by construction. Running it validates assignment, instrumentation, and analysis, catching issues that would otherwise inflate false positives in real tests.

Question 33Conversion Rate Optimization

Which practice most directly reduces the risk of a false positive in an A/B test?

  1. Running several variants simultaneously and reporting the best-performing one
  2. Adding a new secondary metric after the test ends if the primary metric is flat
  3. Pre-registering a single primary metric and the analysis plan before launch Correct answer.
  4. Checking results daily and stopping the test the first time p exceeds 0.05

Explanation

Committing to one primary metric and an analysis plan in advance prevents selective reporting and multiple-comparison inflation. Peeking and metric shopping are the classic routes to spurious wins.

Question 34Conversion Rate Optimization

A test variant shows a 12% relative lift in conversion rate with a 95% confidence interval of +2% to +22%. How should the team most reasonably interpret this result?

  1. The result is invalid because the interval is asymmetric around the point estimate
  2. The test should be rerun with the same sample size until the interval narrows to zero
  3. The variant is guaranteed to deliver exactly a 12% lift in production
  4. The direction of the effect is credibly positive, but the true lift size is still imprecise Correct answer.

Explanation

The interval excludes zero, so the evidence supports a genuine positive effect, but its width shows substantial uncertainty about magnitude. Decisions can proceed on direction while planning follow-up tests to pin down size.

Question 35Conversion Rate Optimization

On a pricing page, a team hypothesizes that visitors abandon because plan differences are hard to compare. Which change most directly tests that specific hypothesis?

  1. Changing the page background color to match the brand palette
  2. Reformatting the plan comparison into an aligned feature matrix with highlighted differentiators Correct answer.
  3. Adding a customer testimonial carousel above the plans
  4. Shortening the footer navigation links

Explanation

A hypothesis about comparison difficulty must be tested with a change that alters comparability, such as an aligned feature matrix. The other changes do not address the stated cognitive barrier and would not isolate its effect.

Question 36Conversion Rate Optimization

A checkout test shows a statistically significant lift overall, but segment analysis reveals the lift occurs only among returning customers while new customers show a slight decline. What is the most defensible next step?

  1. Discard the entire test because one segment did not improve
  2. Roll out to all traffic immediately because the overall result is significant
  3. Report only the returning-customer segment and ignore the new-customer data
  4. Investigate the new-customer experience and consider a targeted follow-up test before full rollout Correct answer.

Explanation

Heterogeneous treatment effects mean an overall average can mask harm to a subpopulation. The responsible path is to examine why new customers responded differently and validate a targeted approach before broad deployment.

Question 37Conversion Rate Optimization

Which scenario most clearly violates the stable assignment requirement of a valid A/B test?

  1. A returning visitor is reassigned to a different variant on each new session Correct answer.
  2. The primary metric is defined before the test launches
  3. The test runs for two full weeks to cover weekly cycles
  4. The test uses a 50/50 traffic split between control and variant

Explanation

Stable assignment ensures a given user consistently sees the same experience, which is necessary for clean measurement and for avoiding contamination of user experience. Re-randomizing per session mixes exposures and muddies attribution of effects.

Question 38Conversion Rate Optimization

A team observes that 70% of mobile users drop off on a form with twelve required fields, while desktop users complete it at a much higher rate. Which diagnostic combination best supports forming a hypothesis?

  1. A paid search impression share report plus a bid strategy review
  2. A backlink profile audit plus a sitemap review
  3. A server log review of crawl frequency plus a robots.txt check
  4. Session recordings of the mobile form plus a field-level abandonment report Correct answer.

Explanation

The observed problem is on-form abandonment concentrated on mobile, so evidence must come from how users interact with the form itself. Recordings and field-level abandonment data reveal where and why users stop, which is the raw material for a testable hypothesis.

Question 39Conversion Rate Optimization

Which sample size consideration is most important when planning an A/B test for a small expected effect?

  1. Larger samples are needed to achieve adequate power than for a large expected effect Correct answer.
  2. Smaller samples suffice because small effects are easier to detect
  3. Sample size is irrelevant as long as the test runs for at least seven days
  4. Sample size should be set to the number of visitors available on the launch day

Explanation

Detectable effect size and statistical power are inversely related: smaller true effects require more observations to distinguish from noise. Planning sample size from the minimum effect of practical interest prevents underpowered tests.

Question 40Conversion Rate Optimization

A test result shows p = 0.04 with alpha set at 0.05, but the team had planned to run three primary metrics and reported only the one that reached significance. What is the main threat to this conclusion?

  1. The variant was shown to fewer users than the control
  2. Multiple comparisons inflate the family-wise false-positive rate, so the single significant result may be spurious Correct answer.
  3. The test duration was insufficient to establish causality
  4. The p-value is too small to be meaningful in any experiment

Explanation

Testing several metrics and reporting the significant one raises the chance that at least one appears significant by chance, which is the multiple comparisons problem. Corrections such as Bonferroni or pre-registering one primary metric address this inflation.

Question 41Conversion Rate Optimization

Which page element is most directly associated with reducing friction at the point of form completion?

  1. Inline validation that flags errors as the user types Correct answer.
  2. Autoplay video with sound on page load
  3. A navigation menu with many top-level categories
  4. A large hero image that pushes the form below the fold

Explanation

Inline validation gives immediate, contextual feedback so users can correct mistakes before submitting, reducing frustration-driven abandonment. The other elements add load, distraction, or navigation complexity rather than removing form friction.

Question 42Conversion Rate Optimization

A team wants to test five variants of a hero section plus a control. Which statement about this design is most accurate?

  1. It cannot be analyzed with standard statistical methods
  2. It eliminates the need for a control group once three or more variants exist
  3. It guarantees a faster answer because more variants collect data simultaneously
  4. It requires a larger total sample and multiplicity control than a two-arm test to reach reliable conclusions Correct answer.

Explanation

Multi-arm tests partition the same traffic among more cells, so each arm accumulates data more slowly, and comparing many arms to control multiplies the false-positive risk. Larger samples and adjusted error control are needed for trustworthy conclusions.

Question 43Conversion Rate Optimization

Which metric is best suited as a guardrail metric in a test aimed at increasing newsletter signups?

  1. Number of variants in the test
  2. Newsletter signup rate
  3. Purchase conversion rate for the same traffic Correct answer.
  4. Time since the test launched

Explanation

A guardrail metric detects unintended harm to important outcomes outside the test's target. If a signup-focused change suppresses purchases, the guardrail catches the tradeoff that the primary metric would hide.

Question 44Conversion Rate Optimization

What distinguishes a one-tailed from a two-tailed hypothesis test in an A/B experiment?

  1. A one-tailed test requires no control group
  2. A one-tailed test is always more conservative than a two-tailed test
  3. A one-tailed test directs all alpha to a single direction of effect, while a two-tailed test splits it across both directions Correct answer.
  4. A two-tailed test cannot use confidence intervals

Explanation

The tail choice determines how alpha is allocated: one-tailed tests look only for improvement (or only for decline), while two-tailed tests detect effects in either direction. Because direction is usually uncertain in CRO, two-tailed tests are the safer default.

Question 45Conversion Rate Optimization

A team runs a test and finds no significant difference. Which interpretation is most appropriate?

  1. The change definitively has zero effect on conversion
  2. The control is proven superior to the variant
  3. The test should be extended indefinitely until significance emerges
  4. The data are consistent with no effect, though an underpowered test may also have missed a real one Correct answer.

Explanation

A non-significant result means the evidence does not rule out the null; it does not prove the null. Before concluding, the team should check whether the test had adequate power to detect the effect it cared about.

Question 46Conversion Rate Optimization

A/B test results are being generalized from a test run during a flash sale week to normal operations. Which validity concern is most prominent?

  1. Internal validity, because the random assignment mechanism was necessarily broken
  2. External validity, because the traffic mix and intent during the promotion may differ from ordinary periods Correct answer.
  3. Statistical conclusion validity, because the p-value formula changes during sales
  4. Construct validity, because conversion rate no longer measures conversions

Explanation

External validity concerns whether results generalize beyond the conditions of the study. Promotional periods attract atypical traffic and intent, so a lift measured then may not transfer to baseline conditions.

Question 47Conversion Rate Optimization

A team applies a Bonferroni correction to four primary metrics tested at alpha 0.05. Which adjusted threshold is consistent with this approach?

  1. 0.05
  2. 0.0025
  3. 0.20
  4. 0.0125 Correct answer.

Explanation

Bonferroni divides the chosen family-wise alpha by the number of tests, so 0.05 / 4 = 0.0125. This controls the probability of any false positive across the family, at the cost of reduced power per test.

Question 48Conversion Rate Optimization

A test shows a significant lift in add-to-cart rate but no change in completed purchases. Which explanation is most plausible?

  1. Purchases are unaffected by anything that happens before checkout
  2. Add-to-cart rate is not measurable in web analytics
  3. The change moved users earlier in the funnel but introduced friction later, so the effect washed out before purchase Correct answer.
  4. A significant result on one metric guarantees an equal effect on all downstream metrics

Explanation

Funnel metrics are linked but not deterministically coupled, so a gain upstream can be offset by losses downstream, such as added steps or confusion at checkout. The pattern calls for diagnosing the later stages rather than assuming the upstream win carries through.

Question 49Conversion Rate Optimization

Which situation best demonstrates Simpson's paradox in a CRO analysis?

  1. A variant wins overall and also wins in every traffic segment
  2. A page's bounce rate falls while its conversion rate also falls
  3. A variant wins overall but loses within both mobile and desktop segments because traffic mix shifted toward the segment where it performs least badly Correct answer.
  4. A test is stopped early because the result looks promising

Explanation

Simpson's paradox occurs when an aggregate trend reverses or disappears within every subgroup, driven by unequal group sizes or mix shifts. Segmenting before concluding protects against acting on a misleading aggregate.

Question 50Conversion Rate Optimization

A team wants to detect a 2% relative lift on a baseline conversion rate of 5% at 80% power and alpha 0.05. Compared with detecting a 10% relative lift under the same settings, what is required?

  1. No traffic estimate is possible without knowing average order value
  2. Substantially more traffic per arm, because the detectable effect is much smaller Correct answer.
  3. Substantially less traffic per arm, because precision requirements are lower
  4. The same traffic per arm, because power and effect size are unrelated

Explanation

Required sample size scales roughly with the inverse square of the effect size, so detecting a 2% relative lift demands far more observations than detecting a 10% relative lift. Teams often cannot reach that volume and must accept larger minimum detectable effects.