A Difficult Test Is Not Necessarily a Good Assessment

There is an intuitive assumption behind many professional assessments: if the test is difficult, it must be rigorous.

A low pass rate can reinforce that impression. Complex questions can make an examination appear selective. Time pressure can make it feel demanding. Candidates may leave believing that the assessment was challenging - and therefore credible.

But difficulty alone proves very little.

An assessment can be extremely difficult and still measure the wrong things. It can demand obscure knowledge that has little relationship with professional performance. It can create ambiguity where none exists in the real work environment. It can reward familiarity with test-taking techniques rather than competence.

The relevant question is not:

How hard was the assessment?

It is:

What evidence does the assessment provide about the person being evaluated?

That distinction separates difficulty from rigor.

Difficulty is a characteristic. Validity is a requirement.

Professional certification exists to provide assurance that an individual meets defined competence requirements.

That principle is reflected in ISO/IEC 17024, the international standard for bodies that certify persons. The standard frames certification as a mechanism for providing confidence that an individual meets the competence requirements established by a certification scheme.

The implication is significant.

The purpose of an assessment is not to create obstacles. It is to produce defensible evidence.

A question can be difficult because it addresses a genuinely complex professional judgment.

But it can also be difficult because:

• the wording is unnecessarily confusing;

• the content is excessively obscure;

• the candidate is required to memorize information that would normally be accessible during professional practice;

• several answers are defensible because the scenario lacks relevant context;

• the examination tests concepts that are only weakly connected to actual performance.

All of these can increase difficulty.

None necessarily increases the quality of the assessment.

A good assessment starts with what it intends to measure

Before designing questions, an assessment must define the competence it intends to evaluate.

That may include technical knowledge.

But professional competence often extends beyond knowledge alone.

A manager may know multiple prioritization frameworks and still struggle to choose between conflicting initiatives.

A finance professional may understand the mechanics of an investment decision and still fail to recognize when assumptions are unreliable.

A leader may know the principles of stakeholder management and still make poor decisions when incentives conflict.

This creates a fundamental challenge for certification design.

If the competence involves application, judgment or decision-making, an assessment focused primarily on recall may generate only partial evidence.

Assessment organizations such as ETS treat validity as a central quality criterion: whether the assessment actually measures what it is intended to measure. Their standards also address reliability and fairness as essential characteristics of technically sound assessment.

A difficult recall question may therefore be less useful than a moderately difficult scenario that reveals how a professional interprets evidence and chooses a course of action.

High failure rates are not evidence of quality

Pass rates are often interpreted as signals of rigor.

A certification passed by only a small proportion of candidates may appear more prestigious than one with a higher success rate.

But this conclusion can be misleading.

A low pass rate can result from a highly demanding competence standard.

It can also result from poor item design, misalignment between the assessment and its stated scope, unclear instructions or an arbitrary passing threshold.

The pass rate itself does not tell us which explanation is correct.

Consider two assessments.

In the first, 35% of candidates pass because the standard requires them to demonstrate several advanced competencies consistently across realistic professional scenarios.

In the second, 35% pass because the test contains unusually obscure questions and requires memorization of details that have little relevance to the work being certified.

The numerical outcome is identical.

The evidence is not.

Rigorous certification cannot define quality by scarcity alone.

The relevant question is whether those who pass have demonstrated the competence the credential claims they possess - and whether those who do not pass have failed to demonstrate that same standard.

The assessment should resemble the competence

There is no universal format that makes an assessment credible.

Multiple-choice questions can be appropriate for evaluating certain types of knowledge.

Constructed responses can reveal reasoning that selected-response questions may not capture.

Performance tasks can approximate actual professional activity.

Scenario-based assessments can test how candidates apply knowledge under constraints.

Each format has advantages and limitations.

Even performance-based assessments are not automatically superior. Research and assessment guidance emphasize that alternative item types must still be evaluated for validity and fairness; a more realistic-looking task can introduce irrelevant factors if it is poorly designed.

The format should therefore follow the competence - not the other way around.

If a professional standard requires the candidate to recognize the correct regulatory requirement, a structured knowledge question may be sufficient.

If the standard requires the candidate to prioritize competing objectives under uncertainty, simply asking for the definition of prioritization is unlikely to provide meaningful evidence.

The closer the assessment method is to the competence being claimed, the stronger the inference that can reasonably be drawn from the result.

Ambiguity should come from the profession, not from the question

Professional environments are often ambiguous.

Assessment scenarios sometimes need to reflect that.

But there is a difference between productive ambiguity and poorly designed ambiguity.

Productive ambiguity exists when a professional must interpret incomplete information, balance conflicting interests or make a decision without certainty.

Poor ambiguity exists when the candidate cannot understand what the question is asking or when several responses are equally defensible because important information was omitted unintentionally.

The first can reveal judgment.

The second introduces noise.

A well-designed scenario may intentionally withhold information because deciding what matters is itself part of the competence.

But that choice must be deliberate.

The candidate should be challenged by the professional problem - not by the assessment designer's inability to express it clearly.

Reliability matters even when judgment is involved

One concern frequently appears when assessments move beyond simple right-or-wrong questions:

Can judgment be evaluated consistently?

It can - but doing so requires structure.

Assessment criteria need to make clear what dimensions of performance matter.

Scoring rules need to distinguish stronger reasoning from weaker reasoning.

Equivalent responses should receive equivalent treatment.

And assessment processes should be designed so that results do not depend excessively on who evaluates the response.

This is why rigorous assessment design requires more than creating interesting questions.

It requires a measurement system.

ETS, for example, publishes specific guidance for constructed-response and performance assessments precisely because the development and scoring of these formats require systematic evidence supporting the interpretation of results.

The objective is not to eliminate judgment from the assessment.

It is to ensure that the judgment being assessed belongs to the candidate - not to an inconsistent evaluation process.

Fairness is part of rigor

There is another misconception that can emerge from the pursuit of difficult assessments: reducing accessibility can be mistaken for increasing selectivity.

A rigorous assessment should distinguish candidates based on the competence being measured.

It should not unintentionally distinguish them based on unrelated characteristics.

If unnecessarily complex language makes an operations assessment partly a reading-comprehension test, the assessment may be measuring more than intended.

If a scenario assumes cultural or organizational experiences that are irrelevant to the target competence, candidates may face disadvantages unrelated to their actual capability.

Assessment organizations explicitly treat fairness as part of technical quality, alongside validity and reliability. Fairness reviews are intended to identify aspects of tests that could prevent candidates from demonstrating the relevant knowledge and skills appropriately.

Fairness does not mean making an assessment easier.

It means ensuring that difficulty comes from the competence standard itself.

The purpose of rigor is confidence

Ultimately, a professional assessment serves a trust function.

A candidate wants the credential to represent something meaningful.

An employer wants to understand what can reasonably be inferred from it.

A certification body needs to defend the standard behind the result.

That confidence does not come from the number of pages in an exam, the complexity of its vocabulary or the percentage of people who fail.

It comes from alignment.

Alignment between the competence being claimed and the competence being tested.

Alignment between the questions and real professional expectations.

Alignment between scoring criteria and the standard being certified.

Alignment between the result and the conclusion that others are expected to draw from it.

A difficult assessment may be rigorous.

A rigorous assessment may also be difficult.

But the two concepts are not interchangeable.

The objective of professional certification should never be to make passing unnecessarily hard.

It should be to make the evidence of competence sufficiently strong that passing actually means something.

References used in the article

• ISO/IEC 17024 - Conformity assessment - General requirements for bodies operating certification of persons (ISO).

• ETS Standards for Quality and Fairness and related publications on validity, reliability, fairness, constructed-response and performance assessment.