Concepts and Terms

FAE and AInspector for Firefox use the OpenA11y Evaluation Library the following concepts are important in understanding the terms and relatiohnships of the rules, rulesets and evaluation results of the library. The OpenA11y Evaluation Library is based on the requirements

Basic Concepts

Term Definition
Rule A low-level accessibility compliance requirement with numerous components, the most basic of which are a definition, an evaluation function, and result messages. When a rule is evaluated, it is assigned a rule result value.
Ruleset A set of rules that embodies specific standards compliance goals. Each rule within a ruleset is designated as either required or recommended, according to its relationship to the compliance goals of the ruleset.
Evaluation The application of all evaluation functions of all rules in the selected ruleset to their corresponding target resources in a web page, and the subsequent collection of all evaluation results and messages, at both the rule and element levels, into a dataset.
Rule result The aggregate result of a rule evaluation with respect to its target resources (all of its target elements and/or the entire page).
Element result The result of a rule evaluation with respect to one of its target elements.
Rule Scope
  • The scope helps the developer understand what they need to consider in satisfying a rule.
  • Element level rules require looking at element's features for content, labeling, properties and states.
  • Page level rules require looking at the page to help users understand the purpose and structure of the content on a page.
  • Website level rules require looking at the set of pages in a website for consistent ordering and labeling of content and making sure users understand the unique content of each page through effective titling of the pages.

Rule and element result values

Value Abbrev Meaning
Violation V One or more target elements fail the requirements of a required rule
Warning W One or more target elements fail the requirements of a recommended rule
Manual Check MC Unable to programmatically determine a result - evaluation requires human inspection and knowledge of the associated WCAG 2.0 requirement.
Pass P All rule requirements were satisfied
Hidden H Target elements were not evaluated because they are hidden (either visually or from assistive technologies)
Not Applicable N/A or ‘-’ Rule was not applied because there were no relevant target elements

More about rules, target resources and result values

  • A rule tests target resources for specific accessibility features.
  • Some rules are fully automated and return pass or fail results and messages.
  • Other rules can only identify the need for human judgement to determine if an accessibility requirement has been met. These return manual check results and messages.
  • If target resources are not found on the page, the rule result is designated as not applicable.
  • If target resources for a rule are hidden, the rule result is designated as hidden. The hidden resources are included in the element results list to enable developers to consider their accessibility if and when they become available to users.
  • Rules also include information on accessibility techniques and links to related resources to help developers understand how to meet the requirements of a rule.

Implementation Score Definitions

  • The implementation score is based on the rule results across all pages and ranges from 0 to 100.
  • '-' means there were no rule results for that group (e.g. rules were not applicable).
  • The following messages give a general idea of the overall implementation of the rules for a website.
Abbrev Score Status Description
C 100 Complete Complete means all rules have passed. There were either no manual checks or all manual checks have been resolved.
R 0 Required Manual Checks
  • Manual checks are required to verify accessibility. Markup has been identified that needs to be reviewed to determine if accessibility requirements have been met.
  • Start with the website and page level manual checks since they affect website and page navigation and identification of content and then start on the element level manual checks.
AC
AC-R
95-99 Almost Complete
  • Almost Complete means that you seem to understand the accessibility requirements of the rules, and are close to fully implementing their requirements on all pages within the website.
  • "-R" means there are required Manual Checks. Markup has been identified that needs to be reviewed to determine if accessibility requirements have been met.
PI
PI-R
50-94 Partial Implementation
  • Partial Implementation means that you may understand at least some of the accessibility requirements.
  • "-R" means there are required Manual Checks. Markup has been identified that needs to be reviewed to determine if accessibility requirements have been met.
  • Please re-read the accessibility requirement and review the techniques before trying to improve the accessibility.
NI
NI-R
0-50 Not Implemented
  • Incomplete means that you do not understand the accessibility requirements of the rules or did not consider accessibility in the design of the website.
  • "-R" means there are required Manual Checks. Markup has been identified that needs to be reviewed to determine if accessibility requirements have been met.
  • Please read the accessibility requirements and review the techniques before trying to improve the accessibility.
na - Not Applicable No markup was found that identifed a known or possible accessibility issue.

Note: Future versions of FAE will allow you to mark manual checks as, pass, fail or not applicable.

Implementation Score Computation

Implementation Score = P / (P + F + MC)

Each variable represents the number of element level results for a particular rule, where:

  • P = elements that passed
  • F = elements that failed
  • MC = elements that require a manual check

The score ranges in value from 0 to 100, that is from no implementation to all accessibility requirements met. In the future, FAE will allow you to change manual checks to pass, fail or not applicable, and will recompute the Implementation Score based on the updated manual checking results.

Note: When a manual check is changed to not applicable that element result is removed from the implementation score computation.

Implementation Score Combinations

Implementation scores are rule-based and are computed for the following combinations:

Pages Rules Calculation
1 1 Element level results determine the implementation score for a rule on a specific page.
1 All The implementation score for a page is the average of all implementation scores for all rules on the page, with all rules being equally weighted.
All 1 The implementation score for a rule across all the pages in a website is based on the aggregation of all element results for the rule across all pages.
All All The implementation score for a website is the average of the website rule implementation scores.

Example Rule Implementation Scores on Individual Page

Description Pass Fail Manual Check Implementation Score
A rule with scope of element, like "Image must have alt text". 8 2 0 80
A rule with scope of page like "Page must have main landmark". 0 1 0 0
A rule with scope of element with elment results of pass, fail and manual checks that may occur for the color contrast rule. 134 29 37 67

Example Rule Implementation Score for All Pages in a Website

This is a hypothetical example for a website

Page results for Rule X Pass Fail Manual Check Implementation Score
Page 1 element results for rule X. 23 12 4 59
Page 2 element results for rule X. 12 4 0 75
Page 3 element results for rule X. 45 0 0 100
Website element results for Rule X 80 16 4 80