Accessibility Assessment Automation Testing Using Selenium & HTML Code_Sniffer

Accessibility assessment automation is a piece of code that can evaluate the web controls on a web page for accessibility. Accessibility assessment automation tools are useful because they can save a significant amount of time. The larger the site or the more complex the layout, the higher the chance of missing errors by manual testing. So accessibility automation can be used as an excellent way to generate a list of accessibility issues. There are various errors that can be caught by automation tools and do not need human inspection. A good example is in a case where images do not have any alt text, an accessibility assessment is an efficient way to document each instance one by one when an automatic means of finding is available.

In a project I worked on recently, there was a lot of content to test, so it was a good approach to use automation. It increased the speed of our task and made it more efficient and optimized. The use of a good automated tool can mean a significant improvement in productivity and more accurate results.

In the beginning of our project, we were doing a full manual assessment of accessibility. But later we realized that some of the portions could be automated, and that it would save our time. We had many challenges with heading structure, having alt text for images, form fields association, and missing table headers in the application, so we thought it would be best to automate this part.

We did some research on the many tools in the market and found that HTML Code_Sniffer can help us implement accessibility assessment automation with some coding in Selenium/Java. The best part of this initiative was that everything was available for free on the market. Therefore, we did not face many challenges to convince our client to implement accessibility assessment automation.

In this blog, we will take you through how we achieved automation in accessibility assessment.

Tools used in Accessibility Assessment Automation

We used the following tools for our accessibility assessment automation:

Selenium-WebDriver

Selenium is a popular open-source web-based automation tool. It provides a more stable approach to automating the browser. More about Selenium can be found on the Selenium website: http://docs.seleniumhq.org/docs/03_webdriver.jsp. We used Selenium WebDriver to login to the application and to invoke the HTML_Code Sniffer on Web Page.

TestNG Framework

TestNG is a testing framework that is used with Selenium WebDriver. We used the TestNG framework for different types of annotations and to generate various type of reports (e.g. test output report)

Java

Java is used as a scripting language. It is used with Selenium WebDriver to login to the application and inspect the web controls on the web page.

Eclipse

We used Eclipse IDE (an editor) to write the code using the scripting language Java.

HTML CodeSniffer

The backbone of our accessibility assessment automation process. You can simply drag it from ‘http://squizlabs.github.io/HTML_CodeSniffer/’ and it will be added to your Bookmark. HTML_CodeSniffer is a client-side JavaScript that can check HTML code and detect WCAG 2.0 A/AA/AAA or Section 508 violations. For more information, please visit HTML_CodeSniffer. We did not invoke it from the bookmark; instead, we called the HTML Code_Sniffer JavaScript in Selenium code.

How the Accessibility Assessment Automation Process Works

The steps given below will help you out to understand the accessibility automation using Selenium WebDriver and HTML_CodeSniffer.

Generate the Automation Report

Fetch the URLs and update in Excel sheet – (Excel Creation)

Fetch the URLs that you want to test in your application. We fetched the URLs manually, but any tool like Sitamp can be used. Our automation framework is test-data-driven, so we need to update all of the URLs in an Excel sheet as per the pre-defined format.

We created a workbook named DataProvider.xlsx with two sheets – ‘ApplicationLogin’ and ‘ApplicationInfo.’ The first sheet contains the login credentials of the application on which the accessibility assessment needs to be performed. The second sheet contains the URL of the page and accessibility standards that need to be tested against the application. The automation process will test that standard against the application.

Open the Page in browser – (Selenium Script)

The script will help in logging into the application in the browser. After logging in, it will pick the first URL from the Excel sheet and redirect to that URL in the browser. Once the page is open, we can begin to identify the accessibility issues.

This is the code to open the browser in Firefox:

public static WebDriver methodForFirefox() throws Exception
{
driverForApplication = new FirefoxDriver();
driverForApplication.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
driverForApplication.manage().window().maximize();
return driverForApplication;
}

This is the Selenium code snippet to login to the application:

public static WebDriver methodForFirefox() throws Exception
{
driverForApplication = new FirefoxDriver();
driverForApplication.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS);
driverForApplication.manage().window().maximize();
return driverForApplication;
}

This is the method used to navigate to the URLs mentioned in the data sheet:

public void navigateToURL(String pageUrl) throws AutomationException
{
driver.get(pageUrl);
logger.info("Page is navigated to : " + pageUrl + "  and waiting for 10 seconds to load all Web Elements");
try {
Thread.sleep(10000);
} catch (InterruptedException e)
{
throw new AutomationException(e.getMessage(), e);
}
}

Invoke ‘HTML Code Sniffer’ – (Selenium Script)

Invoke the ‘HTML Code Sniffer’ JavaScript. Select the guidelines for the standard dropdown to show errors, warnings, and notices category. Through the script and Excel sheet configuration, errors or warnings or both can be taken into consideration. In our automation process, we coded the Selenium script and configured our Excel sheet in such a way that either or both of them can be taken.

The following method is used to invoke HTML CodeSniffer on the open URL:

((JavascriptExecutor) driver).executeScript("javascript:(function() {var _p='//squizlabs.github.io/HTML_CodeSniffer/build/';var _i=function(s,cb) {var sc=document.createElement('script');sc.onload = function() {sc.onload = null;sc.onreadystatechange = null;cb.call(this);};sc.onreadystatechange = function(){if(/^(complete|loaded)$/.test(this.readyState) ===  true){sc.onreadystatechange = null;sc.onload();}};sc.src=s;if (document.head) {document.head.appendChild(sc);} else {document.getElementsByTagName('head')[0].appendChild(sc);}}; var options={path:_p};_i(_p+'HTMLCS.js',function(){HTMLCSAuditor.run('WCAG2AA',null,options);});})();");
/*this method is used to select WCAG Standard and total count for Error and Warning */
logger.info("Selecting technique WCAG2AA.");
selectStandard(2); //Select WCAG Technique from Excel sheet notation for which WCAG2AA is set as ‘Yes’
int totalWarning = 0;
if (!sliderStatusDisable) {
totalWarning = Integer.parseInt(warningCount.getText());
}
int totalError = Integer.parseInt(errorCount.getText());
int errorCountInt = totalWarning + totalError;
logger.info("Total Error count is : "+errorCountInt);
logger.info("Clicking on View Report.");
viewReport.click();
for (int i = 1; i < errorCountInt + 1; i++) { if (vrFirstPage.isDisplayed()) { issueTitle.click(); Thread.sleep(1000); summ = titleSummaryText.getText(); tech = techniqueText.getText(); tecHerf = techniqueText.getAttribute("href"); techAndUrl.put(tech, tecHerf); cs = codeSnippentText.getText(); errorWar = isErrorWarning.getAttribute("title");                             } else { int k = i - 1; logger.info("Move to next Error/Warning ----> " + k);
reportNextButton.click();
Thread.sleep(1000);
summ = driver.findElement(By.xpath("//li[@id='HTMLCS-msg-detail-" + k + "']/div[1]/div")).getText();
WebElement prinTechs = driver.findElement(By.xpath("//li[@id='HTMLCS-msg-detail-" + k
+ "']/div[1]/div[2]"));
List tech1 = prinTechs.findElements(By.tagName("a"));
for (int j = 1; j < tech1.size(); j++) {
techAndUrl.put(tech1.get(j).getText(), tech1.get(j).getAttribute("href"));
}
errorWar = driver.findElement(By.xpath("//*[@id='HTMLCS-msg-detail-" + k + "']/div[1]/span")).getAttribute("title");
}

Create Automation Report – The bug report (Selenium Script)

As you saw, we have already invoked HTML Code_Sniffer and it started finding the issues. Our Selenium code/script fetches the defect information (Issue Summary, Principle, WCAG technique, Code Snippet) from HTML Code_Sniffer and puts it into the Automation Report simultaneously. The Automation Report was a single workbook in which multiple worksheets were created based on the number of Page URLs that we needed to test. This excel report contains fields like Summary, Page URL, Page Number, Techniques, and Success Criteria.

Use the following to generate an automation report:

ReportExcelFormat.setCellData( i + rowCount, summ, errorWar, techAndUrl, cs, "WCAG2AA", ApplicationInfo.pageUrl, ApplicationInfo.pageNumber,screenshotName);
techAndUrl.clear();
summ=null;
if (i == errorCountInt) {
rowCount += errorCountInt;
logger.info("Clicking on Code sniffer Home Icon.");
codeSnifferHomeIcon.click();
Thread.sleep(3000);
}

Mail to the user (optional) – (Selenium Script)

After generating the Automation report in previous step, an e-mail can be sent to users.

Use this code to mail the generated report to the user:

public void sendReportMail() throws AutomationException {
try {
Thread.sleep(5000);
if(!ReportExcelFormat.allAttachments.isEmpty()){
DateFormat format = new SimpleDateFormat("MMM-dd-yyyy_HH:mm:ss");
String timeStamp = format.format(new Date());
logger.info("Sending mail to all users");
ApplicationMailMonitoring applicationMail = new ApplicationMailMonitoring();
applicationMail.sendMail(ApplicationMailConfig.server, ApplicationMailConfig.from,
ApplicationMailConfig.to, "Application " +ApplicationMailConfig.subject
+ BrowserOsDetails.getOSName() + "   "           + ReportExcelFormat.browsersWithVersions + " ==>"                       + timeStamp, K12MailConfig.messageBody);
logger.info("Successfully Sent Automation report mail to Users ");
}else{
logger.info("No Attachment is present.");
}
} catch (InterruptedException | MessagingException me) {
logger.warn(me.getMessage());
throw new AutomationException(me.getMessage(), me);
}finally{
System.exit(0);
}
}

Log issues in JIRA

After generating the Automation Report (Bug report), our aim was logging these issues in JIRA. We achieved this through Jersey Client. This API helps to create issue in JIRA using JSON Object.

public static boolean createIssueInJira(String screenshotName, String summay, String pageUrl, String primaryTechnique1, String groupName) throws AutomationException, UniformInterfaceException, ClientHandlerException, JSONException {
try
{
Client client = Client.create();
client.addFilter(new HTTPBasicAuthFilter(getJiraUserName(), getJiraPassword()));
WebResource webResource = client.resource(jiraURL + "/jira/rest/api/2/issue/");
ClientResponse response = webResource.type("application/json").accept("application/json").post(ClientResponse.class,
createIssueMetaData(summay, pageUrl, primaryTechnique1, groupName));
int status = response.getStatus();
String output = response.getEntity(String.class);
//System.out.println(output.toString());
if (addAttachmentToIssue(getIssueKey(output), screenshotName))
logger.info("Attachment is added successfully to issue.");
else
logger.info("Attachment is not added.");
return true;
}  catch (IOException e){
throw new AutomationException("Issue is not Created in JIRA.");
}
}

Limitations

There are some limitations in the Automation Accessibility process as well because some of the cases can not be verified.

Elements

Limitations

ImagesProcess cannot verify if the alt text is meaningful or not. Manual intervention will be needed to judge it.
Logical Reading orderLogical order cannot be judged by the code. Manual intervention will be needed.
Navigation/KeyboardKeyboard navigation cannot be verified by the automation process. Manual intervention will be needed.
HeadingsProcess cannot verify which text should be heading.
Form Error/Warning MessageProcess cannot verify errors or warnings and whether they were handled in the application or not
Table HeaderIf <TH> is not present in the table, the table header cannot be verified by the process.
Table Association Between Header and CellsProcess cannot verify the association between <th> and <scope>
Table used for layout purposeIf the table is used for the purpose of application page layout, this can not be verified by the automation process
Special Characters (Sigma, Math Operators, Chemistry Formula, Math Equations)  read by Screen ReaderSome of the special characters cannot be read by the screen reader and the automation process cannot verify this.

Although we achieved a 29-30% automation accessibility assessment, the automation process helped us to achieve higher quality and to deliver the accessibility testing on time. It saved a lot of time that would normally be used to log bugs in JIRA. Most of the straightforward bugs were logged directly through automation and we put a manual effort into logical bugs in which pure manual intervention was required.

I hope this will help you to understand the accessibility assessment automation process we did in our project. A beautiful part of our accessibility assessment automation process is that you can use it for any application because coding is application-independent. With some small modifications/changes, it can be used for any type of application.

Sayyed Akram Ali

Sayyed Akram Ali

Module Lead QA Engineer

Sayyed Akram Ali is a Module Lead QA Engineer in 3Pillar’s Noida office. He has around 7 years of experience in Software Testing in various domains such as finance, e-commerce, and Learning Management Systems. Sayyed has extensive experience in software testing with quality assurance, as well as great knowledge of Accessibility, WCAG 2.0 Guidelines, Section 508 Standards, VPAT and extensive knowledge of assistive technology and screen readers like NVDA JAWS. He has worked in all phases of the software development life cycle, including requirements analysis, design, development, and integration. Prior to joining 3Pillar, he worked for AonHewitt India.

One Response to “Accessibility Assessment Automation Testing Using Selenium & HTML Code_Sniffer”
  1. Shivam Garg on

    Hi,
    I want to implement the same in our framework using C#. Can you help me the same to implement this.
    In place of excel I want to use .csv files for URL which I take care off.
    Thanks

    Reply
Leave a Reply

Related Posts

How to Test a Website on Mobile for Accessibility Despite the popularity of mobile devices, few people know to use the Accessibility features available to them on their device. It's not easy for someo...
Take 3, Scene 27: The Pros & Cons of Machine Learning &... Julia Slattery sits down with Kathryn Rosaaen and Jesse Vizcaino to define exactly what machine learning is, and explore the pros and cons of incorpor...
Take 3, Scene 10: The Importance of Accessibility This episode of Take 3 centers around a discussion of the importance of accessibility on the web and beyond. 3Pillar's Jessica Hall and Jenna Warren j...
Creating Infrastructure on the Cloud Using AWS CloudFormatio... While working on digital media software a while ago, I came across a requirement to build the whole infrastructure (server, database, and others) to r...
Mobile Automation Testing 101 – Tools, Frameworks and ... In the fast paced software development industry, mobile automation testing has become indispensable. The real value of mobile automation testing is re...