A Framework For All Levels Of Testing?

Thinking purely in terms of Selenium RC for a moment, I find it hard to find anyone who does not believe that they have a test automation framework. One minute someone has a Selenium IDE test suite and then suddenly they have a Selenium RC framework.

Often all they really have is a test suite usually written in Java for JUnit. So is this really a test framework? I would say; not without qualification. Unit test frameworks are not easily extended to cover tests at all test levels, so remain primarily and solely "unit" test frameworks.

Unless you think automating unit tests is sufficient, you will need a framework that is good for all test levels or different ones focused for particular test levels.

Almost all Selenium RC implementations are fairly technical and have been conceived within the development domain. Most have been written in Java and most of them written for JUnit. The things I would highlight at this stage, is the words "unit" and "development".

Unit test frameworks are there to support testing at the unit test level. Automation conceived exclusively with a view to support development objectives are unsurprisingly focused on doing so.

However, the biggest things determining the success of test automation projects comes down to what it produces in test assets and how in engages test stakeholders.

Frameworks are enabling and there are a whole range of capabilities a framework can enable. Which makes it easy to call anything a framework - even those that do very little.

Test automation frameworks have to fully enable the usability of test assets and the continuous involvement of it's non-developer stakeholders. Without this a framework can enable reporting, debugging, test data abstraction but still remain deficient.

Unit test frameworks, even good ones like TestNG, are enabling to developers and development methodologies. Development methodologies are neutral to automation and test automation methodologies and practices.

So can test automation conceived, built and operated in the development domain, like 95% plus of Selenium RC implementations really achieve all the goals of a test automation framework? You know my answer, but what do you think?

Customer Expectations

The part the customer plays in determining the success or failure in an automation project is critical. After all they are the ones that make all the decisions.

There are no shortage of IT projects that have made serious commitments, financial and otherwise to test automation - yet struggle during implementation.

There are of course many reasons why test automation initiatives appear prone to failure. It is all too easy to blame large suppliers and their failings are well documented.

However, the test automation arena suffers heavily from unrealistic customer expectations. Some suppliers find themselves with no alternative but to pander to them or lose out to a competitor who will.

The “record and playback” paradigm is perhaps one of the most seductive marketing tools in IT. The clear over selling of it’s usefulness has seriously infected the beliefs of people who buy test automation.

The present climate is such, that suppliers which do not offer zero code solutions and test suites built in minutes find it increasingly difficult to get a hearing.

Before making any commitment to test automation, customers should at least consider the following as a tasklisk:

• Set realistic goals around your test automation project
• Draw up evaluation criteria that fit your needs
• Evaluate using a range of suppliers
• Pilot your automation project
• Implement and continually review
  

Traditional Suppliers

The test automation software industry, formally known as the ASQ market has consolidated through numerous acquisitions. It is now dominated by four big suppliers:

• HP Mercury Interactive (WinRunner/QuickTestPro)
• IBM Rational (Rational Robot)
• Compuware (TestPartner)
• Borland Segue Software (SilkTest)

Fully evaluating software provided by these vendors can be expensive. Some of their software certainly has merit and if you have very deep pockets and have made a massive commitment to automation, you probably already have evaluated the main suppliers latest offerings.

Most who reject propriety software do so on cost. However, their biggest problem is one of design.

Advanced automation frameworks need the flexibility and portability of open source software. Test software suites provided by these vendors are closed source with a fixed design. They do not interface with other third party test software that you may wish to use.

Instead they provide suites of software, usually containing components that may well have some value, but packaged up with a collection of others that clearly do not. Many of these components are from an era that lacked support for proper code design, development, test and maintenance.

Using the best tools available for source control, software development, defect tracking and test management is vital to achieve advanced productivity. This ability to choose and integrate the best tool is not an option with propriety test tools.

Consequently, the end product lacks the advanced design features available to OSS frameworks.

Test suites built through the main propriety platforms can end up mostly derived from their “record and playback” features. They end up bloated with massive amounts of code often written in obscure propriety languages, which adher to no recognised methodology or standard.

The big four have no incentive to offer open access to their automation software.

Instead they offer propriety test suites whose primary function is to tie customers to their suites of products. Customers usually end up over committed and very disappointed.

Secondly, the principle of Automation For All means, at the very least, that customers should have some control and flexibility over their financial commitment to test automation. The financial entry level of propriety software is considerable and never ending. Their licensing arrangements can be very restrictive and organisations find themselves limiting its use.

Role-based Framework

The cult of the amateur is a major problem within the test automation industry. Major vendors of test automation software have over sold the involvement of the inexperienced.
Where ever you find test automation projects that fail you will also find test automation novices operating without a methodology or framework.

Traditionally, a testers involvement in automation usually means wearing at least two hats - that of test engineer and test analyst. On very rare occasions this can deliver exceptionally well. As a general practice it is a disaster.

Most test automation efforts merge the implementation of test automation with it's specification. Those that rely on the "record and playback" paradigm are an extreme example of this.

Role-based testing acknowledges the specialisms that exist within the test function. Many test automation projects struggle, simply due to under appreciating the need for specialisation. They deliver neither good analysis nor good engineering.

Only a framework that recognises role based testing can deliver high productivity. This is when test automation engineers and test analysts deliver what they do best. To fully realise this benefit frameworks have to support this distinction by design.

The W3QA Framework is designed to fully support role-based testing.This is one of the advanced features of the W3QA Framework.

Test Analysts write and maintain test specifications and verify failure. Test Engineers write code and schedule the execution of automated test frameworks.

The HRMES Framework is unified across test levels and test assets are common, shared and transferrable, but the entire process supports specialisation and domain expertise.

Executable Test Specifications

The most advanced automation software such as Selenium, is aimed directly at the development community. This severely restricts its broader usability.

For any automation solution to be fully effective, the direct and continuous participation of business domain experts is a must. This is a great challenge.

One of the biggest pitfalls of test automation is one generally known as the pesticide principle. Test specifications may start off relevant but quickly degrade when an application, risk or business priority changes. If they stay in place as originally defined they will quickly lose their value. Specifications have to stay relevant and must be maintainable by those who need to change them.

The W3QA Framework is enabling to non technical testers and Business Analysts. All tests are specified using a non programming narrative, through the Human Readable - Machine Executable Specification (HRMES - pronounced hermes) format.

HRMES has a very clear and intuitive structure derived from FIT and Seleniums Fitnesse formats.

Business domain experts can develop and maintain the definition of test scenarios through Executable Specifications.

Designing usable automation frameworks usually means compromising on power and extendibility. Much like implementations of Selenium Core, which is highly usable but hard to build in advanced features. Selenium users at some point make a decision to migrate from Selenium Core to Selenium RC (SRC). If they want their test framework to easily control the flow of a test case or interact with databases and filesystems they have little option. Selenium allows for the easy migration from their html based IDE into any mainstream programming language. So migration is straightforward.

However, SRC is code driven and migrating comes with a cost of usability and will severely limit the continuing involvement of non-engineers. Which is some cases means the majority of domain experts and business stake holders.

HRMES works very much in the way Selenium IDE/Core works, but HRMES is based on SRC. SRC is very powerful but means rendering all your test specifications into code. HRMES preserves the usability of Selenium IDE/Core without the constraints on it's extensibility. You really do have power and usability with HRMES.

HRMES has the code abstraction and simple narrative of SIDE but without its functional limitations. This abstraction and independence from the automation code deliveries a very high level of test automation maturity. Archiving this criteria is widely recognised, by independent test specialists as one of the highest level of design criterion.

This separation of implementation and specification is pretty much unachievable with propriety automation software. Only an open source framework could deliver this flexibility in its design. However, OSS test software like Selenium do not come with frameworks. HRMES is a execution ready framework.

Low maintenance profile

The HRMES Framework breaks the link between writing more tests and having to write more code. This offers a highly maintainable solution.

Maintenance changes to HRMES are:

• Available instantly to system test and have no code integration and release cycles
• Implemented when the change is known rather than when the code is delivered
• Without involving very costly development resource
  

Business users can change the specification within HRMES without it requiring any changes to the automation framework.

Test specifications will harness the power of Selenium, without having to be written in code. This is as close to a code free paradigm as is possible.

In terms of classification, the W3QA Framework is a Action Based Test (ABT) framework. The W3QA Framework works through implementation mappings to the Selenium API.

ABT frameworks offer a high degree of usability and specification abstraction. They usually come with the cost of custom building the entire mapping between the specification layer and application layer.

However, as the W3QA Framework uses Selenium, all action mappings are open sourced and have been fully mapped to the W3QA Framework using the Seleniums API. This solution is available out of the box and under a free license.

The HRMES format is a simple, flexible and accessible XML based datasource that can be loaded via commonly used spreadsheet packages such as MS Office or OpenOffice.

Making use of OpenQA open source XUL based UI, this format can be fully generated via the Selenium IDE.

The W3QA Framework, including the HRMES format is open source and a free license is available upon request.

Flexible Test Resourcing

One constant phenomenon observed throughout the IT industry is that testing has extremes of activity. Flexible resourcing suits the nature of testing, but is not always possible. Outsourcing can facilitate this.

Even if you are committed to Agile, you still need to weigh the real benefits of outsourcing against co-locating all your system and system level regression tests.

Benefits of outsourcing with W3QA

• Fixed costs turn into variable costs
• Access to specialist technical expertise “on demand”
• Increased productivity
• Allows you to focus on core strengths
• Access to a secure, reliable, scalable and integrated infrastructure
• Opportunity to explore new ways to work
• Interact collaborate and communicate with a test automation pioneer
• Provide complete accountability and reporting on all test execution

Consider the following examples:

A project is using an agile methodology and going through four weekly incremental deliveries. The cumulative affect of regression testing is diverting test resource away from focusing on the current increment. Although the increments are planned, regressing testing is often overlooked during planning.

Another incremental project is geared to weekly scheduled releases to the live environment. There is usually a period of two to three days of system testing before being released. Here full time test resources switch between days of relative inactivity to being required to become supermen and wonderwomen when the application is ready to test.

In both cases, managers have little option but to over resource their test teams or compromise on quality. They find themselves having to retain highly skilled and expensive test specialists during unutilised time.

In an ideal world the tap of test resource would be turned on and off to match the intermittent demand. This is of course not always possible. Often domain expertise prevents simply moving testers around where they are required.

Using the W3QA Framework offers the benefit of having domain expertise involved during test specification but not during execution. This role-based design presents an opportunity for successfully outsourcing a specialist activity that requires no business domain expertise. This enables clients to keep business domain expertise and project management resource on-site.

The scalability offered by W3QA means that any outsourcing can be staged and controlled. Devote Agile practicioners can take cautionary steps whilst evaluating whether they have really lost anything - and hopefully keep some time to assess what they have gained.

Geographically distributed teams (GDT) are becoming common - even within the Agile community. The keenest advocators of co-location are recognising its limitations. Almost as much as those that promote outsourcing are starting to see a retreat from poorly conceived distribution. There are certainly limitations with following the extremes of any practice.

Distributed Working

If there is something holding back the advance of distributed working it is not the workforce. Some degree of remote working has long figured on most peoples wish list.

Now technology looks set to become equally enthusiastic about distribution. The advent of Cloud Computing and WebOS are a clear indication of a very strong trend to support light distributed technology delivered through web virtual infrastructures.

The web enabling of corporate IT applications has been phenomenal.

W3QA does employ communication tools that minimise the downside of distribution and does enable more and frequent communication.

Communication

Some of the tools we use in this area include:

• Instant messaging (IM)
• Web conferencing tools

These do not replace face-to-face communication and are not intended to.

Power with Usability

Software

Web applications are becoming more advanced and more technically demanding to test. This complexity is driven by a strategic move to web enabled corporate IT applications.

The testability of Rich Internet Applications (RIAs) will depend primarily on test automation software and methods keeping pace. The proliferation of SOA and AJAX present enormous challenges to test practices and technologies.

The W3QA Framework is powered with the industry’s most acclaimed and advanced test automation software: Selenium and Soap UI.

People

W3QA understands the need to empower complete teams and preserve the full input of test and business analysis skills in any automation solution. Technical automation solutions need to stay focused on fulfilling test requirements. This can only be successfully achieved when a framework offers clarity and usability to non technical users throughout the implementation lifecycle.

Process

The W3QA Framework is an approach to test automation where test engineering and test analysis deliver a unified test function across test levels.

It avoids the common pitfalls of implementing test automation and offers:

• High Scalability
• Low Maintenance
• Broad Usability
• Affordability