Recommanded Free YOUTUBE Lecture: <% selectedImage[1] %>

Testing & QA in the Web Services World by Damian V. Rinaldi

SOA standard finding broad acceptance as distributed computing platform, and it helps drive IT and business together

“Test early, test often” has long been the mantra for traditional software quality assurance (QA) and testing professionals. However, with the proliferation of Web services-enabled applications and their deployment in Service-Oriented Architectures (SOAs) or event-driven architectures, what used to be a low murmur in the background, is rising to a roar.

Describing one of the forces driving the rising volume, Christopher Lochead, Mercury Interactive’s chief marketing officer, observes, “What most people don’t understand is that what’s going on is the biggest platform shift since mainframe to client/server… . We’re moving from a relatively small number of large apps to a relatively large number of small apps. Without the ability to manage or control these small apps, these composite pieces, today’s solutions will become tomorrow’s problems. The key point is that this is different, and that it is a very significant shift that people are missing because it is happening very gradually.”

Sandy Carter, VP for strategy, channels and marketing for Websphere at IBM, points to another key change in the marketplace. “We started seeing a couple of things around SOA that drove us to take a look at this. Customers weren’t asking what SOA was; they were asking how to get started. Unlike in the past, where the business process guy was asking for a solution, and IT would pour cement over it, today we see business and IT joining hands.”

Another factor is that as Web services adoption rises, more developers are doing more testing. “Developer testing and developer-level testing is getting pretty hot now,” says Parasoft’s Wayne Ariola, VP of corporate development. “Early and often in QA is a lot different than early and often in development.”

QA isn’t likely to disappear, however. Ken Cowan, DevPartner product-line manager at Compuware, thinks that QA isn’t going to be ultimately responsible for testing individual services in isolation. “By virtue of SOAs, developers will have to be developing higher-quality units — but a lot of the asynchronous nature of the business process services will make the QA jobs less grunt work and more of an interesting challenge,” he explains.

A new standard

At a fundamental technology level, the industry is finally seeing broad acceptance of a distributed computing standard, and that is also helping drive the proliferation of Web services. Arthur Ryman, a Rational desktop tools developer at IBM and a leader on Eclipse’s Web tools platform project, described the technology evolution this way: “To start, Web services are a distributed computing technology. Previously, none had become dominant — they were too focused on architecture or programming interfaces as opposed to protocols. Take CORBA, for example. The ‘A’ in CORBA stands for ‘architecture,’ but it deals with design issues, not interoperability. It took a while, but even when IIOP came out to address interoperability, it was too tied to CORBA.”

Ryman continues, “There were others: DCE, RMI (which dealt with JAVA-to-JAVA interoperability — though the versions/implementations of JAVA had to be very close), and Microsoft’s DCOM. But the reality is that the Internet is highly heterogeneous, with no one dominant element. The only way to advance interoperability is to throw away programming interfaces and focus on the protocols (valid exchange patterns) and format (sequence of bytes by which the message is represented). Web services’ evolution began in 2000 with SOAP and WSDL; we’re now in the midst of ‘service-ification’ of the Web.”

“The hype in Web services started early before standards, specifications and tooling were ready. The early notion of automated linking of services without direction or orchestration doesn’t make sense. It has to be driven in a more rigorous way. Before implementation, there has to be a lot of process and infrastructure planning,” says Rob Cheng, director of development solutions at Borland. “People have been experimenting and prototyping — developers have had to wait while architects and CIOs have been working on process and planning.”

From an architectural standpoint, as the use of Web services increases, application failures can be caused not only by the breakdown of individual Web service components in a composite application environment but also by the breakdown of the interactions between and among multiple Web service components and/or the legacy applications with which they coexist and interact. A single failure may no longer be easily isolated in a monolithic application operating behind a firewall, or in a self-contained business environment. It might touch dozens of applications and drive multiple processes inside and beyond the corporate firewall. So testing is critical.

Exploring testing issues

Compuware’s Cowan believes “the biggest issue in testing Web services is ensuring their functional quality, because when you string together a set of services, you introduce many more opportunities for error or failure. Unit testing to verify function is more important than security.” But arguably, security is itself a functional requirement.

Security testing, however, is complex. Theresa Lanowitz, research director at Gartner, says, “Right now, probably fewer than 10% of development organizations within enterprise IT shops understand the need for vulnerability testing — that the code they’re producing is secure code and that they’re not introducing any vulnerability into the source code. The effort has to start with security and vulnerability managers and application developers, including QA, working together to make vulnerability testing part of the overall test plan. The situation today is that there’s some sort of wall that exists between security/vulnerability and application developers. Security starts with the requirements. You can’t just wait until the end of the cycle.”

Parasoft’s Ariola notes, “Today, in 90 percent of organizations, vulnerability testing is considered part of the audit, and that’s far too late. There hasn’t been the big event that made the architect group or CIO aware that these things must happen, but organizations are starting to be aware that security has to be part of the QA issue. Developers are still being coached to produce code that performs, not code that’s secure.”

Demographically, development organizations are operating at a disadvantage when it comes to security. “The number of developers who are well versed in security is extremely small as a percentage of the developer population. It’s analogous to what’s involved with the small number of architects who are overseeing a large development effort,” says Borland’s Cheng.

Serge Lucio, product strategist at IBM/Rational, adds, “Unit testing in traditional software is focused on invoking a sequence of operations in a work flow. In an SOA, at the unit test level, the selection of business data that you’re going to pass to the service and the effort to try to understand the transformation that occurs is much more important. Traditional unit testing data is more fine grained than it is for Web services.” Coarser-grained data will require higher-level tools for developers.

Also, there are different usage models. When you’re testing a traditional application, the tools invoke the application and take a look at what it returns. Lucio points out that “with Web services, you have much more synchronous invocation of [multiple] services, which creates more challenges.”

From a performance testing standpoint with Web services, developers should probably be doing performance testing early on, because as you compose services and deploy them, the performance of that service is going to roll out across the application environment,” Lucio adds.

And from a functional testing standpoint, there is one big issue: QA engineers will typically not have the skill set to test services. Functional testers have more business skills, but not the technical skills to deal with the Web services environment. “When you expose them to the interfaces, they’re going to tend to be scared by that,” says Lucio. Also, at the level of the business process, functional testing has to span a number of technologies. Changes must be orchestrated, because it is much more important to translate the knowledge of the business process definition into a language QA can understand. If a business process modeling tool is used, business process definitions created with that tool must be the source for the tester to act on and from which to implement the functional test.

And there are other sources of risk. Look at a corporation’s application portfolio from any stakeholder’s perspective. It is clear that the risks associated with unintended consequences of application failure are larger and more visible than ever before. As businesses strive for flexibility and agility in response to strategic initiatives, to competitive or other market pressures and to customers’ demands, applications have become and continue to become more dynamic.

As a result, if they are poorly implemented or inadequately tested, applications significantly more fragile. Also, they more directly represent the face that a business presents to its employees, partners and customers and more explicitly enable and embody the practices and processes that drive the “expressions” on that face.

Lost in translation

Web Services Testing Timeline

Phase One (2002-2004): Internally Focused Testing Testing SOAP messages Testing WSDL files and using them for test plan generation Web service consumer and producer emulation

Phase Two (2004-2007): Testing Service-Oriented Architectures Testing the publish, find and bind capabilities of an SOA Testing the asynchronous capabilities of Web services Testing the SOAP intermediary capability Quality of Service monitoring

Phase Three (2005 and beyond): Testing Dynamic Runtime Capabilities Web services orchestration testing Web services versioning testing

Source: Zapthink

And there are cultural issues too. “One of the things we’re seeing in software development is the difficulty in communication among different parts of the organization. Services content feeds back to what the requirements specified; the people getting the requirements and the architects and those responsible for models and development all have to stay in synch,” says Borland’s Cheng. “The requirements are changing, but the risk is a kind of ‘meta’ risk. In other words, in software development you always have functional risk, but this is kind of a project-based risk — it demands a more integrated and rugged process.”

Still, there are barriers. “There are still silos in IT, silos across IT and business, and across the board, we’re not seeing a complete shattering of walls,” says Gartner’s Lanowitz. “But while customers are saying they know they need to do it, they don’t have the funds to do all the work they need to. They still have to balance the competing concerns of the quality triangle: cost, schedule and quality.”

Jason Bloomberg of Zapthink adds, “The silo problem is more complex than just operational security people on one hand and developers and QA on the other. The way people are approaching security is changing. You can no longer rely on a barrier-type metaphor — inside trusted, outside not trusted. That approach no longer works. The traditional barrier approach operates on the OSI stack, and that’s entirely inadequate for dealing with SOA security. XML can come in, but it’s just text. The traditional firewalls just don’t know what’s inside the message. The security issue impacts content-aware networking — message assembly, encryption, decryption, forwarding, etc. — that is part of the SOA architecture.” And all of that needs to be tested.

One key element of an integrated process is that developers have to collaborate more closely and more effectively businesspeople. Active Endpoint’s co-founder and CEO, Fred Holahan, observes, “The person doing this type of testing has to really understand how a composite application was put together. That’s the bad news, but the good news is that the process is likely an automation of an existing process, so some organizations will really be able to involve the business users in crafting the test scenarios. They’ll be able to sit side by side with a developer and provide input.”

Worth the work

Despite the quality and testing challenges raised by the proliferation of Web services, they offer tremendous promise, so developers and QA professionals have begun to address some of the limitations of their existing tools and methods, though it may well be that the methods require more adjustment than the tools do.

“Aside from interoperability, Web services are just like any standard application component,” opines IBM’s Ryman. “Therefore, interoperability tests and verification are the single most important aspect of testing. Initially, the interoperability specifications weren’t standards — both SOAP and WSDL had a lot of soft spots and room for interpretation. Then was formed and defined the basic profile that specified what you had to implement and what elements you had to implement. That was the focal point of initial Web services testing activities.”

Zapthink’s Bloomberg observes that adoption of Web services and service-oriented architectures has taken longer than expected. In his view, customers are still struggling with testing in SOA environments. “If all you’re thinking about is Web services themselves (standards-based interfaces) — you send it test data, load-test it, etc. — that’s a no-brainer and involves traditional testing disciplines. But when you talk about SOAs, that’s something altogether different.” Back in 2002 Bloomberg and Zapthink developed a timeline outlining their expectations for what would be required. (See the updated version of the timeline, reflecting slower-than-anticipated adoption rates.)

Parasoft’s Ariola elaborates. “First, you have testing the message, making sure that the document is valid, that it is being consumed and that it is doing what it’s supposed to do. You also have to do conformance testing to the WSI profile. The tools themselves don’t guarantee compliance/conformance, because standards are still evolving and that really leads into governance and into a credible regression suite. If you have one of these reusable Web service assets, you need an interoperability check, but you also need to make sure that this asset is ready and available for reuse.”

Bloomberg notes that governance has two sides. “If you have an SOA, you want to handle governance in a service-oriented fashion, but you have to build the right governance policies as well.” Governance has an IT operational perspective as well as a business perspective.

Rajesh Radhakrishnan, senior director of product marketing at Mercury, identifies some of the areas governance touches. “We have created risk in three areas. There is business-level exposure via APIs or other interfaces, and that’s because services are being used by many different applications.” The third area is not intuitively obvious, and is more IT-centric and IT relevant than the first two: “You might have a scenario where you have one version of a Web service in deployment, and then the service is upgraded in some manner. If one user of that service is not prepared to upgrade, you may have multiple versions of the same service in production,” Radhakrishnan explains.

Identifying all the versions of a particular service in use at any one time, and which applications are using them, becomes another requirement for the development and QA teams — and for the IT and business stakeholders, who are ultimately responsible for applications once they have been deployed. Ian Muir, CTO of Standard Life Assurance, in describing Web services proliferation in his organization, says that after a multiyear effort dating back to 1999, Standard Life now has a catalog of approximately 300 reusable XML-based business services. “The most common service is used in over 19 different applications.”

That means that impact analysis is critical at both the level of system performance and the business performance layer. Services have to be time stamped by the business owners so that they know — and agree — that a new service is safe to be put into production.

If, as IBM/Rational’s Lucio suggests, “business owners may demand that services won’t be deployed without QA,” it may mean that “test early, test often” will become a way of life, instead of just a mantra for a few.

Damian V. Rinaldi, formerly a Wall Street software sector analyst, IT market researcher and trade publication editor, is now an independent writer and consultant. He can be reached at