Software Product Line Testing: Coverage, Adequacy and Combinatorial Methods
- Lecturers:
Myra Cohen (University of Nebraska),
Matthew Dwyer (University of Nebraska)
Software product line (SPL) modeling has received a great deal of attention for its potential in fostering reuse of software artifacts across development phases. Research on the testing phase, has focused on identifying the potential for reuse of test cases across product line instances. While this offers reductions in test development effort for a given product line instance, it does not focus on and leverage the fundamental abstraction that is inherent in SPLs - variability.
This lecture will present emerging results that leverage the explicit representation of the variability in SPL models. We illustrate how rich SPL modeling notations can be mapped onto an underlying relational model that captures the feasible product line instances. This relational model serves as the semantic basis for defining a family of coverage criteria for testing of a product line, as opposed to a single product line instance. These criteria make it possible to accumulate test coverage information for the product line itself over the course of multiple product line instance development efforts. Cumulative coverage, in turn, enables targeted testing efforts for new product line instances. A relational encoding of SPL models allows emerging combinatorial interaction testing methods and tools to be applied to generate sets of test configurations that achieve a desired level of coverage; we survey several such methods and available tools. Finally, we describe a set of open questions in the area of SPL testing that will be fruitful for follow on work.
- Supporting Material:
- Reading Material
- Constructing test suites for interaction testing, M.B.Cohen, C.J. Colbourn, P.B. Gibbons and W.B. Mugridge, Proceedings of the International Conference on Software Engineering , May 2003, pp. 38-48.
- Coverage and Adequacy in Software Product Line Testing, Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. (forthcoming)
- Lecture Slides
- Reading Material
Model-driven Development of Certified Avionics Systems with SCADE
- Lecturer:
Jakob Gaertner (Technical Director Central Europe, Esterel Technologies)
- Supporting Material
Model-driven Development of Component-based Systems in Cadena
- Lecturer: John Hatcliff (Kansas State University)
The use of component models such as Enterprise Java Beans and the CORBA Component Model (CCM) in application development is expanding rapidly. Even in real-time safety/mission-critical domains, component-based development is beginning to take hold as a mechanism for incorporating non-functional aspects such as real-time, quality-of-service, and distribution. To form an effective basis for development of such systems in large-scale industrial contexts, we believe that a variety of modeling and analysis capabilities will be required including meta-modeling facilities for describing product-line architectures, light-weight formal specifications, behavioral and dependence analysis, and code generation facilities.
This lecture will present an overview of Cadena -- a sophisticated integrated development environment for building and modeling large-scale component-based systems of systems using widely-used component models such as EJB, CCM, and domain-specific component frameworks. The talk will focus on Cadena's multi-level type-based meta-modeling framework for capturing component-based product-line architectures and various notions of architectural refinement. In this framework, type signatures define contracts for Cadena plug-in points at which different analysis, visualization, and code generation capabilities can be plugged into Cadena to customize the tool environment to particular development scenarios. The architectural refinement relations give rise to a rational framework for plug-in re-use across all architectures whose type signature conforms to the architecture at which the plug-in was originally defined.
Cadena is implemented within IBM's Eclipse IDE, and it has been used by research engineers at Boeing, Lockheed Martin, and Rockwell Collins to demonstrate the effectiveness of model-driven component-based product-line development for avionics and and command-and-control systems.
- Supporting Material:
- Cadena home page
- Reading Material
- CALM and Cadena: Metamodeling for Component-Based Product-Line Development. Adam Childs, Jesse Greenwald, Georg Jung, Matthew Hoosier, and John Hatcliff. IEEE Computer, Feb. 2006, Vol. 39, No. 2, ISSN: 0018-9162. (pdf)
- Cadena: An Integrated Development, Analysis, and Verification Environment for Component-based Systems. John Hatcliff, William Deng, Matthew Dwyer, Georg Jung, Venkatesh Ranganath. ICSE 2003, Portland, Oregon, May 2003. (pdf)
- Lecture Slides
Service Engineering
- Lecturer: Tiziana Margaria (University of Potsdam)
"Service Engineering" indicates the systematic, engineering-oriented design and development of service applications. Its central characteristics are a component based approach, the use of methods and tools to improve the reliability of the products and to increase the efficiency of the development process, and the consequent orientation to the needs of application experts. This last point is achieved by directly enabling application experts to define and design the services and their properties. Technically, this requires concentrating at the model level on the "what" of a service, before one discusses the "how", as well as a continuous, traceable modelling chain from the requirements to the testing and maintenance phases. This lecture will overview service engineering and provide perspective on design and implementation of services.
- Supporting Material
Domain-specific Model Checking with Bogor
- Lecturer: Robby (Kansas State University)
Model checking has proven to be an effective technology for verification and debugging in hardware and more recently in software domains. We believe that for model checking technology to be effectively applied to software, model checking tools must directly support language constructs found in modern object-oriented languages like Java and C# and underlying checking algorithms must make use of sophisticated analyses similar to those found in optimizing compilers to reduce model checking costs. Moreover, recent trends in both the requirements for software systems and the processes by which systems are developed suggest that domain-specific model checking engines may be more effective than general purpose model checking tools.
The Bogor model checking framework is an extensible and highly modular explicit-state model checking framework that aims to (a) overcome limitations of existing model checking tools that fail to provide direct support modeling software, and to (b) enable more effective incorporation of domain knowledge into verification models and associated model checking algorithms and optimizations. This lecture will cover topics related to Bogor's design principles, graphical user interface and visualizations, internal architecture and APIs, working with architectural elements to create customized domain-specific model-checking engines, and example deployments of Bogor in large scale development/verification environments.
- Supporting Material:
- Bogor home page
- Reading Material
- Bogor: An Extensible and Highly-Modular Model Checking Framework Robby, Matthew Dwyer, John Hatcliff. In the Proceedings of the 2003 Symposium on the Foundations of Software Engineering (FSE 2003) (pdf)
- Lecture Slides
Use of Models in Analysis and Design
- Lecturer:
Sriram Rajamani
(Microsoft Research, Bangalore)
Models are abstractions of reality. All branches of engineering rely on models to conquer complexity, and to focus on the right issues, one at a time, while ignoring others. The use of models in main-stream software development is rare. Most main-stream software development happens with the mantra, that "code is the truth, and the only truth". I will present two situations where I have seen models being used:
- Analysis: Automatically extracting models from source code, to do verification. I will present my experiences from the SLAM and Behave! projects at Microsoft Research.
- Design: Describing software at higher levels of abstraction, and using models to generate parts of the so do code generation. I will describe my assessment of the state of the art, and fundamental difficulties this approach has.
- Supporting Material
Behavioral Specifications and Checking in Component-based Systems
- Lecturer Jakob Rehof (Microsoft Research, University Dortmund, Fraunhofer ISST)
- Supporting Material
Software Architecture and Product-line Development
- Lecturer: David Schmidt (Kansas State University)
``Software architecture'' refers to the premises, structure, and pragmatics that underlie a software system. This lecture introduces the topic from a historical perspective, presents standard terminology and approaches, and surveys the evolution of the field into new areas.
Within the core topic of software architecture, we present the evolution of programming-in-the-large into connectors-and-components-style programming, approaches to architectural description, standard styles and patterns of software architecture, application of design patterns to architectural structure, architectural views, and architecture design languages.
From application and evolutionary areas, we cover domain-specific design and languages, product-line development, generative programming, middleware-based approaches, and aspect-oriented design and programming.
- Supporting Material
- Suggested background reading:
- M. Shaw and D. Garlan, Software Architecture. Prentice Hall, 1996.
- K. Czarnecki and U. Eisenecker, Generative Programming. Addison-Wesley 2000.
- Lecture Slides
- Suggested background reading:
A Framework for Agressive Model-Driven Design
- Lecturer:
Bernhard Steffen (University of Dortmund)
Aggressive model-driven development (AMDD) aims at extending the model driven design paradigm to explicitly include the application expert in the design process. This is done be introducing a particularly simple way of graphical modelling, hiding technical details. In particular, this style of model-driven development, strictly separates compatibility and migration issues from model/functionality composition.
JavaABC is a framework for supporting the AMDD process by providing modelling and anaylsis means helping to bridge the gap between the models and the final implementation, and to let application expert take part in the system design process. The lecture will present the following features and their realization in JavaABC and discuss their practicality:
- fully hierarchical, but simple graphical modelling,
- model-level validation,
- model interpretation and code generation,
- testing and monitoring.
Of course, AMDD will never replace genuine application development, as it assumes techniques to be able to solve problems (like synthesis or technology mapping) which are undecidable in general. On the other hand, more than 90% of the application development costs arise worldwide at a much simpler development level, during routine application programming or software update. There, the major challenges are software quantity and evolution rather than conceptual complexity and algorithmic issues, and automation should be largely possible. AMDD is intended to address (a significant part of) this 90% "niche".
- Supporting Material