Feature-Focused Test Generation

Specification-based test generators use formal specifications of input languages to produce valid test inputs for complex systems. They can thus be used to cover the entire input space of the system under test. This broad generation capability, however, can become a problem when one wants to focus test generation on a particular subset of the input space, say, a feature whose implementation has recently changed or is particularly critical.

In this paper, we propose Feature-Focused Test Generation (FFTG), an approach that treats specification-based test generation as a configuration problem. FFTG is deeply rooted in the theories of feature-oriented software development, enabling testers to select and combine features while resolving feature interactions. From the input specification, FFTG first extracts a feature model that represents alternate and optional expansions as possible feature values. After the tester selects which features to focus upon, FFTG specializes the specification accordingly, yielding what we name a slice that generates inputs in which only the focused features vary: “Please generate PNG inputs that have a fixed 8-bit color depth and an Adam7 interlacing, yet explore arbitrary animations.”

We implement FFTG in a prototype named Compás and evaluate it on ten real specifications. We find that Compás-generated slices (1) effectively focus test generation on specific features, preserving validity with 100% of generated inputs accepted by the parser; (2) exercise unique code locations associated with the targeted features and thus enable traceability of features; and (3) enable test suite minimization, with a small set of slices retaining the aggregate coverage of the full suite while requiring fewer tests. To the best of our knowledge, this is the first work to apply techniques from product line engineering to specification-based testing.