Artículo

Estamos trabajando para incorporar este artículo al repositorio
Consulte el artículo en la página del editor
Consulte la política de Acceso Abierto del editor

Abstract:

Code artefacts that have non-trivial requirements with respect to the ordering in which their methods or procedures ought to be called are common and appear, for instance, in the form of API implementations and objects. Testing such code artefacts to gain confidence that they conform to their intended protocols is an important and challenging problem. This paper proposes conformance testing adequacy criteria based on covering an abstraction of the intended behaviour's semantics. Thus, the criteria are independent of the specification language and structure used to describe the intended protocol and the language used to implement it. As a consequence, the results may be of use to black box conformance testing approaches in general. Experimental results show that the criteria are a good predictor for fault detection for protocol conformance and for classical structural coverage criteria such as statement and branch coverage. They also show that the division of the domain derived from the criterion produces subdomains such that most of its inputs are fault revealing. Copyright © 2015 John Wiley & Sons, Ltd.

Registro:

Documento: Artículo
Título:Behaviour abstraction adequacy criteria for API call protocol testing
Autor:Czemerinski, H.; Braberman, V.; Uchitel, S.
Filiación:Departamento de Computación, FCEyN, Universidad de Buenos Aires, Buenos Aires, Argentina
Department of Computing, Imperial College London, London, United Kingdom
Departamento de Computación, Pabellón 1, Ciudad Universitaria, Intendente Güiraldes 2160, Ciudad Autónoma de Buenos Aires, C1428EGA, Argentina
Palabras clave:adequacy criteria; API call protocol; software testing; Abstracting; Computational linguistics; Fault detection; Semantics; Software testing; Specification languages; Adequacy criteria; API calls; Black boxes; Branch coverage; Conformance testing; Coverage criteria; Protocol testing; Sub-domains; Black-box testing
Año:2016
Volumen:26
Número:3
Página de inicio:211
Página de fin:244
DOI: http://dx.doi.org/10.1002/stvr.1593
Título revista:Software Testing Verification and Reliability
Título revista abreviado:Software Test Verif Reliab
ISSN:09600833
CODEN:JTREE
Registro:https://bibliotecadigital.exactas.uba.ar/collection/paper/document/paper_09600833_v26_n3_p211_Czemerinski

Referencias:

  • Frase, G., Arcuri, A., A sound empirical evidence in software testing (2012) ICSE '12, pp. 178-188. , Zürich, Switzerland
  • Xiao, X., Xie, T., Tillmann, N., De Halleux, J., Precise identification of problems for structural test generation (2011) Proceedings of the 33rd International Conference on Software Engineering, ICSE '11, pp. 611-620. , Honolulu, Hawaii, USA
  • Hierons, R.M., Bogdanov, K., Bowen, J.P., Cleaveland, R., Derrick, J., Dick, J., Gheorghe, M., Zedan, H., Using formal specifications to support testing (2009) ACM Computing Surveys, 41, pp. 91-976
  • Beckman, N.E., Kim, D., Aldrich, J., An empirical study of object protocols in the wild (2011) ECOOP '11, pp. 2-26. , Lancaster, UK
  • Groce, A., Zhang, C., Eide, E., Chen, Y., Regehr, J., Swarm testing (2012) ISSTA '12, pp. 78-88. , Minnesotaj, Minneapolis, USA
  • Grieskamp, W., Kicillof, N., Stobie, K., Braberman, V., Model-based quality assurance of protocol documentation: Tools and methodology (2011) STVR, 21 (1), pp. 55-71
  • DeLine, R., Fähndrich, M., Typestates for objects (2004) ECOOP '04, pp. 465-490. , Oslo, Norway
  • Selic, B., The pragmatics of model-driven development (2003) IEEE Software, 20 (5), pp. 19-25
  • Utting, M., Legeard, B., (2007) Practical Model-based Testing: A Tools Approach, , Morgan Kaufmann: San Francisco, CA, USA
  • Beck, K., (2003) Test-driven Development: By Example, , Addison-Wesley Professional: Boston, MA, USA
  • Korel, B., Koutsogiannakis, G., Tahat, L.H., Model-based test prioritization heuristic methods and their evaluation (2007) A-MOST '07, pp. 34-43. , London, UK
  • Offutt, J., Liu, S., Abdurazik, A., Ammann, P., Generating test data from state-based specifications (2003) STVR, 13 (1), pp. 25-53
  • Hierons, R.M., Testing from a Z specification (1997) STVR, 7 (1), pp. 19-33
  • Dick, J., Faivre, A., Automating the generation and sequencing of test cases from model-based specifications (1993) FME '93: Industrial-Strength Formal Methods, pp. 268-284. , LNCS, Odense, Denmark
  • Pecheur, C., Raimondi, F., Brat, G., A formal analysis of requirements-based testing (2009) ISSTA '09, pp. 47-56. , Chicago, Illinois, USA
  • Pretschner, A., Prenninger, W., Wagner, S., Kühnel, C., Baumgartner, M., Sostawa, B., Zölch, R., Stauner, T., One evaluation of model-based testing and its automation (2005) ICSE '05, pp. 392-401. , St. Louis, Missouri, USA
  • Rutherford, M.J., Carzaniga, A., Wolf, A.L., Evaluating test suites and adequacy criteria using simulation-based models of distributed systems (2008) TSE, 34, pp. 452-470
  • Heimdahl, M.P.E., George, D., Weber, R., Specification test coverage adequacy criteria = specification test generation inadequacy criteria? (2004) HASE '04, pp. 178-186. , Tampa, Florida, USA
  • Rothermel, G., Harrold, M.J., Ostrin, J., Hong, C., An empirical study of the effects of minimization on the fault detection capabilities of test suites (1998) ICSM '98, pp. 34-43. , Bethesda, MD, USA
  • Mouchawrab, S., Briand, L.C., Labiche, Y., Di Penta, M., Assessing, comparing, and combining state machine-based testing and structural testing: A series of experiments (2011) TSE, 37 (2), pp. 161-187
  • Weissleder, S., Simulated satisfaction of coverage criteria on uml state machines (2010) ICST '10, pp. 117-126. , Paris, France
  • Drira, K., Azéma, P., Soulas, B., Chemali, A.M., Characterizing and ordering errors detected by conformance testing (1993) IWPTS '92, pp. 67-78. , Montreal, Canada
  • Miller, R.E., Chen, D.L., Lee, D., Hao, R., Coping with nondeterminism in network protocol testing (2005) TESTCOM '05, pp. 129-145. , Montreal, Canada
  • Cihan Yalcin, M., Yenigun, H., Using distinguishing and uio sequences together in a checking sequence (2006) TESTCOM '06, pp. 259-273. , New York, NY, USA
  • Hierons, R.M., Testing from a nondeterministic finite state machine using adaptive state counting (2004) IEEE Transactions on Computers, 53 (10), pp. 1330-1342
  • Dallmeier, V., Knopp, N., Mallon, C., Hack, S., Zeller, A., Generating test cases for specification mining (2010) ISSTA '10, pp. 85-96. , Trento, Italy
  • Liu, L., Meyer, B., Schoeller, B., Using contracts and boolean queries to improve the quality of automatic test generation (2007) TAP '07, pp. 114-130. , Zurich, Switzerland
  • Visser, W., PǍsǍreanu, C.S., Pelánek, R., Test input generation for Java containers using state matching (2006) ISSTA '06, Portland, Maine, USA, pp. 37-48
  • De Caso, G., Braberman, V., Garbervetsky, D., Uchitel, S., Program abstractions for behaviour validation (2011) Proceedings of the 33rd International Conference on Software Engineering, ICSE '11, pp. 381-390. , Honolulu, Hawaii, USA
  • Jeng, B., Weyuker, E., Some observations on partition testing (1989) ACM SIGSOFT Software Engineering Notes, 14 (8), pp. 38-47
  • Field, J., Goyal, D., Ramalingam, G., Yahav, E., Typestate verification: Abstraction techniques and complexity results (2005) Science of Computer Programming, 58 (12), pp. 57-82
  • Fink, S.J., Yahav, E., Dor, N., Ramalingam, G., Geay, E., Effective typestate verification in the presence of aliasing (2008) TOSEM, 17 (2), pp. 91-934
  • Bierhoff, K., Aldrich, J., Modular typestate checking of aliased objects (2007) SIGPLAN Notice, 42 (10), pp. 301-320
  • Bierhoff, K., Beckman, N.E., Aldrich, J., Practical API protocol checking with access permissions (2009) Proceedings of the 23rd European Conference on ECOOP 2009 - Object-Oriented Programming, Genoa, pp. 195-219
  • Lee, D., Yannakakis, M., Principles and methods of testing finite state machines - A survey (1996) Proceedings of the IEEE, 84 (8), pp. 1090-1123
  • De Caso, G., Braberman, V., Garbervetsky, D., Uchitel, S., Validation of contracts using enabledness preserving finite state abstractions (2009) Proceedings of the 31st International Conference on Software Engineering, ICSE '09, pp. 452-462. , Vancouver, British Columbia, Canada
  • Mori, M., Vuong, S.T., On finite covering of infinite spaces for protocol test selection (1995) PSTV '94, pp. 237-251. , Hyderabad, India
  • Inozemtseva, L., Holmes, R., Coverage is not strongly correlated with test suite effectiveness (2014) ICSE '14, pp. 435-445. , Hyderabad, India
  • Andrews, J.H., Briand, L.C., Labiche, Y., Namin, A.S., Using mutation analysis for assessing and comparing testing coverage criteria (2006) IEEE Transactions on Software Engineering, 32 (8), pp. 608-624
  • Andrews, J.H., Zhang, Y., General test result checking with log file analysis (2003) IEEE Transactions on Software Engineering, 29 (7), pp. 634-648
  • Briand, L.C., Labiche, Y., Wang, Y., Using simulation to empirically investigate test coverage criteria based on statechart (2004) ICSE '04, pp. 86-95. , Edinburgh, Scotland, United Kingdom
  • Caso, G.D., Braberman, V., Garbervetsky, D., Uchitel, S., Enabledness-based program abstractions for behavior validation (2013) ACM Transactions on Software Engineering and Methodoly, 22 (3), pp. 251-2546
  • Henzinger, T.A., Jhala, R., Majumdar, R., Permissive interfaces (2005) ACM Sigsoft Software Engineering Notes, 30, pp. 31-40
  • Pacheco, C., Lahiri, S.K., Ernst, M.D., Ball, T., Feedback-directed random test generation (2007) Proceedings of the 29th International Conference on Software Engineering, ICSE '07, pp. 75-84. , Minneapolis, Minnesota, USA
  • Ma, Y.S., Offutt, J., Kwon, Y.R., Mujava: An automated class mutation system: Research articles (2005) Software Testing Verification and Reliability, 15 (2), pp. 97-133
  • Offutt, A.J., Pan, J., Automatically detecting equivalent mutants and infeasible paths (1997) Software Testing Verification and Reliability, 7 (3), pp. 165-192
  • Gligoric, M., Groce, A., Zhang, C., Sharma, R., Alipour, M.A., Marinov, D., Comparing non-adequate test suites using coverage criteria (2013) ISSTA '13, pp. 302-313. , Lugano, Switzerland
  • Jia, Y., Harman, M., An analysis and survey of the development of mutation testing (2011) TSE, 37 (5), pp. 649-678
  • Namin, A.S., Andrews, J.H., The influence of size and coverage on test suite effectiveness (2009) ISSTA '09, pp. 57-68. , Chicago, IL, USA
  • Andrews, J.H., Groce, A., Weston, M., Xu, R.G., Random test run length and effectiveness (2008) ASE '08, pp. 19-28. , L'Aquila, Italy
  • Arcuri, A., Briand, L.C., A practical guide for using statistical tests to assess randomized algorithms in software engineering (2011) ICSE '11, pp. 1-10. , Honolulu, Hawaii, USA
  • Weyuker, E.J., Ostrand, T., Theory of program testing and the application of revealing subdomains (1980) IEEE Transactions on Software Engineering, 6, pp. 236-246
  • Frankl, P.G., Weyuker, E.J., A formal analysis of the fault-detecting ability of testing methods (1993) TSE, 19, pp. 202-213
  • Schneider, S., (1999) Concurrent and Real Time Systems: The CSP Approach, , 1st edn. John Wiley & Sons, Inc.: New York, NY, USA
  • Magee, J., Kramer, J., (2006) Concurrency - State Models and Java Programs, , 2nd edn. Wiley: New York, NY, USA
  • Andrews, J.H., Briand, L.C., Labiche, Y., Is mutation an appropriate tool for testing experiments? (2005) ICSE '05, pp. 402-411. , St. Louis, Missouri, USA
  • Veanes, M., Campbell, C., Grieskamp, W., Schulte, W., Tillmann, N., Nachmanson, L., Model-based testing of object-oriented reactive systems with Spec explorer (2008) Formal Methods and Testing, LNCS, 4949, pp. 39-76
  • Tonella, P., Evolutionary testing of classes (2004) Proceedings of the 2004 ACM Sigsoft International Symposium on Software Testing and Analysis, ISSTA '04, pp. 119-128. , Boston, Massachusetts, USA
  • Andrews, J.H., Menzies, T., Li, F.C.H., Genetic algorithms for randomized unit testing (2011) IEEE Transactions on Software Engineering, 37 (1), pp. 80-94
  • Inkumsah, K., Xie, T., Improving structural testing of object-oriented programs via integrating evolutionary testing and symbolic execution (2008) Proceedings of the 2008 23rd IEEE/ACM International Conference on Automated Software Engineering, ASE '08, pp. 297-306. , L'Aquila, Italy
  • Thummalapenta, S., Xie, T., Tillmann, N., De Halleux, J., Su, Z., Synthesizing method sequences for high-coverage testing (2011) OOPSLA '11, pp. 189-206. , Portland Oregon, USA
  • Sharma, R., Gligoric, M., Arcuri, A., Fraser, G., Marinov, D., Testing container classes: Random or systematic? (2011) FASE '11, pp. 262-277. , Saarbrücken, Germany
  • Czemerinski, H., Braberman, V., Uchitel, S., Behaviour abstraction coverage as black-box adequacy criteria (2013) Proceedings of the 6th International Conference on Software Testing, Verification, and Validation, ICST '13, pp. 222-231. , Luxembourg
  • Gannon, J., McMullin, P., Hamlet, R., Data abstraction, implementation, specification, and testing ACM Transactions on Programming Languages and Systems, 3, pp. 211-223
  • Doong, R.K., Frankl, P.G., The ASTOOT approach to testing object-oriented programs (1994) ACM Transactions on Software Engineering and Methodology, 3, pp. 101-130
  • Bernot, G., Gaudel, M.C., Marre, B., Software testing based on formal specifications: A theory and a tool (1991) Software Engineering Journal, 6 (6), pp. 387-405
  • Weyuker, E., Goradia, T., Singh, A., Automatically generating test data from a boolean specification (1994) IEEE Transactions on Software Engineering, 20 (5), pp. 353-363
  • Gargantini, A., Heitmeyer, C., Using model checking to generate tests from requirements specifications (1999) Software Engineering - ESEC/FSE '99, pp. 146-162. , 1687, Nierstrasz O. Lemoine M. (eds), Lecture Notes in Computer Science. Springer: Berlin / Heidelberg
  • Bochmann, G., Petrenko, A., Bellal, O., Maguiraga, S., Automating the process of test derivation from SDL specifications (1997) SDL Forum '97, pp. 261-276. , Evry, France
  • Petrenko, A., Boroday, S., Groz, R., Confirming configurations in EFSM testing (2004) IEEE Transactions on Software Engineering, 30, pp. 29-42
  • Dan, H., Hierons, R.M., Conformance testing from message sequence charts (2011) 2008 International Conference on Software Testing, Verification, and Validation, pp. 279-288. , Berlin, Germany
  • Ammann, P.E., Black, P.E., A specification-based coverage metric to evaluate test sets (1999) HASE 99, p. 239. , Washington, DC, USA
  • Hong, H., Lee, I., Sokolsky, O., Ural, H., A temporal logic based theory of test coverage and generation (2002) Tools and Algorithms for the Construction and Analysis of Systems, pp. 151-161. , Grenoble, France
  • Muccini, H., Inverardi, P., Bertolino, A., Using software architecture for code testing (2004) TSE, 30 (3), pp. 160-171
  • Frankl, P.G., Weiss, S.N., An experimental comparison of the effectiveness of branch testing and data flow testing (1993) TSE, 19, pp. 774-787
  • Tretmans, J., Conformance testing with labelled transition systems: Implementation relations and test generation (1996) Computer Networks and ISDN Systems, 29 (1), pp. 49-79
  • Bochmann, G.V., Petrenko, A., Protocol testing: Review of methods and relevance for software testing (1994) ISSTA '94, pp. 109-124. , Seattle, WA, USA
  • Mariani, L., Pezzè, M., Riganelli, O., Santoro, M., Autoblacktest: A tool for automatic black-box testing (2011) ICSE '11, pp. 1013-1015. , Honolulu, Hawaii, USA
  • Grieskamp, W., Gurevich, Y., Schulte, W., Veanes, M., Generating finite state machines from abstract state machines (2002) ISSTA '02, pp. 112-122. , Roma, Italy
  • Ostrand, T.J., Balcer, M.J., The category-partition method for specifying and generating functional tests (1988) Communications of the ACM, 31 (6), pp. 676-686

Citas:

---------- APA ----------
Czemerinski, H., Braberman, V. & Uchitel, S. (2016) . Behaviour abstraction adequacy criteria for API call protocol testing. Software Testing Verification and Reliability, 26(3), 211-244.
http://dx.doi.org/10.1002/stvr.1593
---------- CHICAGO ----------
Czemerinski, H., Braberman, V., Uchitel, S. "Behaviour abstraction adequacy criteria for API call protocol testing" . Software Testing Verification and Reliability 26, no. 3 (2016) : 211-244.
http://dx.doi.org/10.1002/stvr.1593
---------- MLA ----------
Czemerinski, H., Braberman, V., Uchitel, S. "Behaviour abstraction adequacy criteria for API call protocol testing" . Software Testing Verification and Reliability, vol. 26, no. 3, 2016, pp. 211-244.
http://dx.doi.org/10.1002/stvr.1593
---------- VANCOUVER ----------
Czemerinski, H., Braberman, V., Uchitel, S. Behaviour abstraction adequacy criteria for API call protocol testing. Software Test Verif Reliab. 2016;26(3):211-244.
http://dx.doi.org/10.1002/stvr.1593