DTM Tool
Graag brengen we je weer op de hoogte van de ontwikkelingen rondom de DTM tool.
DTM nieuwe versie!
Wij zijn afgelopen tijd weer druk bezig geweest met het inventariseren van de feedback en dit te verwerken en te prioriteren in onze DTM Product Backlog. Dit heeft geresulteerd in een nieuwe versie van de DTM tool, versie 2.0.0.
Deze nieuwe versie zal Maandag 4 maart 2013 online staan.
Uiteraard hebben we alle nieuwe zaken ook weer opgenomen in de manual, die te vinden is in het menu onder ‘Help contents’ en hebben we een nieuwe instructie video in het menu opgenomen.
Wat is er verbeterd?
Hieronder een overzicht van de meest belangrijke verbeteringen t.o.v. de vorige versie:
Bugs die zijn gemeld, zijn gefixt.
Undo en Redo functionaliteit opgenomen (onder rechtermuisknop en in hoofdmenu).
Copy & paste functionaliteit opgenomen onder rechtermuisknop.
Overige functies opgenomen onder rechtermuisknop, zoals delete.
De testpaden worden elke keer opnieuw berekend en genummerd bij genereren test cases.
Voor elk model wat wordt geopend, wordt een tab boven het canvas geopend, zodat je snel kunt schakelen tussen de diverse geopende modellen.
Naast de huidige manier van het verbinden van 2 symbolen, is er een nieuwe manier van verbinden toegevoegd. Klik hiervoor in het connectiepunt van waaruit je wilt verbinden en sleep de muis naar het connectiepunt waar je op wilt aansluiten en laat de muis los (een preview van de lijn is zichtbaar bij het slepen met de muis).
Mogelijkheid toegevoegd om kleurwijziging aan te geven per symbool.
Symbolen worden automatisch groter naar gelang er meer tekst wordt ingetypt.
Mogelijkheid om symbool groter of kleiner te maken d.m.v. expand-icon.
Mogelijkheid om vanuit file Explorer de DTM tool te openen.
Laatst geopende projecten zichtbaar en te openen.
Bij openen project vanuit de DTM tool worden alleen de projectfiles .dtmprj opgelicht.
Taal NL en ES toegevoegd.
Connectielijnen worden beter gepositioneerd.
Connectielijnen lichten op t.b.v. betere selectie.
Mogelijkheid om meerdere symbolen tegelijk te verplaatsen d.m.v. pijltjestoetsen.
Tenslotte
Geregeld plaatsen wij informatie over nieuws DTMtool, interessante websites, interessante blogs, etc. op onze facebook pagina. Misschien vindt je het leuk om ook op de hoogte te blijven door ons te volgen via https://www.facebook.com/DtmTestTool?ref=hl . De laastste blog is onlangs ook gepubliceerd op de community Model Based Testen, http://model-based-testing.info/
Uiteraard blijven wij bezig met de verdere ontwikkeling van de tool en zijn we gestart met de ontwikkeling van de SaaS versie. Daarom zouden wij het erg op prijs stellen om van jullie de nodige input te ontvangen als het gaat om verbeteringen en nieuwe features.
Bedankt voor alle enthousiaste reacties en feedback, blijf ons bestoken met je feedback en vragen!
Drawing test model
· Drag and drop elements to the canvas
· Connect elements (two different ways to connect)
· Copy-paste functionality
· Undo-redo functionality
· Editing text in elements
· Expanding elements
· Snap to grid (elemenst will be lined out)
· Change colour of one or more elements
· Delete elements and/or connector lines
· Delete whole test model
· Add name and description to test model
Generate test cases
· Generate test cases on two different algorithms
· Automatically numbering of the test paths
· Visability whole test path in test model
· Add comments to test case
· Add priority to test case
· Add result status to test case (OK / NOK)
· Automatically checks on test model (warning during generation)
Generate test design
· Generate test design and export to Excel or PDF
· Generate test design based on selected priority
General functions
· Auto save file
· Snap to grid
· Change language (NL, EN, ES)
· Set default test coverage
· Export test model to JPG
· Help (instruction manual and video)
· Save test model and project
For more information or a trial version, please visit our website www.dtmtool.com
Model Based Testing nicely deals with modern development methodologies. But also in traditional development projects, this way of testing has many advantages over other testing methods. However, this is highly dependent on the way we organize our MBT testing process and what role is played by the tester. Should the tester blindly assume an already established model or should the tester be responsible for the preparation of the test model? In this paper, an explanation.
Increasingly, we see that Model-Based Testing (MBT) is used for software testing. This is a logical step to keep our value as a functional tester high. As a tester we simply can no longer carry out testing as we previously did which was accustomed to the traditional development processes. No, especially if we look at the contemporary agile development processes, as a tester we must anticipate the new ways of working. Project properties such as interactive, iterative, incremental and multidisciplinary must therefore be consistent with the current test approach. MBT deals nicely with that, provided that this new way of testing is applied correctly.
The benefits of applying MBT are now known. Thus we can, through the proper use of MBT, address any ‘open ends’, ambiguities, inconsistencies and errors in the requirements, at a very early stage. This will shorten the duration of the project and improve the quality of the software. Another great advantage of MBT is the flexibility and adaptability of the test set. If changes occur in the requirements, we merely need to adjust the model and then we can generate a new test set automatically. This automatic generation of the test set is another great advantage of MBT. The condition is that we have the proper tool.
With regards to the above, I would emphasize that MBT is the modern way of testing and has many advantages over other testing methods, but this is highly dependent on the way we organize our MBT testing process and what tool we are using.
V-model
If we look at the traditional development processes, we see that the tester has an independent view on the test basis. Think of the V-model, it shows that the basis for technical testing and development is the technical design and that the tester will use the functional design to set up the test set for the functional test. Precisely this will effectuate an independent view of the tester and any interpretation errors in the requirements will be observed. The tester can make no assumptions and all the required information to establish a thorough test set is collected by the tester himself.
But how does this work in MBT? Is a tester also able to maintain an independent view and does the tester have a clear field to complete the test basis, without making assumptions? The answer is yes, as long as the tester is able to draw the test models himself.
We too often see that within MBT the models that are used to generate the test set, are too technical, static or poorly readable and these models have not been produced by the tester itself. Who can say the author of these models has made no mistakes in the interpretation? Who takes care of updating the models after an update in the requirements? Do all project members, such as the tester, product owner, etc., understand these models? Are all test situations captured in these models? And perhaps more importantly, are these models so arranged that for the generation of the test set, a test algorithm can be used, based on the desired test coverage?
DTMtool
The solution to the above-described problems is simple. In MBT, the functional tester must draw the test models. The tester should then evaluate these models with the team members and business and where necessary adapt the model. Just as the functional tester composes the test set within traditional development processes. With this, the functional tester preserves an important independent perspective, these models are world-readable, these models are easy to adjust by the tester itself and these are really test models. From out of these models, now the test set can be generated automatically based on a test algorithm, taking into account the desired test coverage.
The tool, which will be used, must be adapted to this. The tester itself must be able to draw the test models in the tool and the tool must automatically generate the test set based on different test algorithms (test coverage). The DTMtool makes this possible!
The conclusion is that for functional testing, MBT perfectly fits to modern development processes, but that the value of the functional tester is even greater, with a correct insertion of this method.
For more information about our DTMtool, please visit www.dtmtool.com
Silvio Cacace
Test Consultant, Dialogues Technology
Silvio.cacace@dialoguestechnology.nl
Instruction video DTM test tool (in Dutch).
In the past period we have been busy identifying improvements and features desired by the DTM users. It is very nice to hear all the feedback and see that the DTM tool is so widely used.
Last week we’ve implemented a new version of DTM (version 1.4). This new version includes the ability to generate test cases based on two different algorithms. This allows users to generate test cases with a test medium coverage and a high test coverage. In addition a number of improvements have been made with regard to drawing the test model.
Soon we will launch version 1.5. In this new version a number of issues will be resolved and there will be the ability to generate the test design based on the pre-specified priority level. Also this priority level and specified comments will be includes in the test design, both in the Excel and the PDF test design.
Also we are busy preparing the tool on a so-called SaaS deployment. This deployment is based on the latest version of DTM.
We are proud to announce that at this time a number of new clients using the DTM tool and it is also nice to report that there are several developments abroad.
Naturally, we are concerned with the further development of the tool and therefore we would be very grateful to receive any necessary input when it comes to improvements and features.
Thanks for all the enthusiastic response and feedback, keep bombarding us with your feedback and questions!
Model-Based Testing in Early Software Development Phases
Silvio Cacace and Tanja E.J. Vos
Dialogues Technology, Foppingadreef 22, 1102 BS Amsterdam - The Netherlands Universidad Politecnica de Valencia, Camino de vera s/n 46022 Valencia - Spain
Abstract. Despite the clear advantages of using test models, drawing them is not common practice in industry. This is not in the last place because, up to date, no easy to use tool existed that enables the creation of test models and implies less maintenance when requirements change. In this paper we will discuss a tool that does do these things, yet is very simple, light-weight, easy to learn and does not require experience and or knowledge of difficult formal methods. Basically the tool helps you draw a model, by providing a drawing canvas with a range of shapes and the possibility of connecting them. Secondly, the coverage algorithms underlying the tool, will extract all the combinations of the connections between the elements, and display a minimum set of test cases that can be used for functional or acceptance testing purposes.
1 Introduction
In the eighties, Boris Beizer published his book on blackbox testing techniques [1]. The ideas in this book are simple: draw a model of the expected behaviour of the software system, and cover it. In the book, Beizer advocates that such test suite design activities should start the sooner the better. The presence of the tester in the initial development phases increases the rate of error identification, since ambiguities and unknown issues will be made clear in a very early stage and can help the business to clarify the requirements with the involvement of the stakeholders. Moreover, when testing starts early, the tester can also define the acceptance criteria in that early stage.
Using models or graphs during the early phases of testing has many identified advantages [1, 4, 6]:
– Reviewing a model is more manageable and less error-prone, than reviewing hundreds of test cases.
– Since a model is created from the requirements, it gets easier to trace the model and test cases back to
the requirements.
– Models represents the tester’s understanding of what the software should do In this way these models
can serve as a tool for improving the communication with other stakeholders about the functionalities
of the system.
– Many times, just creating a model from the requirements and showing it to the client will already find
important errors in the requirements.
2 Cacace and Vos
- Using a model to review requirements, takes the ’personal element’ out of reviewing. With traditional reviews, people might feel attacked about the quality of their work. A review based upon a model that the tester himself has made, reduces this element since his own model is being questioned.
- If we have a model, test case generation can be automated. If the model changes, you just generate them again.
- The scope and coverage of a test suite generated from a model becomes clear.
- A model is easier to maintain than test cases when requirements change.
However, despite the clear advantages of using test models, drawing them is not common practice in industry[2]. This is not in the last place because, up to date, no easy to use tool existed that enables the creation of test models. According to a survey done by Binder last year [2], most Model-Based Testing tools that are around and used by industry are found to be too complex, too formal and have a steep learning curve. In this paper we will introduce a tool called DTM (Dialogues Testing Method) [5] a very simple, light-weight and easy to learn tool that enables testers to eas- ily create models that can be used for effective communication and automated test case generation. Basically the tool helps you draw a model, by providing a drawing canvas with a range of shapes and the possibility of connecting them. Secondly, the coverage algorithms underlying the tool, will extract all the combi- nations of the connections between the elements, and display a minimum set of test cases that can be used for functional or acceptance testing purposes. In the following sections we will describe DTM’s methodology, the control flow models, test case generation. Finally, Section 5 concludes with future work. The length of this paper only permits illustration of the models and the generated test cases through very simple examples. However, DTM has been used on various real-world software systems and their requirements.
2 Model-Based Testing Methodology
The MBT methodology behind the DTM tool consists of the following steps:
1. Use the requirements documents that are available to create a model. This is an iterative process that consists of:
-
(a) Understand the requirements and make a mental model of the system
-
(b) Translate the mental model in a flow model and leave open ends for the items that were unclear, inconsistent, ambiguous, and need more explanation from business people, designers or other stakeholders.
-
(c) If there are open ends, organize a brainstorm meeting with the necessary stakeholders to close them.
-
(d) Update the requirements and update your test model with the obtained information and clarifications.
-
(e) If there are no open ends left, review the model with the stakeholders.
-
(f) If the model is found complete during the review meeting, continue with
step 2. If not, go back to step 1d.
Model-Based Testing in Early Software Development Phases 3
Fig. 1. A simple DTM model for calculating a bonus based on amount of years worked.
-
Generate test cases from the model using a preferred coverage criteria.
-
Execute the tests, debug and fix if defects are found, and update the require-
ments and the model if necessary.
-
If the requirements do not need to be updated, the test suite can be saved
for regression testing.
3 Creating a model with DTM
The models used in the DTM tool are control flow graphs. The DTM tool only offers 6 symbols that you can use to draw your test model:
-
– start (a circle with the word start in the centre), is used for indicating the beginning of the control flow.
-
– end (a circle with the word start in the centre), is used for indicating the ending of the control flow.
-
– decision (a diamond) for making binary decisions. The right-arrow is the YES option, the down-arrow is the NO option.
-
– action (an oval), is used to model actions or states.
-
– result(awavedrectangle),isusedforstoringvaluesinvariablesordatabases.
-
– reference (a pentagon), is used to reference another test model.
An example of a simple model is in Figure 1. The model does not contain actions, nor references and serves merely as an example for the next Section.
4 DTM’s Test Case Generation capabilities
When the test model is drawn, abstract test cases can be generated just by one- click. The DTM tool will automatically place numbers beside the arrows coming out from each decision symbol and all the possible test cases will be generated based on a decision coverage algorithm that ensures that each combination of two decisions will be tested at least one time. If you generate your test cases on a test model which is not correct, e.g. the model is missing an end symbol, a
4 Cacace and Vos
path is not complete, etc., you will be notified by a pop up message. The format of the test cases is in plain text and can be exported to Excel (see Figure 2 for the test cases generated form the model in Figure 1).
Fig. 2. DTM test case descriptions in Excell
5 Conclusions and Future Work
We have briefly introduced the philosophy, the methodology, the control flow models and the test case generation capabilities of the DTM tool. When using models during early phases of a development project, the mere fact of creat- ing a model can reveal inconsistencies, ambiguities and incompleteness in the requirements. This saves time and money, as Boehm [3] indicated in 1976.
Future work that is being contemplated for extending the tool is related to sensitization of the abstract test cases with concrete test data, additional coverage criteria for generating test cases, integration and connection with tools for test management and bugtracking, and empirical evaluations of the usability (i.e. effectiveness, efficiency and subjective satisfaction) of the tool.
References
-
Boris Beizer. Black-box testing - techniques for functional testing of software and systems. Wiley, 1995.
-
Robert V. Binder. Model-based testing user survey: Results and analysis, 2011.
-
B. W. Boehm. Software engineering. IEEE Trans. Comp., 25(12):1226–1241, 1976.
-
Alan Richardson. Practical experiences with graph based software testing. In Star-
East Software Testing Conference, 2003.
-
DTM tool. http://www.dtmtool.com/, last visited 27 april 2012.
-
B. Visser and B. Vrenegoor. First model, then test! Testing Experience, 3, 2012.
Last Wednesday, March 28, 2012, we gave our 1st foreign training of the DTM test tool at BULL S.A in Spain (Valencia).
After the last trainee had arrived by train from Madrid, we started the training at 9:30.
The first part of the training was totally dedicated to the concept of Model Based Testing (MBT). Through various slides, we explained and illustrated this concept of testing.
After the coffee break we continued with explaining the testing technique PGA which we use within Dialogues Technology, to test our SCRUM projects. This testing technique is a proven technique, has a very high test coverage and the basis of this technique is used for the algorithm within the DTM test tool.
We explained how we used to work with the testing technique without using a tool. Here we made clear the many advantages of this test technique, but also the disadvantage that this technique is very labor-intensive. Particularly when the specifications are changing.
After the trainees worked out a number of assignments on paper, it was time for a demo of the DTM test tool. It soon became clear what benefits to be gained by using the DTM test tool.
Through a series of slides, the trainees become acquainted with the possibilities that the DTM test tool offers. At the end of the demo, the trainees were able to independently use the tool. At the end of the course there was opportunity for independent practice with the tool and of course there was time to answer the questions.
We found this training session in Valencia very pleasant and want to thank everyone for the positive feedback we have received.
Also interested in the training of our DTM test tool or do you want more information, see our website www.dtmtool.com
Dialogues Test Method (DTM) is focused on functional testing to ensure testing is initiated at the very start of a project, even before any line of code has been written.
The core of DTM is that within Agile processes, like scrum, you can start testing at the beginning of the process and that you are able to execute the testing process in a profound and structured manner during the whole process (static and dynamic tests). The goal of this is to strengthen the business case in an early stage and to be able to identify possible contradictions, unclarities and flaws in the requirements. In this way possible recovery costs will be low (Barry Boehm).
The DTM tool ensures that you can automatically generate all test cases over and over again (with advancing insight), even with a minimal set of requirements. This means a considerable reduction in manual labour to adapt and update the test cases. Besides this it is also possible to generate test cases based on executed risk analysis.
Yesterday we released the new version of the DTM test tool. In this new version some bugs are fixed and some new improvements were added. Also we implemented a totally new interaction design. With this new interaction design now it is even easier to draw a test model and connect the different symbols. Below a brief explanation.
To draw a test model, you have to open the ‘Designer’ tab. After opening this tab, you will see 6 symbols that you can use to draw your test model.
The 6 symbols are: start, end, decision, action, result and reference.
Just by dragging a symbol and dropping this on the canvas, you can draw your test model.
To connect two symbols with an arrow, click in the first symbol and then click in the connection point from where you will connect the line (when clicking in the first symbol, the connection points appears). Then release the mouse and click on the preferred connection point in the second symbol (when hovering the connection points appears) and the two will be connected. See example below.
When we think about functional testing the most people think of executing test cases in the system under test. It does not matter in what kind of a system we execute the test cases. Whether it’s an SAP system, a web application or a batch processing system, the test cases should be executed and the test result gives us insight into the quality of the tested functionality.
However, functional testing is more than just executing test cases, namely developing the test cases in a structured way. If not, we never can say something about the test coverage of the executed test?
It appears a logical story, but more and more we see new test tools that only can be used in the executing phase of the functional test, like the capture & playback tools. But if we do not develop the test cases in a structured manner and with a predetermined test coverage, what are we automate with these kind of tools? Yes, we often hear that these tools generates test cases and generates the script for capture & playback. But are this real test cases or are this just the steps that the tester or end user intends to follow in order to test the functionality?
We are convinced that test cases need to be developed based on one or more test specification techniques. Without using a test specification technique it is not possible to mention about test coverage and we never can be sure that we test the happy flow and the non happy flow.
Within the DTM tool, we therefore integrated a test specification technique. The test cases which will automatically be generated, are based on an algorithm. This algorithm ensures us that we will test each combination of 2 test paths out of a decision symbol in the test model, at least one time.
At this moment we look even for the possibility to expand the DTM tool with multiple algorithms, so that we can generate test cases based on multiple test coverage’s.
For more information about our Agile test tool, see the website www.dtmtool.com
Since the implementation of version 1.0 of the DTM tool, we have been busy with developing new features. One of these new features is the ability to export the test design to an Excel format. This is because an existing customer of the DTM Tool, documents the test requirements into HP Quality Center. This was for us the reason to give this feature a high priority. After a short test phase the current version 1.1. of the DTM Tool is implemented with this new feature included. Therefore, it is now possible to export the test designs to an Excel format that can be read by HP Quality Center (see example below this blog post).
Right now we are busy developing new features and doing some bug fixes. An important new feature is to make drawing a test model more simple. In particular, to draw the connecting line between two components. Simply by clicking on the first component and then the second component, the line is drawn.
We still have many plans to expand the DTM Tool with new features. We also think about connections with existing test tools. We will write about this in a later blog post.
