SmartSoft and SmartMDSD Toolchain User Study

This is an excerpt of:

Dennis Stampfer, Alex Lotz, Matthias Lutz and Christian Schlegel. “The SmartMDSD Toolchain: An Integrated MDSD Workflow and Integrated Development Environment (IDE) for Robotics Software”. Special Issue on Domain-Specific Languages and Models in Robotics, Journal of Software Engineering for Robotics (JOSER), 2016. Link

Over the last six years, there was a total of 19 public releases of the SmartMDSD Toolchain under an open source license. To date, 36 components are publicly available. Including non-public components, we have about 64 components available for immediate composition. The toolchain has been “demonstrated in operational environments” (technology readiness level (TRL) 6 according to [2] as acknowledged in [3].

A user study was conducted to evaluate the experience in using the SmartMDSD Toolchain, in applying the proposed workflow and to evaluate the perceived benefits among active users. This section will describe the initial results of this user study.

We designed a questionnaire totaling 44 questions of which 30 questions related to the core topic, the other questions for personal background and skills. It took about 20-30 minutes to answer all questions. Results of the questionnaire were collected anonymously. For each question, the users were asked to rate a given statement based on their level of agreement. The questions were designed in five-level Likert [1] items with answers ranging from “strongly agree” to “neutral” to “strongly disagree” and similar (Fig. 1). The participants were asked to skip a question, giving no answer, if the question was not understood.

The user study was conducted with our partners in current research collaborations. Although there were few participants (18 responses), the user study can be considered representative since it is ensured that all participants had spent a good amount of time working with the proposed tools and methods and therefore can give a valuable evaluation.

The user study was organized in several sections, starting by collecting personal experience to set answers in a broader context. Then, follow-up questions specifically assessed the experiences and benefits perceived with the concepts in general and how the SmartMDSD Toolchain helped in applying and using these concepts. Most importantly, we asked about the experience during integration of the project demonstrator during integration meetings.

All 18 participants were using SmartSoft and the SmartMDSD Toolchain for the first time within their research project. About half of them were from industry (45%) or academia (55%). They work in domains such as embedded systems, automotive, robotics, artificial intelligence, sensors and control. Our users are quite experienced with the kind of system development as undertaken in these research projects: The majority is very experienced with software engineering (55%) but have little experience (60% with less than a year) in using software models or model driven methods. Almost all participants are familiar with Eclipse-based IDEs (94% with more than 1-2 years of experience).

When asked about the development methodology and tools that the participants used prior to the SmartMDSD Toolchain, we observed that there is a very heterogeneous set of separated development tools and methods of integration. The user study revealed that using an integrated IDE for development is not yet standard and that the integration methodology is focused on class-based integration. Reuse is made on the level of libraries, but very few component-based approaches are applied.

Most of the participants use the SmartMDSD Toolchain for developing components (94%). They use the functionality they expect from IDEs, such as auto-completion, syntax-checks and warnings on all levels, collected documentation, role-specific views, execution and debugging of the robotics application. As such, 82% state they “can work productively using the toolchain”.

The seamless combination of otherwise separated tools and the seamless transition including handover of artifacts (models) between development steps and roles is easy for most participants (81%). Also, 81% of participants (see Q1 in Fig. 1) agreed that “the toolchain provides the necessary technical workflow-support and aids all developers along the concepts of SmartSoft by guiding them through the overall development process.”

The adequateness of textual and graphical modeling is not easy to find and often a controversial question in MDSD. When asked about the adequateness of textual and graphical modeling for the workflow steps in our toolchain (system design, component modeling, configuration, composition and deployment modeling), all participants in general observe it as adequate, although, there is a slight shift towards “more graphical” modeling.

The participants especially liked the fast development of working applications. Participants noted: “being able to have a prototype working in short time” and “Fast project development, in all phases, is something [we are] looking for.”

SmartSoft and the SmartMDSD Toolchain are a “fundamental contribution to the composition of software building blocks (components) in order to effectively build new applications” (94%, see Q2 in Fig. 1).

For most (67%) of the participants, it “was possible to compose software components to applications without further inspection or even reading source code. The information given through the models is determined to be sufficient for the interface descriptions”. Only 12% disagreed with this statement.

All participants were able to compose new applications from existing components. 56% rated this benefit with “high benefit”, 45% as “beneficial”. Asked for the effort for composition, only two participants felt that the effort is too high.

As with many software development approaches, it takes effort to get used to tools and methods. 56% of the participants needed several weeks while 33% needed several days to get used to it. However, the benefit as perceived by the participants is rewarding: 70% stated that the ``initial effort pays off, especially with larger systems“.

Almost all participants (88%) stated that they “were able to focus on [their individual] field of expertise and core contribution to the project”, no one disagreed (Q3 in Fig. 1). This statement is supported by 88% saying that they ``think that the needs, tasks and responsibilities of [their] specific role were clear and helpful” (one disagreed) and comments by the participants such as: “[the approach] helps to draw boundaries [between roles]”.

The approach presented in this paper clearly “helped to structure project collaboration towards demonstrators” (93%) and provided guidance for almost all participants (81%). Participants especially made “benefit of clear definition of services in system design” (94%) which also successfully “supported the separation of development in space and time” (Fig. 2}). One participant commented that “that developing prototypes is easier since you can easily avoid communication problems that normally arise at the beginning of a project.”

Compared to other approaches, participants noted that they made less errors in system integration (69%) when using our toolchain. In case they encountered “errors or problems, it was easy to identify the specific architectural element that caused it. The problem or error was fixed within that element alone without further influencing others” (77%). To correct problems or errors, “it was easy to identify who (component or partner) is affected and needs to become active” (94%).

During integration activities, most of the actual problems that came up were algorithmic issues (41%). Since we know that 94% of the participants were also component suppliers who need to implement algorithms, we interpret this as a confirmation that the participants were able to focus on their specific role (algorithmic implementation of components). This is supported by a direct question, where almost all participants agreed that they were able to focus on their contribution to the projects (Q3 in Fig. 1). Most participants rated the effort for integration via composition of components as low (50%) while only 17% felt that the effort is high.

Lower flexibility might be observed as a weakness when applying Freedom from Choice. However, participants stated that they “did not feel this as a limitation but rather a very helpful guidance”. While most agreed with this statement (44%), some strongly agreed (25%) and only 19% disagreed.

When asked about weaknesses or ideas for improvements, many answers included tool-documentation in the sense of a user manual, enhancements for graphical tooling, suggestions for more intuitive use and hints on some minor bugs. Methods for structural debugging in the tooling and security capabilities of the approach were mentioned as possible next steps. Finally, one user reported about “not [being] friend of Eclipse, since it is too complex and overloaded”.

Below are several direct comments from the participants:

• “A great approach when collaborating with different partners, from different countries, as it allows us to work in parallel.”
• “I like the way it is organized in chunks (components). This is useful in simplifying a complex problem by providing a higher level of abstraction.”
• “Easy modification and switching of compositions / deployments between different test-runs.”
• “Traditional approach based in libraries/packages may be seen as more flexible, but implies a great effort in integration. The approach provided by SmartSoft / SmartMDSD is clearly more efficient, especially in collaboration projects.”
• “There is a steep learning curve at the beginning where it is not much fun to use SmartSoft, but after investing some time it starts to be fun.”
• “The SmartSoft ecosystem is a great solution to easily analyze new services and features, that can be easily deployed and evaluated.”
• “[SmartSoft] can help project people in order to realize their tasks in communication intensive software systems in an easy way.”

[1] R. Likert, “A technique for the measurement of attitudes.” Archives of Psychology, vol. 22, no. 140, pp. 1–55, 1932.

[2] euRobotics aisbl, “Robotics 2020 Multi-Annual Roadmap,” Dec 2015.

[3] “ITEA2 FIONA Project Review: Framework for Indoor and Outdoor Navigation Assistance,” October 2015, (non-public document).