A summary of Design and validation of precooked developer dashboards

Ivanov et al. Posted in the ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2018 DOI [1]

For the summary of the paper, go to the Summary section of this article.

Table of Contents

• A summary of Design and validation of precooked developer dashboards
  • Table of Contents
  • First Pass
    • Problem
    • Motivation
    • Category
    • Context
    • Contributions
  • Second Pass
    • Background Work
    • Figures, Diagrams, Illustrations, and Graphs
    • Clarity
    • Relevant Work
    • Methodology
    • Author Assumptions
      • Correctness
    • Future Directions
    • Open Questions
    • Author Feedback
  • Summary
  • Summarization Technique
  • Citations

First Pass

Read the title, abstract, introduction, section and sub-section headings, and conclusion

Problem

What is the problem addressed in the paper?

This paper addresses the questions surrounding what developers want to see of their dashboards. As dashboard’s are growing in popularity, their usefulness within the engineering community needs to be addressed and identified.

Motivation

Why should we care about this paper?

The authors of this paper conducted a survey with engineers about what they want to see on their dashboards, as well as introducing a Goal-Question-Metric (GQM) approach to designing developer dashboards.

Category

What type of paper is this work?

This is a survey paper and an problem introduction paper. In other words, this paper only identifies problem and solutions that developers want, but does not provide an implementation of these solutions.

Context

What other types of papers is the work related to?

This paper is related to survey papers and similar problem introduction papers.

Contributions

What are the author’s main contributions?

The authors main contributions are the results of their survey, an understanding of how developers want to use their dashboards, and a GQM approach to identifying why metrics that the survees identified as important are important.

Second Pass

A proper read through of the paper is required to answer this

Background Work

What has been done prior to this paper?

Previous work has been done in identifying metrics that engineers think are important, as well as methods of evaluating the metric’s usefulness. Dashboards have also been created prior to this study and typically follow one of three different patterns:

1. Stategic
2. Operational
3. Analytical

Finally, work has been done in understanding the usefulness of metrics presented on dashboards using GQM.

Figures, Diagrams, Illustrations, and Graphs

Are the axes properly labeled? Are results shown with error bars, so that conclusions are statistically significant?

The tables are easy and clear to read, however, all of the figures use a font or graphic that is too small to understand.

Additionaly, the captions of all figures and tables are sparse.

Clarity

Is the paper well written?

The paper is well written.

Relevant Work

Mark relevant work for review

• Response bias, social desirability and dissimulation [2]
• Common Method Biases in Behavioral Research: A Critical Review of the Literature and Recommended Remedies [3]

The following relevant work can be found in the Citations section of this article.

Methodology

What methodology did the author’s use to validate their contributions?

The author’s interviewed engineers from several different companies to collect their data. Additionally, prior to the interviews, the questions asked went through the GQM method to validate their usefulness.

Author Assumptions

What assumptions does the author(s) make? Are they justified assumptions?

The author’s solely rely upon the GQM method for designing the dashboard. While a fine method to do so, they do not provide evidence as to why this is the superior method to do so.

Correctness

Do the assumptions seem valid?

No.

Future Directions

My own proposed future directions for the work

I’d like to take their results and bake them into future dashboards that I design.

Open Questions

What open questions do I have about the work?

Why weren’t other design philosophies outside of GQM evaluated?

Author Feedback

What feedback would I give to the authors?

I’d like to have seen more evaluation of other GQM like methods.

Summary

A summary of the paper

Design and validation of precooked developer dashboards by Ivanov et al. describes a survey and study that the authors conducted on engineers working at different companies about what would make for an effective dashboard. Their results include utilizing the GQM method for dashboard design, statistics as to what metrics engineers care most about, and a design philosophy regarding dashboards for engineers and their direct management.

Summarization Technique

This paper was summarized using a modified technique proposed by S. Keshav in his work How to Read a Paper [0].

Citations