Finding Out What Might Cause Poor Usability in Open Source Software Products

https://wiki.oulu.fi/download/attachments/34308420/ossd_2013_ylipelto_nissinen_parviainen.pdf?version=1&modificationDate=1384798539000&api=v2

Finding Out What Might Cause Poor Usability in Open Source Software Products, by more fins: Matias Ylipelto, Henna Nissinen & Eero Parviainen

Raza and Capretz (2012) have conducted a study about how the developers collect users’ feedback and how they meet the expectations of the end users. They surveyed the developers of different sized open source projects. According to their results, only 30% of the developers consider usability as the most important quality attribute in their projects and only 42% collected some form of user feedback for their projects. In addition, 77% of the respondents did not consult usability experts for their projects and only one­third of the respondents who consulted an expert actually followed the advices given and modified their project. (Raza & Capretz, 2012.)

Even though their sample size was quite small (72 respondents), their results indicate that the developers do not seem to perceive the importance of usability and that methods to improve the usability, such as collecting users’ feedback and consulting usability professionals are not utilized in most OSS projects. Andreasen et al. (2006) also found out that usability evaluation is not considered as a priority in many OSS projects. Their study indicates that OSS developers are actually interested in usability, but in general, they do not have professional usability practices. Usability experts’ evaluations are appreciated as long as they do not interfere the decision­making about changes and priorities. One reason for OSS developers not being interested in focusing usability issues is that in general, they are seeking for challenges. They want to improve their skills and want to be intellectually stimulated. (Andreasen et al., 2006.)

And a Comment From Eric Raymond

There’s a closely related issue, however that I don’t know how to solve yet without a big player with a lot of money, which is doing systematic user interface end-user testing. We’re not very good at that yet, we need to find a way to be good at it. –

Eric Raymond, Why Open Source will Rule

http://www.zdnet.com/article/eric-raymond-why-open-source-will-rule/

Another Interesting Article on Moving Beyond the Developer-centric Approach

https://tampub.uta.fi/bitstream/handle/10024/81866/gradu04510.pdf

by Finnish student Olli Savolainen

Some quotes (indented). My notes follow some quotes:

Usability work is problematic in open source software development on many levels. There is often no documentation or explicit research of user goals to drive design, no user experience vision to direct development, no usability testing, and typically no resources allocated for usability work. No usability practitioners are usually employed, and doing usability work is hard due to the community value system being focused on concrete results. An attitude of “code is cheap” leads to a tendency to skip user research, design and validation before implementation. However, developers in open source communities do get feedback from actual users.


Usability work in open source software projects is challenging. Pemberton (2004) states the problem being as follows: either open source developers need to become interested in ordinary users, or the users’ needs have to be brought to developers by mediators. Indeed, it seems that not only open source communities have a lack of systematic thinking about users, but it is also challenging for usability practitioners to approach open source communities.

I have a small problem with the direction here, and there is something of a tension in the entire thesis with regard to this point: is the ‘fix’ one of tacking on usability to the end of a process (ie. solve it in the last mile). Or is the fix deeper – in the way products are designed in OS. I think the latter, which is why the Cabbage Tree Method is as it is. I don’t mean to mischaracterise the document as there is quite a bit of examination of these two approaches (and some solutions that appear in-between these two poles). However, it is worth noting that ‘usability’ is quite a different problem to how things are designed and how solutions are imagined.


Projects are often oriented in a straight-to-the-action manner: the user research and design phases are largely skipped


Defining what the software should be like happens iteratively, as development progresses, and replaces the initial planning, requirements analysis, and design phases of conventional development. As such, there may not be an explicit system design, a project plan, a schedule, a list of deliverables, or defined processes (Warsta & Abrahamsson, 2003). The idea of a user interface getting explicitly designed is thus in a sense antithetical to the idea of open source. “Code is cheap” is a general attitude, and the real action happens in programming code. Satisfaction comes from practical problem solving, not from applying abstract usability concepts (such as scenarios or personas). Users cannot be reduced to points in a to do list, so they, too remain abstract (Nielsen & Bødker, 2007) and distant.

I take issue with ‘code is cheap’. I would rather characterise it as code being ‘gold’ in the OS world. It is the only thing of value. UX and design culture (etc) does not have a value and hence does not appear in OS processes or thinking in general.

I don’t disagree with the point being made above, just how it is expressed.


It is said that developers care about the quality of their work more, because they only work on things for which they have a real passion (Raymond, 2001). An open source project is typically a meritocracy (Trudelle, 2002) where status (and trust) in the community is gained by tackling problems considered challenging. There is a bias for improvements in functionality instead of usability: Nichols et al. (2001) state that “hard” algorithmic problems may have a greater perceived value in the “reputation market” than “soft” usability related issues, being also easier to specify, evaluate and modularise than usability issues.

I’m also not entirely convinced by this. I think, with the development of libs like Bootstrap, developers think they can also be designers ie good UX is just a matter of putting the right things in the right place.

But once again, this misplaces the real problem. The problem is in how we create (design) solutions and how communities can be formed that have a diversity of the many skills required to develop kick ass software.


Still, developer communities can appear inward-looking to users and the community hacker culture closed and idiosyncratic (Nichols et al., 2001). Depending on the application domain, there can be a language barrier between developers and end users. Current systems for reporting software issues problems are reported to be difficult to use and biased towards bugs that can easily be expressed textually or with single screenshots. Usability problems and their interrelations are often hard to express in such issue trackers. In addition, analysis of complex usability problems is difficult with these tools and meaning is often lost (Nichols & Twidale, 2003; Viorres et al., 2007). Users can also be seen as co-developers (Raymond, 2001) – if a user has a usability issue, they may be encouraged to fix the application themselves. Alternatively, developers may find it more convenient to encourage the user to read the documentation in order to work around an issue, perceived to be a problem of the user instead of one of the software.

I, like Jonas, rally against requiring users to become programmers. See http://blog.jonasoberg.net/how-free-software-is-failing-the-users/


Twidale and Nichols (2003, 2005) differentiate between objective and subjective usability bugs. It is easy to agree on the severity of objective usability bugs, such as a button not fitting a window in a given situation (rendering the application dysfunctional). On the other hand, developers may be tempted to tag subjective usability bugs as “works for me” in the bug tracking system.

Yes. ‘Works for me’. Says it all really. I want that on a t-shirt.


In May 2007, I proceeded to present my prototype to the community in the forum of the Quiz module 25 . I explained the problems with the then-current UI and offered justifications for the new solutions.

The response I got in the forum thread was mostly negative. A considerable number of community members were not even convinced there were any problems with the UI. Instead of changing the UI, many community members saw improving the documentation a more viable way to approach the difficulties users were facing. Tim Hunt, the maintainer of the Quiz module, was worried about the consistency of my proposition with the rest of the Moodle UI.

I was dumbfounded. I spent days trying to find an appropriate reaction to what they were saying. I had spent some weeks creating a prototype and testing it, and I felt was being treated like I had attacked the community. Where not ignored, I thought I had received little relevant feedback on the actual work presented. From the first responses I could read virtually no understanding or seemingly not even an attempt to understand what I was trying to express. There was no counter argumentation to the actual design, just brief comments bypassing my effort as uncalled for. I used quite a bit of personal support from a colleague, friends and family before going back and continuing the discussion to see just how bad the misunderstandings were – either on my side, or theirs.

Not an uncommon story. Developers being hostile to ‘non-coders’. See this saddening discussion: https://ask.slashdot.org/story/14/12/02/007206/ask-slashdot-non-coders-why-arent-you-contributing-to-open-source


Hunt’s initial response to me in Spring 2008 was that I did not have sufficient experience in the Moodle project to carry out such big changes. In other words, he referred to the fact that as an OSS project, Moodle is a meritocracy where previous evidence of quality code determines whether one is trusted to work further. I sidestepped the meritocracy by stating that the contribution I had to make was not one of programming, but one of usability work. Even if I had had a history of quality code, that would not have proven my skills in usability work. Hunt also proposed making a group of smaller fixes at first, instead of the complete redesign that I was proposing, which exemplifies the fact that in OSS, iteration appears in actual small steps in source code, whereas in UCD it is usually seen that iteration is best done before any implementation. Suggesting small, incremental changes might also be seen connected with thinking that usability can be added to a UI after development.

I find the above paragraph particularly interesting because it shows the problems of trying to introduce new metrics into an established meritocracy. When doing this you essentially have no currency in this environment by definition, which makes cultural processes very hard to change.


Essentially Moodle is still closed source in terms of design (like its proprietary rivals). None of the higher-level thinking about the design of the system is readily available in a manner that would make it possible for anyone to further develop the UI. Such documentation would be critical so that one could know that when doing a redesign, the user needs that were taken into account in while creating the original UI, are still supported. Documenting any of user goals, conceptual models, workflows, personas and scenarios could provide such constraints to base design on.

The above is also interesting in that it suggests that design docs, which usually exist in the head of the BDFL should exist in the open. Else it is a closed, gated, and inaccessible process (the design process). Interesting point.


The methods need to be reconceptualized to fit the OSS culture of practice and the corresponding value system, fostering and taking advantage of the OSS social relationships, since these ”motivate attention to usability issues in day-to-day development” (Terry et al., 2010). UCD being unfit as a process for OSS seems a likely reason why usability practitioners often have hard time gaining trust in OSS communities.


Thus, the notion of a solution encouraging users to further participate in the design instead of being observed is a very tempting one.

You bet.

However, Iivari (2009) states that in the OSS project she studied, users do not have actual decision-making power regarding the OSS but are left in a consultative role. This seems to be true of most OSS projects. Ultimate decision-making is left to developers only, to whom a very limited amount of understanding about the users of the software is available in the forums of the community (Iivari, 2009). In OSS projects, user involvement is thin from the perspectives of both UCD (no usability practitioners to represent users) and participatory design (the few users who do participate have no true power over the eventual software).

Solved!

See also Ollis recent article on the topic:

https://medium.com/interaction-reimagined/user-experience-design-inviting-the-users-25ccd00711c7

Big Ups to Steve Pemberton

I just found this article from 2004 that reflects exactly on an issue I am writing about for Opensource.com (to be published shortly).

http://homepages.cwi.nl/~steven/vandf/2004.1-itch.html

Open Source software is produced by programmers. Programmers are very different from the general public (a far greater proportion of programmers are intuitives than in the general public, for instance). This means that when programmers produce open source software, since they are largely scratching their own itch, they will tend to produce the software for themselves, and in particular be perfectly content with the (programmer-oriented) user interface.

If Open Source software is to make inroads into the general public, something will have to be done about this. The typical response to a complaint about a piece of open source software is “I’ll show you where the source is.” But by definition, the people with this particular itch won’t be programmers, and they won’t know how to fix the problem: the general public will have an itch they can’t scratch; the programmers won’t have that itch, and so won’t scratch it.

This means that if Open Source Software is to appeal to people that are not producing it, the programmers are going to have to start learning to scratch other people’s itches, or people who are able to mediate for the non-programmers are going to have to find a way to tickle the programmers, so that they will scratch it.

I just wrote to Steven and thanked him for writing this great post. He replied with a link to an article he wrote on the topic:

http://interactions.acm.org/archive/view/january-february-2004/scratching-someone-elses-itch1

Unfortunately, his article is behind a paywall, so I can’t get to it. But good to know he has written more on the topic. He also did this W3C talk on the topic:

https://www.w3.org/2010/Talks/06-24-steven-open-source/

Building Alliances

Over the last so-many years, I’ve worked to put together many alliances and consortia, starting with the Open Source Streaming Alliance I co-founded in Amsterdam in 2000 or so with Drazen Pantic, Robert Geus, Jaromil and others, through to helping establish the Substance Consortium recently.

So, it shouldn’t be surprising that I’m involved in a new initiative to develop a new group, this time around open source and open science. The group is called The Open Source Alliance for Open Science. It was previously called ‘Open Source Super Friends’, which I preferred, but you win some and you lose some.

We are putting on a first event in Portland in May. It is invitation-only but you can request an invite here. We also have $ to sponsor travel.

The alliance is put together to help form a network of good-faith practitioners who wish to learn from each other to build a healthy ecosystem of tools and approaches to solving all the crazy problems in open science. My interest is in the ‘publishing’ (sharing) side of this, but when you get down to it, open science is, at its core, really a sharing issue. Open science is, in a sense, a publishing (small ‘p’) issue.

So if this is of interest to you, please apply. If you get turned down, it is probably because we consider you to be from an organisation that is not a good-faith actor in this sector. All proprietary projects fit that description, but also there are some open source projects out there that fit that description. If you are an open source project and get turned down, then I recommend you look deep inside your soul… you are being kept out for a reason.