Weird Scenes Inside the Vivliostyle

Strange days… We use Vivliostyle at Coko, but it has been a fraught experience. Vivliostyle is an open source javascript library for automating typesetting of HTML/CSS content. We use it in Editoria for automating the production of book-formatted PDF from an HTML source. You can read a little about how this works here.

The source code is open source, but it was apparent from the beginning that Vivliostyle-the-company were not into collaborating unless we could pay. We tried very hard to get inside their bubble and contribute, but were pretty well cold-shouldered. We offered meeting resources for community building, and developer and designer time, to help move it all on. Coko went to quite some effort to promote what they were doing (eg https://www.adamhyde.net/why-vivliostyle-is-important/ + the posts on PagedMedia.org by Coko’s Julien Taquet. But they wanted none of it. I was also pretty sure they were being paid to extend Vivliostyle but not putting those changes back into the common pool. Fauxpen, as they say.

We weren’t the only ones to feel things were odd – I spoke to many people who felt the same way and who had had similar experiences. As a result, to de-risk ourselves I founded another initiative – pagedmedia.org – so we would not be reliant on Vivliostyle. My thoughts on this were very similar to what I recently wrote about editors ie. if an open source community is not open to building community, something is wrong and you should carefully consider whether you should be involved with them. They could at any time change direction, and if there was no community at play, then you could be left high and dry. Sometimes you have to make hard decisions and we made it…

No one really wants to build an automated typesetting solution from zero unless they have to. After trying to solve this problem for the last 10 years or so (with open source) I wanted to see it done…so… sigh… if it (unfortunately) wasn’t going to be Vivliostyle after all, then what choice did I have? So I committed to this new endeavor (which is going very well! some cool updates soon) and we launched at MIT Press earlier this year with a community-first approach. We are, similarly, currently considering our choice of editor libraries because of similar concerns.

So, as it happens, I guessed it right and moved at the right time… as that is exactly what has happened with Vivliostyle. A few weeks ago they mysteriously posted a new company name (Trim-Marks Inc) and site under the old URL and pointed vivliostyle.com to vivliostyle.org. No further information was forthcoming. Now there is an announcement about it all on vivliostyle.org… you can read it here:
https://vivliostyle.org/blog/2018/03/26/a-new-beginning/

Essentially Vivliostyle-the-company went off to form trim-marks.com.
I have no idea what they do, but it isn’t open source. Vivliostyle, the open source project, is apparently continuing under the new .org site, led by Shinyu Murakami and Florian Rivoal. I know them and like them both. Very talented and committed people. I met Shinyu when he was thinking about how to tackle this problem – we met at Books in Browsers a long time ago and talked about strategies for solving this problem. So, I wish them all the best, but I retain some initial skepticism – until I see them actively building community, I won’t be terribly interested in going down that path again… They are good folks, so who knows… fingers crossed (the more projects active in this space the better).

Open Source Successes

Believe it or not there is a bit of an active campaign out there to discredit Open Source in the publishing world. I’m pretty sure this is because the nay sayers are feeling a little threatened, else why comment on it at all?

So, just for the record…here are some examples of successful open source projects…

1. The Internet

Yup. The underlying infrastructure for the internet is all open…TCP and IP, also known as TCP/IP, the governing communications protocols among all computers on the Internet, are both open. Don’t forget also that the Berkeley Internet Name Domain, otherwise known as BIND, is open source, and is by far the most common way to publish and resolve Domain Name Service (DNS) queries. Without it, you would type ‘http://google.com’ into your browser location bar and nothing would happen…

2. The Web

Let’s not forget that the internet and the web are not the same things. The internet is that hidden substructure that enables information to ferret around all over the world. The web is what surfaces that info up to you to consume in the browser and other mechanisms. It’s no surprise that the vast market share of browsers are open source now that IE (closed) has fallen to a wee 12-15% or so of the market and Chrome (open) has risen to hold the majority share (somewhere around 60%). Firefox is coming in around 12-15% also. I got my stats from here – https://www.netmarketshare.com/browser-market-share.aspx

Why, do you wonder, did IE fall so far from grace? Could it be that Open Source enables a faster adoption rate?…cause…y’know …it’s free …maybe…

3. Your Computer

Whats that? I hear you say…”I don’t use Linux!”…. yeahyeah…  but do you use Apple? The under-the-hood OS that enables the pretty bells and whistles is the BSD Kernel (the core of the operating system) and that is licensed under BSD, one of the most permissible and liberal Open Source licenses. In fact, Open Source now beats Windows (the only major proprietary OS) in total user-facing operating system market share (including mobile devices, see below) (https://en.wikipedia.org/wiki/Usage_share_of_operating_systems).

4. Your Phone

And your tablet! In the world of mobile devices, Open Source is just killing closed source operating systems hands down (https://en.wikipedia.org/wiki/Mobile_operating_system).

5. Your Website

Yep. over 25% of all websites in the world are run by…WordPress…that humble, non-threatening, simple blogging engine is now an internet monster. It is also by far the most popular CMS in the world. See https://om4.com.au/wordpress/market-share/

Also, don’t forget that your WordPress site is probably delivered to the world by an Apache web server which is open source and holds 50% of the market share (http://www.perfectleads.com/marketshare/apache) which, together with nginx, means open source holds about 65% of the web server market.

And more…

Well, so much more…. Publishers have been skeptical of open source for a very long time but they don’t seem to realise their businesses are built on top of it… They literally could not operate without open source…  It’s like a homeowner complaining that they don’t think plumbing, as an approach to delivering water, works. Or that houses are a bad idea. I’m at a bit of a loss to what to say to such folks…

Designing for All

At Coko we are tackling the long unsolved problem of design in Open Source. I have written a bit about this elsewhere, and it is a hard problem to solve, mainly because it is not solved anywhere – there are no good community models for this and very little discourse about the issue.

Design, of course, covers many things. I am chiefly talking about UX, although we have also solved the problem of designing for users by getting users (I prefer to call them ‘use case specialists’) to design their own systems. But this is not about that. I want to talk just about community processes for UX that will yield common best practices, while also enabling each organisation to realise their desired aesthetics ie achieve the look and feel they want that best reflects their organisation. As reflected in the following screenshots of xpub-Hindawi, xpub-eLife, and xpub-collabra:

b

c

a

There are several mechanisms that we have employed to achieve this. The first is that we have ‘elevated’ design to the same level of value in the community as code. I say ‘elevated,’ not because we had to raise its value in the Coko community, but rather the estimation of the value of design in the open source community at large is very low. So, against this, as a baseline, at Coko we have chosen to value design more than in your typical open source project. Also, I don’t want that to imply there are other disciplines in the software development process that do not rank as highly as these two – documentation, feedback, testing and so on… these are all equally valued alongside design and code, but for now I want to focus just on UX design.

To ‘elevate’ design, it took just the simple step of stating it to be so. As the co-founder of this community I have some sway over the direction and form of the culture. So I cashed out some of this capital to make a very simple statement when I was facilitating a community meeting that we must value design as much as we value code. It was this simple statement that set a placeholder for building processes that put design in the value-center of the community.

Next, when there was a call for more processes to assist communication across the various orgs, I made sure we had two work groups which each consisted of one person from each collaborating organisation. These two foundational WorkGroups were the Dev WorkGroup and the Design WorkGroup.

The Design WorkGroup then went ahead and started discussing how we might contribute to a platform where each org could have the look and feel they wanted. It was a process of first understanding the problem and then working through possible solutions. It was necessarily like this, a process of discovery before implementation, because there are not very many established or discussed processes like this in the open source world that we could follow (as mentioned above). So we have to discover the problem, and then invent a solution.

The problem boils down to small levels of abstraction. Interfaces are, after all, made up of many small parts. The problem therefore resided in how to account for different stylistic approaches to, what we now call, atoms (see Brad Frost’s article on Atomic Design), and how atoms fit together to form ‘molecules’ ie. we needed a level of abstraction that would allow any org to have the look and feel  they wanted for an (eg) button, and also how that button was situated with other elements.

Julien Taquet (Coko) and Nick Duffield (eLife) met with Yannis Barlas (Coko) in Athens for three days to thrash this out.

image-2

They came up with an agreed common set of standardised CSS attributes per UI element, and associated reusable CSS variables that could be used across any UX element.

You can see these lists here…

  1. Color Variables
    https://gitlab.coko.foundation/pubsweet/design/issues/1
  2. Typography Variables
    https://gitlab.coko.foundation/pubsweet/design/issues/2
  3. Form Variables
    https://gitlab.coko.foundation/pubsweet/design/issues/3
  4. Spacing Variables
    https://gitlab.coko.foundation/pubsweet/design/issues/4

That is the simple starting point. In effect, this means that if any organisation designs a feature using UI elements (eg a login form) they can use these shared style names and populate them with their own values. This means another org’s stylesheet will apply equally well to inherited/reusable components as it does to the ones they build themselves.

Which seems simple: it is… but understanding the problem space, and where the commonality lies, was half the battle.

The group has now also identified a common set of UI elements that can be reused and are iterating through this list, designing them as they go. See https://gitlab.coko.foundation/pubsweet/design/issues/21

This is required because even the simplest UI element may have several contexts it has to account for. Take the example of a ‘simple’ button, one of the simplest UI elements of them all…. however, there are actually 5 different button states that must be accounted for – the default state, active, hover, pressed and disabled. In each case, the style variables must be applied appropriately. See here for more information – https://gitlab.coko.foundation/pubsweet/design/issues/19

So, it’s not quite as easy as it might first seem.. the good news is that the teams are very committed to working through these issues, as without doing this groundwork, components are really not reusable.

dsc05093

The result is you can grab components that another org has made, be it a page-based component (eg. Dashboard), or a molecule (eg a Staff Assignment dropdown) and plug it into your system and it will look like it is part of your system.

We are making good progress on all of this, including now integrating common best practices for accessibility into the design process. Also, since this groundwork has come quite far, new orgs joining the Coko community can benefit a lot from the work already done and pick up components from the shared/common UX stylebook and integrate them… pretty cool….

Coko and Open Source

We’ve tried to build software for other people before, but it wasn’t until we worked with the Coko Foundation that we realised how important a “community-first” approach is. The benefits from the conversations, before any code is even written, is obvious when you start writing and even more obvious when you see the quality of the finished product.
– Paul Shannon, Head of Technology, Elife

Paul Shannon is a great advocate for the way we work in the Coko community. eLife was one of the first to see the potential in the way we work, and I remember clearly sitting in a pub talking to Paul about it with Kristen. He was enthusiastic and genuinely supportive when we explained in straight forward terms that we were trying to establish *real* collaboration and not collaboration in name only which is unfortunately so often the case.

We work closely with Paul and the eLife team, as well as the Hindawi team, and in many cases we are each extending the others team. That is what close collaboration is all about. As Paul states:

I love the pleasant surprise when one of your collaborators solves a problem for you, or when you find a problem you thought was just unique to you being discussed elsewhere. For us, the main benefit is knowing that what we’re building is useful to others and is solving the important problems – things get built more quickly and more correctly which reduces the burden on everyone.

This is very true. In a community when you share the burden of a problem you have the luxury of bringing in a bunch of active minds who have a desire to solve the same thing but are not on your payroll. So, you can leverage expertise in areas that your team doesn’t necessarily have. Software development in publishing these days covers a huge scope – everything from workflow, to file formats, OAuth, to containerised deployments and much more. Its hard for any one team to have that scope of expertise. But we have so far found, even with our relatively small foundational group, that we have always been able to find someone in our community that knows a lot about any given problem space.

So, extending your own team and being able to reach out to others with specific expertise is a huge advantage of working in the Coko community. But it doesn’t stop there. It is often the case that someone will be able to improve on your approach to a specific problem, and sometimes it is also true that someone in the community will offer to build it since they have the same need.

dsc04921

This leads to, as Paul says, things getting built more ‘quickly and correctly’. I absolutely agree, as does Andrew Smeall of Hindawi who also collaborates with us:

[Coko] have marshalled a diverse group of publishers towards a shared goal. Together, we’ve implemented a radically open process of constructive collaboration, where each member benefits from the strengths of the others. The result has been rapid progress towards an elegant, maintainable product.
– Andrew Smeall, Head of Strategic Products, Hindawi

This is awesome stuff. Imagine not having to solve all the problems of building a publishing system by yourself?

Of course there is a cost to this. To communicate this I sometimes use the phrase ‘The cost of collaboration is conversation”. This is very true. In order to benefit from collaboration in an open source project, you have to be communicative. We have several mechanisms to help with this, the mattermost chat channel is one. If you visit the Coko GitLab (where the code is stored) you will also find it is very active with conversations in the commits and the issues (eg https://gitlab.coko.foundation/XSweet/XSweet/issues/106) . We also have a lightweight RFC process (eg https://gitlab.coko.foundation/pubsweet/pubsweet/issues/341), and we also have smaller WorkGroups for developers and designers. These WorkGroups consist of one person per collaborating organisation each meeting and is facilitated by Coko. This keeps the group small and the conversation focused, lowering the overhead for all.

The result of all this is that we can keep the overhead low, and produce better products than if we were working alone.  We can also produce them faster, learning from each other as we go. As Andrew also says:

An improvement for one becomes an improvement for all. Individual communities can focus on core areas of expertise — peer review, hosting, discovery — knowing that their innovations will improve the entire system.  The result is more creativity, a more diverse set of solutions, and, ultimately, faster progress.

There is a secondary benefit too… communities are in themselves motivating, as Paul Shannon says:

Having other people build components to solve the problems you are facing is a great benefit of a common infrastructure, but also knowing that what you’re building is useful to more than just your team really adds energy and purpose to our teams of developers and designers.

All in all, its hard to argue against this kind of value in collaboration. I’m not currently seeing any downside except (for me) lots of travel! But thats in my DNA now, so I can handle it….would be good though if there was better surf in London and Athens…

We will add more organisations to the mix as we go…some announcements coming shortly. But here too we are being careful. Its important that we add the right orgs, ones that have a willingness to put in as well as take out…we don’t want to host parasites… this ecosystem can only work if there is close communication and a dedication to working together. This requires trust and good faith and if an org enters this tight community with the intention of free riding they will soon find out they are not welcome. For this to work for you, you need also to work for it. It is not a one way street. If you can accept that and get into the right mode, you will benefit greatly AND you will have a great time… communities are also fun when done right and so far our community meets are super fascinating and filled with awesome folks. It is an honor and a joy to hang out with them and solve problems together.

dsc05118

One of the additional benefits, which I touch on above, is learning from each other. In the last community meet, we had a few weeks ago in London each of the three foundational orgs – Coko, eLife, Hindawi – presented where they were with their journal platforms built on top of PubSweet. Each had taken slightly different paths to solve similar problems. There was much overlap of course, but where I learned the most was where someone had tackled the same problem slightly differently…this was really the first time this had occurred in our community and it was tremendously exciting.

As it happens, there seem to be quite a few orgs wanting to jump into the community. So, from here, it is all about scaling gracefully. We will work out processes to manage and maintain this good faith and good will as we grow. The foundations are there, now it’s all about the mechanics of growth…

Workflow Tweaking at Coko

Yannis and the Athens crew have been tweaking our workflow for the products built on top of PubSweet. This includes (at present) Editoria (book production system) and xpub-collabra (Journal system).

Essentially, after the initial design and build process we slip into faster iterations where I step out of the way and the ‘client’ and developers work in very fast iterations to tweak features and fix bugs. We loop in Julien for UX as necessary. Then we move into a stabilization phase for landing the product.

Right now both Editoria and xpub-collabra are in the tweaks and bugs phase, and we have started adding a ‘roadmap’ to each of the READMEs to list what we are working on now. See:

https://gitlab.coko.foundation/xpub/xpub#roadmap

https://gitlab.coko.foundation/editoria/editoria#roadmap

We’ll be adding a similar table to XSweet and other products. It is a simple, readable, format.

GitLab as Back up

GitLab has a great feature for backing up git repos located anywhere. You can just provide a URL and it will clone that repo and poll every hour for changes and update the mirror accordingly. Info on it here:

https://docs.gitlab.com/ee/workflow/repository_mirroring.html

Super easy to use. I’m using it as an additional backup (we have other back up processes) of our main Coko GitLab repo.

screenshot-2018-3-24-projects-%c2%b7-dashboard

It Always Amazes Me…

Book Sprints is something that popped out of my brain and a lot of experimentation. It’s now its own thing and it constantly surprises me.. .here for example is a post about a Book Sprint last week, where my friend and colleague Barbara Rühling facilitated Cisco to produce a book in 5 days. It had 100 illustrations (!) and will be printed next week…incredible..

https://www.booksprints.net/en/blog/catalyst-9000-done/

I currently don’t facilitate Book Sprints as my Shuttleworth Fellowship requires me to focus on Coko, which is fine with me. It is in good hands 🙂

Rolling Birthday…

I had a rolling birthday over a few weeks. Covering NZ and Berlin, maybe soon birthday drinks in SF….First was  bbq at my place in NZ with the neighbors and Nokome, Jure and Yannis… then dinner with Ma and some old family buddies, and then drinks in Berlin… It was cool for me to see people from different parts of my life hanging together. some photos…

dsc04828

dsc04833

dsc04838

dsc04844

dsc04856

dsc05187

dsc05189

dsc05197

dsc05200

img_0573

p1010457

p1010458

p1010459

p1010462