Crossing the Line…

10 years ago I started on a new adventure – I was trying to work out how to produce books fast. Along the way came numerous open source book production tools and platforms, and the Book Sprint methodology.

Now, for the first time, I crossed the line from facilitator to participant in a Book Sprint about the Coko PubSweet product. It was a cool experience… at first I thought I was going to be trouble since I have facilitated facilitators before and it’s no fun. When you facilitate facilitators they can all ‘see’ the process, or they think they can, and they try and scramble madly to get meta on it. They can’t just be subjects to it, they need to get above it and can’t help suggesting better ways to do things. Its a real pain.

So I was trying very hard to not be that and I think I did pretty well. I asked our facilitator (Barbara, CEO of Book Sprints) if I could or should do stuff, and I mostly did what she said ūüôā Mostly…but no more or less compliant than your typical participant…I was a little proud of myself for that. I didn’t want to be a Diva.

So, from the other side I noted a couple of things. First of all… Book Sprints are fun. Second… they are tiring… but actually, its more tiring to be a facilitator than a participant. Also, as a participant you can time out from time to time which helps recharge… there is no such luxury for a facilitator…

Other things… good facilitation appears from the outside like there is nothing going on… Barbara is an excellent facilitator, as are all the Book Sprint team…but we (I own the Book Sprint company) see people come out of a Book Sprint thinking they can do the next one themselves. It doesn’t actually happen very often but when it does it fails. What I can understand now a little better is why they think they can do it… it is because they just can’t see the process. They think the facilitator is just some friendly person pointing them in the right direction every now and then… but there is way more going on…. but I can see how they can think there isn’t.

Also, I had some nice bonding experiences. I didn’t expect it, but it happened. When you are tired and in a room working side by side with people for many hours, your defenses just evaporate and I found myself being way more open to letting people get to know me than I usually am.

Also, watching the book you are creating improve as you go is very satisfying. Especially when the illustrations and book design start to land… it’s very cool…

Lastly, there are four other issues that were big take aways… number one – in one session me and two others (Bogdan and Peter) spent an hour hammering out a problem. We essentially were tackling a similar problem from different directions and not quite able to understand each other. The result was that we all got a much clearer shared understanding and together developed a ‘new’ way of describing the problem… that was very exciting and the model we came up with is something I will continue to use. Two – the design process gave us a whole lot of icons that we will also continue to use. That was wholly unexpected and crazy rewarding. Three – books really matter. One of the things I wrote about in the book was something I have been working on for the last months (Workflow Sprints) but the book now instantiates it as ‘a thing’… thats soooo interesting….and Four – the chapters I wrote on Workflow Sprints were added to and edited by others and as a result the description of the process is miles better and I have a much clearer understanding of the process too…

So, it’s interesting to me that I have always witnessed these takeaways in other sprints but I was an outsider to ‘the value’. But being on the inside, these things loom large in my mind, and I feel, walking away from the Book Sprint, that the value I took with me was greater than that printed on the pages of the book…

Whats Involved in Building Editoria

We have a webinar coming up soon for Editoria. In anticipation of that I thought I’d write a little overview of what was involved in building the product. So… Starting with a little hand drawn schematic…

ed
The above shows all the moving parts that we had to consider, and build, to make Editoria what it is. From the user’s perspective, it looks like one app – the Editoria interface – where they can upload a docx file (or files) and start work on producing¬†a book.

From an ‘under the hood’ perspective, it looks like the following…

xsweet

As it happens, we had to build all of the above to make it possible… So, here is a list of all the parts starting from ingest…

Ingest

Ingest consists of 2 parts:

  1. INK
  2. XSweet

INK is a job processing workhorse. Essentially it will run any kind of job you want and ‘steps’ are written for specific kinds of jobs/processes. These are a kind of ‘wrapper’ around executable code. INK has an API that enables platforms like Editoria to send files to it and request a specific set of steps to be run, and then return the result. In this instance, INK runs a set of steps that wrap up XSweet.

XSweet is a set of scripts that take a docx file and convert it to HTML. It is very modular and can be pretty easily extended and improved. It can also be customised per use case.

So, on the ingest stage it looks like the following to the user:

  1. user logins in to Editoria
  2. they create a new book from the book dash
  3. they vlivk on the new book
  4. from the book builder interface they choose ‘upload multiple word files’ or ‘upload word’.
  5. they select the word file(s) from a system dialog
  6. these files are automagically appear ‘in the book’ and can be edited
    1. note : in the case of multiple docs files, if the files have been named according to a simple convention, Editoria will create each new chapter and automatically populate the structure of the book and load the content in the right place

From the tech side the following happens:

  1. when a word doc file or files are selected, the files are sent to INK via the INK API, with a request to convert to docx using the XSweet steps
  2. INK runs the docx file(s) through the XSweet steps, creating HTML
  3. when the job is done, INK signals Editoria that the files are good to go
  4. Editoria grabs each file when it is ready and puts in the the right chapter, creating the chapter if multiple word files are uploaded at once

So… we had to build all of the above. First, INK exists because, crazy as it sounds, we could not find an elegant open source pipeline manager to manage jobs like this. So we decided to build from zero. In addition, which also sounds crazy, there is not a comprehensive open source library out there for converting docx to HTML that was easy to extend or customise for different use cases. We certainly know of many scripts for docx->html (many are listed on the XSweet site), but they are either unmaintained, or difficult to extend. So, unfortunately, we had to build that also.

Editoria App

The Editoria App is what people think of ‘as Editoria’. Everything the user interacts with is, on the surface, Editoria interfaces in the browser. However, Editoria comprises of 3 main parts:

  1. PubSweet
  2. the Editoria interfaces
  3. Wax

PubSweet is the ‘under the hood’ CMS that enables apps like Editoria to be built on top. PubSweet is a Node/React application written entirely in Javascript and it sits on the server and provides a ‘wrapper’ for the Editoria interfaces. You can think of PubSweet as a ‘headless’ CMS that thinks like a publishing system. Once again, when we started (and still today) there was nothing out there that could realise multiple publishing use cases, as diverse as micropubs, books, journals, content aggregation etc, built on top of the same CMS. So we had to build it ourselves. PubSweet is now pretty mature and indeed supports multiple publishing platforms – at last count I think that totals 6 platforms built on top of PubSweet (and it is early days).

The Editoria interfaces are the ‘web pages’ that make up the Editoria application. This includes the dashboard, the book builder etc… these are all custom code written on top of PubSweet to realise the Editoria use case / platform.

Wax is an online editor, it is so sophisticated we prefer to call it a web-based word processor. It is actually an Editoria interface, except that it is so complex and such a critical part of any online publishing tool, that it is a product in its own right. Nothing like this existed either, so we built our own (typical WYSIWYG editors are not sophisticated enough for the publishing use case).

PagedJS

From Editoria, you can press a button and have a book appear in the browser. If you print from the browser, you will then get a PDF ready to take to the printers with all the typographical bells and whistles that books need. It looks magical. We currently use Vivliostyle for this but we have hit many limitations with the software. We reached out to the org that produces the code but they weren’t ready to collaborate to improve the product. Hence, unfortunately, we had to build our own. Soon we will replace Vivliostyle with this new offering. The speed of dev has been incredibly fast on paged.js (working title) and so watch this space for some exciting news.. there are some early demos in the pagedjs repo if you are interested.

Summary

…. So, as you can see, Editoria is more complex than it appears to the eye of the user. Also, the crazy thing is that most of these parts should have existed already,¬†but, crazy as it is, they didn’t exist. There has been no full page, open source, editor out there gives the source control and extensability required for the publishing use case. The docx conversion tools available do not yet do a good enough job of the conversion, and are not easy to customise. There is no headless CMS that ‘thinks like a publisher’ and would help us build Editoria; and we had to create a pagination engine (paged.js) because the one open source option couldn’t give us what we needed and we failed at trying to help them improve the product.

Its not like we wanted to build all this, but the sad fact is for the publishing industry the web is still an innovation they haven’t embraced (except as a distribution medium). So while all of these parts (or many of them) should exist, the sad truth is they did not exist. We did a pretty good look around to see if we could save ourselves the effort before embarking on these projects. We did, for example, bank on Substance Lens Writer for our editor, but it was not maintained and proved unstable, and our eventual assessment was that it was better to start again – hence Wax. We also have gone a long way down the path of using and promoting Vivliostyle, until it became apparent that neither the technology nor the technology providers could get us to where we needed to be… consequently, after implementing Vivliostyle (it is still baked into Editoria) we started a new approach to this problem – Pagedjs.

So, we had to build all of this ourselves… the good news is, that all of these moving parts are reusable in other scenarios… so you can take whatever you like from the above, and add it to your own product and hopefully that will fuel the publisher sectors ability to innovate and transform their products and workflows.

Aperta Halted

A part of my personal history is a platform called Aperta which was previously called Tahi. It was a PLoS project and I was hired to design and build it. I quit when the PLoS Board decided to close the repositories, effectively making it a closed project. The repos remained closed, and as far as I know, are still closed today. Ironically after I left, they renamed the project ‘Aperta’ – Italian for ‘open’. A really silly marketing move to reassure everyone that despite what they may have heard, the project¬†was still open…that was perhaps true, albeit (ironically and literally) in name only.

Now, it seems, the platform dev has been halted. Feels good to me. From what I heard (and I didn’t hear much), PLoS didn’t take the project in a good technical direction and generated a significant amount of bad faith and market confusion while trying to develop it behind closed doors.

To quote the new CEO Allison Mudditt (who I respect very much, Coko worked with Alison when she was at UCP):

Part of this initiative will involve changes around the workflow system ‚Äď Aperta‚ĄĘ¬†‚Äď we set out to develop several years ago with the goal to streamline manuscript submission and handling. At the time we began, there was very little available that would create the end-to-end workflow we envisioned as the key to opening research on multiple fronts. But the development process has proved more challenging than expected and as a result, we‚Äôve made the difficult decision to halt development of Aperta. This will enable us to more sharply focus on internal processes that can have more immediate benefit for the communities we serve and the authors who choose to publish with us. The progress made with Aperta will not be wasted effort: we are currently exploring how to best leverage its unique strengths and capabilities to support core PLOS priorities like preprints and innovation in peer review. This will be part of our planning for 2018.

http://blogs.plos.org/plos/2017/12/ceo-letter-to-the-community-mudditt/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+plos%2FBlog+%28Blogs+-+PLOS%29

I hope that PLoS releases¬†the technologies that have been developed for Aperta (there was a lot more than just the submission system) into the open… with¬†both open repositories and open licenses AND, more importantly, an open heart. Collaboration and openness is more to do with how people are than what open license they choose and several of the practices, including asking potential collaborators to sign Non-Disclosure Agreements (NDAs) before getting a demo of the system, were ridiculous and ungenerous.

Having said that, it would be awesome to see all that work released into the open, in open repos with open licenses, and no more blurring of the word ‘open’. Afterall the systems developed that included Tahi were all paid for by researchers. The PLoS Article Processing Charges fuels PLoS and they committed some of this revenue to the development of Tahi. When I was there, no external funding was secured for developing the system. Pedro Mendes made a good point in response to the announcement:

ha

There is some merit to this, but I do applaud PLoS for being adventurous, and if it had worked then the result would have been APCs could be lowered, not just for PLoS, but for any Journal out there reliant on expensive and dysfunctional Manuscript Submission Systems. Allison also notes this in a discussion below the post mentioned above:

…the original idea was that Aperta would allow us to eliminate or speed up the slowest steps between a finished work and its publication in order to reduce the cost of our publishing services

That is true, and it was an admirable goal. However, whatever the journey was between then and now, the project should have always have been out in the open as a public asset. Open for science, open for access, open for source, open for all Рand the fact researchers paid for it but it was turned into closed project mid-flight is reprehensible and in the end it worked against PLoS, in particular, it severely weakened PLoS claims to supporting all things open. What a mess.

But, it can’t be ignored that Tahi is about 5 years old now, which is old in software years. A entire generation of technologies that are better suited to solving these problems has arisen in that time. The system is now not much more than a still (just) relevant but outdated approach. That is the risk you take when you develop things behind closed doors. By the time you release it (or don’t in this case), it is out of date. That said, it would still be good to release it, but there are better technologies and approaches out there now.

So I look on with interest to see what will happen next.  I sincerely hope PLoS can return to cutting a path through publishing and exploring and enabling a viable Open Access model that others can follow. With Allison at the helm I am betting things are going to take a much needed turn for the better, not just with this project, but on all counts.

As for me, I learned a lot from designing Aperta (I prefer to call it Tahi). The design process was an introduction to scientific journal publishing for me. I learned a great deal. Tahi gave me, at the time, an unencumbered dream time to imagine something new. It had a lot of interesting innovative approaches and if I had stayed with the project it would have ended up close to where PubSweet is now as I wanted to completely decouple the ‘spaces’ (a concept important to Tahi). It would not have been as good as PubSweet at doing this as a complete ‘decouple’ really has to be imagined from the start, and isn’t as clean if retrofitted. Still, the system would have been a lot more flexible and reusable.

But that wasn’t to be. Don’t get me wrong – I don’t think Tahi was the perfect platform, but it was a pretty good starting point with some significant innovations. At the time, I was looking forward¬†to¬†shaping Tahi with use and to mature it into¬†an excellent system. The good news is, the next platform you design is always better.¬† I took a lot of what I learned (I have now been involved¬†in instigating around a dozen publishing systems) to my next development, and worked hard to re-conceptualise a new system that avoided some of the mistakes I made with Tahi, and took some of the good¬†parts a whole lot further. That new project is PubSweet and it is looking awesome,¬†and leverages modern technologies and approaches to the max – mainly thanks to the bunch of amazing folks working on it within the Coko team (particularly Jure Triglav) and also now, increasingly, from the collaborators we work with (at this stage mainly eLife, YLD, Hindawi and ThinSlices). Also a huge thanks to the Shuttleworth for backing me, especially because it was at a time (I had just quit PLoS) when it was very much needed. Their backing meant Coko was possible, and consequently, PubSweet and everything else we have done.

Anyways… it was past time PLoS moved on too from Aperta and congratulations to Allison for making the right call, especially given that it¬†would have been a difficult one given the¬†cultural forces at play inside of PLoS.

Newsbot

Tamlyn Rhodes from YLD in London has developed a cool¬†robot for our instance of Mattermost. It is a newsbot. Essentially, anyone in the Coko Townsquare channel can message ‘@newsbot subscribe’ and will be subscribed to the Coko community newsletter.

The newsletter is an automatically prepared email, compiled once a week. The robot looks through all posts that start ‘#weekly’ (we use this for weekly updates) or ‘:cokobot:’ (tags with a cokorobot icon) and adds these items to the weekly image newsletter (bot icon by Julien Taquet)….we are trying it out this week! Cool stuff. You can see Tamlyn in the pic below, center stage in orange. Photo taken last week at the London PubSweet meet.

dsc04103

Getting Design and User Experience Right in Open Source

So, I’ve thought about Open Source and design… I’ve even written some articles on opensource.com (https://opensource.com/users/adam-hyde) about the subject, and created a methodology for bringing the use-case specialist (user) into the center of the process, along with designers and everyone else…

I’ve also brought this subject up a number of times in Shuttleworth Foundation meetings and¬†received some invaluable advice and insights from fellow Fellows and Shuttleworth staff… many of whom have heard my whacky ideas several times over now and are still patiently listening and offering advice! Forever grateful to y’all… especially Helen, Sean, Arthur, and Andrew for ideas and feedback.

But what I didn’t expect, is that I’d be part of a wider community where these ideas could form the basis of the culture. This is what I saw happen this week in London as part of the PubSweet Global meet Coko hosted (& I facilitated).

There were about 25 of us¬†coming together to discuss all things PubSweet with particular emphasis on building Journals. Present were many folks from eLife, Hindawi, Ubiquity, and Coko. We got to the topic of ‘Technical Council’ and I tabled the idea that we need some kind of process in place so that all stakeholders feel they are getting a say in, and are being heard, the future of PubSweet – since it is their technology too.

dsc03891
CoFounder of Coko, Kristen Ratan, kicking off the meeting

When I tabled the idea that we need some form of technical council, Catriona MacCallum, who I used to work with at PLOS, asked the very salient question Рand what about the users?

Catriona on left.
Catriona on right.

I’m very grateful to Catriona for that question as it gave me the opportunity to open up the concept and talk about how there are very few communities in open source that treat software development as anything other than just a technical problem, and further that we should take this opportunity to experiment in making this community strong on solving ‘user needs’ and design… it was a great discussion and I’m also grateful to eLife’s head of product –¬†Giuliano Maciocci – for having a strong voice in favor of this and really stepping into, what looks to be, an emergent leadership role with regard to design in the community.

dsc04133
Giuliano.

As a result, we formed a Dev Council¬†and a Design Council. These are oversight/communication groups of 5 people each. So, they don’t ‘govern’ but the choice by the community to form these two groups is a¬†testament to how seriously the community is to making beautiful products that solve real problems in publishing for real people…

dsc04029
Whiteboard from the session showing our decisions. Dev/Design council structure at the bottom, also showing their relationship to community (supporting) and Coko (facilitated by).

All in all, a pretty fantastic 3 days.