Crossing the Line…

10 years ago I started on a new adventure – I was trying to work out how to produce books fast. Along the way came numerous open source book production tools and platforms, and the Book Sprint methodology.

Now, for the first time, I crossed the line from facilitator to participant in a Book Sprint about the Coko PubSweet product. It was a cool experience… at first I thought I was going to be trouble since I have facilitated facilitators before and it’s no fun. When you facilitate facilitators they can all ‘see’ the process, or they think they can, and they try and scramble madly to get meta on it. They can’t just be subjects to it, they need to get above it and can’t help suggesting better ways to do things. Its a real pain.

So I was trying very hard to not be that and I think I did pretty well. I asked our facilitator (Barbara, CEO of Book Sprints) if I could or should do stuff, and I mostly did what she said 🙂 Mostly…but no more or less compliant than your typical participant…I was a little proud of myself for that. I didn’t want to be a Diva.

So, from the other side I noted a couple of things. First of all… Book Sprints are fun. Second… they are tiring… but actually, its more tiring to be a facilitator than a participant. Also, as a participant you can time out from time to time which helps recharge… there is no such luxury for a facilitator…

Other things… good facilitation appears from the outside like there is nothing going on… Barbara is an excellent facilitator, as are all the Book Sprint team…but we (I own the Book Sprint company) see people come out of a Book Sprint thinking they can do the next one themselves. It doesn’t actually happen very often but when it does it fails. What I can understand now a little better is why they think they can do it… it is because they just can’t see the process. They think the facilitator is just some friendly person pointing them in the right direction every now and then… but there is way more going on…. but I can see how they can think there isn’t.

Also, I had some nice bonding experiences. I didn’t expect it, but it happened. When you are tired and in a room working side by side with people for many hours, your defenses just evaporate and I found myself being way more open to letting people get to know me than I usually am.

Also, watching the book you are creating improve as you go is very satisfying. Especially when the illustrations and book design start to land… it’s very cool…

Lastly, there are four other issues that were big take aways… number one – in one session me and two others (Bogdan and Peter) spent an hour hammering out a problem. We essentially were tackling a similar problem from different directions and not quite able to understand each other. The result was that we all got a much clearer shared understanding and together developed a ‘new’ way of describing the problem… that was very exciting and the model we came up with is something I will continue to use. Two – the design process gave us a whole lot of icons that we will also continue to use. That was wholly unexpected and crazy rewarding. Three – books really matter. One of the things I wrote about in the book was something I have been working on for the last months (Workflow Sprints) but the book now instantiates it as ‘a thing’… thats soooo interesting….and Four – the chapters I wrote on Workflow Sprints were added to and edited by others and as a result the description of the process is miles better and I have a much clearer understanding of the process too…

So, it’s interesting to me that I have always witnessed these takeaways in other sprints but I was an outsider to ‘the value’. But being on the inside, these things loom large in my mind, and I feel, walking away from the Book Sprint, that the value I took with me was greater than that printed on the pages of the book…

Whats Involved in Building Editoria

We have a webinar coming up soon for Editoria. In anticipation of that I thought I’d write a little overview of what was involved in building the product. So… Starting with a little hand drawn schematic…

ed
The above shows all the moving parts that we had to consider, and build, to make Editoria what it is. From the user’s perspective, it looks like one app – the Editoria interface – where they can upload a docx file (or files) and start work on producing a book.

From an ‘under the hood’ perspective, it looks like the following…

xsweet

As it happens, we had to build all of the above to make it possible… So, here is a list of all the parts starting from ingest…

Ingest

Ingest consists of 2 parts:

  1. INK
  2. XSweet

INK is a job processing workhorse. Essentially it will run any kind of job you want and ‘steps’ are written for specific kinds of jobs/processes. These are a kind of ‘wrapper’ around executable code. INK has an API that enables platforms like Editoria to send files to it and request a specific set of steps to be run, and then return the result. In this instance, INK runs a set of steps that wrap up XSweet.

XSweet is a set of scripts that take a docx file and convert it to HTML. It is very modular and can be pretty easily extended and improved. It can also be customised per use case.

So, on the ingest stage it looks like the following to the user:

  1. user logins in to Editoria
  2. they create a new book from the book dash
  3. they vlivk on the new book
  4. from the book builder interface they choose ‘upload multiple word files’ or ‘upload word’.
  5. they select the word file(s) from a system dialog
  6. these files are automagically appear ‘in the book’ and can be edited
    1. note : in the case of multiple docs files, if the files have been named according to a simple convention, Editoria will create each new chapter and automatically populate the structure of the book and load the content in the right place

From the tech side the following happens:

  1. when a word doc file or files are selected, the files are sent to INK via the INK API, with a request to convert to docx using the XSweet steps
  2. INK runs the docx file(s) through the XSweet steps, creating HTML
  3. when the job is done, INK signals Editoria that the files are good to go
  4. Editoria grabs each file when it is ready and puts in the the right chapter, creating the chapter if multiple word files are uploaded at once

So… we had to build all of the above. First, INK exists because, crazy as it sounds, we could not find an elegant open source pipeline manager to manage jobs like this. So we decided to build from zero. In addition, which also sounds crazy, there is not a comprehensive open source library out there for converting docx to HTML that was easy to extend or customise for different use cases. We certainly know of many scripts for docx->html (many are listed on the XSweet site), but they are either unmaintained, or difficult to extend. So, unfortunately, we had to build that also.

Editoria App

The Editoria App is what people think of ‘as Editoria’. Everything the user interacts with is, on the surface, Editoria interfaces in the browser. However, Editoria comprises of 3 main parts:

  1. PubSweet
  2. the Editoria interfaces
  3. Wax

PubSweet is the ‘under the hood’ CMS that enables apps like Editoria to be built on top. PubSweet is a Node/React application written entirely in Javascript and it sits on the server and provides a ‘wrapper’ for the Editoria interfaces. You can think of PubSweet as a ‘headless’ CMS that thinks like a publishing system. Once again, when we started (and still today) there was nothing out there that could realise multiple publishing use cases, as diverse as micropubs, books, journals, content aggregation etc, built on top of the same CMS. So we had to build it ourselves. PubSweet is now pretty mature and indeed supports multiple publishing platforms – at last count I think that totals 6 platforms built on top of PubSweet (and it is early days).

The Editoria interfaces are the ‘web pages’ that make up the Editoria application. This includes the dashboard, the book builder etc… these are all custom code written on top of PubSweet to realise the Editoria use case / platform.

Wax is an online editor, it is so sophisticated we prefer to call it a web-based word processor. It is actually an Editoria interface, except that it is so complex and such a critical part of any online publishing tool, that it is a product in its own right. Nothing like this existed either, so we built our own (typical WYSIWYG editors are not sophisticated enough for the publishing use case).

PagedJS

From Editoria, you can press a button and have a book appear in the browser. If you print from the browser, you will then get a PDF ready to take to the printers with all the typographical bells and whistles that books need. It looks magical. We currently use Vivliostyle for this but we have hit many limitations with the software. We reached out to the org that produces the code but they weren’t ready to collaborate to improve the product. Hence, unfortunately, we had to build our own. Soon we will replace Vivliostyle with this new offering. The speed of dev has been incredibly fast on paged.js (working title) and so watch this space for some exciting news.. there are some early demos in the pagedjs repo if you are interested.

Summary

…. So, as you can see, Editoria is more complex than it appears to the eye of the user. Also, the crazy thing is that most of these parts should have existed already, but, crazy as it is, they didn’t exist. There has been no full page, open source, editor out there gives the source control and extensability required for the publishing use case. The docx conversion tools available do not yet do a good enough job of the conversion, and are not easy to customise. There is no headless CMS that ‘thinks like a publisher’ and would help us build Editoria; and we had to create a pagination engine (paged.js) because the one open source option couldn’t give us what we needed and we failed at trying to help them improve the product.

Its not like we wanted to build all this, but the sad fact is for the publishing industry the web is still an innovation they haven’t embraced (except as a distribution medium). So while all of these parts (or many of them) should exist, the sad truth is they did not exist. We did a pretty good look around to see if we could save ourselves the effort before embarking on these projects. We did, for example, bank on Substance Lens Writer for our editor, but it was not maintained and proved unstable, and our eventual assessment was that it was better to start again – hence Wax. We also have gone a long way down the path of using and promoting Vivliostyle, until it became apparent that neither the technology nor the technology providers could get us to where we needed to be… consequently, after implementing Vivliostyle (it is still baked into Editoria) we started a new approach to this problem – Pagedjs.

So, we had to build all of this ourselves… the good news is, that all of these moving parts are reusable in other scenarios… so you can take whatever you like from the above, and add it to your own product and hopefully that will fuel the publisher sectors ability to innovate and transform their products and workflows.

Aperta Halted

A part of my personal history is a platform called Aperta which was previously called Tahi. It was a PLoS project and I was hired to design and build it. I quit when the PLoS Board decided to close the repositories, effectively making it a closed project. The repos remained closed, and as far as I know, are still closed today. Ironically after I left, they renamed the project ‘Aperta’ – Italian for ‘open’. A really silly marketing move to reassure everyone that despite what they may have heard, the project was still open…that was perhaps true, albeit (ironically and literally) in name only.

Now, it seems, the platform dev has been halted. Feels good to me. From what I heard (and I didn’t hear much), PLoS didn’t take the project in a good technical direction and generated a significant amount of bad faith and market confusion while trying to develop it behind closed doors.

To quote the new CEO Allison Mudditt (who I respect very much, Coko worked with Alison when she was at UCP):

Part of this initiative will involve changes around the workflow system – Aperta™ – we set out to develop several years ago with the goal to streamline manuscript submission and handling. At the time we began, there was very little available that would create the end-to-end workflow we envisioned as the key to opening research on multiple fronts. But the development process has proved more challenging than expected and as a result, we’ve made the difficult decision to halt development of Aperta. This will enable us to more sharply focus on internal processes that can have more immediate benefit for the communities we serve and the authors who choose to publish with us. The progress made with Aperta will not be wasted effort: we are currently exploring how to best leverage its unique strengths and capabilities to support core PLOS priorities like preprints and innovation in peer review. This will be part of our planning for 2018.

http://blogs.plos.org/plos/2017/12/ceo-letter-to-the-community-mudditt/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+plos%2FBlog+%28Blogs+-+PLOS%29

I hope that PLoS releases the technologies that have been developed for Aperta (there was a lot more than just the submission system) into the open… with both open repositories and open licenses AND, more importantly, an open heart. Collaboration and openness is more to do with how people are than what open license they choose and several of the practices, including asking potential collaborators to sign Non-Disclosure Agreements (NDAs) before getting a demo of the system, were ridiculous and ungenerous.

Having said that, it would be awesome to see all that work released into the open, in open repos with open licenses, and no more blurring of the word ‘open’. Afterall the systems developed that included Tahi were all paid for by researchers. The PLoS Article Processing Charges fuels PLoS and they committed some of this revenue to the development of Tahi. When I was there, no external funding was secured for developing the system. Pedro Mendes made a good point in response to the announcement:

ha

There is some merit to this, but I do applaud PLoS for being adventurous, and if it had worked then the result would have been APCs could be lowered, not just for PLoS, but for any Journal out there reliant on expensive and dysfunctional Manuscript Submission Systems. Allison also notes this in a discussion below the post mentioned above:

…the original idea was that Aperta would allow us to eliminate or speed up the slowest steps between a finished work and its publication in order to reduce the cost of our publishing services

That is true, and it was an admirable goal. However, whatever the journey was between then and now, the project should have always have been out in the open as a public asset. Open for science, open for access, open for source, open for all – and the fact researchers paid for it but it was turned into closed project mid-flight is reprehensible and in the end it worked against PLoS, in particular, it severely weakened PLoS claims to supporting all things open. What a mess.

But, it can’t be ignored that Tahi is about 5 years old now, which is old in software years. A entire generation of technologies that are better suited to solving these problems has arisen in that time. The system is now not much more than a still (just) relevant but outdated approach. That is the risk you take when you develop things behind closed doors. By the time you release it (or don’t in this case), it is out of date. That said, it would still be good to release it, but there are better technologies and approaches out there now.

So I look on with interest to see what will happen next.  I sincerely hope PLoS can return to cutting a path through publishing and exploring and enabling a viable Open Access model that others can follow. With Allison at the helm I am betting things are going to take a much needed turn for the better, not just with this project, but on all counts.

As for me, I learned a lot from designing Aperta (I prefer to call it Tahi). The design process was an introduction to scientific journal publishing for me. I learned a great deal. Tahi gave me, at the time, an unencumbered dream time to imagine something new. It had a lot of interesting innovative approaches and if I had stayed with the project it would have ended up close to where PubSweet is now as I wanted to completely decouple the ‘spaces’ (a concept important to Tahi). It would not have been as good as PubSweet at doing this as a complete ‘decouple’ really has to be imagined from the start, and isn’t as clean if retrofitted. Still, the system would have been a lot more flexible and reusable.

But that wasn’t to be. Don’t get me wrong – I don’t think Tahi was the perfect platform, but it was a pretty good starting point with some significant innovations. At the time, I was looking forward to shaping Tahi with use and to mature it into an excellent system. The good news is, the next platform you design is always better.  I took a lot of what I learned (I have now been involved in instigating around a dozen publishing systems) to my next development, and worked hard to re-conceptualise a new system that avoided some of the mistakes I made with Tahi, and took some of the good parts a whole lot further. That new project is PubSweet and it is looking awesome, and leverages modern technologies and approaches to the max – mainly thanks to the bunch of amazing folks working on it within the Coko team (particularly Jure Triglav) and also now, increasingly, from the collaborators we work with (at this stage mainly eLife, YLD, Hindawi and ThinSlices). Also a huge thanks to the Shuttleworth for backing me, especially because it was at a time (I had just quit PLoS) when it was very much needed. Their backing meant Coko was possible, and consequently, PubSweet and everything else we have done.

Anyways… it was past time PLoS moved on too from Aperta and congratulations to Allison for making the right call, especially given that it would have been a difficult one given the cultural forces at play inside of PLoS.

Newsbot

Tamlyn Rhodes from YLD in London has developed a cool robot for our instance of Mattermost. It is a newsbot. Essentially, anyone in the Coko Townsquare channel can message ‘@newsbot subscribe’ and will be subscribed to the Coko community newsletter.

The newsletter is an automatically prepared email, compiled once a week. The robot looks through all posts that start ‘#weekly’ (we use this for weekly updates) or ‘:cokobot:’ (tags with a cokorobot icon) and adds these items to the weekly image newsletter (bot icon by Julien Taquet)….we are trying it out this week! Cool stuff. You can see Tamlyn in the pic below, center stage in orange. Photo taken last week at the London PubSweet meet.

dsc04103

Getting Design and User Experience Right in Open Source

So, I’ve thought about Open Source and design… I’ve even written some articles on opensource.com (https://opensource.com/users/adam-hyde) about the subject, and created a methodology for bringing the use-case specialist (user) into the center of the process, along with designers and everyone else…

I’ve also brought this subject up a number of times in Shuttleworth Foundation meetings and received some invaluable advice and insights from fellow Fellows and Shuttleworth staff… many of whom have heard my whacky ideas several times over now and are still patiently listening and offering advice! Forever grateful to y’all… especially Helen, Sean, Arthur, and Andrew for ideas and feedback.

But what I didn’t expect, is that I’d be part of a wider community where these ideas could form the basis of the culture. This is what I saw happen this week in London as part of the PubSweet Global meet Coko hosted (& I facilitated).

There were about 25 of us coming together to discuss all things PubSweet with particular emphasis on building Journals. Present were many folks from eLife, Hindawi, Ubiquity, and Coko. We got to the topic of ‘Technical Council’ and I tabled the idea that we need some kind of process in place so that all stakeholders feel they are getting a say in, and are being heard, the future of PubSweet – since it is their technology too.

dsc03891
CoFounder of Coko, Kristen Ratan, kicking off the meeting

When I tabled the idea that we need some form of technical council, Catriona MacCallum, who I used to work with at PLOS, asked the very salient question – and what about the users?

Catriona on left.
Catriona on right.

I’m very grateful to Catriona for that question as it gave me the opportunity to open up the concept and talk about how there are very few communities in open source that treat software development as anything other than just a technical problem, and further that we should take this opportunity to experiment in making this community strong on solving ‘user needs’ and design… it was a great discussion and I’m also grateful to eLife’s head of product – Giuliano Maciocci – for having a strong voice in favor of this and really stepping into, what looks to be, an emergent leadership role with regard to design in the community.

dsc04133
Giuliano.

As a result, we formed a Dev Council and a Design Council. These are oversight/communication groups of 5 people each. So, they don’t ‘govern’ but the choice by the community to form these two groups is a testament to how seriously the community is to making beautiful products that solve real problems in publishing for real people…

dsc04029
Whiteboard from the session showing our decisions. Dev/Design council structure at the bottom, also showing their relationship to community (supporting) and Coko (facilitated by).

All in all, a pretty fantastic 3 days.

1+1 Design Method for Workflow

So, I think I’m formulating something of a design strategy for publishing workflows. I’ll call it, for now, the 1+1 Method. It is not a facilitation methodology like the Cabbage Tree Method, but more of an approach for encapsulating workflows within the simplest and most flexible system possible.

This is very much a work in progress.

The basic idea comes down to this – workflows seem complex when you map them out. There are many forks and eddies and cul de sacs, as well as seemingly ad hoc tasks and unique circumstances. When you map out a workflow like this, you tend to get a complex route with many optional routes determined by a variety of variables and dependencies.

If you build a system that ‘hard codes’ that logic, then you face a number of problems, the most important being:

  1. you tend to create a system that reflects the present with only slight optimisation
  2. the system is prescriptive and inflexible and does not provide a path to future optimisation or, even possibly, radicalisation of workflow

So, how to avoid this? The answer is to design a system that is simple and can be optimised easily. I’m guessing you are thinking this is not really a very helpful answer! True enough… so let me attempt to break this down a little. What we need to do is first understand the organisational workflow, and then design a system that will enable this using the concept of ‘workflow spaces’. We need to create as few of these spaces as possible, and reuse as many of these spaces as possible. Further, we need to move people into these spaces at the moment that something is required of them.

I’ll get better at explaining this over the next months, I’m sure…I’m still working out a language and conceptual framing of this that is easy to communicate and understand….

But let’s look at a simple example… say, a Journal publishing workflow. In this example the workflow looks like this (keeping it relatively simple):

  1. an Author creates a new submission
  2. they fill out the required information and submit
  3. an Editor checks the submission and rejects the submission or assigns a Handler
  4. the Handler checks the submission and invites reviewers
  5. Reviewers accept or reject the invitation
  6. the Handler invites new Reviewers if necessary
  7. Reviewers write reviews and submit
  8. the Handler reviews the reviews and writes a decision to accept (goto (11)), reject (falls out of system), or ask for a revision
  9. the Author reads the decision and changes information if necessary and resubmits
  10. the Handler checks the revision, if all is ok it passes to production, else go to (4).
  11. Production
  12. Publishing

This is relatively a relatively simplified and generic Journal workflow, but you can see some forks in the paths… In my experience, most Journals have the above but with additional organisation-specific forkings and nuances. So the above is very much an ideal simplified state.

So… if you were to develop a system that takes the object, in this case a journal article, through this process, then you have a pretty complex conditional system. For example, there is two-fold circularity at play. First, there may be more than one round of reviews so the system has to be able to cope with this, second more than one round of reviews requires the same amount of decisions. That’s pretty complex already. The mistake most systems make is to try and program this path into the system in a very prescriptive and linear manner – which is difficult to do and, more importantly, hard to change once done.

So… how to get around this? Well… the answer is to use this emerging 1+1 Method I’m trying to articulate here. It is a simple way to encapsulate workflow, leaving the door open for further optimisation and innovation. To achieve this, you must walk an article (or whatever object it is) through all the steps of the workflow, but while doing so keep in mind the following:

  1. for each step, you are only allowed a maximum of 2 spaces. A common Dashboard, and one other space.
  2. push all light, single issue, tasks to the common Dashboard (space 1)
  3. if necessary, push the other tasks for that step to another space (space +1)
  4. reuse as many +1 spaces as possible, merge as many +1 spaces as possible
  5. indicate to each player/role they have something to do through Dashboard status/notifications, providing links to +1 spaces only at the appropriate moment (when they have a reason to use that space)

That’s pretty much it. What we are doing is telling a story to each participant (those that have something they need to do at a certain time), that there is something they need to do now and they have to go here (Dashboard or +1 space) to do it. When they have completed that task, the next participant gets their notification that something needs to be done and they are provided with the means to do it (an action provided on the Dashboard, or access to a +1 space).

So the journey of the article is managed through notifications at the appropriate time to the right people. This means that if we need to optimise the workflow, by either collapsing some steps or adding some steps, we don’t need to redevelop the whole prescriptive, linear workflow logic of the system. We simply need to tweak the order of notifications. Additionally, if we decide that some new innovative process should be introduced, we can do so by either tweaking an existing +1 space, or adding a new one (introducing it to the relevant participant at the relevant time through Dashboard notifications).

That keeps the system intact, and allows us to tweak and optimise/radicalise, as we go.

In the case of the Journal workflow above…. a 1+1 solution might look something like this:

  1. an Author creates a new submission  – Dashboard
  2. they fill out the required information and submit – Submission Page
  3. an Editor checks the submission and rejects the submission or assigns a Handler – Dashboard (notification) + Submission Page (to check the Submission) + Dashboard (to accept or reject an assign Handler)
  4. the Handler checks the submission and invites reviewers – Dashboard (notification) + Submission (to check submission) + Reviewer Assignment Page (to manage Reviewer Assignment)
  5. Reviewers accept or reject the invitation – Dashboard
  6. the Handler invites new Reviewers if necessary – Dashboard + Reviewer Assignment Page
  7. Reviewers write reviews and submit – Dashboard + Review Page (includes submission)
  8. the Handler reviews the reviews and writes a decision to accept (goto (11)), reject (falls out of system), or ask for a revision – Dashboard + Decision Page (includes Submission and Reviews)
  9. the Author reads the decision and changes information if necessary and resubmits – Dashboard + Submission Page
  10. the Handler checks the revision, if all is ok it passes to production, else go to (4) – Dashboard + Submission Page
  11. Production
  12. Publishing

Of course, the above could be simplified still, or you could move some elements around. The reviewer assignment could, in some workflows, occur from the Submission Page. Or, perhaps, the reviews could be placed on the Submission Page…. these decisions are dependent on your organisational needs, what information needs to be shown to who etc…

The point is, the entire workflow above, with all its loops and eddies, has been created with just the following spaces:

  1. Dashboard (primary space for everyone)
  2. Submission Page
  3. Reviewer Assignment Page
  4. Review Page
  5. Decision Page

Just 5 spaces.

Now, the above is still in need of a better explanation. If you want to get a feel for how it works, I suggest you think of a workflow that you are involved in. Simplify it a little, since this is just a first exercise, and then write down all the steps in order. Now… work through this 1+1 method. Start with the Dashboard, rely on notifications to keep everyone in the flow (exposing to them to the right Dashboard action or space at the right time), and conservatively add a new space only when you are sure that:

  1. that action couldn’t easily be handled in the Dashboard
  2. there isn’t another space that could ‘house’ this action sensibly

Walk through it once. You could even do it with Post-it notes representing the spaces if you like. Then sit back, look at it and ask yourself could it be simpler?

Then play with it a little. When you have a good system you could even play with some innovative new spaces – put them into the steps and ask yourself what that space would look like and who should it appear to and when…. you’ll see how easy it is to innovate with a system that enables this kind of workflow….

Needless to say…. that is why we created PubSweet… There might be more technologies like it, but if there are I’m not aware of them (let me know if you know of any). Also, if you have any feedback or thoughts on the above, especially on how to improve the explanation, then please let me know!

Coko and Hindawi

for immediate release…

https://about.hindawi.com/opinion/hindawi-partners-with-the-collaborative-knowledge-foundation-to-develop-open-science-tools/

and

Which means we are partnering with key Open Access publishers eLife and Hindawi…pretty cool!