Mapping Workflows

One of the things I do a lot, is help people think through their workflows and how to optimise them. We do this for all kinds of projects – book workflows, journals, micropubs, pre-print services etc prior to developing a software with them that will help them do what they do better, enable efficiencies, while leaving the door open for continual optimisation of how they work and for possible future transformations of how they work. Drawing from my 10 or so years of facilitation experience I facilitate these processes, and Kristen, drawing from her 20+ years in publishing, is ’embedded’ in the group to be part of the conversation and foster discussion and ideas from ‘within’ the group. We make a pretty good combo.

To do this we must help people ‘think in web-based workflows’. While most journal platforms out there are accessible via the browser they are not what I would call ‘web-based’. They take no advantage of what a networked digital environment can enable. Instead, the web is used as a common way to access what are, primarily, cumbersome database ‘ledger’ systems where workflow is tracked more than it is enabled. Books are even more interesting in that book production staff primarily work directly on the file system with MS Word files. There is little proof the web exists in many publisher’s book production workflows except that they use the network to email collections of files to each other.

So, how do we get people whose primary work environment is anything but the web, to start thinking about how they might work in the web? Well, we essentially follow 3 stages:

  1. document the current workflow
  2. discuss an ‘optimum’ workflow
  3. map this workflow into browser views (we call them ‘collaborative spaces’ or, more commonly, ‘spaces’)

Each of these steps takes several hours. We allow minimum one day for the entire process detailed below. 1.5 – days would make it more comfortable.

Document the Current Workflow

It is important to start the process with all major stakeholders in the same room. Clear out a few hours, or a day, and bring along a lot of paper, post-its, pens and, preferably, some large whiteboards.

Talking through the Journal of Creative Technologies workflow (Auckland, NZ)
Talking through the Journal of Creative Technologies workflow (Auckland, NZ)

It is important to make sure all the major stakeholders are there (or representatives if some groups are large) because when you go through the current workflow you will discover that no one in the room knows the whole story. It is also often true that the organisers of the event have some fundamental assumptions about the workflow that are wrong, and that most people do not understand the knock-on effect of their actions on a co-worker’s job. All this is important to reveal through discussion, while ignoring the claim that ‘we all know what each other does’ – I haven’t ever found this to be the case so far.

This discussion should start right at the very beginning of the workflow. In the case of a Journal Manuscript Submission System, for example, the process usually starts with an author with a manuscript that they wish to submit. That is your starting point. Make sure you really do identify where it all starts as most people will tend to suggest that the early stages are known and understood and simple and consequently they may try to skip this part or deal with it superficially. But your job is to have a discussion that covers the entire workflow and get exact details of what happens, by whom, and when. So keep asking really annoying pedantic questions to get to the clarity you need. Don’t let assumptions leave anything uncovered – I have often found that items everyone assumes are understood reveal hidden, not widely understood, processes when the right, very straightforward, clarifying questions are asked. For this process, you will have to get over your fear of looking stupid! Ask as many pedantic clarifying questions as possible.

Work through the workflow step by step from start to finish. As you do this, document it clearly – a shared large space (eg whiteboard) is best. As you draw it, ask for affirmations that it is correct. Work through all the eddies and conditional forks in the workflow to their conclusion and document them. In the end, make sure you summarise this in one simple document. Theexample below is a document from the Journal of Creative Technologies workflow discussion.


This is a useful process for exposing the actual (vs assumed) workflow to all players, stimulating ideas on how things could be better, and creating a ‘source of truth’ for the current workflow (people often forget details down the road).

While this part of the process is good for raising ideas on how things could be better, don’t let the group get stuck in the future weeds. Planning an optimum workflow comes next and if you let them stray too far into future thinking, they will either not finish documenting the current workflow, or they will confuse how things are now with how they want things to be. But don’t entirely kill these ‘future thinking’ moments either – let them be aired but if it looks like they are going to evolve into deep, nuanced, discussion of one part of the workflow and how it could be improved, then move discussion along. Also, make sure that all people are paying attention through this entire process. Don’t allow side conversations as these take those participants out of the wider discussion as well as distracting everyone else.

Discuss an Optimum Workflow

Next, we lead the group through an open-ended ‘pie in the sky’ discussion about possible utopian workflow futures. We start this discussion very broad – asking anyone to jump in with an idea. The discussion starts broad and we allow it to roam around a little. Often throughout this process, we are taking note of where there is agreement or ‘energy’ around a given idea or approach.

Sometimes people whiteboard ideas but most of the time it is pure discussion.

Discussing workflow possibilities with Collabra Pyschology Jourmal

After a while, perhaps 30 mins, maybe more than an hour, we start bringing back some of the ideas into the discussion that we know the group was interested in and ask them to expand it together. Sometimes we drop in small hints that a particular idea might be a good one but is technically unfeasible – this starts to give the group a sense of what is possible, or what might be technically achievable in the short, medium, or long term. It’s important not to squash ideas as soon as they emerge but to ‘sober them up’ if they get too fantastical. A little bit of realism doesn’t hurt at this stage but, more importantly, you want to get the ideas flowing. Nothing is nailed down or committed to at this point, it’s just exploration and discussion.


Now we get to mapping these ideas onto a web-based platform. The group is going to engage in systems design without knowing they are doing so. We do this by first explaining a little about the web – some of the things that are possible. We might show some examples of sophisticated web-based editors like Wax, or in-browser pagination using Vivliostyle. Perhaps we might look at Trello/wekan or kanban examples of workflow management. We might look at Stencil.a or any number of examples of interesting platforms and approaches the web can offer.

We also talk about the efficiences of the web and how that is enabled – mostly this comes down to unpacking why a single-sourced content environment is so powerful, why ‘everyone working on the same object’ is important, how collaboration and concurrency can change the way we do things. As well as preparing them for the idea of designing a new system that is native to the web, we are introducing some new ideas that might reshape how they think about how they work. Why, for example, do we email MS Word files around when we can simply all see the same document, at the same time, in the browser?…these ideas will mean different things to different groups and how they play with them depends on how ready they are to explore new possibilities, combined with how much of their ‘old ways’ they are prepared to let go, plus how much they understand about the web. So it is important to let this conversation go at its own pace across whatever topics the group feels they need to explore. It is important that the discussion evolves of its own accord from the ideas introduced. We ‘shape’ the conversation, point it in a certain direction and signpost with certain interesting examples, rather than provide a soliloquy or monologue about how we would like things to be done. This is important, because unless the group internalises these ideas, they won’t end up exploring them when it comes to design time.

Next, we give a little rundown about ‘thinking in collaborative spaces’. Basically what we are trying to do is to take some of the ideas that emerged in the ‘optimal workflow’ discussion and start prompting them to think how this might be realised ‘inside the browser’. Most commonly we can do this by talking about very concrete things like browser windows, which everyone knows, and asking ‘how could this part of the workflow happen in a browser window?’ For example, in a journal system, we might ask ‘how might submission happen for an author in a browser window’? As a starter example, we could discuss this and draw a box on a whiteboard and draw some basic ‘UI’ elements in that box. This makes everything very very concrete. You are reducing systems design to something they understand without them realising that they are about to start designing a system.

The next step can take several directions, but the most common is that we split the group into smaller groups. 4 or 5 to a group is usually a good number. We then give them large sheets of paper and a bunch of thick pens, and send them to acoustically separated spaces to draw up proposals for ‘several spaces’ that would represent the workflow in the browser. The time we give them depends on the time at hand. You want more than 15 mins, but 45 is probably too long. Sometimes we might also ask them just to focus one one part of the workflow, usually the first part (however you design this is up to you).

Erich van Rijn (UCPress) pitching some ideas to the group while Dan Morgan (Collabra) documents, and Paul Shannon (eLife), and Yannis Barlas (Coko) listen.

It is very important when you send them away to do this that you ask them to draw the interfaces. One big page of paper, for example, per ‘space’ (browser window). If you don’t do this some groups, especially those used to writing requirements docs, will write lists. The problem with that is that these lists don’t tend to be scoped or connected very well. They just turn into a list with as many items relevant to (for example) ‘submission’ as you can imagine. Better is if they draw the spaces as if it was a browser window, putting in UI elements with basic drawing.

Oliver Buchtala ( holding up his group’s proposal at Erudit, Montreal.

No one has to be an artist, it just needs to be a rough proposal of what that space would enable, and one most people will understand if they were just to look at it.

John Chodacki (CDL) puts some finishing touches on a proposal he designed together with Andrew Smeall (Hindawi), and Jure Triglav (Coko).

We circle around and make sure everyone is getting on with the task and in good time. At the end, we bring everyone back into the room and ask them to present their proposals to the rest of the group.

Lia Schiff (CDL) presenting a proposal to the rest of the group.

We give them a short amount of time to make the presentations. Perhaps 10 mins, more if the scope of the workflow is larger.


What you will be surprised at, I can guarantee it, is that all the proposals are *great* AND they will all be very similar. We have done this many times and the results of each group are rarely very far apart. When they are far apart, it’s usually because a group has locked onto one idea or another as a starting point and designed everything around that. These ideas usually have something very worthwhile that can be combined with the other proposals to improve them.

As each group finishes presenting a space or group of spaces we ask the group for questions or comments. We don’t let this go too deep into the weeds as time is tight at this point, but some discussion is always useful.

Another proposal at the Erudit workshop.

Then at the end, you discuss these approaches and agree on a common approach. We call this an ‘architecture’ which it is to an extent. It is the bird’s-eye view of how their new workflow will fit snugly into a browser based workflow. The following being an early ‘architecture’ or high-level systems view of a journal platform for Collabra.


Handily each of these new spaces can then be built as a PubSweet component! The following is the architecture showing the components we need to design and build – this diagram was the result from early sessions with the staff from UCP I facilitated to produce the Editoria monograph production platform.


This process is a very specific description of the first part of the Cabbage Tree Method. The second part of the Cabbage Tree Method involves facilitating the design of each of these components.

The Road to PubSweet 1.0

We are pretty close to our PubSweet 1.0 with the RFC now out for PubSweet 2.0, and a PubSweet dev site release next week.


It has been an amazing effort, particularly by Jure Triglav, the lead dev for PubSweet at Coko, but also fantastic work from Richard Smith-Unna, Alf Eaton, Yannis Barlas, and Christos Kokosias. Also more recently some great contribution from Alex Georgantas.


So, we are pretty much there and I’m presenting in San Francisco this week as part of a small Coko event to reflect on the future of the framework and discuss the RFC. For this purpose I’d thought I’d write a post to help me think through the thinking that got us here.

So…the thinking behind PubSweet started when I came back from Antarctica around 2007 or so (I was there setting up an autonomous base for artist-scientist collaborations).


I decided I wanted to give up the art world and try something new. The something new turned out to be FLOSS Manuals – a community writing free manuals about free software. I started it when I was living in Amsterdam somewhere around 2007. In order to execute on this mission I needed to get a couple of things sorted. Namely, learn how to build community, work out processes for rapid book production, and work out the tooling.

The tooling started with me scratching around with TWiki. A wiki written in Perl that happened to have the best plugins for rendering PDF. I scratched around, writing some Perl and cutting and pasting a whole lot more, and added some crazy .htaccess URL rewriting to produce a basic system for producing books. It was pretty scratchy but it actually worked. Later a buddy helped extend the system and later still I was able to pay him and others to extend it.

At the time it pretty much comprised a page (per book) for creating a table of contents.


and an interface to edit the content (chapters). I ripped out the native wiki markup editor and replaced it with a WYSIWYG editor, I think it was TinyMCE…


As you can see Right-to-Left content (in this case, Farsi) was also supported. There were also some basic things in place for keeping track of the status of a chapter, the version number, side by side diffs, side by side translation interfaces, and, later, dynamic table of contents organisation and edit locks.

Coupled with some basic PDF rendering stuff and a way to push the content from the ‘draft’ to the publishing front end and we were away.


It actually had some other pretty cool stuff, such as side by side translation interfaces…



..a built in live chat for talking with collaborators…


and even a way to send books between different instances (eg for sending a book from the FLOSS Manuals French community site to FLOSS Manuals Finnish for translation)….

We could even render book formatted PDF and push to the print on demand services. I just now checked and some of the books are still there!

Not bad for a Perl-based system, built on top of a wiki that wasn’t supposed to do this kind of thing, and built very with very few resources. The TWiki extensions were contributed back upstream to the TWiki repo and it was all open source but it was pretty hard to rebuild and no one I knew actually had a similar use case.

After this, I embarked on a journey to replace the system with a custom built solution specifically for book production. I can’t remember exactly when this started, maybe 2008 or 2009 or something. It was originally called Booki…


…which later became Booktype. Booki (and later Booktype) replaced the FLOSS Manuals tooling, although you can still see the working old tool here. That ole Perl code is still functional with no maintenance after 10 years, I can hardly believe it. The docs on how to use it also still exist.

Booki was built with Django (python) and pretty much had all the same stuff. Although the look and feel was changed quite a bit in the transition. There aren’t too many images around of Booki although I did find these screenshots of Booki taken by someone using it on the OLPC XO! (FLOSS Manuals did all the docs for OLPC/Sugar OS etc).



It was hard to get financial support for it. Internet Archive gave us $25,000 at the time which seemed like a fortune. The evolution of Booki to Booktype represented me taking the project to a buddy’s in Berlin (I was living there at the time) based org (Sourcefabric) and parking it there so I could get more resources to build it out.

Booki/Booktype pretty much had, and has, the same stuff as the FLOSS Manuals system, just purpose built. So it had, a table of contents manager


And a book (chapter) editor…




And the other stuff. Perhaps the only new features (compared to the FLOSS Manuals system) were a dashboard…




and an interesting way to have Twitter-like messaging to pass snippets from chapters to other users.


Before I left Sourcefabric I wanted to get some other innovations built but didn’t get there. I did build some prototypes though. There was a task editor…


…and live in-browser book design…


Booktype is still going strong, now it is its own company (based in Berlin) and they also run the Omnibook commercial service using the software.

I left because John Chodacki and Kristen Ratan from PLoS invited me to come work for PLoS to design a new web-based journal submission system. I agreed…

But, before I leave the book story behind for a bit..I had set up Book Sprints as a company and put a small amount of my own money into building two new book production systems somewhere between leaving Sourcefabric and starting at PLoS. These two systems were PHP-based and Juan Gutierrez built them over some months.


I wanted to do this because I was a little frustrated by Booktype not moving forward and also the platform was becoming more difficult to use. We were using it for Book Sprints but after I left the product took a new UI direction and I was finding Book Sprints participants were not enjoying using the system. So I built a Book Sprints specific system called… PubSweet… the namesake of the current Coko system which has eventually turned into something of a prototype for the new PubSweet… this new system was a lot simpler and easier to use than Booktype. It was initially meant to be modular but I think we lost that somewhere along the way. Cleanly modular systems take a lot of extra effort and time to produce so we gave in for speed of development’s sake.

The old PubSweet had a dashboard….


..table of contents manager…


and editor. Just like before!


We also introduced some new innovations including visualisations of the book production process…


Plus annotation (using Nick Stennings annotator software)…



and other stuff…I think threaded discussions, outline views, review page, an in-browser book renderer, book stats and I can’t remember what.

Anyway …I also built a platform on top of this old PubSweet for the United Nations Development Project. It was called Lexicon. Lexicon was pretty interesting as it opened my mind for the first time to the idea that an editor is not an editor is not an editor. Different content types (in a book) may require different editors or production environments.

Lexicon was produced to collaboratively produce a tri-lingual (Arabic, French, English) lexicon of electoral terms for distribution in Arabic regions.


Lexicon had all the same stuff as the old PubSweet but with one major innovation, you could create chapters that were WYSIWYG based, or you could create a chapter which enabled you to add and sort individual terms and provide translations.


It was a pretty interesting idea and we were able to make a really cool book which the UNDP printed and distributed across many Arabic-speaking countries. I still have the book on my bookshelf.

The other interesting thing was that the total cost for building this on top of the old PubSweet was $10,000 USD. This was mostly because we could leverage all the existing stuff and just build the difference…interesting idea!

Ok, so then I dropped book production systems around 2013 or so for a while and went to work for PLoS on a system that was called Tahi and then became Aperta. The name Tahi came from the name of the street I was living on in New Zealand before I had a US work visa and was designing the system – Reotahi Road (cool road). Reotahi means ‘one voice’ and ‘tahi’ means ‘one’ in Maori. It was built on Rails with Ember. Essentially the front end and backend were decoupled although it was really pushing the technology at the time to do this. I designed the system and moved to San Francisco to manage the team to build it.

Tahi (Aperta) had a dashboard (surprise!) and editor, just like the book production systems but I introduced two major innovations – Cards, and card-based workflow management interfaces. Unfortunately, while I was asked to come and build an open source system, things went a little weird at PLoS and they closed the repos, effectively making it a closed platform. So I quit. That also means I don’t have any screenshots to show you. Pity. If you sign an NDA with PLoS I believe they might show it to you.

However, you can picture it a little – imagine something like Trello, or Wekan – these are card based kanban systems. But imagine if you could custom make cards to do anything. Effectively cards were first class citizens of the platform and could access the db, perform system operations, make external calls, do validations, whatever you wanted. In hindsight, I think they were as close to an idea of an ‘app’ that you could have in a browser platform, although that wasn’t the way I thought about them at the time. Additionally, cards were imported into the system since each card was actually a gem file. This meant any publisher could custom make their own cards to do whatever they wanted and place them within the kanban-like workflow space (task manager). Pretty neat.

So, cards could be surfaced and used anywhere in the system. We used them for authors to enter submission data, but also for production staff to perform operations, for reviewers etc etc etc. They could also be placed on a kanban board to make a workflow. Cards could be moved around the workflow and deleted or new ones added at any time.

To manage all this my other idea was to let these cards flow through a TweetDeck-like interface. So you could sort cards, per role, per user, at volume.

Tahi essentially had four spaces – a dashboard, a submission page (which displayed the manuscript in an editor, and submission data could be entered through cards), a task manager (workflow for the article, using cards as tasks), and a ‘flow manager’ (the TweetDeck-like interface for sorting all your cards across all your articles). While the FLOSS Manuals, Booki and Booktype platforms were pretty much monolithic systems, the old PubSweet was sort of modular. However, Tahi did decouple the front end and back end but I wanted to also break these four spaces into discreet components. That would have given the system enormous flexibility but unfortunately I wasn’t able to do this before I left.

Anyways, Tahi/Aperta is a little old now but it was pretty cool. I don’t know what happened to Aperta but I believe it is now being used for PLoS Biology.

After I left PLoS I was offered a Fellowship by the Shuttleworth Foundation to continue on the mission to reform publishing. So I started Coko with Kristen Ratan (who was the publisher at PLoS)….


So there are some themes from building the past 7 or 8 publishing systems (depending on how you count it… there were also some other interesting experiments in between). First, the next system you build is always better. That is for sure. It’s an important thing to realise because when I developed the FLOSS Manuals system I thought that was it. Nothing could be better! But I was wrong. Then Booki/Booktype and I felt the same thing. I was so proud of it and nothing could be better! haha… you get the picture. The reason why it’s important to understand this is because I think it gives someone like me a bit of freedom. I can take some risks with systems knowing you get some stuff right, you get some stuff wrong. But the next system will get that bit you got wrong, right. Taking this attitude also takes the pressure off and you can have more fun which is good for your health, the team you are working with, and the system.

As far as technical lessons learned… well… after looking back at all these systems when we started Coko, I realised that the idea of independent ‘spaces’ for publishing workflows had a heap of currency. How many systems did I have to build with baked in dashboards, task managers, editors, table of content managers, etc etc etc before I could realise it doesn’t make sense to do this over and over. I wanted to take the idea of these kinds of spaces forward and not have to build them again and again… so some kind of system where you could include whatever spaces/components you wanted would be ideal… This would have two very important side benefits:

  1. I could learn so much because if the next system you build is always better, what about a framework that would allow you to easily build a whole lot of systems at once! Or build a lot quickly over a short amount of time… just imagine how much you could learn…
  2. It would open the door for others to innovate. I have since given up the idea that my system (so to speak) was the best ever and no one could top it. That’s just the testosterone talking. I’m kinda over it (sorta). I want other people to be able to make better stuff than what I have produced so far, to bring in innovations I never thought of. I want to make that easy for them and now I understand a whole lot better how publishing workflows actually work I’m in a very good position to do that.

That was a lot of the thinking behind the new PubSweet – PubSweet 1.0. But there is some other stuff too. Through my time at PLoS, I came to understand just how many variables affect workflow choices in journal publishing and that each publisher has slightly different conditions and roles that affect this. That means that the access control is complex. We might think there are various roles – author, editor, reviewer etc that shepherd an article through a process but it’s not that simple. Any number of conditions can affect who gets to see or do what and when. So we need to have a very sophisticated way to set and manage this.

There was a lot of other stuff to take into account to but I mention these two specifically because recently when I was talking to Jure (lead PubSweet dev) about PubSweet 1.0 and reflecting on how far we came he nailed it, he identified the two major innovations of the system being:

  1. reusable/sharable components (spaces)
  2. attribute-based access control

I agree entirely. I think I might add another:

  • developer experience

It is pretty easy, and getting easier, for developers to develop publishing platforms/workflows (call them what you will) with PubSweet. I think it is pretty astonishing and I think these 3 characteristics put together enable us to build multiple publishing systems fast and in parallel (with small teams) as well opening the door for other to do the same and huge opportunities for innovation.

If we are successful at building community this will be a huge contribution to the publishing sector.

In a future post, I’ll break PubSweet spaces / components down in more detail. There were also a lot of other similar stories regarding technical innovations on the way (eg Objavi->iHat->INK), but I’ll break them down into posts on another day.

I meant to also talk about Editoria here, the monograph production system built on top of PubSweet, and xpub – the PubSweet-based Journal system.


They are both pretty amazing and leverage so much more than the previous systems identified above.

Login page for our first Journal platform.

I think the main thing with them is that we are working extremely closely with publishers using the method I developed – the Cabbage Tree Method.

Editoria Design Session

This means that I am no longer involved in building, what I would call, naive publishing systems. Naive in the sense that publishers could use, for example, Booktype, but it’s not really built for publishers. It’s a general book production system built by someone who didn’t know much about publishing at the time. That’s great of course, there is a place for it. However, Editoria is not a naive system. It is designed by publishers for publishers and the difference is enormous.

But I will leave a longer rant about this for another post.

I do however, want to say that I didn’t, of course, build any of the above systems by myself. There were many people involved and I have credited them elsewhere in this blog. I’m not going to do another roll call here except for Jure Triglav.

Jure and I sat down just over 18 months ago to discuss some of the lessons I learned as explained above. We jammed it out over post-its, whiteboards, coffee, and food in Slovenia and you can read a little more about that process in the PubSweet 2.0 RFC. But Jure trusted me, and I trusted him, and he took these ideas and, with a small team in very good speed, made them a reality. As a result, I think PubSweet is an exciting system and will only get better. Congratulations Jure, you deserve special thanks and recognition for the absolutely amazing job you have done.


Coko’s Efforts getting Out ‘n About

It seems the Coko meme is getting out there. Just these last few weeks we have seen the following: – we put some money into assisting ‘, including flying the founder Nokome over to San Francisco and introducing him to funders. The result is that Nokome just landed a very nice grant from the Sloan Foundation. We don’t lay any claim to Nokome’s great work, but it’s good to see that our help was instrumental in helping this great project along the way.

HTML Typescript – Recently (last week) Wendell Piez, who we work with for Docx to HTML file conversion, presented at the well known JATSCon about his work with Coko. The preliminary proceedings are available online for Wendell’s talk and I hear the video will be available soon too.

Texture – if one slot at JATSCon wasn’t enough… Texture was also presented. It is an online JATS editor produced by the Substance Consortium which we co-founded. Proceedings here.

Penguicon, Carnegie Mellon – I recently presented Coko at these two events and upcoming we will be presenting Coko and various projects at SSP, AUP, Open Source Lisbon, Wikicite, OSEHRA, Open Source Albania and more!

March for Science – Coko CoFounder Kristen Ratan was lead organiser for the San Francisco March for Science. I couldn’t be there (was in the desert) but it apparently was a wonderful event.

Open Source Alliance for Open Science – we have 28 open source projects for open science coming to join in a day working out how to work together. Starts tomorrow in Portland!

Lots happening 🙂