Anxious

I’m so anxious about the quakes in NZ. Some channels to follow:

https://www.geonet.org.nz/quakes/felt

serious_earthquakes_14-15nov2016

https://twitter.com/search?f=tweets&vertical=news&q=%23eqnz

http://www.radionz.co.nz/

A good friend and Coko collaborator Nokome Bentley is there in Kaikoura with his family. Kaikoura was worst struck. All wishes that he and his family are ok.

UPDATE: Since heard Nokome and his family are ok! 🙂

 

Background map data copyright OpenStreetMapc ontributors, CC-BY-SA 2.0. GeoNet content is copyright GNS Science and is licensed under a Creative Commons Attribution 3.0 New Zealand License

On creating a methodology

The first 'real' Book Sprint. Inkscape, Paris, 2008.
The first ‘real’ Book Sprint. Inkscape, Paris, 2008.

Some years ago, I developed the Book Sprints methodology. It’s now pretty well defined and is being used very successfully to develop many books for all types of organisations. I am now working on another methodology – Collaborative Product Development. So, this time round, it’s sooo much easier and I thought perhaps I should write down some of the journey from zero to methodology. It will certainly help me to do so and maybe someone else out there will find it useful.

First of all, methodologies are odd things. Book Sprints is a methodology but, at the same time, I often wonder exactly what this means? I mean…you can’t just take a written template and turn it into a successful facilitated process. It just doesn’t work. The reason is because facilitation is a little, if you pardon the strange analogy, like administering the law. The law exists on paper but every time it is administered by a judge it is new. The context requires the law to be interpreted to enable it to be applied.

I think facilitation is like this. You can take a method and apply it but the context is everything. You must always make the application of the method new each time it is applied. Further, you need to extend it and break it when necessary and you have to do it effectively, other wise you break the process and, in the facilitation game, process is everything.

So…you must first be a facilitator to apply a facilitation methodology. Even then you can still screw it up. For example Book Sprints, and Collaborative Product Design, employ all the same skills that you need for ‘unconference’ facilitation plus more. The more is specifically the skills required to drive people effectively along a finite timeline to completion of the thing you are making. The need to drive product in this manner often freaks unconference purists out because you wield a lot of ‘do it now’ power else you will never get there. ‘Do it now’ power is only subtly used in unconferences, in product-driven facilitation you use it like a hammer.

Anyways… the point being that a methodology on paper is nothing. You need to define it, so out comes the pen and paper.

Defining a methodology is actually pretty strange and, in my experience, the newer the ideas in the methodology are to you, the longer it takes to define. Book Sprints, for example, took me about 4 years to work out and I only really knew it had a definition when I had to train someone else. CPD, on the other hand, took me a day to outline on paper. I’m still refining it but that’s normal, the body of CPD came into existence in one day-long session. What’s more, it appears to be working. The bits that are taking the longest to work out are the bits I haven’t experienced before. We haven’t taken a full product to market with CPD, so, consequently, the production implementation phase still needs to be tested and refined. The rest of it, however, seems pretty well constructed.

It took a long time for me to work out a method for Book Sprints. I was really just trying stuff completely blind. It didn’t start that way, as it happens. I first went to people in the publishing industry and asked them how I should do it. I figured they knew how to make books and surely they would have some ideas on how to make it faster. Hah. Was I ever wrong. My experience was that publishing people screwed up the process with their own process. Too much process. So I had to, finally, after a long period of angst, decide I was going to have to work this thing out myself. I didn’t want to do it. I didn’t even know what it was I was trying to do. So, sigh, I relented and proceeded to slug it out.

It was like slugging it out too. I mean, I had no experience in making books. I didn’t want to do anything I considered ‘publishing’. I have an aversion to cathedrals – which is what publishers looked like to me. A deeply institutionalised sect. So I had to actually learn to trust myself and do it my way. It was quite scary at first. I mean, I was inviting a bunch of people to come to some place to work for a week to develop a book using a methodology that actually didn’t exist. I didn’t have much idea what it might look like either. Geez… I must actually be pretty stubborn when I look back on it.

So I went about inviting people to Book Sprints and I challenged myself to trust myself. To listen to myself and observe. I came to understand that the only thing that could enable or prevent a successful Book Sprint was me. There was no other factor. That was pretty frightening at times. I spent many nights during Book Sprints completely terrified that I was about to send this nice group of people into an unproductive deep abyss. They might return from that dark hole but I was pretty sure I wouldn’t.

On this journey, however I had some starting points. I don’t know what triggered it but very early I came to understand that rapid book development of the kind that Book Sprints represents could not be about publishing processes – it was about unconference processes. Luckily I had a mentor, who had, for reasons unknown to me, taken me under his wing. At the time I didn’t know how lucky I was. That was Allen Gunn (Gunner). I had seen him in action many times as a facilitator and I had a few clues to a process. It wasn’t much but it was a start.

So I tried my hand at facilitation. I would, at the time, tell myself that Book Sprints were more about facilitation of people than the production of a book. To say so out loud sounded odd. It didn’t sound true. It sounded like I just made it up. But I kept saying it and trusting it. I kept challenging myself to trust myself. It wasn’t easy but as I became used to it, the process became easier.

Anyway, I still couldn’t call this thing a methodology. All I knew was that I was trying things out and formulating some kind of Book Sprints worldview. I think it’s fair to say the worldview was nothing I could articulate. I worked through gut and felt that everything was instinctual. However, I was learning useful tricks. For example, I remember Gunner saying he tried something new each time he facilitated an event. So I tried something new each time. I learned that if you failed with the something new, the trick was to make out like it was a success and, further, an expected success. This way I created a fallback framework that would allow me to experiment, to try and fail, to learn on the job and to get away with it.

So, I iterated forward. Rather slowly really. I mean, I had to convince people that this non-existent Book Sprint thing was not only possible, but they should do it, while I had little to point to as proof. Yeesh. Talk about a hard sell. But some people saw the possibilities and tried it out. I am forever grateful to Leslie Hawthorn who was working at Google Open Source Programs Office at the time. Leslie (introduced to me by Gunner) backed me and sponsored the first 3 Book Sprints. Amazing. I got something I could start with and that was critical. Later, with a few under my belt, it became easier to convince others.

So I did this for many years and felt I something taking shape. It was actually working. The books were getting better which seemed to indicate I was improving the process. I was starting to construct an embedded framework and the things I did were tested against it. It was filling out and becoming more consistent. I could finally start becoming a little more playful with the process. Still, when I started to talk about it as a method it was a query more than a statement. I wasn’t even sure what a methodology was. I wasn’t sure if Book Sprints was a method or even a stable set of practices. Others also said they same thing. David Berry and Michael Dieter, two good friends and critical theorists, would ask me what I thought it was. I had no idea how to articulate it. I asserted at time that it was a methodology but I wasn’t even sure I convinced myself.

A critical break through came when David and Michael (who had both been to 2 Book Sprints by this time) wrote a post about the process:
http://ausserhofer.net/2013/02/the-method-of-book-sprints/

I was amazed. They saw things that I didn’t. It was stunning to me. They had given me my own framework. From this point I started to construct a much better understanding of what I was doing.

The next step that finally defined the process was when I actually had to train someone to do it. I had other work offered to me (designing platforms) and I had a Book Sprints client base to keep happy. So I needed to scale. I needed me times 2. So I brought in Barbara Rühling (as introduced to me by Zara Rahman). It was at this moment, during a Book Sprint in Malta, and Barbaras first training session, that I started to make up a language to explain what a Book Sprint was. It flowed out of me like I knew it all along. I was very lucky in that Barbara trusted my narrative. It allowed me to expand notions as they came out of my mouth. I could define and refine a language and process as I spoke. It was fantastic and it was the final moment in the long process of defining a method. Hard won experiences were suddenly made salient and coherent though in situ explanation.

Now when I look at Book Sprints it appears pretty concrete. I laugh when people talk about it as if its a ‘known thing’. How can it be? I just made it up! 🙂 Of course, I didn’t just make it up. It is the result of a lot of trial and error and observation, risk, stress, unbelievable elation when things worked out, and the inability to give up. I still don’t know why I didn’t give up. I like to think if I did it again I’d give up. It might be a better way to live my life 😉

Some publishing systems I have developed

developed_floss

I recently went over some publishing systems with Yannis and Christos from Coko. We looked at various systems and discussed them. As we did this, I realised that I’d actually built quite a few! Although we weren’t just talking about those I had built, I began to think through what I had done right and wrong when building those earlier systems. Each development is a learning process and you will always get things both wrong and right at the same time. The trick is to get less wrong the next time round…

In the ten years that I have been building these systems, I have worked with some pretty talented people, including Luka Frelih, Douglas Bagnall, Alexander Erkalovic, Johannes Wilm, Remko Siemerink, Juan Gutierrez, Lotte Meijer, Fred Chasen, Michael Aufreiter, Oliver Buchtala, Nokome Bentley, Andi Plantenberg, Mike Mazur, Rizwan Reza, Gina Winkler and many others including the entire team of Coko – the most talented bunch of them all. Coko team:

Kristen Ratan – CoFounder, San Francisco
Jure Triglav – Lead PubSweet Developer, Slovenia
Richard Smith-Unna – PubSweet Developer, Kenya
Yannis Barlas – PubSweet Frontend Developer, Athens
Christos Kokosias – PubSweet Frontend Developer, Athens
Charlie Ablett – INK Lead Developer, New Zealand
Wendell Piez – XSL-pro, East Coast USA
Julien Taquet – UX-pro, France
Henrik van Leeuwen – Designer, Netherlands
Kresten van Leeuwen – Designer, Netherlands
Juan Guteirez, Sys Admin, Nicaragua

Alex Theg – Process, San Francisco

All systems except, unfortunately, one (see below) are open source.

FLOSS Manuals

top_read

The first publishing system I designed and built didn’t have a name really. It was the glue behind the FLOSS Manuals English site. FLOSS Manuals was, and is still, a community focussed on building free manuals about free software. I started the development in English but the system needed to be useful to a number of different language communities, of which Farsi was the most interesting. Today, only the French and English communities are still operational.

I built the FLOSS Manuals system in Amsterdam in about 2006. It was based on TWiki, an open-source Perl-based wiki. I chose TWiki because back then it was the only wiki around that had good PDF-generation support – I think this came from some of its plugins. TWiki had a good plugin and template system and I came to think of it as a rapid prototyping application – it was pretty malleable if you knew how. I was a crap programmer so I cut and pasted my way to a system that became usefully functional.

developed_twiki

I leveraged the account creation and permission systems of TWiki, and ripped out the wiki markup editing and replaced it with an HTML WYSIWYG editor (I think it was TinyMCE). So in wiki world terms I had committed something of a crime. Throwing out wiki markup at the time was unheard of in the post-2004 euphoria of Wikipedia. But JavaScript WYSIWYG editors were coming along just fine, and I thought wiki markup no longer made any sense (in fact it had been invented to make making HTML easier than writing raw HTML). But y’know… wiki markup was no easier than using a WYSIWYG editor, despite the sacred status of Wiki markup in the Wikimedia community. And despite my heresy,the Wikimedia did give me a barnstar for my efforts, which was nice of them:)

developed_star

After I reached the limit of my coding skills, a friend, Aco (Alexander Erkalovic), helped build the next bits. I found some money to pay him and that is when things started to move forward. I can’t remember all parts of the system, although it’s still in up and running for some FLOSS Manuals language sites. The core of the system was the manual overview page. This contained a list of all chapters in a manual. You could also set the status, add new chapters, view overall progress etc. from this one page. We had a separate mechanism for creating an ordered table of contents (index) for a manual.

developed_fmindex

Index builder, essentially a dynamic drag and drop mechanism for arranging chapters

developed_fm_overview

The overview page

Essentially, you added a chapter on the overview page and then opened the index builder and arranged the chapters. When saved, the (refreshed) overview page displayed the correct (new) order of chapters. We had to do it like this because back then, in the era of the ‘page refresh,’ it wasn’t possible to synchronise dynamic elements across multiple clients. So we couldn’t have one ‘shared’ index builder that would dynamically update all user sessions simultaneously. Nevertheless, it worked pretty well.

From the overview page, you could choose a chapter to edit. When you did so, you locked the chapter and, through some JS trickery Aco cooked up, we wereable to do this across multiple clients so everyone could see in real time who was editing what.

developed_blog_chapterlist

When editing a chapter, you could save the document and then, when ready, publish it. This way you could have an ‘in progress’ state of a chapter and a published state. At publish time the chapter was copied across to the system’s web delivery cache.

We also added chapter status markers, as you can see from the above image. It was pretty basic but nifty.

Next I hacked in a live chat, initially using IRC (freenode) for a global FLOSS Manuals-wide chat:

developed_irc

developed_irc2-irc-en

Then I hacked a fancy AJAX script into the chapter edit interface so each manual could have its own chat with the interface present while you were editing a chapter. It also looked a little nicer than the IRC channel. From the beginning, I tried to make everything look like it was meant to be there, even if it was a fiddly hack.

developed_adamwindow

FLOSS Manuals also had a lot of other interesting goodies. We had federated content, for example. This was established so one language site could import an entire manual from another and set up a translation workflow.

developed_xchange

We also had a simple side-by-side editor set up for translation that worked pretty well for translators.

developed_fm_metaphorIn addition, we had a remix system for generating new versions using mixed content from other manuals. This was useful for workshops and for making personal manuals, for example.

develped_remix_dropdown

developed_renixed2

One of the cool things about the remix is that you could output in many formats such as PDF and zipped HTML, add your own styles through the interface, PLUS you could embed the remix in your own website 🙂 The embed worked similarly to methods used today to embed YouTube or Vimeo videos in websites (by cut and pasting a simple snippet). The page delivery was ‘live,’ so any updates to the original manual were also displayed in the embed. I thought that was pretty cool. No one used it of course 🙂

developed_remix3

The system also had diffs so you could compare two versions of a chapter. It was good for seeing what had been done and by whom. In addition, it was possible to translate the user interface of the entire system to any language using PO files and a translation interface we custom built:
developed_translateBut by far the most interesting thing for me was building FLOSS Manuals to support Farsi. It was interesting because, at the time, no PDF-renderer I could find would do right-to-left rendering. Behdad Esfahbod advised me to just use the browser to do it. Leslie Hawthorn from Google Open Source Programs Office introduced me to him after I went on a naive search in the free software world for ‘someone that knew something about RTL in PDF’. I was very lucky. The guy is a generous genius. He just suggested an approach a new way to think about it and later I worked with Douglas Bagnall (see below) to work out how to do it. The basic idea being that HTML supported RTL and the PDF print engines rendered it nicely…so…it was my first realisation that the browser could be a typesetting engine.

Implementing RTL in the FLOSS Manuals system was so very tricky, and I was unfortunately on my own for working out Farsi regex .htaccess redirects and other mind-numbing stuff. Just trying to think in right-to-left for normal text did my head in pretty fast, but somehow I got it working.

developed_fm_farsi

Outputs of all language books were PDF and HTML, later also EPUB. I initially used HTMLDoc for HTML-to-PDF conversion. It was pretty good but didn’t really think like a book renderer needs to. This was my first introduction to the overly long wait for a good HTML-to-PDF typesetting engine. Later I found some money and employed Luka Frelih and then Douglas Bagnall to make a rendering engine separate from the FLOSS Manuals site (see Objavi below). Inspired by my chats with Behdad (above) Douglas introduced some clever PDF tricks with the Webkit browser engine to get book formatted PDF from HTML. I can’t remember exactly how he did it but essentially he used xvfb frame buffer to run a ‘headless browser’. In short, and for those that don’t know those terms, he came up with a very clever way to run a browser on a server to render PDF. It did it by rendering HTML to PDF in pages (using the browser PDF renderer) and then rendered another set of slightly larger blank PDF with page numbers etc and embedded one within the other. Wizard. Hence we were able to make PDF books from HTML. It also supported RTL 🙂 I think I need to say here that this whole process was immensely innovative and we did it on a dime. Also, because we refused to use proprietary systems we were forced to innovate. That was a very good thing and I welcomed the constraint and the challenge.

Later we also used WKHTMLTOPDF to make PDF. It worked and we worked with that for a long time. I even tried to start a WKHTMLTOPDF consortium with Jacob Trulson, the founder of the project, it got some way but not far enough (I am very happy WKHTMLTOPDF is still going strong!).

I played a lot with Pisa and Reportlab for generating PDF and finally cracked it when I started book.js (more on this below). Actually, when it comes down to it I think I used everything that wasn’t proprietary to make book formatted PDF from HTML. It was the start of a long love affair with the approach. I promoted this approach for a long time, even calling for a consortium to be formed to assist the approach:
http://toc.oreilly.com/2012/10/the-new-new-typography.html
http://toc.oreilly.com/2012/11/gutenberg-regions.html

As it happens, all this time later I’m still doing the same thing 🙂
http://www.pagedmedia.org

We integrated FLOSS Manuals with the Lulu API (now defunct). This allowed us to generate books automatically from HTML using Objavi (below) and they would be automagically uploaded to the Lulu print-on-demand marketplace for sale…that was amazing! Ah…but also no one used it. Lulu shut down the API as soon as they realised no one was using it.

We made many many manuals with this setup. Many of them printed from the auto-PDF magic and were distributed as paper books. Many of the books were in Farsi. The One Laptop Per Child community even built a FLOSS Manuals app that was distributed on all OLPC laptops with the documentation made with FLOSS Manuals.

developed_books

The system produced loads of manuals and printed books about free software. All free content.

developed_stripofbooks

The FLOSS Manuals platform didn’t have a name. It was a hack of TWiki. While you could get all the plugins online, it would have been a nightmare to deploy. I did deploy it many times but I essentially copied one install to another directory and then cleaned out all the content and user reg etc. and went to town rewriting all the .htaccess redirects. It sounds stupid now, but I spent a lot of time doing URL redirects to ditch the native TWiki URLs (which were extremely messy) and make them readable. Hence the system was a pain to deploy or maintain. It was feeling like we needed a standalone, dedicated, system….

Booki

Booki was the next step. It was clear we needed something more robust and also it was clear no web-based book production system existed, hence the hackery Twiki approach. So it was about time to build one. At the time, though, I remember it being very difficult to convince anyone that this was a good idea. I didn’t have access to publishers, and everyone else thought books were just soooo 1440. What they didn’t realise is that we were building structured narratives and that, IMHO, will have a lot of value for a long time. Call it a book or not. Anyway, we built Booki on Django (a Python framework) and the first time we used it was a Book Sprint for the Transmediale Festival in Berlin.

developed_collab

Aco literally would be building the platform as it was being used in the Book Sprint – restarting when everyone paused to talk. It was an effective trick. We learned a lot working with the tool and building it as it was being used.

At the time I couldn’t imagine book production being anything other than collaborative. FLOSS Manuals collaboratively built community manuals. Book Sprints were short events building books collaboratively. So Booki was meant to be all about collaboration. Booki also was run as a website (booki.cc) which was freely available for anyone to use.

develope4d_booki1

Many groups used it which was cool.

developed_booki2

Booki.cc run from with the OLPC laptop

Booki had all the basic stuff the FLOSS Manuals system had and we advanced the feature set as our needs became more sophisticated, but the basics were really the same.

developed_booki3

The main differences were that we had a dynamic ‘table of contents’ where you could add and rearrange chapters etc and the updated ToC would be dynamically updated across all user sessions. Hence the ToC became a kind of ‘control panel’ for the book.

developed_bookitoc

We also experimented with data visualisations of book production processes.

developed_civicrm

developed_seo

We did some cool stuff with the visualisations. For example, during the Open Web Book Sprint (also in Berlin) we worked in the Hungarian Embassy. They had a huge window that you could backwards project onto so people could see the projections on the street. So we made a visualisation of text from the book being overlayed as we wrote it. Below is an image shot with us standing in front of the projector…I mean..c’mon…how cool is that? 🙂

developed_hungary

developed_hungary2

Booki also had federation. You could enter the target URL of a book from another install and Booki 1 would make a portable archive (booki.zip) and send it to Booki 2. Booki 2 then unpacked the zip and populated the book structure complete with images etc. I liked the idea very much of using EPUB instead as a transport technology instead and was later able to do so for Aperta and PubSweet 1 (see below).

In general, Booki didn’t advance much further from the FLOSS Manuals set up. It kind of did ‘more of the same’. I think the only stuff that went further than the previous system was the dynamic table of contents plus it was easier to install and maintain. Having groups was also new, but that wasn’t used much. It was in some ways just a slightly different version of the previous system.

Booki was also used for producing a tonne of books.

developed_2books

developed_collabfutures

Objavi 1 & 2

Alongside the development of the FLOSS Manuals system and Booki, I headed up the development of Objavi. Objavi is basically a separate code base that was used as a file conversion workhorse. Objavi 1 was a little bit of a hacky maze but it did a good job. It would basically accept a request and then pipe that through a preset conversion pipeline. It did a good job. What I found most useful from this is that each step left a dir that I could open in a browser to inspect the conversion results. So if the conversion needed several steps, this was very helpful in troubleshooting.

Objavi 2 was meant to be an abstraction. However I don’t think it really got there, and Booki, which later became Booktype, came to internalise these conversion processes after I left the project. I always thought internalising file conversion was a bad idea because file conversion is resource-intensive, making it better to throw it out to another external service. And having a separate conversion engine enables you to completely overhaul the book production code without changing the conversion code. Hence FLOSS Manuals could migrate to Booki but still use the same external conversion engine. This was a HUGE advantage. Further, all the FLOSS Manuals instances, as well as booki.cc could use a single Objavi install for their conversions.

Objavi was actually also the magic behind the federated content in both the FLOSS Manuals system and Booki. All content would be passed through Objavi for import and export so Objavi became the obvious conduit for passing a book from one system to another. This gave me a lot of ideas about federated publishing which I have written about elsewhere and archived here.

Booktype

developed_booktype1

Booktype was really Booki taken to market. I was frustrated by not getting much uptake, so I took Booki to Sourcefabric in Berlin and headed up the development there. Booktype gained a UX cleanup. The editor was changed after an event I organised in Berlin called WYSIWHAT. The event was meant to catalyse energy around the adoption of a shared editor for many projects. It was of many things I did in pursuit of the perfect editor. At that event, we chose the Aloha contenteditable editor. I don’t think that was as useful a change as expected at the time, but back then contenteditable looked like the way to go despite there being few editors that supported it. Since then TinyMCE, CKEditor and others all have contenteditable support, further econtenteditable became a bit of a lacklustre implementation in browsers.

Booktype was literally the same code as Booki but rebranded. So many of the same features persisted for quite some time until Booktype eventually took on a life of its own.

developed_booktype2

Displaying ‘diffs’ (differences) between 2 different versions of a chapter

developed_booktype3

 

Activity stream of a book

I prototyped some interesting things in Booktype but not much of it got built. For example, I was keen on editing content in the style of the final output. The example below is using the CSS layout of Open Design Now which I mentioned below with reference to BookJS.

developed_booktype4

I also made a task manager prototype based on kanban cards but it was never integrated into Booktype.

developed_booktype5

I think there are only really three parts of Booktype’s development that occurred while I was in charge of the project. First was the integration of a short messaging system into a user’s dashboard and into the editor. Essentially you could highlight text in a chapter, click on the msg widget and a short message could be sent with that text to whoever you wanted (in the system). It was intended for fact checking or editing snippets etc. The snippets were loaded into an editor to assist with this kind of use case.

developed_booktype6

A good idea but seldom used.

In addition, Booktype supported groups, so users could form groups which were populated by users and books. The idea behind this was that you could form a group to work on a collection of books collaboratively.

Lastly, the renderer was integrated in a more sophisticated way so you had more control of the output from within Booktype. Essentially you could choose a number of output options and style them from within Booktype.

developed_booktype7

These were all interesting additions, but in the end, Booktype was really only Booki taken a step further as a product without offering much that was new. I think we should have probably actually removed a lot of things rather than adding new things that didn’t get used.

Booktype developers added some interesting stuff after I left. I particularly like the image editor and the application of themes to the content.

The image editor enables you to resize and effect an image from within the chapter editor.

developed_booktype8

The theme editor allows you to choose from an array of styles/themes and edit them.

developed_booktype9

developed_booktype_10

Booktype 2 has since been released. It has become a standalone business and is doing good work. Also, the Omnibook service is a ‘booki.cc’ online service based on Booktype 2.

Booktype has been used by many organisations and individuals to produce books.

developed_cryptoparty

developed_booktypeangola

I think Booktype 2 is good software but I didn’t particularly like the direction of the Booktype UX after I left the project – it was becoming too ‘boxy’ and formal. ‘Good UX’ is not necessarily good UX. So I eventually developed another system with a much simpler approach, specifically for using with Book Sprints (but it has had other uses as well). More in this in a bit.

book.js

One innovation, and a further exploration of using the browser as a typesetting engine, that resulted from my time with Booktype is book.js. Essentially I had been looking for a new way to render books from HTML using the browser. I wanted to understand how Google docs could have a page in the browser and then render it to PDF with 1-to-1 accuracy. Surely the same could be done with books? However, no one could tell me how it was done. So I researched and eventually found out about the nascent CSS Regions that would allow you to flow text from one box to another in HTML. I worked with Remko Siemerink at a workshop in Amsterdam to explore PDF production from CSS. We made an interesting book with some of the Sandberg designers.

developed_bookjs

After more research and breaking down what I thought could happen, Remko worked with CSS Regions (& js) to replicate the page design of a book called “Open Design Now.” It was amazing. He got the complex design down plus it was all just HTML and CSS, it looked like a page AND when printed from the browser it retained a 1:1 co-relation. Awesome.

The following summer I was able to employ someone for Booktype to work on the tech and I hired a developer (Johannes Wilm) to work on the PDF rendering. From there we eventually had BookJS that enabled in-browser rendering of books which could then be exported to print-ready PDF by just printing from the browser. Whoot!

developed_contents

After time, BookJS (original site still available) could also formulate a table of contents etc. It was, and is, pretty cool and IMHO is the right way to do this. Unfortunately, Google Chrome decided to discontinue CSS Regions so if you now want to use BookJS you have to use a very old version of Google Chrome. However, better technologies have come along that support the same approach, namely Vivliostyle (which Johannes later worked on).

Github Editor

During the time with Booktype, I also did some experiments in other processes. One was using Github as the store for an EPUB and using the native zip export that Github offered to output EPUB (since EPUB is just a zip formatted in a specific way). Juan Gutierrez put together a quick demo and I published about it here:
http://toc.oreilly.com/2013/01/forking-the-book.html

Sadly the demo is no longer available but later a good buddy, Phil Schatz, adopted the idea and built something similar and (I think) much better:
http://philschatz.com/2013/06/03/github-bookeditor/

PubSweet 1

Since Booktype was going its own way, I wanted to develop a new, lighter, system for Book Sprints. Hence Juan and I worked out PubSweet, a simple PHP-based system. This would later be reformulated as a JavaScript system (see below).

developed_pubsweet1

PubSweet was very simple. Essentially 3 components – a dashboard, a table of contents manager, and an editor. It would later evolve to include more features but it was really the same as the systems I had developed earlier, though simpler. I wanted to get to a cleaner interface and bare basic features. I also wanted to retain some book elements, hence the table of contents manager looks a little like a book table of contents (except the garish colors ;).

PubSweet 1 is still in use by the Book Sprints team and functions well. It uses BookJS for rendering paginated books and for PDF rendering straight out of the system. It can also generate EPUB directly. Apart from that, it is pretty simple and effective. I used a basic card-based task manager for this, based on an earlier prototype I made when working with Booktype. It was a simple kanban type board but we never properly integrated it.

We included annotation using Nick Stennings Annotator project:

developed_pubsweet2

PubSweet 1 has been involved in producing more books than I can count, for everyone from OReilly books to Cisco, to the World Bank, UNECA, IDEA etc etc etc

developed_oreilly

developed_labcraft

 

developed_cisco

developed_f5

PleigOS

Somewhere along the way, I developed a simple system for creating Pleigos – one-page books created by folding a single piece of paper which has text and images. The system places text and images so that when you fold a single page after printing, a small book is formed. It was more an artistic experiment than anything but it was fun.
http://www.pliegos.net/

Note: the Pleigos project was by my good friend Enric Senabre Hidalgo, I just developed the initial Pleigos software.

Lexicon

The UNDP approached me to build software for developing a tri-lingual lexicon of electoral terminology. The languages to be supported were English, French, and regional Arabic. It sounded interesting so I used PubSweet as the basis for this.

The main difference between Lexicon and PubSweet was that you could choose to create a chapter from 2 different types of editor. Editor 1 (‘WYSI’) was a typical WYSIWYG editor. This was used for producing chapters with prose. The second type of editor (‘lexicon’) allowed you to create a list of terms, each with three different translations – English, French, and Arabic. This opened my eyes to the possibilities of having different types of editors for different content types, a strategy I hope to use again.

developed_lexiconThe ‘lexicon’ editor in action

The final print output looked pretty good:

developed_lexicon2

The system enabled a dozen or so people from different Arabic regions to discuss translations and work collaboratively through a list while at remote locations. We built a specific discussion forum for them as part of the system and had up and down voting etc. I liked this project a lot because it was very different from any other project I had worked on yet it employed many of the same strategies.

Aperta

Along the way I was approached by John Chodacki of PLOS to build a Journal system for them. I knew nothing about scientific journals, so I went through a process of re-education 🙂 It sounded interesting! I spent a year-and-a-half heading up a team to design and develop this system. This is pretty much the first time I wasn’t scraping together pennies to build a system.

developed_aperta

Journal systems aren’t that different from book production systems, in fact doing this project helped me realise that the production of knowledge, in general, follows a particular high-level conceptual schema:

  1. produce
  2. improve
  3. manage
  4. share

Each artifact (journal article, chapter, book, issue, legislation, grant application etc) follows its own kind of path, with its own unique processes, through these four stages. I have written more about this elsewhere in this blog so won’t go into more detail about it here.

Research articles, (Aperta wasn’t initially intended to deal with Journal Issues which are a collection of articles) come into a Journal as (predominantly) MS Word. They then need to follow a process of checks (eg. to make sure the article is right for the journal etc) before going through the hands of various editors (handling editors, academic editors, etc) and reviewers, including a back-and-forth with the author(s) and a final pass through a production team and/or external vendor to prep the files for publication. The biggest difference I found from my previous experience with book production systems is that there is more to-and-fro involved. Simply put, there is more workflow. So Aperta addresses this with a simple workflow engine based on the Trello/Wekan kanban card-based model.

Aperta was built in Rails with Ember JS. The front end was pretty much decoupled from the backend. It was pretty ambitious and we decided to use the Wikimedia Foundations Visual Editor at the heart of the project. It was the most sophisticated editor available at that moment. I have come to learn that you have to take what you have at the time and work with it. We committed to adopt it and make contributions. I hired the Substance.io chaps (Michael and Oliver) to work on the editor. They did a great job. (Subsequent development of their own editor libs has enabled us to work with them for Coko (more below). )

Aperta’s approach was to simplify the submission process that was managed by Aries Editorial Manager. We leveraged the OxGarage project for MSWord-to-HTML conversion and imported the manuscripts into Aperta. Then workflow templates could be set using kanban cards (as per above). These cards then enabled a flexible method for gathering information (submission data) from the authors at submission time. A lot of time was spent on getting good clean, simple, UX. The kanban card system was very modular – the cards themselves were their own applications, enabling anyone to build a card to their own requirements. This made the cards extremely powerful as they were essentially portable applications.

Further, I developed a model of sorting a lot of cards into columns much like Tweetdeck’s saved-columns model. It was all rather neat. A ton of innovation. Unfortunately, I don’t have any screenshots apart from what I see PLOS has on their website:
http://journals.plos.org/plosbiology/s/aperta-user-guide-for-authors
http://journals.plos.org/plosbiology/s/file?id=d45b/Aperta-Quickstart.pdf
http://journals.plos.org/plosbiology/s/aperta-user-guide-for-reviewers

Aperta is in production now for PLOS Bio but unfortunately the code still isn’t available.

I think Aperta radically rethought how Journal workflows might be managed. I learned a lot through this process and it informed decisions and architecture for the next platform I was to work with – PubSweet 2, developed by Coko.

I also think taking a step into another realm where many concepts were transferable but the use case was different was very good for me. It helped me abstract a few notions. It was also good to build a sophisticated framework in another language and to have the resources to try things out. It was an excellent incubation period where I learned a lot while building a pretty good platform.

During this period I also developed a kind of Objavi 3 but I’m not sure now what it is called or if it is used.

PubSweet 2

The following systems are a result of a team of very talented people at Coko. This has been the first group of people that I have worked with that enable the systems to get close to what I believe is an ideal state for the problems at hand, the time we live in, and with the technology available. If I have learned one thing over the past years, it is that, generally speaking, development teams often prevent good solutions from happening. The Coko team is a rare case where the solutions can flourish and become what they need to be. I’m very lucky to be working with such people.

PubSweet 2 is not actually a publishing platform, it is a de-coupled JavaScript framework that enables the production of multiple publishing platforms.

developed_pubsweet_2dev

With all the other platforms I have been involved with I have always learned something. So the next platform is always better. Strangely enough, because when I look back I remember, for example, thinking that Booki was the best platform ever, and perhaps the best I could do. But not so. Each subsequent platform has been much better. So… why not build a framework that enables the rapid development of many platforms at once? good idea! Imagine what you could learn… Indeed!

Using PubSweet, we have developed two platforms to date. Science Blogger and Editoria. For the purposes of this blog post I’ll just focus on Editoria as Science Blogger is more of a reference implementation. However, all these front components can be developed independently of each other, and independent of the core framework. Hence we have released some PubSweet NPM modules (sort of like front end plugins) online:
developed_node

If you would like to read more about PubSweet check here:
http://slides.com/eset/ismte

Editoria

https://editoria.pub/blog/

developed_editoria

Editoria Book Builder component

What is interesting about Editoria is that I didn’t design it. I facilitated the staff at the University of California Press to design it. And what is really interesting is that it is a better book production software than any of the above that I did design…. That’s not to say that I finally realised I was a crappy designer 😉 , rather I came to realise that given the right guidance and parameters, the people with the problems can solve the problem with more deep understanding and nuance than I could as an outsider to their processes. It was a great thing to realise.

In general, Editoria follows the same model as PubSweet 1. It is a simple collection of 3 interfaces – dashboard, book builder (in PubSweet 1 we refer to it as a table of contents manager) and a chapter editor. Very similar to PubSweet 1 and also these components fit into the conceptual schema I mentioned above of:

  1. produce – this happens outside the system and we convert authored MS Word files to HTML
  2. improve – styling, editing, reviewing, all occurs in the editor component
  3. manage – basic workflow managed by the Book Builder component
  4. share – when done we export to book formatted PDF, and EPUB

Editoria is a very simple and elegant solution. It’s the only one of the systems I have worked with that is not in production but I’m looking forward to seeing it up and running soon.

It uses the Substance.io libraries to build a custom made and elegant editor. Finally with some real $ to spend on moving this field forward, and as part of my 2015 Shuttleworth Fellowship I committed $60,000 USD (10k a month) or so to Substance, Coko followed it by a similar amount, so they could focus on the development of their libraries to 1.0. Coko also put considerable effort into founding a consortium around Substance (although I’m not convinced it is really working well yet).
http://substance.io/consortium/

Editoria brings some nice implementations to the table including the use of Vivliostyle to render PDF from HTML. Plus MSWord-to-HTML conversion, front and back matter divisions plus body content. Pagination information (left/right breaking) etc. In addition, we are building into Editoria various tools in the editor including its own annotation system and a host of publisher-specific markup options for different styles (quotes, headings etc). Coko has employed Fred Chasen for some part time work to contribute to the Vivliostyle development (although we are still working out what we should work on).

In many ways, Editoria is the system I always wanted to be involved in. It is better than any other system I have seen or been involved in and this is a result of two critical factors:

  1. the technologies have matured, including Vivliostyle and Substance.io, that enable critical solutions to be solved
  2. the people who need the system have designed it

There is more to come from Editoria, so stayed tuned…

INK

INK is like Objavi on steroids. It is possibly an Objavi 4 😉 but done the right way. INK is a Rails-based web service which is primarily built to manage file conversions. However, it can actually be used for the management of any job you wish to throw at it. These jobs are what we call steps, and steps can be compiled into recipes. For example, you might have a step ‘convert MS word to HTML’ and then a second step ‘Validate HTML’. Hence INK enables you to chain together these steps.

develop3ed_ink

Additionally, these steps are plugins written as Gems. A gem is a Ruby-based plugin architecture. Accordingly, you can develop gem steps and distribute them online for others to use. We hope in time there will be a free (as in beer) market around these steps.

Summary thoughts

I think I learned a lot from writing this personal ‘sense making’ piece. I didn’t realise just how strong some of the themes were that I pursued until a wrote them down…for example, I pursued collaborations from inter-organisational consortiums a number of times and none of them have worked. Huh. Interesting. I’ve also seen various practices evolve from an idea to being mature thoughts, prototypes, workable solutions and eventually, eventually, adopted. But over many more years than I expected. Also interesting.

It is also obvious that there are some big ticket technological problems that still need to be solved for good to really move forward. They are mostly there but still in need of work, the top two being:

  1. browser as typesetting engine
  2. sophisticated editor libraries

I add a third which I haven’t noted much in the above. I have worked on this since Aperta onwards and it is necessary only in the publishing world where content is authored in the legacy ‘elsewhere’ (ie MSWORD):

  • reusable, sensible, MSWord to HTML conversion

This has been solved many times but has not yet been solved well.

These all need to be open source solutions. At Coko we are trying to move all these on and we are making good contributions. We, as a sector, are nearly there. Although it would be good to work more together to solve these problems by either making contributions to the existing efforts, or using their technologies. This is really the only way things move forward.

Finally, in the tech world people sometimes quote “Being too far ahead of your time is indistinguishable from being wrong,”. I’m not sure who said it but when I look through the evolution of technologies I have written about above, seeing various things evolve from idea to a production implementation many years later, I’m also thinking that perhaps sometimes waiting might be indistinguishable from being right 🙂

The beauty of small teams

athens_meetup

I have been involved in setting up a team for the Collaborative Knowledge Foundation (Coko). We are about 12 all up now and spread around the globe. At our core we have :

Jure – Lead PubSweet Developer Slovenia
Richard – PubSweet Developer Kenya
Yannis – PubSweet Frontend Developer Athens
Christos – PubSweet Frontend Developer Athens
Charlie – INK Lead Developer New Zealand
Wendell – XSL-pro East Coast USA
Julien – UX-pro France
Henrik – Designer Netherlands
Juan – Sys Admin Nicaragua

and then we have myself, Kristen Ratan (co-Founder), and Alex (process manager) in San Francisco.

So… just looking at how development works… in the dev side of things we are pretty well distributed over the world and across a large variety of time zones, in fact, NZ, EU, USA is about as bad as it can get when setting up meetings at reasonable hours for all affected… so how do we manage to make this work?

Well, firstly we obviously rely on some remote tools. We all coalesce around Mattermost – an open source version of Slack (By the way, don’t use Slack – https://clearchat.com/blog/forget-applevsfbi-slack-gmail-have-backdoors/). This is a good avenue for remote chatting. We use it for both chit chat, y’know, hanging out and taking bullshit, and also work. Work mostly gets done on side channels, I wrote something about it here.

Okok..so that’s Mattermost. It’s a good tool but if we were just a bunch of people scattered around the world with a chat channel then it probably wouldn’t work as effectively.

So, I wanted to say something about creating space for people to operate and how remoteness can actually play into that rather nicely.

First of all, we are an organisation that likes people to like what they do. So we like to trust them and give them a bit of room to move. All people are creative and I believe all folks need challenges where they can exercise that creativity. Nothing new here – if you want to read more about this, with special reference to development teams, then read Peopleware (terrible title, great book – very old school… you can stop reading when they talk about optimum cubicle layouts, but the rest is awesome).

What I have found over the years is that responsible people work responsibly. So, I am assuming you work with some people like this 🙂 Otherwise return to zero and start again…. the thing about responsible people is that they like to get their head down into a flow and get on with it. They love hard problems and they love the trust given to them to solve these problems. But sure sure, this can happen in real space in a shared office…but…can it? I think there is an argument to made here that remote teams enable a more optimal working environment for this kind of team.

In our case, for example, we actually only have two people in the same place (not counting San Francisco). They are Yannis and Christos. Very nice chaps if you ever happen to be fortunate enough to meet them. Yannis and Christos work on developing the front-end components for the PubSweet publishing framework that Richard and Jure are building. We have an advantage here that we have carefully chosen to create a technical architecture that supports the autonomous development of front-end components which is separate to the backend that they rely on (the backend is known as PubSweet or sometimes we refer it to PubSweet core or ‘the backend’). Anyways… note to self, how you build what you build is a determining factor on how your team is structured and where they are… but… back to remoteness…

When we develop front-end components for clients, I go with Alex and/or Kristen to the client’s locale and then work through their design in a Collaborative Design Session. We then write this up and meet with Julien, Yannis and Christos remotely with a online whiteboard tool (BigBlueButton). And we have another collaborative session where we nut out the details. This is all a little by-the-by but just to say that we can segment our work quite nicely and the communication that does take place remotely is either:

  1. bullshit and fun
  2. high value and short

These whiteboard sessions are type 2 from the above. We set 2 hours to have the complete design sorted. When that meeting is done, everyone knows what they need to do. After that, we leave Yannis and Christos to get on with it and let us know when they are done. Any miscellaneous stuff gets sorted out bit-by-bit between the people involved. If, for example, some more UX mocks are needed, or some CSS detail, then Yannis or Christos will ping Julien on Mattermost and sort it out. No project management, no scheduled meetings. Short and sweet as per type (2).

At the same time, Jure and Richard are sorting out the backend. They communicate amongst themselves and solve the complex problems that such an abstracted system requires. All over Mattermost, too. Of course, there are various breakouts to real-time calls when required but it’s a minimum. Watching Jure and Richard tumble through difficult problems in Mattermost is a sight to behold. Both firing off each other and throwing ideas back and forth.

My job is largely to give each of these folks room to move.

So what about when each team needs something from the other? Being right up on the coal face of the client needs, I can see what things we need to address a little ahead of time and I call out to those affected and work out how we can sync the various threads. For example, we need to integrate PubSweet with INK (more about this below) for importing Word docs into PubSweet as HTML. So, seeing this, I talk to Jure mostly, and think through the issues. Jure interprets my partial developer-/partial user-speak and thinks about it. He then discusses it with all those necessary and they work out the best approach. Then we work out when it all needs to happen by and the order of play. Everyone then orientates around that timeline and away we go…in otherwords…the parallel teams ‘touch’ only when necessary – when they need something from each other. That’s not to say we try and stop people from talking to each other 🙂 Quite the opposite, we don’t gate any communication at all and leave people to sort it out. What we don’t do is build things before we need them or layer on complex management overhead. These are smart responsible people, let them work it out 🙂

athens_whiteboard

the above image drawn with the fabulous MyPaint (open source) on a touchscreen Thinkpad running Ubuntu.

We are also achieving the same process with INK. Charlie is down in the Southern hemisphere tinkering away. She’s like a deep sea diver in Ruby and coming up to explain to us surface dwellers what’s going on down there. You saw whaat? …an uncontrolled multi-threaded waaaaaat!?oh good, you killed it Just-in-Time???…phew….. Charlie is building a system for managing file conversion pipelines but it is very abstracted. Essentially it is a job-agnostic queue manager. You can throw anything at it and it will do it. We are first using it for MS Word to HTML conversion. Wendell and Alex are working together to get the conversion process into shape. Wendell is writing the code and Alex is testing. Then we are making this into what we call a ‘step’ – essentially a plugin to INK. So Charlie can build the architecture for INK and then we need to make a light INK wrapper for the conversion scripts Wendell is making and then Alex can (not happening yet but soon) actually use INK to test the whole sheeeeebang. Cool. So, they can all operate pretty well remotely to each other building the same machine.

The cool thing about all this is that, sure sometimes details are hard to transmit, but mostly what happens is that people have most of the day, every day, to work on what they need to work on. I think this is helped by being remote. We have to be efficient. We don’t get in each other’s way. We have to get clear understandings fast. Remote comms aren’t nice for long conversations. We need them short, sweet, and efficient. Then everyone can just get on with what they want to do most…solve interesting problems.

I’m not going to say we are perfect. I’m also not going to say this process is perfect. We see the issues. We plan to have at least one full in-person team meet every 0.8 years so everyone can hang out and fill up the camaraderie tank (important)…the last one was in Athens and it was a blast

athens_meal

athens_pause

…and I make sure I go and see everyone every few months to make sure we are all on the same page and feeling connected. This setup needs these additional things. But it works, and when I travel around to see everyone, I see it working well and I see people performing exceedingly well doing things they want to do.

More research

Looking into the dynamics of Open Source communities and questioning if there is indeed such a thing as devcentricity. Looking for evidence for or against. Looking for anything that can describe relationships and workflow between a diverse group of stakeholders with differing skills.

https://halshs.archives-ouvertes.fr/file/index/docid/860579/filename/IJHSC-Barcellinietal.pdf

Major OSS projects are highly hierarchical and meritocratic communities (Gacek and Arief, 2004; Mahendran, 2002). Five different statuses are generally distinguished in these projects, according to the distinctive rights and power of the participants. Some participants can modify the source code and participate directly in the design process and in decisions regarding the software:
– the project leader (generally the creator of the project such as Guido Van Rossum for Python, or Linus Torvalds for Linux);
– the core team or administrators, who have to maintain the code base, the documentation;
-the developers or contributors who participate in the evolution of the OSS and maintain some of its parts.
Others participants are called users. In an OSS context, users may be highly skilled in computer science, and thus far from the classical notion of “end-users”.
– They are called active users if they participate in mailing-list discussions as informants for newcomers, by reporting or correcting bugs with patches, and by proposing new modules. These active users in a particular OSS project may be developers in another project
– Other users are called passive users as they only use the software or lurk on the discussion and documentation spaces of the project (Preece et al., 2004).
It is possible to evolve between these statuses by acquiring and proving one’s technical skills and ability to engage and maintain online discussions: that is to say that roles emerge and are actively constructed within the community
(Ducheneaut, 2005; Mahendran, 2002).

and

The literature on OSS clearly identifies that active users take part in the evaluation phase of design (bug reporting and patching, e.g. Ripoche and Sansonnet, 2006) and that the project leader, administrators and developers participate in the design process itself, i.e. generating and evaluating solutions and taking decisions (Barcellini et al., 2005).

and

Open issues are still to characterize the role of users regarding the design process itself and the role of all the active participants (project leader, administrators, developers and active users) during the elicitation of the needs and requirements phase. Despite the idealistic picture that users may intervene freely in the process, we will question whether users who are neither administrators nor developers in the core Python community can really have an impact on the design choices and decisions.

and

The coding of activities is inspired by previous studies on collaborative software design activities (d’Astous et al., 2004; Détienne et al., 2004; Olson et al., 1992; Stempfle and Badke-Schaub, 2002) and by the coding scheme developed in our previous paper (Barcellini et al., 2005), which we have extended.

and

We found that users mostly tend to participate in the user-oriented list, python-list. Here, they provide references mostly on usage and personal experience, computer science, and code and examples. They also tend to provide more personal experience and end-user references than others in python-list. Even if the users’ contribution seems important in order to specify usage needs, their participation remains local to the user-oriented community and does not guarantee that these needs will be taken into account in the actual design.

and

We found that cross-participants perform an emerging role of boundary spanners, which guarantees that usage is linked to design and that the boundary between the user community and the developer community is crossed.

and

According to our results, OSS design does not seem to be participatory in the strict sense of the definition, i.e. user involvement in “design” activities. Even if users of OSS may potentially be involved in all the phases of the OSS design process (elicitation of needs and requirements, design and implementation), we found that their participation remains mostly local to the user community in the PEP process we analysed. We found that the design-use mediation is supported rather by a number of key participants who act as boundary spanners and who are not necessarily users themselves: two of them were users but the three others were administrators and developers

Dev Centricity

More partial thoughts. I have been pondering the whole dev-centric nature of Open Source.

Open Source projects that put the dev at the centre of the culture see all problems as technical. Which means that if a problem is presented then a dev-centric approach will see the entire problem and, consequently, the entire solution, as technical. That routes around seeing the problem for what it really is.

I think this leads to:

  1. misunderstanding the real problem at hand
  2. producing solutions that don’t fix the problem

So we need to address this, however, the way that Open Source projects avoid seeing that they claim the users don’t know what they want.

We are kind of left with the scenario that all problems are technical and only the developers can fix them. Doesn’t that feel like a little broken?

This is the result of putting faith in technical meritocracy. A developer-centric sector that mostly judges your value by your ability to program. The signifiers are all there – naming a sector after code (‘Open Source’) signifies this, as does dismissively labelling those that don’t program as ‘non-coders’.

That’s not to say developers are bad people! Some of my best friends are developers! 😉 But it is to say that we need to see, discuss, and rework this legacy power imbalance as it really doesn’t help in making good solutions. We need diversity of all kinds – gender, ethnicity and roles in Open Source – bringing all voices in at the appropriate moments, and with even power dynamics brought about through skilled facilitation. Only this will lead to unleashing the real power of collaboration and sharing and make OS beat all those brain-dead VC funded techno-meritocratic proprietary ‘solutions’.

Some notes on CPD

At its heart, CPD is three things:

  1. Political critique
  2. A problem-solving methodology
  3. A diffusion strategy

Political critique

I like this part of Collaborative Product Design most of all. Open Source is a developer-centric solution model. Essentially we have created a clear distinction between people with problems and people with solutions. The later are called Developers, the former, Users.

The idea is that “Users don’t know what they want”. Which might also be remapped to “Developers don’t know what Users want” given that so many of these developers have created ‘solutions’ that fail. Somewhere in this model, something is broken.

I believe placing the developer at the centre of the solution design process is the problem. We need to put the right people in the right place. Users should be at the centre of designing solutions for user-facing problem spaces: developers should be at the centre of designing solutions for developer-facing problem spaces.

This means we need to do the following:

  1. first, change the language. We don’t have Users and Developers. We have People with Problems that need solving. Some have workflow problems that need solving, some have technical problems that need solving.
  2. People with workflow problems design solutions for these problems
  3. People with technical problems design solutions for those problems
  4. don’t mix up 2 & 3

Mixing up 2 & 3 is generally the default ‘Open Source’ model and is cast as:

  1. Developers solve workflow problems for Users
  2. Developers solve problems for Developers

Number 2 is fine, number 1 is what gets Open Source in trouble.

Where did this cultural default come from? Well…Open Source is famous for a horrible book known as the Cathedral and the Bazaar which the self-appointed bishop of open source anthropology, Eric Raymond, states as cannon

Every good work of software starts by scratching a developer’s personal itch.

He also states as cannon:

The next best thing to having good ideas is recognising good ideas from your users. Sometimes the latter is better.

This is just wrong. Good software comes from collaboration across specialist domains. If someone has an itch, we talk to them, possibly call in people with similar itches, some specialist medical experts, maybe some researchers if it is a new kind of itch, and whoever else is affected (infected? hoho)… and we learn from each other and collaboratively solve the problem. Raymond’s text, which has had a huge influence on Open Source culture, places the ‘user’ as someone who might have some ideas worth considering but really it’s up to the developer to make that decision. The developer is still, even in the most liberal interpretation of these statements, the arbiter of the design.

We need to get everyone to work together, talk to each other, and work out who is best at solving each part of the problem.

By the way, I am not saying that ‘developers’ have no value when creating solutions for user-facing problems. They do have value in this process. However, it is their ability to empathise and join in a collaboration that is of value in this moment, not an ability to code.

…and…if you wish to explore further gems from the genius of Eric Raymond’ please put aside a quiet weekend to dive into Sex Tips For Geeks….

this is a scratch pad…more to come…

Notes on Collaborative Product Development

So, during the Shuttleworth Gathering this week I facilitated a session about CPD and asked for feedback. I got some great information, most of which is targeted at what people would need to know in order to understand the process better. I am currently writing a ‘book’ about the methodology so this feedback will help shape that. Starting with:

  • challenges on the name and if there is something better
  • timescales need to be illustrated and discussed
  • implementation phase needs to be unpacked and described
  • testing cycles
  • bring out the fact that the product must be adaptable
  • explain how this will change ‘your’ organisation
  • importance of institutional buy-in
  • case studies and stories
  • maybe a hardware case study?
  • with this method some change is cultural
  • how would you measure cultural change?
  • how do you better ensure cultural change?
  • case studies would be very useful
  • simple illustrations of why this method is better
  • front load the political critique of Open Source
  • with the front loading highlight that most ‘client centered’ solutions are actually developer-centric solutions hidden behind a paywall

Also an interesting idea to make a current workflow image into a poster ‘this is what we are trying to avoid’

Arthur Attwell came up with this lovely way to describe the advantages of collaborative design:

footpath-oraSolution designed by the city council

footpath2-oraActual traffic

Here  you can see the dissonance pretty clearly.

I think I will structure the book something like this:

  • what is Collaborative Product Design
  • What does a hosting organisation need to think about
  • the basics
  • the setup
  • the design session
  • facilitation
  • the build
  • implementation