PagedMedia Initiative

PagedMedia.org is launching a new community-led development at MIT Press on January 9, 2018. The project will develop a suite of Javascripts to paginate HTML/CSS in the browser, and to apply PagedMedia controls to paginated content for the purposes of exporting print-ready or display- friendly, PDFs from the browser.

This will be an Open Source initiative, appropriately licensed with the MIT license.

The January meeting will be the first meeting of the project and attending will be:

  • Adam Hyde (Coko/PagedMedia.org/Shuttleworth Fellow)
  • Dave Cramer (Hachette/PagedMedia.org/W3C Publishing Work Group)
  • Nellie McKesson (Hederis/W3C PWG)
  • Terry Ehling (MIT Press)
  • Erich van Rijn (University of California Press)
  • Kathi Flectcher (OpenStax/Shuttleworth Fellow)
  • Hugh McGuire (PressBooks/W3C PWG)
  • Arthur Attwell (Fire and Lion/Shuttleworth Fellow)
  • Tzviya Siegman (Wiley/W3C PWG)
  • Travis Rich (pubpub)
  • Fred Chasen (PagedMedia.org/Future Press/W3C PWG)
  • Julie Blanc (PagedMedia.org)
  • Phil Schatz (OpenStax)
  • Julien Taquet (Coko/PagedMedia.org)
  • Ned Zimmerman (PressBooks)
  • Carly Strasser (Coko)
  • Wendell Piez

For further information please contact: adam@coko.foundation

PagedMedia.org is funded by the Shuttleworth Foundation.

Also posted here – https://www.pagedmedia.org/paged-media-open-source-initiative/

Aperta Halted

A part of my personal history is a platform called Aperta which was previously called Tahi. It was a PLoS project and I was hired to design and build it. I quit when the PLoS Board decided to close the repositories, effectively making it a closed project. The repos remained closed, and as far as I know, are still closed today. Ironically after I left, they renamed the project ‘Aperta’ – Italian for ‘open’. A really silly marketing move to reassure everyone that despite what they may have heard, the project was still open…that was perhaps true, albeit (ironically and literally) in name only.

Now, it seems, the platform dev has been halted. Feels good to me. From what I heard (and I didn’t hear much), PLoS didn’t take the project in a good technical direction and generated a significant amount of bad faith and market confusion while trying to develop it behind closed doors.

To quote the new CEO Allison Mudditt (who I respect very much, Coko worked with Alison when she was at UCP):

Part of this initiative will involve changes around the workflow system – Aperta™ – we set out to develop several years ago with the goal to streamline manuscript submission and handling. At the time we began, there was very little available that would create the end-to-end workflow we envisioned as the key to opening research on multiple fronts. But the development process has proved more challenging than expected and as a result, we’ve made the difficult decision to halt development of Aperta. This will enable us to more sharply focus on internal processes that can have more immediate benefit for the communities we serve and the authors who choose to publish with us. The progress made with Aperta will not be wasted effort: we are currently exploring how to best leverage its unique strengths and capabilities to support core PLOS priorities like preprints and innovation in peer review. This will be part of our planning for 2018.

http://blogs.plos.org/plos/2017/12/ceo-letter-to-the-community-mudditt/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+plos%2FBlog+%28Blogs+-+PLOS%29

I hope that PLoS releases the technologies that have been developed for Aperta (there was a lot more than just the submission system) into the open… with both open repositories and open licenses AND, more importantly, an open heart. Collaboration and openness is more to do with how people are than what open license they choose and several of the practices, including asking potential collaborators to sign Non-Disclosure Agreements (NDAs) before getting a demo of the system, were ridiculous and ungenerous.

Having said that, it would be awesome to see all that work released into the open, in open repos with open licenses, and no more blurring of the word ‘open’. Afterall the systems developed that included Tahi were all paid for by researchers. The PLoS Article Processing Charges fuels PLoS and they committed some of this revenue to the development of Tahi. When I was there, no external funding was secured for developing the system. Pedro Mendes made a good point in response to the announcement:

ha

There is some merit to this, but I do applaud PLoS for being adventurous, and if it had worked then the result would have been APCs could be lowered, not just for PLoS, but for any Journal out there reliant on expensive and dysfunctional Manuscript Submission Systems. Allison also notes this in a discussion below the post mentioned above:

…the original idea was that Aperta would allow us to eliminate or speed up the slowest steps between a finished work and its publication in order to reduce the cost of our publishing services

That is true, and it was an admirable goal. However, whatever the journey was between then and now, the project should have always have been out in the open as a public asset. Open for science, open for access, open for source, open for all – and the fact researchers paid for it but it was turned into closed project mid-flight is reprehensible and in the end it worked against PLoS, in particular, it severely weakened PLoS claims to supporting all things open. What a mess.

But, it can’t be ignored that Tahi is about 5 years old now, which is old in software years. A entire generation of technologies that are better suited to solving these problems has arisen in that time. The system is now not much more than a still (just) relevant but outdated approach. That is the risk you take when you develop things behind closed doors. By the time you release it (or don’t in this case), it is out of date. That said, it would still be good to release it, but there are better technologies and approaches out there now.

So I look on with interest to see what will happen next.  I sincerely hope PLoS can return to cutting a path through publishing and exploring and enabling a viable Open Access model that others can follow. With Allison at the helm I am betting things are going to take a much needed turn for the better, not just with this project, but on all counts.

As for me, I learned a lot from designing Aperta (I prefer to call it Tahi). The design process was an introduction to scientific journal publishing for me. I learned a great deal. Tahi gave me, at the time, an unencumbered dream time to imagine something new. It had a lot of interesting innovative approaches and if I had stayed with the project it would have ended up close to where PubSweet is now as I wanted to completely decouple the ‘spaces’ (a concept important to Tahi). It would not have been as good as PubSweet at doing this as a complete ‘decouple’ really has to be imagined from the start, and isn’t as clean if retrofitted. Still, the system would have been a lot more flexible and reusable.

But that wasn’t to be. Don’t get me wrong – I don’t think Tahi was the perfect platform, but it was a pretty good starting point with some significant innovations. At the time, I was looking forward to shaping Tahi with use and to mature it into an excellent system. The good news is, the next platform you design is always better.  I took a lot of what I learned (I have now been involved in instigating around a dozen publishing systems) to my next development, and worked hard to re-conceptualise a new system that avoided some of the mistakes I made with Tahi, and took some of the good parts a whole lot further. That new project is PubSweet and it is looking awesome, and leverages modern technologies and approaches to the max – mainly thanks to the bunch of amazing folks working on it within the Coko team (particularly Jure Triglav) and also now, increasingly, from the collaborators we work with (at this stage mainly eLife, YLD, Hindawi and ThinSlices). Also a huge thanks to the Shuttleworth for backing me, especially because it was at a time (I had just quit PLoS) when it was very much needed. Their backing meant Coko was possible, and consequently, PubSweet and everything else we have done.

Anyways… it was past time PLoS moved on too from Aperta and congratulations to Allison for making the right call, especially given that it would have been a difficult one given the cultural forces at play inside of PLoS.

Getting Design and User Experience Right in Open Source

So, I’ve thought about Open Source and design… I’ve even written some articles on opensource.com (https://opensource.com/users/adam-hyde) about the subject, and created a methodology for bringing the use-case specialist (user) into the center of the process, along with designers and everyone else…

I’ve also brought this subject up a number of times in Shuttleworth Foundation meetings and received some invaluable advice and insights from fellow Fellows and Shuttleworth staff… many of whom have heard my whacky ideas several times over now and are still patiently listening and offering advice! Forever grateful to y’all… especially Helen, Sean, Arthur, and Andrew for ideas and feedback.

But what I didn’t expect, is that I’d be part of a wider community where these ideas could form the basis of the culture. This is what I saw happen this week in London as part of the PubSweet Global meet Coko hosted (& I facilitated).

There were about 25 of us coming together to discuss all things PubSweet with particular emphasis on building Journals. Present were many folks from eLife, Hindawi, Ubiquity, and Coko. We got to the topic of ‘Technical Council’ and I tabled the idea that we need some kind of process in place so that all stakeholders feel they are getting a say in, and are being heard, the future of PubSweet – since it is their technology too.

dsc03891
CoFounder of Coko, Kristen Ratan, kicking off the meeting

When I tabled the idea that we need some form of technical council, Catriona MacCallum, who I used to work with at PLOS, asked the very salient question – and what about the users?

Catriona on left.
Catriona on right.

I’m very grateful to Catriona for that question as it gave me the opportunity to open up the concept and talk about how there are very few communities in open source that treat software development as anything other than just a technical problem, and further that we should take this opportunity to experiment in making this community strong on solving ‘user needs’ and design… it was a great discussion and I’m also grateful to eLife’s head of product – Giuliano Maciocci – for having a strong voice in favor of this and really stepping into, what looks to be, an emergent leadership role with regard to design in the community.

dsc04133
Giuliano.

As a result, we formed a Dev Council and a Design Council. These are oversight/communication groups of 5 people each. So, they don’t ‘govern’ but the choice by the community to form these two groups is a testament to how seriously the community is to making beautiful products that solve real problems in publishing for real people…

dsc04029
Whiteboard from the session showing our decisions. Dev/Design council structure at the bottom, also showing their relationship to community (supporting) and Coko (facilitated by).

All in all, a pretty fantastic 3 days.

The Road to PubSweet 1.0

We are pretty close to our PubSweet 1.0 with the RFC now out for PubSweet 2.0, and a PubSweet dev site release next week.

pubsweet

It has been an amazing effort, particularly by Jure Triglav, the lead dev for PubSweet at Coko, but also fantastic work from Richard Smith-Unna, Alf Eaton, Yannis Barlas, and Christos Kokosias. Also more recently some great contribution from Alex Georgantas.

9

So, we are pretty much there and I’m presenting in San Francisco this week as part of a small Coko event to reflect on the future of the framework and discuss the RFC. For this purpose I’d thought I’d write a post to help me think through the thinking that got us here.

So…the thinking behind PubSweet started when I came back from Antarctica around 2007 or so (I was there setting up an autonomous base for artist-scientist collaborations).

5jan2007-2

I decided I wanted to give up the art world and try something new. The something new turned out to be FLOSS Manuals – a community writing free manuals about free software. I started it when I was living in Amsterdam somewhere around 2007. In order to execute on this mission I needed to get a couple of things sorted. Namely, learn how to build community, work out processes for rapid book production, and work out the tooling.

The tooling started with me scratching around with TWiki. A wiki written in Perl that happened to have the best plugins for rendering PDF. I scratched around, writing some Perl and cutting and pasting a whole lot more, and added some crazy .htaccess URL rewriting to produce a basic system for producing books. It was pretty scratchy but it actually worked. Later a buddy helped extend the system and later still I was able to pay him and others to extend it.

At the time it pretty much comprised a page (per book) for creating a table of contents.

developed_blog_chapterlist

and an interface to edit the content (chapters). I ripped out the native wiki markup editor and replaced it with a WYSIWYG editor, I think it was TinyMCE…

developed_fm_farsi

As you can see Right-to-Left content (in this case, Farsi) was also supported. There were also some basic things in place for keeping track of the status of a chapter, the version number, side by side diffs, side by side translation interfaces, and, later, dynamic table of contents organisation and edit locks.

Coupled with some basic PDF rendering stuff and a way to push the content from the ‘draft’ to the publishing front end and we were away.

flossmanuals-1

It actually had some other pretty cool stuff, such as side by side translation interfaces…

flossmanuals-howtotranslate-sideby-en

Remix

..a built in live chat for talking with collaborators…

booksprints-remotecontrib-chat-en

and even a way to send books between different instances (eg for sending a book from the FLOSS Manuals French community site to FLOSS Manuals Finnish for translation)….

flossmanuals-xchange-xchange_1-en
We could even render book formatted PDF and push to the lulu.com print on demand services. I just now checked and some of the books are still there!

Not bad for a Perl-based system, built on top of a wiki that wasn’t supposed to do this kind of thing, and built very with very few resources. The TWiki extensions were contributed back upstream to the TWiki repo and it was all open source but it was pretty hard to rebuild and no one I knew actually had a similar use case.

After this, I embarked on a journey to replace the system with a custom built solution specifically for book production. I can’t remember exactly when this started, maybe 2008 or 2009 or something. It was originally called Booki…

b

…which later became Booktype. Booki (and later Booktype) replaced the FLOSS Manuals tooling, although you can still see the working old tool here. That ole Perl code is still functional with no maintenance after 10 years, I can hardly believe it. The docs on how to use it also still exist.

Booki was built with Django (python) and pretty much had all the same stuff. Although the look and feel was changed quite a bit in the transition. There aren’t too many images around of Booki although I did find these screenshots of Booki taken by someone using it on the OLPC XO! (FLOSS Manuals did all the docs for OLPC/Sugar OS etc).

readingandsugar-booki-en

readingandsugar-booki5-en

It was hard to get financial support for it. Internet Archive gave us $25,000 at the time which seemed like a fortune. The evolution of Booki to Booktype represented me taking the project to a buddy’s in Berlin (I was living there at the time) based org (Sourcefabric) and parking it there so I could get more resources to build it out.

Booki/Booktype pretty much had, and has, the same stuff as the FLOSS Manuals system, just purpose built. So it had, a table of contents manager

images-cms-image-000003634

And a book (chapter) editor…

booktype_version_2_editor_screenshot

…chat…

booktype_2-0_edit_communication_tabs_chat

And the other stuff. Perhaps the only new features (compared to the FLOSS Manuals system) were a dashboard…

booktype_1-6-1_my_dashboard_740x429_1

…groups…

all_groups

and an interesting way to have Twitter-like messaging to pass snippets from chapters to other users.

figure-7_reference

Before I left Sourcefabric I wanted to get some other innovations built but didn’t get there. I did build some prototypes though. There was a task editor…

booktype-1024x531

…and live in-browser book design…

developed_booktype4-1024x768

Booktype is still going strong, now it is its own company (based in Berlin) and they also run the Omnibook commercial service using the software.

I left because John Chodacki and Kristen Ratan from PLoS invited me to come work for PLoS to design a new web-based journal submission system. I agreed…

But, before I leave the book story behind for a bit..I had set up Book Sprints as a company and put a small amount of my own money into building two new book production systems somewhere between leaving Sourcefabric and starting at PLoS. These two systems were PHP-based and Juan Gutierrez built them over some months.

juan

I wanted to do this because I was a little frustrated by Booktype not moving forward and also the platform was becoming more difficult to use. We were using it for Book Sprints but after I left the product took a new UI direction and I was finding Book Sprints participants were not enjoying using the system. So I built a Book Sprints specific system called… PubSweet… the namesake of the current Coko system which has eventually turned into something of a prototype for the new PubSweet… this new system was a lot simpler and easier to use than Booktype. It was initially meant to be modular but I think we lost that somewhere along the way. Cleanly modular systems take a lot of extra effort and time to produce so we gave in for speed of development’s sake.

The old PubSweet had a dashboard….

screenshot-from-2017-07-15-18-37-13

..table of contents manager…

ps1

and editor. Just like before!

screenshot-from-2017-07-15-18-40-08

We also introduced some new innovations including visualisations of the book production process…

developed_seo

Plus annotation (using Nick Stennings annotator software)…

screenshot-from-2017-07-15-18-48-01

 

and other stuff…I think threaded discussions, outline views, review page, an in-browser book renderer, book stats and I can’t remember what.

Anyway …I also built a platform on top of this old PubSweet for the United Nations Development Project. It was called Lexicon. Lexicon was pretty interesting as it opened my mind for the first time to the idea that an editor is not an editor is not an editor. Different content types (in a book) may require different editors or production environments.

Lexicon was produced to collaboratively produce a tri-lingual (Arabic, French, English) lexicon of electoral terms for distribution in Arabic regions.

screenshot-from-2017-07-15-19-09-11

Lexicon had all the same stuff as the old PubSweet but with one major innovation, you could create chapters that were WYSIWYG based, or you could create a chapter which enabled you to add and sort individual terms and provide translations.

screenshot-from-2017-07-15-19-09-34

It was a pretty interesting idea and we were able to make a really cool book which the UNDP printed and distributed across many Arabic-speaking countries. I still have the book on my bookshelf.

The other interesting thing was that the total cost for building this on top of the old PubSweet was $10,000 USD. This was mostly because we could leverage all the existing stuff and just build the difference…interesting idea!

Ok, so then I dropped book production systems around 2013 or so for a while and went to work for PLoS on a system that was called Tahi and then became Aperta. The name Tahi came from the name of the street I was living on in New Zealand before I had a US work visa and was designing the system – Reotahi Road (cool road). Reotahi means ‘one voice’ and ‘tahi’ means ‘one’ in Maori. It was built on Rails with Ember. Essentially the front end and backend were decoupled although it was really pushing the technology at the time to do this. I designed the system and moved to San Francisco to manage the team to build it.

Tahi (Aperta) had a dashboard (surprise!) and editor, just like the book production systems but I introduced two major innovations – Cards, and card-based workflow management interfaces. Unfortunately, while I was asked to come and build an open source system, things went a little weird at PLoS and they closed the repos, effectively making it a closed platform. So I quit. That also means I don’t have any screenshots to show you. Pity. If you sign an NDA with PLoS I believe they might show it to you.

However, you can picture it a little – imagine something like Trello, or Wekan – these are card based kanban systems. But imagine if you could custom make cards to do anything. Effectively cards were first class citizens of the platform and could access the db, perform system operations, make external calls, do validations, whatever you wanted. In hindsight, I think they were as close to an idea of an ‘app’ that you could have in a browser platform, although that wasn’t the way I thought about them at the time. Additionally, cards were imported into the system since each card was actually a gem file. This meant any publisher could custom make their own cards to do whatever they wanted and place them within the kanban-like workflow space (task manager). Pretty neat.

So, cards could be surfaced and used anywhere in the system. We used them for authors to enter submission data, but also for production staff to perform operations, for reviewers etc etc etc. They could also be placed on a kanban board to make a workflow. Cards could be moved around the workflow and deleted or new ones added at any time.

To manage all this my other idea was to let these cards flow through a TweetDeck-like interface. So you could sort cards, per role, per user, at volume.

Tahi essentially had four spaces – a dashboard, a submission page (which displayed the manuscript in an editor, and submission data could be entered through cards), a task manager (workflow for the article, using cards as tasks), and a ‘flow manager’ (the TweetDeck-like interface for sorting all your cards across all your articles). While the FLOSS Manuals, Booki and Booktype platforms were pretty much monolithic systems, the old PubSweet was sort of modular. However, Tahi did decouple the front end and back end but I wanted to also break these four spaces into discreet components. That would have given the system enormous flexibility but unfortunately I wasn’t able to do this before I left.

Anyways, Tahi/Aperta is a little old now but it was pretty cool. I don’t know what happened to Aperta but I believe it is now being used for PLoS Biology.

After I left PLoS I was offered a Fellowship by the Shuttleworth Foundation to continue on the mission to reform publishing. So I started Coko with Kristen Ratan (who was the publisher at PLoS)….

img_9224_792x445

So there are some themes from building the past 7 or 8 publishing systems (depending on how you count it… there were also some other interesting experiments in between). First, the next system you build is always better. That is for sure. It’s an important thing to realise because when I developed the FLOSS Manuals system I thought that was it. Nothing could be better! But I was wrong. Then Booki/Booktype and I felt the same thing. I was so proud of it and nothing could be better! haha… you get the picture. The reason why it’s important to understand this is because I think it gives someone like me a bit of freedom. I can take some risks with systems knowing you get some stuff right, you get some stuff wrong. But the next system will get that bit you got wrong, right. Taking this attitude also takes the pressure off and you can have more fun which is good for your health, the team you are working with, and the system.

As far as technical lessons learned… well… after looking back at all these systems when we started Coko, I realised that the idea of independent ‘spaces’ for publishing workflows had a heap of currency. How many systems did I have to build with baked in dashboards, task managers, editors, table of content managers, etc etc etc before I could realise it doesn’t make sense to do this over and over. I wanted to take the idea of these kinds of spaces forward and not have to build them again and again… so some kind of system where you could include whatever spaces/components you wanted would be ideal… This would have two very important side benefits:

  1. I could learn so much because if the next system you build is always better, what about a framework that would allow you to easily build a whole lot of systems at once! Or build a lot quickly over a short amount of time… just imagine how much you could learn…
  2. It would open the door for others to innovate. I have since given up the idea that my system (so to speak) was the best ever and no one could top it. That’s just the testosterone talking. I’m kinda over it (sorta). I want other people to be able to make better stuff than what I have produced so far, to bring in innovations I never thought of. I want to make that easy for them and now I understand a whole lot better how publishing workflows actually work I’m in a very good position to do that.

That was a lot of the thinking behind the new PubSweet – PubSweet 1.0. But there is some other stuff too. Through my time at PLoS, I came to understand just how many variables affect workflow choices in journal publishing and that each publisher has slightly different conditions and roles that affect this. That means that the access control is complex. We might think there are various roles – author, editor, reviewer etc that shepherd an article through a process but it’s not that simple. Any number of conditions can affect who gets to see or do what and when. So we need to have a very sophisticated way to set and manage this.

There was a lot of other stuff to take into account to but I mention these two specifically because recently when I was talking to Jure (lead PubSweet dev) about PubSweet 1.0 and reflecting on how far we came he nailed it, he identified the two major innovations of the system being:

  1. reusable/sharable components (spaces)
  2. attribute-based access control

I agree entirely. I think I might add another:

  • developer experience

It is pretty easy, and getting easier, for developers to develop publishing platforms/workflows (call them what you will) with PubSweet. I think it is pretty astonishing and I think these 3 characteristics put together enable us to build multiple publishing systems fast and in parallel (with small teams) as well opening the door for other to do the same and huge opportunities for innovation.

If we are successful at building community this will be a huge contribution to the publishing sector.

In a future post, I’ll break PubSweet spaces / components down in more detail. There were also a lot of other similar stories regarding technical innovations on the way (eg Objavi->iHat->INK), but I’ll break them down into posts on another day.

I meant to also talk about Editoria here, the monograph production system built on top of PubSweet, and xpub – the PubSweet-based Journal system.

2016-06-28_15-41-37

They are both pretty amazing and leverage so much more than the previous systems identified above.

Login page for our first Journal platform.

I think the main thing with them is that we are working extremely closely with publishers using the method I developed – the Cabbage Tree Method.

Editoria Design Session

This means that I am no longer involved in building, what I would call, naive publishing systems. Naive in the sense that publishers could use, for example, Booktype, but it’s not really built for publishers. It’s a general book production system built by someone who didn’t know much about publishing at the time. That’s great of course, there is a place for it. However, Editoria is not a naive system. It is designed by publishers for publishers and the difference is enormous.

But I will leave a longer rant about this for another post.

I do however, want to say that I didn’t, of course, build any of the above systems by myself. There were many people involved and I have credited them elsewhere in this blog. I’m not going to do another roll call here except for Jure Triglav.

Jure and I sat down just over 18 months ago to discuss some of the lessons I learned as explained above. We jammed it out over post-its, whiteboards, coffee, and food in Slovenia and you can read a little more about that process in the PubSweet 2.0 RFC. But Jure trusted me, and I trusted him, and he took these ideas and, with a small team in very good speed, made them a reality. As a result, I think PubSweet is an exciting system and will only get better. Congratulations Jure, you deserve special thanks and recognition for the absolutely amazing job you have done.

dsc00516-1

Flash..aaaahaaaaaaaaaa

The Shuttleworth Foundation, of which I am a proud fellow, has a rather beautiful but under-known program they call ‘Flash Grants’. Twice a year they give each fellow (current and alumni) $5,000 USD to give to someone they think is doing good in the world. It is a great program.

My first Flash Grant I gave to Seth Vincent and the second to Zara Rahman. I have known Zara for a long time but I don’t know Seth and only followed his work remotely. He recently wrote up a report about what he did with the 5k and here it is. Seems like a pretty productive use of the money if you ask me. Awesome…

Building Book Production Platforms p2.

Amongst the core requirements for a book production platform are the source file format and the editor, and of course, these are intimately linked. The development team is usually faced with choosing the format first, then the editor.

Choosing a format

The choice is pretty much HTML? or not HTML?

Currently, HTML is the ruling choice of format for a web-based book production platform. HTML is native to the browser and has associated standards-compliant support, such as CSS and javascript. Inversely, not choosing HTML puts you in a bit of a hole and can create a lot of overhead.

It might be interesting to look back a little and learn from some others since there have already been projects in this space that started down non-HTML roads and then gave it up for HTML. Kathi Fletcher, originally the project manager and technical director for Connexions (now OpenStax) which built a custom XML editing environment for academic materials, later researched in-browser XML vs HTML editing environments for her Shuttleworth Foundation-funded OERPUB project. Kathi became convinced HTML was the way to go and did some great work on HTML editor usability with the Aloha HTML editor.

We have chosen to use HTML5 as the canonical format for open textbooks, because developers and tools are more plentiful for web technologies than XML technologies.

http://www.w3.org/2012/12/global-publisher/statements-of-interest/29-oerpub.html

The (closed source) O’Reilly Atlas platform also started with the complex AsciiDoc format (a form of markdown) and eventually awoke to the power of HTML in 2012.

HTML5-based authoring offers a streamlined production workflow for producing both print and digital outputs, facilitates “digital first” content development, and is a perfect fit for creating a WYSIWYG, web-based writing experience.

http://radar.oreilly.com/2013/09/html5-is-the-future-of-book-authorship.html

They then got an extra dose of religion and started a project called HTML Book which is a suggested ‘spec’ for a subset of HTML elements to be used in books.

So far I have not seen a book production platform travel the reverse direction, from HTML to something else. Instead, we are seeing more and more platforms start with, or change to, HTML as a source file format.

Markdown

Markdown is sometimes put forward as the way to go but I’m not going to go into that in too much detail here. I have talked about this elsewhere. The only additional thing I will say is that markdown causes even more issues for book production platforms than those included in that article. Namely, in an in-browser markdown environment, the markdown will most likely be displayed as rendered HTML next to the authoring pane. That is a huge amount of lost screen space and extra UI junk for no apparent gain. Think of the UX cost. If you don’t have that rendered display then you will most likely only see pure markdown in a text field with no rendered display. The user won’t really know if their document looks right until it is rendered somewhere down the line, which is also a tremendous cost to the user for no apparent gain. Markdown: all pain, no gain.

NB: There is a possible good use case for markdown as a helpful add-on for HTML WYSI editors but I will cover that later.

LaTeX

There is a more valid use case for LaTeX in the browser since some scientists and academics will never use anything else, and you’ll never convince them to adopt HTML regardless of the benefits. You are up against the great Church of Knuth and I don’t fancy your chances. If your audience is comprised of LaTeX addicts, then I think you have no choice other than to support that.

Many times I have talked about remedies for unstructured MS Word documents (for scientific manuscripts) only to have someone earnestly comment that if everyone just learned LaTeX we would be in a much better position… They might be right, but I’m pretty sure it’s never going to happen.

The preference for LaTeX is a legacy issue, and problematic, but needs to be dealt with. (Unfortunately, today’s Markdown heroes are growing legacy issues like this with each passing day, and that is going to cost us down the road).

Recently there has been some interesting work on in-browser LaTeX editing including the (closed source) Authorea platform and, most notably the (open source) ShareLatex platform. ShareLatex round trips the LaTeX syntax displayed and edited in a text area (in the browser), renders that to a bitmap on the server, and returns it to the browser for a side-by-side ‘WYSIWYG’. The effect is that you can see a just-in-time rendered view of the LaTeX as you type. It’s a neat trick and effective if you insist on LaTeX in a web-based platform. Then you just have to live with the UI costs. However ,you only need this approach if you wish to support the full LaTeX syntax. If you wish to just support LaTeX equations, you can use an HTML editor with a LaTeX plugin based on MathJax or the Khan Academies KaTeX(and there are some other solutions such as Mathoid).

Incidentally, if you need to support full LaTeX I highly recommend checking out ShareLaTeX over WriteLaTeX. They both have the same approach but WriteLaTeX is proprietary whereas you can pick up the ShareLaTeX code and integrate it straight away. You could even build your own ShareLaTeX-like interface, it’s not too tricky – together with a colleague – Rizwan Reza – and I (Riz did all the hard work) we managed to develop a workable prototype in about 2 days, but there are many gotchas setting up the LaTeX compiler correctly.

Not many book projects need LaTeX, so I will leave this as an interesting edge case. There are solutions if you need it, but not many people need it.

XML

I think I will just leave it to the words of the brilliant Dave Cramer (Hachette Book Group):

So we’ve chosen to describe our content with HTML, and build our production system around HTML.

When I tell people that, they smile condescendingly, and chuckle a bit. “That’s cute. Why don’t you use real XML?”

I then ask them what you can do in Docbook (or TEI, or NLM) that you can’t do in XHTML? I haven’t heard a good answer to that question yet. XHTML is XML, by definition. Calling something “para” rather than “p” doesn’t get you anything, except carpal tunnel syndrome and invoices from consultants

The problem with non-HTML XML is that it is essentially just XML the browser can’t use. Hence you lose all that other good stuff like WYSI editors, CSS design tools, cool tricks with JavaScript, and all the cool tools that are being developed for HTML. XML just can’t compete, plus you are going to need to convert the XML into HTML anyway. So don’t make life more complicated than it already is – continue your love affair with XML as long as it’s XHTML!

HTML

HTML is king in the browser and it gives you all you need to make books. I don’t want to spend a lot of time arguing the merits of HTML in this post as there is a lot to say and I want to bring that in at other points of the conversation. But in brief:

  • HTML is supported by JS and CSS.
  • The DOM is known natively by the browser.
  • HTML is standards-based.
  • It is straightforward.
  • HTML is easy to read and easy to clean.
  • HTML is the most popular file format on the planet.
  • You can use HTML to build structure in documents with assigned class and id values, or microdata formats.
  • HTML is the native file format for EPUB.
  • PDF can be rendered directly from HTML in the browser (more on this later).
  • HTML can be paginated in the browser.
  • CSS is moving towards supporting more and more page based elements.
  • The browser can act as a design environment.
  • You can create real what-you-see-is (WYSI) production environments.
  • Basic editing is built into the format itself.
  • HTML is supported by an enormous number of tools for conversion (in and out).
  • HTML is supported by an enormous repository of examples (the web).
  • HTML is cheap to develop with.
  • Even book designers are getting used to it.
  • Some schools teach it.
  • It has a million free tutorials online to help you use it.
  • A lot of people know HTML.
  • HTML is supported by a rapidly proliferating body of JavaScripts for typography, graph production, animation, interactions, dynamic rendering etc etc etc etc

The basic idea really comes down to this.

  • HTML is the cheapest format of our time.
  • HTML is the most popular format of our time.
  • HTML is the networked document format of our time.

Increasingly HTML is the way stories are told, whether that is in books or on the web. It’s a trite analogy perhaps, but HTML is the paper of our time. As Dave Cramer says:

why start with something other than HTML, when you have to turn it into HTML anyway?

It should be noted that Cramer also turns HTML into paper, and the Hachette Book Group have produced many beautiful paper books using HTML as the source format. Many of these books you will now find in the best-selling sections of your local brick and mortar bookstore.

Other print producers are also using HTML as the source. Print-on-demand services, used to producing very ugly books by ingesting MS Word and dealing with all that ugly conversion, are also adopting HTML production environments. Books on Demand, Germany’s largest Print on Demand service, adopted Booktype so their customers could have an easy in-browser book production environment. The source format is HTML but the users don’t know that, and the books look better. That’s the beauty of HTML.

Finally, helped a lot by the efforts of Dave Cramer and the Hachette Book Group, Sourcefabric, the people at O’Reilly, and others adopting HTML, we might be starting to see the very beginning of the changing of the guard.

HTML is the way to go for Book Production Platforms. If you choose another format you will find you inherit a lot of costs and additional overheads and, sadly, you will soon be left behind. There is just no format going forward at the same speed as HTML. Not even close. So, my advice is to first ask the question – can HTML do what you need? Push your team to answer that question. Will format X give you anything HTML can’t? As an exercise ask your team to prove HTML is a bad choice, and if the answer is not-HTML, then contact me and let me try and talk you into it!

Annotate Me

Annotation is an interesting world. It has survived the many changes in book technologies until, interestingly enough, the net. It’s not that we have never needed it, it’s that we haven’t been able to do a good job of it. There have been some good attempts – CommentPress was one by Bob Stein and the Future of the Book Institute. Comment Press was useful, I installed it myself and used it – it was built on top of WordPress. But Bob and crew learned their lessons and improved the idea with the yet-to-be released Social Book.

On top of that has been Purple Numbers (by Douglas Engelbart – you know! the guy that invented the mouse!) , and the code known as Marginalia, and there have been three of four attempts using Jquery to get this right. While Marginalia did get included into Moodle, which is pretty cool, it didn’t really take off and none of the other attempts got anywhere.

I think that might be about to change with a very nice relatively new project called AnnotateIt. It is built on top of Jquery and is built by the crew behind the Open Knowledge Foundation, which in turn has been supported extensively by the Shuttleworth Foundation.

It’s good stuff. Very simple to use as either a free and centralised service, or you can establish your own annotation server. I am trialling it at the moment wíth FLOSS Manuals. You can create an account at http://annotateit.org/

And then try it on FLOSS Manuals. For example:

http://booki.flossmanuals.net/a-webpage-is-a-book/_draft/

Looking forward to trying it out. Any feedback on both the book and the annotation tool is very much welcomed.