Ghost 1.0

Cool to see Ghost at 1.0. I like Ghost as a blogging system, not to use, but because I love seeing a system that has made some design decisions and stuck to them. In Ghost the takeaway design decisions are minimalism and markdown. They have seen that through to a harmonised, coherent, system and huge congratulations to them for that.

I haven’t changed my opinion on Markdown! but its great to see well made and well thought-through products in the open source world even if you disagree with some of the decisions. Congrats to the Ghost team!

Building Book Production Platforms p2.

Amongst the core requirements for a book production platform are the source file format and the editor, and of course, these are intimately linked. The development team is usually faced with choosing the format first, then the editor.

Choosing a format

The choice is pretty much HTML? or not HTML?

Currently, HTML is the ruling choice of format for a web-based book production platform. HTML is native to the browser and has associated standards-compliant support, such as CSS and javascript. Inversely, not choosing HTML puts you in a bit of a hole and can create a lot of overhead.

It might be interesting to look back a little and learn from some others since there have already been projects in this space that started down non-HTML roads and then gave it up for HTML. Kathi Fletcher, originally the project manager and technical director for Connexions (now OpenStax) which built a custom XML editing environment for academic materials, later researched in-browser XML vs HTML editing environments for her Shuttleworth Foundation-funded OERPUB project. Kathi became convinced HTML was the way to go and did some great work on HTML editor usability with the Aloha HTML editor.

We have chosen to use HTML5 as the canonical format for open textbooks, because developers and tools are more plentiful for web technologies than XML technologies.

http://www.w3.org/2012/12/global-publisher/statements-of-interest/29-oerpub.html

The (closed source) O’Reilly Atlas platform also started with the complex AsciiDoc format (a form of markdown) and eventually awoke to the power of HTML in 2012.

HTML5-based authoring offers a streamlined production workflow for producing both print and digital outputs, facilitates “digital first” content development, and is a perfect fit for creating a WYSIWYG, web-based writing experience.

http://radar.oreilly.com/2013/09/html5-is-the-future-of-book-authorship.html

They then got an extra dose of religion and started a project called HTML Book which is a suggested ‘spec’ for a subset of HTML elements to be used in books.

So far I have not seen a book production platform travel the reverse direction, from HTML to something else. Instead, we are seeing more and more platforms start with, or change to, HTML as a source file format.

Markdown

Markdown is sometimes put forward as the way to go but I’m not going to go into that in too much detail here. I have talked about this elsewhere. The only additional thing I will say is that markdown causes even more issues for book production platforms than those included in that article. Namely, in an in-browser markdown environment, the markdown will most likely be displayed as rendered HTML next to the authoring pane. That is a huge amount of lost screen space and extra UI junk for no apparent gain. Think of the UX cost. If you don’t have that rendered display then you will most likely only see pure markdown in a text field with no rendered display. The user won’t really know if their document looks right until it is rendered somewhere down the line, which is also a tremendous cost to the user for no apparent gain. Markdown: all pain, no gain.

NB: There is a possible good use case for markdown as a helpful add-on for HTML WYSI editors but I will cover that later.

LaTeX

There is a more valid use case for LaTeX in the browser since some scientists and academics will never use anything else, and you’ll never convince them to adopt HTML regardless of the benefits. You are up against the great Church of Knuth and I don’t fancy your chances. If your audience is comprised of LaTeX addicts, then I think you have no choice other than to support that.

Many times I have talked about remedies for unstructured MS Word documents (for scientific manuscripts) only to have someone earnestly comment that if everyone just learned LaTeX we would be in a much better position… They might be right, but I’m pretty sure it’s never going to happen.

The preference for LaTeX is a legacy issue, and problematic, but needs to be dealt with. (Unfortunately, today’s Markdown heroes are growing legacy issues like this with each passing day, and that is going to cost us down the road).

Recently there has been some interesting work on in-browser LaTeX editing including the (closed source) Authorea platform and, most notably the (open source) ShareLatex platform. ShareLatex round trips the LaTeX syntax displayed and edited in a text area (in the browser), renders that to a bitmap on the server, and returns it to the browser for a side-by-side ‘WYSIWYG’. The effect is that you can see a just-in-time rendered view of the LaTeX as you type. It’s a neat trick and effective if you insist on LaTeX in a web-based platform. Then you just have to live with the UI costs. However ,you only need this approach if you wish to support the full LaTeX syntax. If you wish to just support LaTeX equations, you can use an HTML editor with a LaTeX plugin based on MathJax or the Khan Academies KaTeX(and there are some other solutions such as Mathoid).

Incidentally, if you need to support full LaTeX I highly recommend checking out ShareLaTeX over WriteLaTeX. They both have the same approach but WriteLaTeX is proprietary whereas you can pick up the ShareLaTeX code and integrate it straight away. You could even build your own ShareLaTeX-like interface, it’s not too tricky – together with a colleague – Rizwan Reza – and I (Riz did all the hard work) we managed to develop a workable prototype in about 2 days, but there are many gotchas setting up the LaTeX compiler correctly.

Not many book projects need LaTeX, so I will leave this as an interesting edge case. There are solutions if you need it, but not many people need it.

XML

I think I will just leave it to the words of the brilliant Dave Cramer (Hachette Book Group):

So we’ve chosen to describe our content with HTML, and build our production system around HTML.

When I tell people that, they smile condescendingly, and chuckle a bit. “That’s cute. Why don’t you use real XML?”

I then ask them what you can do in Docbook (or TEI, or NLM) that you can’t do in XHTML? I haven’t heard a good answer to that question yet. XHTML is XML, by definition. Calling something “para” rather than “p” doesn’t get you anything, except carpal tunnel syndrome and invoices from consultants

The problem with non-HTML XML is that it is essentially just XML the browser can’t use. Hence you lose all that other good stuff like WYSI editors, CSS design tools, cool tricks with JavaScript, and all the cool tools that are being developed for HTML. XML just can’t compete, plus you are going to need to convert the XML into HTML anyway. So don’t make life more complicated than it already is – continue your love affair with XML as long as it’s XHTML!

HTML

HTML is king in the browser and it gives you all you need to make books. I don’t want to spend a lot of time arguing the merits of HTML in this post as there is a lot to say and I want to bring that in at other points of the conversation. But in brief:

  • HTML is supported by JS and CSS.
  • The DOM is known natively by the browser.
  • HTML is standards-based.
  • It is straightforward.
  • HTML is easy to read and easy to clean.
  • HTML is the most popular file format on the planet.
  • You can use HTML to build structure in documents with assigned class and id values, or microdata formats.
  • HTML is the native file format for EPUB.
  • PDF can be rendered directly from HTML in the browser (more on this later).
  • HTML can be paginated in the browser.
  • CSS is moving towards supporting more and more page based elements.
  • The browser can act as a design environment.
  • You can create real what-you-see-is (WYSI) production environments.
  • Basic editing is built into the format itself.
  • HTML is supported by an enormous number of tools for conversion (in and out).
  • HTML is supported by an enormous repository of examples (the web).
  • HTML is cheap to develop with.
  • Even book designers are getting used to it.
  • Some schools teach it.
  • It has a million free tutorials online to help you use it.
  • A lot of people know HTML.
  • HTML is supported by a rapidly proliferating body of JavaScripts for typography, graph production, animation, interactions, dynamic rendering etc etc etc etc

The basic idea really comes down to this.

  • HTML is the cheapest format of our time.
  • HTML is the most popular format of our time.
  • HTML is the networked document format of our time.

Increasingly HTML is the way stories are told, whether that is in books or on the web. It’s a trite analogy perhaps, but HTML is the paper of our time. As Dave Cramer says:

why start with something other than HTML, when you have to turn it into HTML anyway?

It should be noted that Cramer also turns HTML into paper, and the Hachette Book Group have produced many beautiful paper books using HTML as the source format. Many of these books you will now find in the best-selling sections of your local brick and mortar bookstore.

Other print producers are also using HTML as the source. Print-on-demand services, used to producing very ugly books by ingesting MS Word and dealing with all that ugly conversion, are also adopting HTML production environments. Books on Demand, Germany’s largest Print on Demand service, adopted Booktype so their customers could have an easy in-browser book production environment. The source format is HTML but the users don’t know that, and the books look better. That’s the beauty of HTML.

Finally, helped a lot by the efforts of Dave Cramer and the Hachette Book Group, Sourcefabric, the people at O’Reilly, and others adopting HTML, we might be starting to see the very beginning of the changing of the guard.

HTML is the way to go for Book Production Platforms. If you choose another format you will find you inherit a lot of costs and additional overheads and, sadly, you will soon be left behind. There is just no format going forward at the same speed as HTML. Not even close. So, my advice is to first ask the question – can HTML do what you need? Push your team to answer that question. Will format X give you anything HTML can’t? As an exercise ask your team to prove HTML is a bad choice, and if the answer is not-HTML, then contact me and let me try and talk you into it!

What’s Wrong with Markdown?

Markdown (.MD) is a text format that lazy people use to write HTML. Unfortunately once those same lazies are used to the format, their eyes glaze over and they start to believe .MD is the solution for all the world’s problems. They share a lot in common with githubbies who think github is the solution for book production, open source, bad democracies etc..

.MD files are common in the geek world. Programmers love them. The design of .MD is simple and efficient. If you know the syntax, you can write basic text documents with headers and bullet lists, blockquotes, bold & emphasis etc. pretty quick. That makes it a handy tool for the elite of text workers – programmers – to develop simple text documents, quickly. So it’s a popular format for writing, for example, human-readable README files that tell you a little bit about the software you are about to install. However, that is where the use case ends. .MD is not a useful format for many other cases unless you want to prove to the programmers that you too can do tricky stuff in plain text. For the rest of us, it has little value.

Markdown was originally developed by John Gruber in 2004 and you can read about some of the reasons why he developed it here. The original purpose of .MD is that it can be read without converting it to another format like HTML. Presumably, John Gruber wanted a format that could be read easily by the eye, allowing the user to be able to quickly understand which part of a text is a heading, which part is a list, which part is a paragraph etc .MD is designed not to be rendered for display, it is meant to be the display.

For example, a list written in .MD would look something like this:

* item 1 
* item 2
* item 3

That actually looks like a list. In HTML we would do something similar and it would look like this:

  • item 1
  • item 2
  • item 3

An asterisk looks like a bullet, so there is little cognisant drag here. Pretty readable.

However, things start to fall apart pretty fast. Can we really parse at speed, for example, nested bold and emphasis in .MD like this:

The quick *brown fox* **jumps** *over* the lazy **dog**  

Is that or the following easier to read?

The quick brown fox jumps over the lazy dog

I’m going to say the second is easier to read. Way easier to read. So, that is just the start of the problems; from here on in it goes downhill pretty quick for use cases beyond simple READMEs.

MarkDown isn’t designed for creating HTML

So, let’s assume we can agree Markdown is readable in a very limited number of scenarios – and move on to rationalise (grasp at straws) other needs for the format. That is pretty much where we are today, with the big selling point being that Markdown is an easy way to create HTML. But let’s face it, even programmers don’t like reading raw Markdown and even in the most popular .MD repository of them all – github – the Markdown files are rendered in the browser as nicely formatted HTML. Great! A use case we can stand behind – use Markdown to create HTML.

However .MD is really a pretty bad way to create HTML. Firstly, you need something to convert the .MD into HTML. So if you use just a plain text editor to create .MD files and load it into the browser you will see just plain, boring Markdown. No nicely formatted documents for you. There are tools that programmers like, and so the rest of us are also expected to like them, for converting .MD to .html. After all, according to the technically gifted, converting a .MD file to HTML is “really really easy.”

One of the most common tools for doing this is Pandoc. Pandoc is a great software and extremely useful. However having to install and learn how to run Pandoc – a complex tool at best – to convert a text file to something readable – sounds a little like the long way home. And  that’s not where the rot ends, far from it, the rot has only just set in and the worst is to come. If everyone was to use Pandoc to convert .MD to HTML we would have consistent conversion results. Unfortunately, that’s not what happens. Each to their own, and we have a lot of different tools with which to do these conversions, and hence we have different results created from the same source. Ugh. That is a file format nightmare right there.

And let’s say you want to add a little colour to your text. Perhaps a highlight? Forget about it. Markdown lacks the tools to enable you to do it. Pandoc might help, however – let’s add some colour highlights to a code block with this easy to remember command from the Pandoc manual:

pandoc code.text -s --highlight-style pygments -o example18a.html

So, first of all, do you know what a command is? Do you know what a terminal is? Happy using one? Oh..that’s actually not ok for you? No problems, there are plenty of online introductory courses on the command line. So before you write that funding document, “about” page on your website or scholarly research document – just whip through a quick course on the command line and you’ll be all set! (don’t forget to read the sections about installing software from the command line, you’ll need it to get Pandoc working).

Problems with conversion tools aside – Markdown struggles to find a nice way to represent HTML. It’s just a bad fit. Use Markdown for creating HTML and you will find all sorts of little formatting gotchas that will cause you frustration. It is why many markdown environments/conversion tools also support HTML tags.

All HTML is valid Markdown. If you’re stuck, not able to format your content as you would like (for example using tables), you can always use plain HTML instead of Markdown. http://support.ghost.org/markdown-guide/

So if you want to really write HTML with Markdown you have to, well, write HTML. Klaro.

Markdown was never intended for writing HTML. It wasn’t designed that way and for good reason – it doesn’t do it well.

Codified text

As mentioned above, by design, the original markdown has a very small subset of elements that can be converted to HTML. As John Gruber says in his philosophy:

Markdown is not a replacement for HTML, or even close to it…The idea for Markdown is to make it easy to read, write, and edit prose.

So, Markdown is not actually designed to be a good format for creating HTML. And it lives up to its design. It is for this reason that some Markdown formats ‘extend’ Markdown to include HTML code, and there are also other forks of Markdown that do some really weird stuff that I can hardly explain. For example, Ghost Markdown, the version of Markdown used for the (Open Source) Ghost blogging platform, tries to wrangle image formatting into Markdown. To place an image you have to write the following:

![]()

Intuitive, right? Nope.

The above is really a leap from ‘readable’ to ‘codified’. It is codified text and in order to be able to work with it, you need to know how to de-code the text… I’m sorry, but I just don’t get it. Markdown adds another level of codified complexity which I then need to de-code first (according to non-standardised, and not-standard rules written in some help file somewhere if I’m lucky), so that I can then sally forth and read the content? No thanks.

Say no to codified text.

Non-standardised formats suck

Efforts to take Markdown and extend it to meet a wider variety of formatting needs are actually where the big trouble starts. Markdown has gone off in a hundred-and-one different directions, each with its own syntax.

That means, if I want to write a Ghost blog (I love Ghost by the way, no disrespect to them) in Markdown (their required format) then it is not enough to learn Markdown. I must learn Ghost markdown …their particular reading of what a good markdown format is… So, that leads us to one of the really big problems. Markdown is not standardised.

Can any of us think of another non-standardised text format and where that leads us? Does MS Word and ‘world of pain’ ring any bells? Yes, Markdown is non-standardised and that is a very big no-no. It is, in fact, quite shocking that programmers, big on standards, do not quite see that by advocating Markdown they advocate dropping some central best practices. Can’t say anything more about that really.

But Markdown is structured!

I often hear the term “structured text” when referring to Markdown. For example, the opening lines from the CommonMark pre-amble.:

Markdown is a plain text format for writing structured documents

Sounds good doesn’t it? Sounds very techy and convincing. But what is structured text? Structured text means basically that we can see if something is a heading or something is a bulleted list, or something is a paragraph. Huh? But that describes just about any text document. Structured text is the basic requirement of any text you create – without it, you just have a flat plain-text document with no headings, no bullet lists etc. So… we might as well start every sentence about documents designed to be read as being ‘structured’. I think tomorrow I will go and buy a structured book. Or perhaps I will write a structured narrative on my text-structuring word processor. Excuse me word processor sales person, do you have structured text word processors? Ugh. Meaningless.

What is left?

Markdown is good for limited use cases. Use it for README files on github.

If you have a good dose of Markdown cool aid then don’t let me bring you down from your sugar high. Markdown away. However, if you have heard that there is this cool format available and it cures all your textual needs and it is just really really easy to use and really really quick… then think about the elixir you are being offered and re-read this document before slurping away…

Some comments on this article on Hacker News here:
https://news.ycombinator.com/item?id=8403783

MarkDown, ugh

MarkDown…lets call it what it is – A Less Useful MarkUp (ALUMU?)…We should also invent a new category for both ALUMU and MarkUp. I don’t have a good name for this superset but it can be characterised easily as “non-standardised, codified text”.

Why anyone would want to use non-standardised, codified text is very hard to understand. I thought we had learnt that mistake from MS Word.