What is xPub?

So, there are a lot of product names at Coko – PubSweet, Editoria, XSweet, INK, xPub etc etc etc… so becoming tricky to track, but I wanted it seems there are quite a few people interested in xPub right now.

xPub is not really a product as such, it’s more a group of products – each of which relates to Journal workflows. The names for each product indicyae those relationships: xpub-collabra, xpub-elife, xpub-faraday (Hindawi) and xpub-epmc

Each of these is a platform, so yip, you read it right – there are actually no less than 4 journal platforms being developed in the Coko community, not including the Micropublications platforms (of which there are two – one developed with Wormbase, and the other with the Organisation for Human Brain Mapping).

So, right from the get go our ‘offer’ is not standard. We don’t offer one platform to rule them all – there are many journal platforms in production. All of these are built on top of PubSweet… PubSweet is a kind of ‘headless’ publishing platform….it’s more or less the backend brains in that it is a kind of framework that ‘thinks like a publishing system’ but has not determined a workflow for you. So, each of the xpub-* platforms are actually publishing platforms built to meet a specific workflow and built on top of PubSweet…

Here is a crappy diagram (drawn in haste) to get the basic idea across:

ya

In the above, you see a PubSweet platform (eg. xpub-elife). PubSweet itself is the whole app – it sort of ‘encapsulates’ everything. Really though, you get the PubSweet core and then extend it with front-end components to meet your workflow needs. A typical front-end component might be a login screen, or a dash, a submission info page, a reviewer form etc.

You can also extend PubSweet on the back end as well (eg to enable integration with external services etc). The following is a more accurate but slightly techy architectural overview (you can skip this image and the following paragraph if you want to avoid the slightly techie part of this article).

ps2

In case you are wondering – everything is written in Javascript. Why did we choose JS? It was a deliberate choice to go with a language that was prolific. Almost every dev around these days needs to know some Javascript (it’s the most popular language by far on Github), this makes finding a developer to work on your project is as easy as we could possibly make it. JS is also a phenomenal language these days. Fast, sophisticated and more than capable of supporting large-scale publishing requirements. I mean, if it’s good enough for Paypal, Netflix, Linkedin, Uber and ebay, then it is good enough for you.

So each PubSweet platform has its own collection of front and back end components to meet the workflow of someone’s dreams… the idea is that to achieve the platform of your dreams you can reuse what others have already built and then just build what isn’t already available. In a sense, you can ‘assemble’ your platform from existing parts.

c1e22edf10cd018c9d1a697c29fbd891

The nice thing about this is that each of the organisations building Journal platforms are sharing the following:

  1. the same back end/framework (PubSweet)
  2. various front (and back) end components
  3. lessons learned…

Each organisation has a vision of their ideal Journal workflow. They then design and build this on top of PubSweet, but as they do, they build various components (either page-based components such as a dashboard, or smaller UI components we call atoms and molecules) and they share these components with everyone. Hence, you should check the list of components before you start building in case the component you need already exists.

We have various agreed-upon ways to build and share components (see this as an example). These best practices are continuously evolving but you can read some of the latest discussions about this approach here – https://gitlab.coko.foundation/pubsweet/pubsweet/issues/408

Of course, all code is reusable in the Coko community because it’s all open source. The best practices are there to make the code easy to reuse.

All agreements as per above are made by consensus by the community. It is actually a pretty snappy process – don’t believe every crappy thing you read about how open source is built. Open source community processes can be elegant and fast, and the resulting code can beautiful. Coko is a good example of this – a community of professionals collaborating together to make fantastic open source software.

dsc01017

So… back to the xPub story. To make these decisions on how to share components etc, we have regular meetings with various workgroups (we keep the numbers in each group small so we can move fast), and we also have quite a few in-person meetings. Not only do we have PubSweet community meetings where all of these organisations meet, but we have various get-togethers on various topics if required. For example, we met in Cambridge recently to discuss Libero (the eLife delivery product), before that EBI there was an onboarding session in Athens, next month we have a special designers workgroup in-person meet, anbd so on. All this helps us keep in contact with each other, which helps build trust, but also turns out to lift energy levels and boost production. These meetings are fun.

dsc01022

Also at these meetings, we learn from each other. One of the big problems in this sector, and one of the reasons people ask ‘why are publishing platforms so hard to build?‘ (after a number of high profile failures), is that there has not been a focused effort to share experiences on how to build publishing platforms. So, that is what we are doing – at each meeting we talk about what we have learned, how we are thinking about things, and show each other what we have done. The last meeting, for example, each xPub platform gave a deep dive to everyone in the group in a process known as speed geeking (it wasn’t so speedy as each table had 25-30 mins to go through their platform).

dsc00903

dsc00944

So, xPub isn’t a single platform, it’s more the community that is building journal platforms on top of PubSweet and sharing learnings and components. It is a very cool thing.

As a community, we also produced a book entitled ‘PubSweet – how to build a publishing platform’. You can get this for free here – https://coko.foundation/books/

I can also send you a print copy (they look great!) if you send me a postal address.

dsc00851

The book covers many things – a whole lot of technical documentation for how to build and share components etc, as well as some information about to think about workflow and how to map this into a PubSweet system. Personally, I don’t think the technology is very hard when it comes to building platforms – we knew what we wanted from the beginning with PubSweet and went about and built it (not to downplay the extraordinary job Jure Triglav has done in leading this effort). But the real hard stuff is actually thinking about workflow – not many publishing orgs have had the opportunity to think about designing their workflow to meet exactly what they want (rather than shoe-horning it into an existing system), and so we have had to beat this track for other to follow. It has been quite a journey but the book pretty effectively outlines how to think about workflow (which of course is a process that can be accelerated using Workflow Sprints which is a process I designed to facilitate a publisher to design their own workflow and PubSweet platform in one day).

0e7a0345

In that book you can see some high level ‘architectures’ of each of the xPub platforms such as the following (xpub-collabora):

02-adam-workflow-v3

Or this (xpub-epmc):

ebi

So, you might be asking…. at what stage of development  are each of these platforms? … I’ll leave it to each of the teams to say exactly, but it has been shocking to see how fast things have come along. I mean, the xPub community only really got together for the first time mid-last year, and building started for most late last year and early this year. It looks like we will have a lot of platforms landing by the end of this year. In this sector that is lightning fast. EBI are particularly impressive – they started two months ago and are almost ready to go – if it weren’t for the fact we are all replacing the under-the-hood data model we designed together at the last PubSweet meet, they would be good to go already. They can achieve such speed because they are reusing a lot of the components that the other teams built before them (and because EBI are just very good 🙂 ).

I can speak a little more about xpub-collabra since that is the platform Coko is building for and with the Collabra Psychology Journal. It’s looking pretty nice. We have some work tidying up the UI, replacing the data model and a few other things, but it’s looking rather good. We are also putting some time into making it a generic platform since the Collabra workflow follows a fairly ‘plain vanilla’ model. So we are building in some management interfaces and various bits and pieces to make it more widely useful – for example, Giannis just built in a Submission Info builder – enabling a Journal admin to build their own submission forms. It requires some hand-holding to use right now, but we’ll shape it to be very usable by your general journal adminy-type. We also have to integrate ORCID and DOIs, extend the range of submission file types etc… but it’s pretty close.

Below is a video showing xpub-collabra in action. It is a a version from some months ago, but you can see the workflow pretty well in this demo.

I think most of the xpub community is going to offer their new platforms to the market in various ways. This will be interesting as there are very different approaches at play. Hindawi, for example, is looking to make their platform a multi-tenant platform, eLife will put JATS at the centre of the workflow etc… So look for more news on that also. For our part, Coko will offer xPub through a partnership with a hosting provider – probably with the same organisations we will work with for Editoria hosting services. Since everything we do is open source, we will also be supplying all the Docker, Helm and Kubernetes scripts so that you can set up your own commercial hosting service if that is your cup of tea (or you can extend your offering should you already be a hosting provider). Coko is pretty close to getting our first hosting partnership set up, so look for news of that soon!

One last thing – because of the modular nature of any platform built on top of PubSweet, it is possible to take any of the xPub platforms and customise it to meet your needs. No need to start building from zero. Additionally, the modularity means you can extend the systems with your own interesting new innovation – finally a place for innovation to call home in the publishing platform world….

So… there is a lot going on in the xPub world….we look quite different to the rest of the market because we are not building one platform – we have instead focused on building a community to support the development of multiple platform solutions for journal workflows. You can pick and choose which one you want, or build something else, reusing as much of the other systems as you can to reduce your development costs (we also spend a lot of time onboarding new folks and supporting them as much as we can).

But it’s not just about improving the game today – supporting the optimisation of workflows is one part of what we are trying to do, the other is to support future innovations.

If you want to know more, feel free to jump into the Coko community channel and chat – https://mattermost.coko.foundation

Come and join the party. We are happy to support you and happy to learn from you…. not-for-profit or commercial we don’t care, build a better journal platform on PubSweet than the rest of us and we’d be happy!  We are in it for the mission – come to talk to us! no preciousness here 🙂

The Awesome Paged.js

So, I’ve been pursuing this dream for many years… every since I started rendering books in the browser using an ad-hoc collection of tools around 10 years ago…. then I instigated the book.js project (which unfortunately died due to lack of browser support of CSS Regions), and now… paged.js…

Built by the talented trio of Julie Blanc, Fred Chasen and Julien Taquet – it’s all open source and modular… there is a lot to this story, but we’ll get to that. Full release in a few weeks, this is a sneak peak:

Paged.js – sneak peeks

This project is entirely funded by the awesome Shuttleworth Foundation.

Hanging in Athens

I’m going to try and be in Athens every month now to work with the team, setting dev priorities for the month (roadmaps are stored in our READMEs). Awesome bunch – doing great work and heaps of fun to hang out with. Can you believe we have four devs in this office and they account for 4 publishing platforms? (wax, micropubs, Editoria, xpub)… I like that we are punching above our weight by some considerable margin… awesome…

Some pics from the week. Geeking and graffiti from around the ‘hood…

dsc00337_small

dsc00316_small dsc00324_small dsc00333_small

CodeMirror Embedded in Wax

The Wax (a web based word processor we are developing) lead dev -Christos Kokosias – yesterday embedded another editor, in the editor. We need this for the PubSweet Book Sprint where we will be including code snippets in  the documentation we create. So we need a nice way to manage that… consequently Christos embedded the CodeMirror code editor into Wax. So it has things like syntax highlighting, line numbering, auto complete etc all built into the code blocks…its pretty cool… below is are two short vids Christos made to show it in action:

 

Pretty cool stuff and extremely useful for writing documentation that includes code…ask yourself… can MS Word or Google Docs do that? #opensourceisbetter

Editoria Vid

If you missed the webinar, the video is here (it will be available from the Coko site also)…starts 9min 30s in:

Also, we had 40 or so live questions during the Q&A. We answered them all and Julien decided to do a quick dive into Right to Left (RTL) support. He posted this screenshot with mixed RTL and LTR text to show it works out of the box (still more to test):

archmerge_2018-05-16_10-25-17

XSweet 1.0!

We have 1.0 of the docx -> HTML transformation tool XSweet out today! It also has a new site:

http://xsweet.coko.foundation/

XSweet is a finely crafted tool. It takes docx files, those horrible mangy MS Word files, and translates them into clean, lovely, HTML. XSweet is open source, modular, and very nicely done.

A huge tip of the hat to Wendell Piez (XML guru), and to Alex Theg. As geeky as it sounds, I loved watching these two chat about the issues they encountered making this software. The attention to detail was really unbelievable. Amazing work. XSweet is a finely crafted tool.

More info on Coko https://coko.foundation/announcing-xsweet-1-0/

If you want to do a deep dive into why I think this is important, I wrote this some time ago – https://www.adamhyde.net/typescript-redistributing-labor/

The Age of the Editor Framework

Once upon a time Substance.io was the only framework available for building editors. Now it seems we have entered a new age of editor frameworks… the leading contenders look to be:

  • ProseMirror – see this post featuring some examples plus check out this editor assembled by Atlassian.
  • CKEditor5 – once an embedded WYSIWYG style editor, it now offers an editor framework called CKEditor5. Some examples here.
  • And of course Slate and Draftjs. These two have been around for a while but I’m still getting a handle on them… their examples are a little clunky and I don’t have a sense yet as to how they compare to the above.

It is a new age. Whats interesting to me is that there is an emergent new category of software – editors built on top of editor frameworks… we don’t see many examples at this moment because it is a relatively new field – ProseMirror was 1.0 only late last year, the latest CKEditor5 1.0 beta was also released only a few months ago,  while Substance released a beta over a year ago (they are yet to release a stable 1.0).

But examples will come in time. I think we will see more community develop around some of these projects (ProseMirror seems to be the leading contender to me on this front), and some sophisticated apps built with these tools. The project that develops the following will start leading the field :

  1. a humming community around the libraries that provide feedback and can have input into the overall design
  2. a thriving community building apps using the libs and helping each other to do so
  3. exemplar applications (multiple) emerging out of (2)

We are watching and learning…Thanks to Jure for his research on this recently.

Some interesting stuff from Atlassian

Jure, PubSweet lead dev, found this very interesting editor built with ProseMirror and built by Atlassian:

https://atlaskit.atlassian.com/packages/editor/editor-core/example/full-page

It looks pretty good. The table editor in particular, while perhaps needing some UX love, is pretty awesome.

It is part of the Atlaskit, which is a psuedo open source design kit Atlassian released. I haven’t looked at it closely but large chunks, namely the styles, is closed source (see their styles license) but the rest uses the Apache licence. Their repo is here.

Also interesting is this blog post about their design process. Finally a project (albeit fringe open source) that talks about design workflow. Very interesting… I don’t like the ‘developerificiation’ of design (eg calling the process ‘DesignOps’) but I do like to see a project talking about design workflow. Awesome.

Designing for All

At Coko we are tackling the long unsolved problem of design in Open Source. I have written a bit about this elsewhere, and it is a hard problem to solve, mainly because it is not solved anywhere – there are no good community models for this and very little discourse about the issue.

Design, of course, covers many things. I am chiefly talking about UX, although we have also solved the problem of designing for users by getting users (I prefer to call them ‘use case specialists’) to design their own systems. But this is not about that. I want to talk just about community processes for UX that will yield common best practices, while also enabling each organisation to realise their desired aesthetics ie achieve the look and feel they want that best reflects their organisation. As reflected in the following screenshots of xpub-Hindawi, xpub-eLife, and xpub-collabra:

b

c

a

There are several mechanisms that we have employed to achieve this. The first is that we have ‘elevated’ design to the same level of value in the community as code. I say ‘elevated,’ not because we had to raise its value in the Coko community, but rather the estimation of the value of design in the open source community at large is very low. So, against this, as a baseline, at Coko we have chosen to value design more than in your typical open source project. Also, I don’t want that to imply there are other disciplines in the software development process that do not rank as highly as these two – documentation, feedback, testing and so on… these are all equally valued alongside design and code, but for now I want to focus just on UX design.

To ‘elevate’ design, it took just the simple step of stating it to be so. As the co-founder of this community I have some sway over the direction and form of the culture. So I cashed out some of this capital to make a very simple statement when I was facilitating a community meeting that we must value design as much as we value code. It was this simple statement that set a placeholder for building processes that put design in the value-center of the community.

Next, when there was a call for more processes to assist communication across the various orgs, I made sure we had two work groups which each consisted of one person from each collaborating organisation. These two foundational WorkGroups were the Dev WorkGroup and the Design WorkGroup.

The Design WorkGroup then went ahead and started discussing how we might contribute to a platform where each org could have the look and feel they wanted. It was a process of first understanding the problem and then working through possible solutions. It was necessarily like this, a process of discovery before implementation, because there are not very many established or discussed processes like this in the open source world that we could follow (as mentioned above). So we have to discover the problem, and then invent a solution.

The problem boils down to small levels of abstraction. Interfaces are, after all, made up of many small parts. The problem therefore resided in how to account for different stylistic approaches to, what we now call, atoms (see Brad Frost’s article on Atomic Design), and how atoms fit together to form ‘molecules’ ie. we needed a level of abstraction that would allow any org to have the look and feel  they wanted for an (eg) button, and also how that button was situated with other elements.

Julien Taquet (Coko) and Nick Duffield (eLife) met with Yannis Barlas (Coko) in Athens for three days to thrash this out.

image-2

They came up with an agreed common set of standardised CSS attributes per UI element, and associated reusable CSS variables that could be used across any UX element.

You can see these lists here…

  1. Color Variables
    https://gitlab.coko.foundation/pubsweet/design/issues/1
  2. Typography Variables
    https://gitlab.coko.foundation/pubsweet/design/issues/2
  3. Form Variables
    https://gitlab.coko.foundation/pubsweet/design/issues/3
  4. Spacing Variables
    https://gitlab.coko.foundation/pubsweet/design/issues/4

That is the simple starting point. In effect, this means that if any organisation designs a feature using UI elements (eg a login form) they can use these shared style names and populate them with their own values. This means another org’s stylesheet will apply equally well to inherited/reusable components as it does to the ones they build themselves.

Which seems simple: it is… but understanding the problem space, and where the commonality lies, was half the battle.

The group has now also identified a common set of UI elements that can be reused and are iterating through this list, designing them as they go. See https://gitlab.coko.foundation/pubsweet/design/issues/21

This is required because even the simplest UI element may have several contexts it has to account for. Take the example of a ‘simple’ button, one of the simplest UI elements of them all…. however, there are actually 5 different button states that must be accounted for – the default state, active, hover, pressed and disabled. In each case, the style variables must be applied appropriately. See here for more information – https://gitlab.coko.foundation/pubsweet/design/issues/19

So, it’s not quite as easy as it might first seem.. the good news is that the teams are very committed to working through these issues, as without doing this groundwork, components are really not reusable.

dsc05093

The result is you can grab components that another org has made, be it a page-based component (eg. Dashboard), or a molecule (eg a Staff Assignment dropdown) and plug it into your system and it will look like it is part of your system.

We are making good progress on all of this, including now integrating common best practices for accessibility into the design process. Also, since this groundwork has come quite far, new orgs joining the Coko community can benefit a lot from the work already done and pick up components from the shared/common UX stylebook and integrate them… pretty cool….

Coko and Open Source

We’ve tried to build software for other people before, but it wasn’t until we worked with the Coko Foundation that we realised how important a “community-first” approach is. The benefits from the conversations, before any code is even written, is obvious when you start writing and even more obvious when you see the quality of the finished product.
– Paul Shannon, Head of Technology, Elife

Paul Shannon is a great advocate for the way we work in the Coko community. eLife was one of the first to see the potential in the way we work, and I remember clearly sitting in a pub talking to Paul about it with Kristen. He was enthusiastic and genuinely supportive when we explained in straight forward terms that we were trying to establish *real* collaboration and not collaboration in name only which is unfortunately so often the case.

We work closely with Paul and the eLife team, as well as the Hindawi team, and in many cases we are each extending the others team. That is what close collaboration is all about. As Paul states:

I love the pleasant surprise when one of your collaborators solves a problem for you, or when you find a problem you thought was just unique to you being discussed elsewhere. For us, the main benefit is knowing that what we’re building is useful to others and is solving the important problems – things get built more quickly and more correctly which reduces the burden on everyone.

This is very true. In a community when you share the burden of a problem you have the luxury of bringing in a bunch of active minds who have a desire to solve the same thing but are not on your payroll. So, you can leverage expertise in areas that your team doesn’t necessarily have. Software development in publishing these days covers a huge scope – everything from workflow, to file formats, OAuth, to containerised deployments and much more. Its hard for any one team to have that scope of expertise. But we have so far found, even with our relatively small foundational group, that we have always been able to find someone in our community that knows a lot about any given problem space.

So, extending your own team and being able to reach out to others with specific expertise is a huge advantage of working in the Coko community. But it doesn’t stop there. It is often the case that someone will be able to improve on your approach to a specific problem, and sometimes it is also true that someone in the community will offer to build it since they have the same need.

dsc04921

This leads to, as Paul says, things getting built more ‘quickly and correctly’. I absolutely agree, as does Andrew Smeall of Hindawi who also collaborates with us:

[Coko] have marshalled a diverse group of publishers towards a shared goal. Together, we’ve implemented a radically open process of constructive collaboration, where each member benefits from the strengths of the others. The result has been rapid progress towards an elegant, maintainable product.
– Andrew Smeall, Head of Strategic Products, Hindawi

This is awesome stuff. Imagine not having to solve all the problems of building a publishing system by yourself?

Of course there is a cost to this. To communicate this I sometimes use the phrase ‘The cost of collaboration is conversation”. This is very true. In order to benefit from collaboration in an open source project, you have to be communicative. We have several mechanisms to help with this, the mattermost chat channel is one. If you visit the Coko GitLab (where the code is stored) you will also find it is very active with conversations in the commits and the issues (eg https://gitlab.coko.foundation/XSweet/XSweet/issues/106) . We also have a lightweight RFC process (eg https://gitlab.coko.foundation/pubsweet/pubsweet/issues/341), and we also have smaller WorkGroups for developers and designers. These WorkGroups consist of one person per collaborating organisation each meeting and is facilitated by Coko. This keeps the group small and the conversation focused, lowering the overhead for all.

The result of all this is that we can keep the overhead low, and produce better products than if we were working alone.  We can also produce them faster, learning from each other as we go. As Andrew also says:

An improvement for one becomes an improvement for all. Individual communities can focus on core areas of expertise — peer review, hosting, discovery — knowing that their innovations will improve the entire system.  The result is more creativity, a more diverse set of solutions, and, ultimately, faster progress.

There is a secondary benefit too… communities are in themselves motivating, as Paul Shannon says:

Having other people build components to solve the problems you are facing is a great benefit of a common infrastructure, but also knowing that what you’re building is useful to more than just your team really adds energy and purpose to our teams of developers and designers.

All in all, its hard to argue against this kind of value in collaboration. I’m not currently seeing any downside except (for me) lots of travel! But thats in my DNA now, so I can handle it….would be good though if there was better surf in London and Athens…

We will add more organisations to the mix as we go…some announcements coming shortly. But here too we are being careful. Its important that we add the right orgs, ones that have a willingness to put in as well as take out…we don’t want to host parasites… this ecosystem can only work if there is close communication and a dedication to working together. This requires trust and good faith and if an org enters this tight community with the intention of free riding they will soon find out they are not welcome. For this to work for you, you need also to work for it. It is not a one way street. If you can accept that and get into the right mode, you will benefit greatly AND you will have a great time… communities are also fun when done right and so far our community meets are super fascinating and filled with awesome folks. It is an honor and a joy to hang out with them and solve problems together.

dsc05118

One of the additional benefits, which I touch on above, is learning from each other. In the last community meet, we had a few weeks ago in London each of the three foundational orgs – Coko, eLife, Hindawi – presented where they were with their journal platforms built on top of PubSweet. Each had taken slightly different paths to solve similar problems. There was much overlap of course, but where I learned the most was where someone had tackled the same problem slightly differently…this was really the first time this had occurred in our community and it was tremendously exciting.

As it happens, there seem to be quite a few orgs wanting to jump into the community. So, from here, it is all about scaling gracefully. We will work out processes to manage and maintain this good faith and good will as we grow. The foundations are there, now it’s all about the mechanics of growth…