Team blog: Developers

Long before Slack, Skype, and Hipchat introduced real-time chat to countless work environments, IRC has been a part of the communications backbone of the Net, enabling people to share real-time news even of the collapse of the Soviet Union, or the first Gulf War. It continues to be a popular tool for open source developers, and we use the #lib.reviews channel on irc.freenode.net (which you can access through Freenode’s web gateway if you don’t want to use an IRC client) for occasional real-time discussion.

Thanks to a pull request by Ori Livneh, lib.reviews now has its first IRC bot, ReviewBot. ReviewBot posts real-time notifications of reviews as they are posted to the site. This is accomplished through a webhook using the node-webhooks module. Basically, anytime a review is posted, we can queue up POST requests to a number of services to let them know a review is live. So, in future, we can use this same infrastructure to hook into other chat systems, or let apps that rely on our data know about relevant updates.

Thanks to the efforts of translators NMaia, Gomoko, Jean-Frédéric, Wladek92, and
Vitorvicentevalente, the first additional lib.reviews language editions are now online: Esperanto, French, Brazilian Portuguese, and European Portuguese. Use the selector at the bottom to change your language. We will assume that any content you create is in the language you’ve selected.

Translations in Bengali, Welsh, Spanish, Finnish, Italian, Korean, Luxembourgish, Macedonian, Russian, Swedish, and Simplified Chinese have started. Of those, Swedish, Chinese, and Macedonian are more than 50% complete. You can see the latest statistics here. See our previous post for details on how to start a new translation or help finish an existing one.

Thanks again to the amazing translatewiki.net community for making this happen!

From the beginning, one of the things that’s meant to set lib.reviews apart has been language support. Not only can you view the user interface in multiple languages, changing it also lets you write reviews in your language, or translate them. But I’m only bilingual, which is why we have only English and German so far.

You can change that now, by signing up for an account with translatewiki.net, the open source translation community behind the translation of the Wikipedia user interface. If you’re new to translatewiki.net, they ask you to make 20 test translations before they activate your account. This is to get quality translations and discourage spammers. I just timed it; it takes about 5 minutes, and you can use “Try another” to skip through a difficult message.

After your account is approved, visit this page and select your language in the top right to get started (see screenshot, language selector highlighted).

Run into any problems? Find us on #lib.reviews on irc.freenode.net or #mediawiki-i18n on irc.freenode.net for general issues with translatewiki.net, or tweet to us at @lib_reviews.

Technical background

I wanted us to be on translatewiki.net, because it’s both very successful and has a great existing translator community, and because it’s fully open source, unlike alternatives like Transifex. It’s a bit more DIY though, so we had to sort out some technical issues first. I submitted a couple of patches to the TranslateWiki platform to support the “.” in the project name, and to support the specific syntax for pluralization that our message format uses. These are now merged and the project is available for translation.

You can now see all reviews associated with a given team, and subscribe to them with an Atom feed reader.

Example:

• All reviews by the Edu-by-Video team
• Atom feed of all these reviews

Teams are central to lib.reviews – they will help us organize topics/groups. They won’t be the only mechanism for this, but they combine two important qualities – trust (a team can have its own rules of admission) and subject-matter (a review can be focused on a single topic of interest) – in ways that I think are appealing.

To start a new team or join existing ones, visit https://lib.reviews/teams . You need to be trusted (write a few reviews) to start new teams.

A few days in the making, you can now associate any review you write with any teams you’re a member of. This is a central idea of lib.reviews and it will take a while to get it right. Think of teams like sub-reddits or Facebook groups, places where people who care about the same thing can organize their work.

Next up, to make this more useful, you’ll be able to get Atom feeds as well and see the latest reviews on a team’s page.

If you’re a trusted user (which you should be if you’ve written a review or two), you can now upload image, sound and video files. To do so, visit a page about a review subject (the page you get when you click “More reviews and information”) and click the “Add media” button. You can add metadata for each upload, including a description and the license.

As with everything else, this is versioned data, and it can be translated, though this isn’t exposed through the user interface yet.

We don’t generate thumbnails yet, so if you want to embed images in reviews, it’s a very manual process for now. Once we create automatic thumbnails, you won’t have to worry about that part – we’ll just show whatever media you’ve uploaded alongside your review.

As always, please report any issues on GitHub.

We’ll have basic file upload support (issue 55) live soon, though there’s lots of work still to do to make it really useful and user-friendly. What’s nice is that we’ll have support for batch uploads (more than one file at a time) from the start. A bit more technical detail follows, if you care.

We’re using the multer middleware for Express. It’s interesting to think about all the security issues with uploads. To begin with, ensuring basic cross-site request forgery token validation (CSRF) was a bit tricky. Because upload forms are encoded differently than regular forms, the token that protects against forged requests wasn’t available when it was needed.

The common recommendation is simply to move multer before the CSRF token check. But because multer itself saves the files, that would require all kinds of shenanigans to clean up the stored files if the check fails. Indeed, I suspect that lots of Node devs who follow this recommendation (multer is being downloaded >100,000 times per week) are vulnerable to uploads via CSRF, even though their checks dutifully report token errors! (See this comment for an explanation of the approach I’m using to avoid this.)

Additional complexity comes from multer’s processing of data streams. multer has a convenient function that lets me check whether files should be stored on the server or not. It can check against the MIME type reported by the client, but it doesn’t yet know the file’s contents, because the stream hasn’t finished yet at the time the check is performed.

Why do we need to look at the contents? Because a client might upload a malicious file disguised as an image, for example. Depending on the browser and operating system, that may create vulnerabilities for spreading malware. So to be safe, we’ll need to perform some basic inspections on the file later, and delete it if it’s not kosher. This isn’t implemented yet, which is a main reason I’ve not deployed the upload feature yet.

And there are other security concerns. Filenames are, of course, another vector for injection of HTML, for example – < and > are perfectly valid parts of a filename. We need to ensure that previously uploaded files don’t get accidentally overwritten. And we need to protect against denial of service attacks aiming to fill up the disk. The OWASP page on uploads is a great starting point to learn more about these and other security risks.

As I mentioned previously, we’re using ava for running our tests. As I started adding more tests, earlier tests mysteriously began to break. ava runs all tests concurrently by default, so it tends to uncover issues you don’t see when you’re just testing things locally as a single user of your application. As frustrating as it can be, it is a good thing.

The particular issue was that strings sometimes didn’t show up in the expected language. English text was German, and German text was English. This issue would also have sometimes manifested in production under high concurrency situations, though I don’t know if any user ever saw it. The root cause was the way our internationalization (i18n) library was initialized.

We registered i18n helpers with our templating system on each page request, so that the i18n features are available to the template system but know about the current request’s language. Because the template system itself is initialized only once, under some timing conditions, a later request could interfere with an earlier request that hadn’t yet completed and mess up its language information.

This was tricky to track down, but should now be fixed – though we may well have introduced new bugs in the process. If something doesn’t look right, please be so kind to file a bug - thank you!

The last couple of days have been spent getting our testing infrastructure into shape. We now have the ability to spin up multiple RethinkDB instances in parallel, each bootstrapped and populated with data by the application code, and to run asynchronous tests against them. This means we can run tests reasonably fast while maintaining a pristine environment for each test.

RethinkDB is an interesting database to work with. It’s built to scale as a distributed system, which is something we’re not taking any advantage of yet since we’re still tiny – but it still has a cost in development in terms of additional complexity. For instance, an operation like table creation can result in two tables with the same name and different unique IDs, which then results in application errors. Table and database readiness needs to be validated in order to make sure you can perform write or read queries. And so on.

You can imagine that multiple automated tests hammering multiple databases in parallel will uncover a lot of edge cases, some of which aren’t likely to pose problems in real world situations. But encountering some of the potential pitfalls during testing certainly makes it easier to reason about that class of problems.

I look forward to eventually tapping into RethinkDB’s real-time change feed features, which will be useful once we can tackle the event feed intended to replace the rather dull review feeds we currently show on the front page and on user pages.

For now, here’s what’s on the horizon for the next 1-2 weeks:

• More tests, which will be more fun to write now that we have some scaffolding in place
• Getting our i18n files ready for a translation platform, so we can invite users to expand the number of languages offered
• Polishing the frontpage a bit, to make it less cluttered
• Automating deployment and code reverts with pm2
• APIs, APIs, APIs
• Working through the f-droid issues.

As always, say hello on the #libreviews IRC channel on freenode.net and/or join our mailing list to get involved. :)

The world is pretty simple when you don’t have to worry about content re-use, accessibility, internationalization, and similar concerns. But the moment you prioritize those issues, even relatively simple features become complex.

I wanted to write a review with spoilers, so in the spirit of developing features the moment I need them as an active user of the site, I put something together that extends the markdown parser we use. It works as follows:

::: spoiler
Some content with spoilers
:::

produces:

::: nsfw
Some NSFW content
:::

produces:

::: warning Here there be dragons
Some dangerous content
:::

produces:

The choice of ::: isn’t arbitrary – it’s used by the markdown-it-container plugin which enables block-level extension to markdown-it that follow this pattern. We may use it in future for similar purposes with other block names.

Indeed, the help page for that package lists spoiler warnings as an example. But how you do them in a manner that 1) degrades gracefully, 2) is keyboard-accessible, 3) works in RSS readers and reviews obtained via API calls and inserted into someone else’s webpage?

The easiest choice would be to just expand the content if JavaScript is disabled, and otherwise collapse/expand it via JS. That may seem like a reasonable choice but it’s important to note that the use case 3) will often be affected – i.e., external content re-use will often show the collapsed content expanded, which is not desirable.

If you look around, you’ll find that there are a number of pure-CSS hacks to collapse/expand content. They work, but they rely on problematic choices, such as hidden checkbox inputs that toggle the collapse/expand state of content. Those are problematic because they mess with keyboard-accessibility.

In addition, CSS itself is not a reliable delivery method for content re-users. Pseudo-elements like :checked cannot be used in inline styles, and scoped stylesheets have been removed from the WHATWG standards and are supported only by Firefox.

Enter the <details> element. It is precisely intended to collapse or otherwise conceal some content by default, and to expand it on demand. Unfortunately, it’s not widely supported yet, but in browsers that do support it, this gets us a nice basic collapse/expand feature that works even without CSS or JavaScript! And where it’s not supported, the whole content is just displayed, not hidden.

But the look & feel is determined by the browser. So I added a bit of CSS and JavaScript that effectively overrides the built-in look and behavior when JavaScript is available (including the addition of a slide-in and slide-out animation) – and implicitly shims the feature for browsers that don’t have it at all.

That leaves only the case where 1) JavaScript is disabled, 2) the <details> element is not supported. In that case, the text will just be expanded by default. I tested this on a couple of mobile browsers and RSS readers, and it worked as expected.

Now what about i18n? markdown-ititself doesn’t have the notion of content language, but it has an environment that can be configured for each call of the render function that converts markdown into HTML. We pass the language into that environment, and our plugin picks the appropriate message to insert. For client-side live previews, we expose the necessary UI messages as JavaScript variables.

The message is ultimately stored as part of the content – which seems reasonable, given that it can be customized as in the “Here be dragons” example. But it has the side effect that if you switch the language of this post to German, the English warnings will still be English. A German review would have German messages. I think that’s an acceptable trade-off for UI text that really is part of the content. In any case, we retain the source code of all content, so we can re-render it in a different way in future if we want to.

Let me know if anything’s not working as it should (or file a bug). Check out the commit for the nitty gritty of how this was done.