Team blog: Developers

New screencast up

A lot has happened since our August 2016 screencast: Atom feeds, teams, file uploads, new language versions, and more. So I just recorded a new screencast (YouTube version) that walks you through some of the recent changes. Share & enjoy ;-)

I don’t work off a script for these, but I prepare a few things that I want to show and pick the best of multiple takes. That means quality isn’t perfect, but it keeps the effort at 1-2 hours for a 10 minute screencast, so we can do an updated one every couple of months or so.


Install lib.reviews using Vagrant

If you want to join lib.reviews development, you now have the option of using Vagrant to get a local development environment up and running. This is thanks to a pull request by Ori Livneh, who also pioneered the use of Vagrant at the Wikimedia Foundation.

Essentially, if you have a working installation of Vagrant and Virtualbox, getting a local install up and running is as simple as running the command vagrant up in a directory that contains a complete checkout of our GitHub repository. Give it a few minutes, and you’ll be able to access a fresh lib.reviews install locally at localhost:8080.

This is not necessarily much easier than following the installation instructions, especially if you’re familiar with Node.js and unfamiliar with Vagrant. The key benefit of using an environment like Vagrant is that the difficulty of getting a predictable local development environment up and running does not increase as the number of dependencies goes up. For example, if we end up adding an external message queue, or a search engine like ElasticSearch, Vagrant (if we update our configuration to reflect those additions) will install and configure those dependencies for you.

And although the environment is virtualized, the codebase is accessible from your host system, so you can use your favorite code editor without any performance penalty.

Thanks again to Ori for working on this change to make development more accessible!


Join our IRC channel to get real-time notifications of new reviews

Long before Slack, Skype, and Hipchat introduced real-time chat to countless work environments, IRC has been a part of the communications backbone of the Net, enabling people to share real-time news even of the collapse of the Soviet Union, or the first Gulf War. It continues to be a popular tool for open source developers, and we use the #lib.reviews channel on irc.freenode.net (which you can access through Freenode’s web gateway if you don’t want to use an IRC client) for occasional real-time discussion.

Thanks to a pull request by Ori Livneh, lib.reviews now has its first IRC bot, ReviewBot. ReviewBot posts real-time notifications of reviews as they are posted to the site. This is accomplished through a webhook using the node-webhooks module. Basically, anytime a review is posted, we can queue up POST requests to a number of services to let them know a review is live. So, in future, we can use this same infrastructure to hook into other chat systems, or let apps that rely on our data know about relevant updates.


Four new languages activated

Thanks to the efforts of translators NMaia, Gomoko, Jean-Frédéric, Wladek92, and
Vitorvicentevalente, the first additional lib.reviews language editions are now online: Esperanto, French, Brazilian Portuguese, and European Portuguese. Use the selector at the bottom to change your language. We will assume that any content you create is in the language you’ve selected.

Translations in Bengali, Welsh, Spanish, Finnish, Italian, Korean, Luxembourgish, Macedonian, Russian, Swedish, and Simplified Chinese have started. Of those, Swedish, Chinese, and Macedonian are more than 50% complete. You can see the latest statistics here. See our previous post for details on how to start a new translation or help finish an existing one.

Thanks again to the amazing translatewiki.net community for making this happen!


Want to see lib.reviews in your language? Help us translate :-)

From the beginning, one of the things that’s meant to set lib.reviews apart has been language support. Not only can you view the user interface in multiple languages, changing it also lets you write reviews in your language, or translate them. But I’m only bilingual, which is why we have only English and German so far.

You can change that now, by signing up for an account with translatewiki.net, the open source translation community behind the translation of the Wikipedia user interface. If you’re new to translatewiki.net, they ask you to make 20 test translations before they activate your account. This is to get quality translations and discourage spammers. I just timed it; it takes about 5 minutes, and you can use “Try another” to skip through a difficult message.

After your account is approved, visit this page and select your language in the top right to get started (see screenshot, language selector highlighted).

Screenshot of translation process

Run into any problems? Find us on #lib.reviews on irc.freenode.net or #mediawiki-i18n on irc.freenode.net for general issues with translatewiki.net, or tweet to us at @lib_reviews.

Technical background

I wanted us to be on translatewiki.net, because it’s both very successful and has a great existing translator community, and because it’s fully open source, unlike alternatives like Transifex. It’s a bit more DIY though, so we had to sort out some technical issues first. I submitted a couple of patches to the TranslateWiki platform to support the “.” in the project name, and to support the specific syntax for pluralization that our message format uses. These are now merged and the project is available for translation.


Teams now have feeds of reviews

You can now see all reviews associated with a given team, and subscribe to them with an Atom feed reader.

Example:

Teams are central to lib.reviews – they will help us organize topics/groups. They won’t be the only mechanism for this, but they combine two important qualities – trust (a team can have its own rules of admission) and subject-matter (a review can be focused on a single topic of interest) – in ways that I think are appealing.

To start a new team or join existing ones, visit https://lib.reviews/teams . You need to be trusted (write a few reviews) to start new teams.


You can now associate reviews with teams

A few days in the making, you can now associate any review you write with any teams you’re a member of. This is a central idea of lib.reviews and it will take a while to get it right. Think of teams like sub-reddits or Facebook groups, places where people who care about the same thing can organize their work.

Next up, to make this more useful, you’ll be able to get Atom feeds as well and see the latest reviews on a team’s page.


Uploads are live

If you’re a trusted user (which you should be if you’ve written a review or two), you can now upload image, sound and video files. To do so, visit a page about a review subject (the page you get when you click “More reviews and information”) and click the “Add media” button. You can add metadata for each upload, including a description and the license.

As with everything else, this is versioned data, and it can be translated, though this isn’t exposed through the user interface yet.

We don’t generate thumbnails yet, so if you want to embed images in reviews, it’s a very manual process for now. Once we create automatic thumbnails, you won’t have to worry about that part – we’ll just show whatever media you’ve uploaded alongside your review.

As always, please report any issues on GitHub.


File uploads and security

We’ll have basic file upload support (issue 55) live soon, though there’s lots of work still to do to make it really useful and user-friendly. What’s nice is that we’ll have support for batch uploads (more than one file at a time) from the start. A bit more technical detail follows, if you care.

We’re using the multer middleware for Express. It’s interesting to think about all the security issues with uploads. To begin with, ensuring basic cross-site request forgery token validation (CSRF) was a bit tricky. Because upload forms are encoded differently than regular forms, the token that protects against forged requests wasn’t available when it was needed.

The common recommendation is simply to move multer before the CSRF token check. But because multer itself saves the files, that would require all kinds of shenanigans to clean up the stored files if the check fails. Indeed, I suspect that lots of Node devs who follow this recommendation (multer is being downloaded >100,000 times per week) are vulnerable to uploads via CSRF, even though their checks dutifully report token errors! (See this comment for an explanation of the approach I’m using to avoid this.)

Additional complexity comes from multer’s processing of data streams. multer has a convenient function that lets me check whether files should be stored on the server or not. It can check against the MIME type reported by the client, but it doesn’t yet know the file’s contents, because the stream hasn’t finished yet at the time the check is performed.

Why do we need to look at the contents? Because a client might upload a malicious file disguised as an image, for example. Depending on the browser and operating system, that may create vulnerabilities for spreading malware. So to be safe, we’ll need to perform some basic inspections on the file later, and delete it if it’s not kosher. This isn’t implemented yet, which is a main reason I’ve not deployed the upload feature yet.

And there are other security concerns. Filenames are, of course, another vector for injection of HTML, for example – < and > are perfectly valid parts of a filename. We need to ensure that previously uploaded files don’t get accidentally overwritten. And we need to protect against denial of service attacks aiming to fill up the disk. The OWASP page on uploads is a great starting point to learn more about these and other security risks.


Fun with concurrency

As I mentioned previously, we’re using ava for running our tests. As I started adding more tests, earlier tests mysteriously began to break. ava runs all tests concurrently by default, so it tends to uncover issues you don’t see when you’re just testing things locally as a single user of your application. As frustrating as it can be, it is a good thing.

The particular issue was that strings sometimes didn’t show up in the expected language. English text was German, and German text was English. This issue would also have sometimes manifested in production under high concurrency situations, though I don’t know if any user ever saw it. The root cause was the way our internationalization (i18n) library was initialized.

We registered i18n helpers with our templating system on each page request, so that the i18n features are available to the template system but know about the current request’s language. Because the template system itself is initialized only once, under some timing conditions, a later request could interfere with an earlier request that hadn’t yet completed and mess up its language information.

This was tricky to track down, but should now be fixed – though we may well have introduced new bugs in the process. If something doesn’t look right, please be so kind to file a bug - thank you!


 Older blog posts