Team blog: Developers

lib.reviews is now using Greenlock Express instead of the outdated auto-sni module to provide HTTPS access. As a user of the site you should not experience any difference, but SSL Labs reports an improved security grade for the site (from A- to A).

As part of routine systems maintenance, I’ve updated the following components:

• ElasticSearch to version 7.8 (which adds improved stemming for some languages, but required significant changes to the search code)

• Node to the 12.x LTS series

• Ubuntu on the server from the 16.04 LTS series to 18.04 LTS

There should be no significant user-facing impact of these changes, but if you notice anything no longer working as it did, please don’t hesitate to file an issue.

My first patch for Lib.Reviews went live earlier this month! A small ease of use feature for users. When you follow a link to a lib.reviews page and click sign in or register from that page, you will now be redirected to the page you came from. This feature will be very nice to have as lib.reviews grows and users become more likely to enter the site from arbitrary links.

The second feature went online last week. This patch ia a hidden ease of use feature for non-english users. Users can post links with a signupLanguage query to advertise registration for lib.reviews in any supported language. Say all your friends speak esperanto, you would send them lib.reviews/register?signupLanguage=eo so the website will show up in esperanto without having to search for the language drop down. This should help new users feel welcome right away.

Mastodon is one of the most exciting developments in open source of the last couple of years: a decentralized social network that is actually taking off. If you’re unfamiliar, here is a good concise introduction, and here is a more comprehensive one.

The super-short version is that, like email, Mastodon is spread across many servers that talk to each other. You can follow users on any server, and messages spread through the network as needed. Each server gets a limited view of the total network, based on who the users on that server are interacting with.

This isn’t a new idea: GNU Social and Diaspora are following a similar model (GNU Social and Mastodon are interoperable). But Mastodon is very user-friendly, development is very active, and a rich ecosystem of apps and communities is forming around it. Exciting times!

As far as lib.reviews is concerned, there’s now a quiet little account you can follow on Mastodon. Down the road, I’d love to explore whether we can make lib.reviews actually talk to the federated social media universe. For starters, it shouldn’t be too hard to make it possible to follow lib.reviews users on Mastodon, thanks to the ActivityPub standard. Help, as always, is most welcome.

Supporting OpenStreetMap properly will take many steps (integrated maps for picking review subjects, search by location, etc.), but a first small step is now complete: lib.reviews pulls the labels for review subjects directly from OpenStreetMap for any subject with an OSM way or node URL. For example, this restaurant has an OpenStreetMap link associated with it. If you click the link to OSM, you’ll notice that the name (“Hareburger”) is a property there. We are updating these names for items with OSM URLs every 24 hours.

Next up will be support for opening hours, which are already included for many businesses in OpenStreetMap, and which are an essential property for restaurant reviews.

lib.reviews has offered database downloads almost from the beginning, but until recently, these did not include uploaded media. As of last week, we are now creating an archive of all uploaded media files every week and offering it here. Note that in order to use these files in a license-compliant manner, you will also need the file metadata from the database dumps.

Thanks to user arx, our database downloads are now also mirrored to IPFS here (may take a while to load). IPFS is a cool project to build a distributed web and very much worth checking out.

If you’d like to run a mirror of some or all lib.reviews content, please go ahead—and please let us know through a pull request against this file. One of the huge benefits of free/open licensing is that we can all work together to ensure that the stuff we create doesn’t go away if one particular website does.

I’m investigating the possibility of turning lib.reviews into a platform co-op. The underlying idea is simple: users own the platform and vote democratically on all decisions. I’m a member of social.coop, which uses this model for a social network that is part of the larger Mastodon network (a good intro if you’re unfamiliar), and my experience with it has been very positive.

What would this mean? Nothing for the day-to-day use of the site. Folks who want to participate in decision-making could sign-up and contribute to costs for hosting and development, but also vote on decisions, e.g., which features to prioritize. There would probably be a free tier for co-op membership as well, just to make sure that active contributors can join up even if they can’t contribute financially.

If this is something you’d like to see, here’s a simple thing you can do: star lib.reviews on GitHub. I’d like to use OpenCollective to manage finances for the platform, and they require a threshold of 100 GitHub stars. That seems fair—there should be active user interest before it makes sense to set up a funding/governance model.

freeyourstuff.cc is a separate project from lib.reviews, but with a related purpose. It’s a Chrome/Chromium browser extension that lets you download reviews you’ve contributed to major websites, including Amazon, Goodreads, IMDB, TripAdivsor, and Yelp. You can publish them on the site as a backup, or keep your own copy.

The most popular feature turns out to be the Quora downloader: tens of thousands of Quora answers have been downloaded and re-published under free licenses with it (see the upload directory).

Today I pushed out a week’s worth of updates to the extension. From a user perspective, the main changes are that IMDB and Amazon extraction works again (design changes had caused the plugins to break), Quora extraction should be more well-behaved, and all plugins have a clear “busy” indicator when they’re doing stuff.

Under the hood, the extension now uses async/await functions instead of callbacks to make the download flow a lot more understandable, especially for complex plugins like the Quora one which have to monitor changes to a page dynamically made with JavaScript not under the plugin’s control.

If you’ve contributed reviews to other sites, I encourage you to use this extension to keep your own copy (and please report issues you experience). In future, we’ll make it easy to migrate individual reviews to lib.reviews or other sites, as well.

It’s now possible to upload and insert media files directly from the rich-text editor when writing reviews, blog posts, or anything else. This video is a quick demonstration:

This was a major effort for a few reasons:

• We needed to add an upload API that handles various failure cases (e.g., incorrect MIME type), batches up errors and passes them along to the application. The API supports multi-file uploads as well, but in the editor we only upload a single file at a time.

• We needed to design a dialog that’s quick and easy, while handling entry of all required data without taking up too much space for mobile users. The flip to a second page you see in the video seems like a pretty good solution—you only ever see that page if you need to.

• We needed to add an upload feed so we can keep track of what files are being uploaded.

As you can see in the video, the feature gives credit to the person who created the work you’re uploading, something that tends to go missing on most websites.

Along the way, we’ve also improved the old multi-file upload and the presentation of metadata on review subject pages. For now, you need to switch to “rich text” mode to see the upload button—in future, the plain text markdown editor will get its own toolbar.

We now have support for HTML5 video and audio in reviews and posts. You can insert media by URL using the “Insert media” menu in the rich text editor. Alternatively, you can use the markdown syntax for images — ![alt text](url) — and it will now work for video and audio files as well.

Here’s an example of an embedded audio file:

Keep in mind that this only works for links to files in formats supported by modern browsers (typically mp4/webm/ogv for video, mp3/ogg for audio). YouTube links and such won’t magically work. We may or may not add support for that—YouTube is ubiquitous, but it’s not ideal to have videos that may disappear at any moment embedded in reviews that are meant to be freely reusable forever.

Under the hood, the CommonMark markdown specification does not yet include support for video/audio. As a result, CommonMark compliant parsers like Markdown-It (which we use) don’t support it, either. There was a preexisting plugin, but it had a few issues:

• clocked in at >100KB due to an unnecessarily complex dependency

• did not show any fallback text for older browsers

• included hardcoded English strings

• did not tokenize audio and video differently from images, making it difficult to integrate with rich-text editors

While some of this may be fixed (I sent a pull request), I ended up writing and publishing a new module, markdown-it-html5-media, that is optimized for our use case. Using image syntax tracks the emerging consensus in the ongoing CommonMark discussion about this topic.