Use Firebase Functions to make your Firebase data searchable & deploy them with Glitch

This is the third post of a serie (first one is here, second here) going into some technical details about a migration project for G+ communities — if you want to retrieve G+ data, take a look at this tutorial.

Since I started the project to export G+ content to Firebase, several people asked for a way to search those exports (eg: display all posts containing the word “awesome”), which absolutely make sense but is in fact quite tricky with the databases provided by Firebase…

Searching all posts containing the word “gmail”

Firebase databases and full-text search

Firebase offers 2 not-so-different databases: The Realtime Database and Cloud Firestore:

• The Realtime Database is the main component of the original Firebase product that Google acquired in 2014.
• Cloud Firestore is an update + re-branding of the Google Cloud Datastore aimed at answering limitations of the Realtime Database (like scalability).

For this project, I decided to use the Realtime Database, simply because I have more experience with it and found it perfect for my needs. But then I realized that performing full-text searches on the Realtime Database was a bit tricky… so, is Firestore a better solution in this case? Not really, it suffers from the same limitation.

The recommended solution is to use a third-party search service like Algolia or ElasticSearch. But my goal was to offer a way for people to do exports of G+ content in their own Firebase project and this would add way too much complexity on setting up an export.

Using a Firebase Function to search the database on server-side

So I decided to try to rely on Firebase Functions to perform searches. Indeed, as stated in the Firebase documentation, “downloading an entire collection to search for fields client-side isn’t practical”: you don’t want to download the whole database each time a user wants to make a search. But does it make more sense if we do that on server-side?

The logic is quite simple:

1. When a user enters a search query we write it to the database
2. This triggers a Firebase Function that fetches all posts in the database and searches which posts match our query
3. Results are saved in the database and we can set a listener on client-side to retrieve and display those results

Reading the database from server-side is fast enough, it takes less than 10s to read 10K posts and present results to the user. Plus you can easily create a cache with Functions using a global variable so subsequent searches will be even faster (“If you declare a variable in global scope, its value can be reused in subsequent invocations without having to be recomputed.”).

let posts;
exports.fullTextSearchOnDatabase = functions
  .database.ref('/searchQueries/{pushId}/query')
  .onCreate(async (snapshot, context) => {
    
    if(!posts) {
      console.log("No cache - retrieve posts again");
      posts = await getAllPosts();
    }
    else {
      console.log("Using cache");
    }
...

Here’s the whole code of the function if you want to have a look. Also, not related, but great to be able to use async / await ❤️ in Functions (available since last summer with the Node.js 8 runtime).

Deploying Firebase Functions with Glitch

In my previous post I presented Glitch as a great way to deploy a web app on Firebase Hosting without having to install anything on a computer.

😭 Sadly the “Deploy to Firebase” button in Glitch only works to deploy Database rules & Hosting files. It will throw an error if you also try to deploy code for a Firebase function. I raised this issue with the Glitch support team, but for now, not possible to use it.

😍 Luckily, Glitch also comes with a full shell so you can use the Firebase CLI to deploy everything, including Firebase Functions, from Glitch, again without installing anything on your computer. Obviously it isn’t as user-friendly as the “Deploy to Firebase” button but it provides an easy — and fast — way for people to set up everything (no need to reinstall npm dependencies, the Firebase CLI, etc… as I’ve already done that in the Glitch project that people will copy / “remix”, here’s a quick tutorial showing all steps needed to deploy).

All Glitch projects come with a console to interact with the underlying project container

Some limitations

Of course the more data you have, the less this solution is viable. Because of the time needed to loop through all the content and return results to client side but also because of the price…

This solution can be 100% free as all Firebase products, even Functions come with a free quota. But all data downloaded by a function from the Firebase Realtime Database count toward the “GB downloaded” quota, priced at $1/GB once the free quota is exhausted. So, depending on the size of your database and the number of searches it can quickly cost you real money 💰.

For many use-cases, the Realtime Database is cheaper than Cloud Firestore, but here it might be different as Google doesn’t charge for data transfer between Firestore and Functions…