Full-text Japanese search with Hugo + Lunr

This article was machine-translated from the Japanese version.

What is Hugo?

Hugo is a static site generator framework built with Go.

What is Lunr?

Lunr is a text search engine built with JavaScript. It supports Japanese language search.¹

How to set up the search function

The following content is based on https://gist.github.com/sebz/efddfc8fdcb6b480f567 and https://gist.github.com/eddiewebb/735feb48f50f0ddd65ae5606a1cb41ae.

File structure

The file structure after implementation is as follows. Only the parts related to the search function are listed.

BLOG_DIR
├── layouts
│   ├── _default
│   │   └── index.json
│   └── index.html
└── static
    └── js
        ├── lunr.jp.js
        ├── lunr.js
        ├── lunr.multi.js
        ├── lunr.stemmer.support.js
        ├── search.js
        └── tinyseg.js

1. Setting up search data

Set up the data that Lunr will load for searching. Here, the titles, tags, categories, main text, and hrefs of all articles are exported in JSON format.

• layouts/_default/index.json

{{- $.Scratch.Add "index" slice -}}
{{- range .Site.RegularPages -}}
    {{- $.Scratch.Add "index" (dict "title" (or .Title (.Date.Format "2006/01/02")) "tags" .Params.tags "categories" .Params.categories "contents" .Plain "href" .URL ) -}}
{{- end -}}
{{- $.Scratch.Get "index" | jsonify -}}

The "title" (or .Title (.Date.Format "2006/01/02")) on line 3 is a custom specification for our site; normally, "title" .Title is sufficient.

Run hugo server -Dw to start the server and confirm that the target file is located at http://localhost:1313/index.json.

2. Installing the Lunr source code

Place the five files related to Lunr in static/js/.

You can find lunr.js at olivernn/lunr.js, while lunr.jp.js, lunr.multi.js, lunr.stemmer.support.js, and tinyseg.js are available at MihaiValentin/lunr-languages.

As an alternative to manual installation, you can use npm install lunr and npm install lunr-languages. Additionally, CDN data is available for lunr.js.² This article assumes a manual installation.

3. Installing the search script

Create static/js/search.js to utilise the scripts installed in step 2.

• static/js/search.js

var lunrIndex, $results, pagesIndex;

function initLunr() {
  $.getJSON("index.json").done(function(index) {
      pagesIndex = index;
      lunrIndex = lunr(function() {
        var lunrConfig = this;
        lunrConfig.use(lunr.multiLanguage('en', 'jp'));
        lunrConfig.ref("href");
        lunrConfig.field("title", { boost: 10 });
        lunrConfig.field("contents");
        pagesIndex.forEach(function(page) {
          lunrConfig.add(page);
        });
      });
    })
  .fail(function(jqxhr, textStatus, error) {
    var err = textStatus + ", " + error;
    console.error("Error getting Hugo index flie:", err);
  });
}

function search(){
  $results = $("#results");
  $results.empty();
  var query = document.getElementById('search-query').value;
  if (query.length < 2) {
    return;
  }
  renderResults(results(query));
}

function results(query) {
  return lunrIndex.search(`*${query}*`).map(function(result) {
    return pagesIndex.filter(function(page) {
      return page.href === result.ref;
    })[0];
  });
}

function renderResults(results) {
  if (!results.length) {
    $results.append('<p>No matches found</p>');
    return;
  }

  results.forEach(function(result) {
    var $result = $("<li>");
    $result.append($("<a>", {
      href: result.href,
      text: result.title
    }));
    if (result.contents.length <= 100) {
      $result.append($("<p>", {
        text: result.contents
      }));
    } else {
      $result.append($("<p>", {
        text: result.contents.slice(0, 100) + " ..."
      }));
    }
    $results.append($result);
  });
}

initLunr();

initLunr() loads the target language, index, and search string. Since specifying only ‘jp’ for the language won’t match numeric sequences or English strings, ’en’ is also specified.

search() reads the search query. If the query length is 2 or more, the search begins.

results(query) uses Lunr to search based on the query and returns information about the matching articles. Since Lunr only returns the index as a search result, the article content is retrieved from the search data in step 1 based on this result.

renderResults(results) appends the search results to the search page (described later) based on the output from results(query).

4. Search Page Generation

In this article, it is placed in index.html, but it can be anywhere.

• layouts/index.html

<script src="https://code.jquery.com/jquery-3.3.1.min.js"></script>
<script src="js/lunr.js"></script>
<script src="js/lunr.stemmer.support.js"></script>
<script src="js/tinyseg.js"></script>
<script src="js/lunr.jp.js"></script>
<script src="js/lunr.multi.js"></script>
<script src="{{ "js/search.js" | absURL }}"></script>

<form class="full-text-search-form" action="{{ .URL }}" onkeyup="search()">
  <input id="search-query" placeholder="Full Text Search" name="query" autocomplete="off" search autofocus/>
</form>

<ul id="results">

The first seven lines load jQuery along with the JS files set up in steps 2 and 3. jQuery is used within search.js.

The form is a text box for entering search terms, and search.js’s search() function is executed with every keystroke.

The ul is where the search results are displayed, and it is manipulated by search.js based on the results.

Finally, confirm that the search function is working on the search page at http://localhost:1313/index.html.