The haskell-indexer package provides libs for preprocessing Haskell source code
into a representation for easy entity cross-referencing, as well as a frontend
for emitting entities for the Kythe indexing schema.

This is not an official Google product.

Supported systems

Indexing hosts:

Linux: supported - follow below documentation.
Windows, MacOS: didn't try - backend part likely compiles, wiring and Kythe
frontend likely not (see google#38).

Compilers:

GHC 8.6.5
GHC 8.8.1 (planned)

Stackage:

A recent LTS release corresponding to above compilers is supported.
See stack-ghcXXX.yml files.

Previous compilers were supported at some point. Checkout an old repository
state if interested:

Installation

Stack

Download Stack from http://docs.haskellstack.org

Kythe

If you want to use the Kythe frontend, you'll need to install it either from
source or from the official release. The latter is easier, but the web UI has
been removed in recent versions.

Official Release

Download a Kythe release and unpack
it.

tar xzf kythe-v0.0.26.tar.gz -C /opt/
rm -r /opt/kythe
ln -s /opt/kythe-v0.0.26 /opt/kythe
chmod -R 755 /opt/kythe/web/ui  # It misses permission by default.

Version v0.0.30 is the latest version that includes the web UI. If you want a
newer Kythe than this, you'll need to build from source.

If you want to install Kythe in a different location to /opt/kythe then you
should also set KYTHE_DIR to the location of the installation.

Building From Source

Clone Kythe from its GitHub repo and
follow the Getting Started guide to build
and install it into /opt/kythe. Then, from within the Kythe clone, build the
web frontend and copy its files into their rightful place:

bazel build //kythe/web/ui
mkdir -p /opt/kythe/web/ui
cp -r bazel-bin/kythe/web/ui/resources/public/* /opt/kythe/web/ui
cp -r kythe/web/ui/resources/public/* /opt/kythe/web/ui
chmod -R 755 /opt/kythe/web/ui

Protoc 3

Download the latest Proto compiler 3
release, unpack it and place the
binary in the PATH.

unzip -j protoc-*-linux-x86_64.zip bin/protoc -d /usr/local/bin/

If you use have Nix installed and you use stack --nix, you do not need to do
this.

Haskell Indexer Plugin (ghc >= 8.6 only)

Haskell modules can be indexed with a GHC source plugin while building a
project. Whatever build system is in use, indexing can be achieved by
ensuring that the invocations to ghc include the flags that enable the
plugin.

For instruction on how to install and use the plugin with stack, see
stack-example/README.md.

If you are using some other build system, the following GHC options are
relevant after the plugin is installed.

-package-db <db_path>: Tells the package database where the plugin has
been installed. It may be used more than once if the plugin dependencies
spread through more than one package database.
-plugin-package haskell-indexer-plugin: Tells ghc to expose the package
containing the plugin, so it can be found when needed.
-fplugin Haskell.Indexer.Plugin: Tells to use the plugin when compiling
modules.
-fplugin-opt Haskell.Indexer.Plugin:-o and
-fplugin-opt Haskell.Indexer.Plugin:<output_path>: Tell the plugin where
to place the output of indexing.

Build the project

Use the following to build and run tests:

git clone --recursive https://github.com/google/haskell-indexer.git
cd haskell-indexer
export STACK_YAML=$(readlink -f stack-ghc865.yaml)
stack build && stack test
# To test Kythe frontend:
pushd kythe-verification; stack install && ./test.sh; popd

To test all supported stack configurations, do ./run-ghc-tests.sh.

Demo

To index a few packages, run:

export INDEXER_OUTPUT_DIR=/tmp/indexer-output
./build-stack.sh mtlparse cpu

The script adds a wrapper for the GHC compiler used by Stack (stack path --compiler-exe), does the indexing when ghc --make is specified on the command line to build a package. You can run build-stack.sh multiple times.

To serve the index at http://localhost:8080:

./serve.sh localhost:8080

If you get empty index, look at $INDEXER_OUTPUT_DIR/*.stderr files about
possible indexing errors. Also, make sure that the *.entries files are not
empty. If they are, it indicates that ghc_kythe_wrapper failed to index.

Indexing using Docker

If you plan to use the Dockerized build feature of stack, please install
Docker. It is also advised to set up a docker wrapper script by following the
instructions at the stack Docker
security
section.

The docker image has all C library dependencies so it's possible to use it to
index the whole Stackage snapshot. See stack-build-docker.sh for a
comprehensive example of indexing a Stackage snapshot, and serving a Kythe
index.

jinwoo/haskell-indexer