SQLite indexer
Installation
Environment variables
These environment variables need to be provided to the indexer to work:
Type | Variable | Meaning | Sample value (using anvil running on the host) |
---|---|---|---|
Required | RPC_HTTP_URL | The URL to access the blockchain using HTTP | http://host.docker.internal:8545 (opens in a new tab) (when running in Docker) |
Optional | RPC_WS_URL | The URL to access the blockchain using WebSocket | |
Optional | START_BLOCK | The block to start indexing from. The block in which the World contract was deployed is a good choice. | 1 |
Optional | DEBUG=mud:* | Turn on debugging | |
Optional | STORE_ADDRESS | Only index tables from this World | |
Optional | SQLITE_FILENAME | Name of database | indexer.db |
Optional | PORT | The port on which the indexer listens | 3001 (the default) |
Using npx
To run the indexer directly on your computer using SQLite:
-
Start a
World
to index. An easy way to do this is to use a TypeScript template in a separate command line window. -
Set
RPC_HTTP_URL
.export RPC_HTTP_URL=http://127.0.0.1:8545
-
Run the indexer. If necessary, install it first.
npx -y -p @latticexyz/store-indexer sqlite-indexer
Note: The indexer.db
is persistent if you stop and restart the indexer.
If that is not the desired behavior (for example, because you restarted the blockchain itself), delete it before starting the indexer.
Docker
The indexer Docker image is available on github (opens in a new tab).
There are several ways to provide the environment variables to docker run
:
- On the command line you can specify
-e <variable>=<value>
. You specify this after thedocker run
, but before the name of the image. - You can also write all the environment variables in a file and specify it using
--env-file
. You specify this after thedocker run
, but before the name of the image. - Both Docker Compose (opens in a new tab) and Kubernetes (opens in a new tab) have their own mechanisms for starting docker containers with environment variables.
The easiest way to test the indexer is to run the template as a world in a separate command-line window.
The command to start the indexer in SQLite mode is pnpm start:sqlite
.
To index an anvil
instance running to the host you can use this command.
docker run \
--platform linux/amd64 \
-e RPC_HTTP_URL=http://host.docker.internal:8545 \
-p 3001:3001 \
ghcr.io/latticexyz/store-indexer:latest \
pnpm start:sqlite
However, this creates a docker container with a state, the SQLite database file. If we start a new container with the same image and parameters, it is going to have to go back to the start of the blockchain, which depending on how long the blockchain has been in use may be a problem. We can solve this with volumes (opens in a new tab):
-
Create a docker volume for the SQLite database file.
docker volume create sqlite-db-file
-
Run the indexer container using the volume.
docker run \ --platform linux/amd64 \ -e RPC_HTTP_URL=http://host.docker.internal:8545 \ -e SQLITE_FILENAME=/dbase/indexer.db \ -v sqlite-db-file:/dbase \ -p 3001:3001 \ ghcr.io/latticexyz/store-indexer:latest \ pnpm start:sqlite
-
You can stop the docker container and restart it, or start a separate container using the same database.
-
When you are done, you have to delete the docker containers that used it before you can delete the volume. You can use these commands:
docker rm `docker ps -a --filter volume=sqlite-db-file -q` docker volume rm sqlite-db-file
Note: You should do this every time you restart the blockchain. Otherwise your index will include data from multiple blockchains, and make no sense.
Using the indexer
See here how to use the indexer with a MUD client.
Viewing events
You can run this command to verify the indexer is working correctly.
curl 'http://localhost:3001/api/logs?input=%7B%22chainId%22%3A31337%2C%22address%22%3A%220x8d8b6b8414e1e3dcfd4168561b9be6bd3bf6ec4b%22%2C%22filters%22%3A%5B%5D%7D' | jq
The result should be nicely formatted (and long) JSON output with all the data change events that happened in that World
.
Where does this URL come from?
The URL has these parameters:
Parameter | Value | Explanation |
---|---|---|
Server | http://localhost:3001 (opens in a new tab) | By default the indexer listens on port 3001 |
Path | /api/logs | Read log events |
input | %7B%22 ... %5D%7D | See below |
The input is the JSON filter that tells the server what we need. It is URL encoded (opens in a new tab), you can decode it using an online calculator (opens in a new tab).
{
"chainId": 31337,
"address": "0x8d8b6b8414e1e3dcfd4168561b9be6bd3bf6ec4b",
"filters": []
}
Meaning that the query is for all events in the World
at address 0x8d8b6b8414e1e3dcfd4168561b9be6bd3bf6ec4b
, on the chain with chain ID 31337
.
Reading database information
If you want to use the indexer database for other purposes, you can use sqlite3
(opens in a new tab).
-
Start
sqlite3
in read-only mode.sqlite3 -readonly indexer.db
-
Get the list of available tables.
.tables
-
In SQLite tables are named
<world address>__<namespace>__<name of table>
. For example, this is how you get the task descriptions when using the React template (opens in a new tab).select description from '0x8d8b6b8414e1e3dcfd4168561b9be6bd3bf6ec4b__app__tasks';
-
Exit from the command line interface.
.exit
Clearing the information
If you restart the blockchain, you need to clear all the information stored by the indexer otherwise you'll have an inconsistent state.
When using npx
Remove the database file (by default indexer.db
).
When using docker
Remove the docker volume:
docker volume rm sqlite-db-file