Mapping

2021-11-24 About 7 min

# Mapping

Mapping functions define how chain data is transformed into the optimised GraphQL entities that we have previously defined in the schema.graphql file.

  • Mappings are defined in the src/mappings directory and are exported as a function
  • These mappings are also exported in src/index.ts
  • The mappings files are reference in project.yaml under the mapping handlers.

There are three classes of mappings functions; Block handlers, Event Handlers, and Call Handlers.

# Block Handler

You can use block handlers to capture information each time a new block is attached to the Substrate chain, e.g. block number. To achieve this, a defined BlockHandler will be called once for every block.

import {SubstrateBlock} from "@subql/types";

export async function handleBlock(block: SubstrateBlock): Promise<void> {
    // Create a new StarterEntity with the block hash as it's ID
    const record = new starterEntity(block.block.header.hash.toString());
    record.field1 = block.block.header.number.toNumber();
    await record.save();
}
1
2
3
4
5
6
7
8

A SubstrateBlock (opens new window) is an extended interface type of signedBlock (opens new window), but also includes the specVersion and timestamp.

# Event Handler

You can use event handlers to capture information when certain events are included on a new block. The events that are part of the default Substrate runtime and a block may contain multiple events.

During the processing, the event handler will receive a substrate event as an argument with the event's typed inputs and outputs. Any type of event will trigger the mapping, allowing activity with the data source to be captured. You should use Mapping Filters in your manifest to filter events to reduce the time it takes to index data and improve mapping performance.

import {SubstrateEvent} from "@subql/types";

export async function handleEvent(event: SubstrateEvent): Promise<void> {
    const {event: {data: [account, balance]}} = event;
    // Retrieve the record by its ID
    const record = new starterEntity(event.extrinsic.block.block.header.hash.toString());
    record.field2 = account.toString();
    record.field3 = (balance as Balance).toBigInt();
    await record.save();
1
2
3
4
5
6
7
8
9

A SubstrateEvent (opens new window) is an extended interface type of the EventRecord (opens new window). Besides the event data, it also includes an id (the block to which this event belongs) and the extrinsic inside of this block.

# Call Handler

Call handlers are used when you want to capture information on certain substrate extrinsics.

export async function handleCall(extrinsic: SubstrateExtrinsic): Promise<void> {
    const record = new starterEntity(extrinsic.block.block.header.hash.toString());
    record.field4 = extrinsic.block.timestamp;
    await record.save();
}
1
2
3
4
5

The SubstrateExtrinsic (opens new window) extends GenericExtrinsic (opens new window). It is assigned an id (the block to which this extrinsic belongs) and provides an extrinsic property that extends the events among this block. Additionally, it records the success status of this extrinsic.

# Query States

Our goal is to cover all data sources for users for mapping handlers (more than just the three interface event types above). Therefore, we have exposed some of the @polkadot/api interfaces to increase capabilities.

These are the interfaces we currently support:

These are the interfaces we do NOT support currently:

  • api.tx.*
  • api.derive.*
  • api.query.<module>.<method>.at
  • api.query.<module>.<method>.entriesAt
  • api.query.<module>.<method>.entriesPaged
  • api.query.<module>.<method>.hash
  • api.query.<module>.<method>.keysAt
  • api.query.<module>.<method>.keysPaged
  • api.query.<module>.<method>.range
  • api.query.<module>.<method>.sizeAt

See an example of using this API in our validator-threshold (opens new window) example use case.

# RPC calls

We also support some API RPC methods that are remote calls that allow the mapping function to interact with the actual node, query, and submission. A core premise of SubQuery is that it's deterministic, and therefore, to keep the results consistent we only allow historical RPC calls.

Documents in JSON-RPC (opens new window) provide some methods that take BlockHash as an input parameter (e.g. at?: BlockHash), which are now permitted. We have also modified these methods to take the current indexing block hash by default.

// Let's say we are currently indexing a block with this hash number
const blockhash = `0x844047c4cf1719ba6d54891e92c071a41e3dfe789d064871148e9d41ef086f6a`;

// Original method has an optional input is block hash
const b1 = await api.rpc.chain.getBlock(blockhash);

// It will use the current block has by default like so
const b2 = await api.rpc.chain.getBlock();
1
2
3
4
5
6
7
8

# Modules and Libraries

To improve SubQuery's data processing capabilities, we have allowed some of the NodeJS's built-in modules for running mapping functions in the sandbox, and have allowed users to call third-party libraries.

Please note this is an experimental feature and you may encounter bugs or issues that may negatively impact your mapping functions. Please report any bugs you find by creating an issue in GitHub (opens new window).

# Built-in modules

Currently, we allow the following NodeJS modules: assert, buffer, crypto, util, and path.

Rather than importing the whole module, we recommend only importing the required method(s) that you need. Some methods in these modules may have dependencies that are unsupported and will fail on import.

import {hashMessage} from "ethers/lib/utils"; //Good way
import {utils} from "ethers" //Bad way

export async function handleCall(extrinsic: SubstrateExtrinsic): Promise<void> {
    const record = new starterEntity(extrinsic.block.block.header.hash.toString());
    record.field1 = hashMessage('Hello');
    await record.save();
}
1
2
3
4
5
6
7
8

# Third-party libraries

Due to the limitations of the virtual machine in our sandbox, currently, we only support third-party libraries written by CommonJS.

We also support a hybrid library like @polkadot/* that uses ESM as default. However, if any other libraries depend on any modules in ESM format, the virtual machine will NOT compile and return an error.

# Custom Substrate Chains

SubQuery can be used on any Substrate-based chain, not just Polkadot or Kusama.

You can use a custom Substrate-based chain and we provide tools to import types, interfaces, and additional methods automatically using @polkadot/typegen (opens new window).

In the following sections, we use our kitty example (opens new window) to explain the integration process.

# Preparation

Create a new directory api-interfaces under the project src folder to store all required and generated files. We also create an api-interfaces/kitties directory as we want to add decoration in the API from the kitties module.

# Metadata

We need metadata to generate the actual API endpoints. In the kitty example, we use an endpoint from a local testnet, and it provides additional types. Follow the steps in PolkadotJS metadata setup (opens new window) to retrieve a node's metadata from its HTTP endpoint.

curl -H "Content-Type: application/json" -d '{"id":"1", "jsonrpc":"2.0", "method": "state_getMetadata", "params":[]}' http://localhost:9933
1

or from its websocket endpoint with help from websocat (opens new window):

//Install the websocat
brew install websocat

//Get metadata
echo state_getMetadata | websocat 'ws://127.0.0.1:9944' --jsonrpc
1
2
3
4
5

Next, copy and paste the output to a JSON file. In our kitty example (opens new window), we have created api-interface/kitty.json.

# Type definitions

We assume that the user knows the specific types and RPC support from the chain, and it is defined in the Manifest.

Following types setup (opens new window), we create :

  • src/api-interfaces/definitions.ts - this exports all the sub-folder definitions
export { default as kitties } from './kitties/definitions';
1
  • src/api-interfaces/kitties/definitions.ts - type definitions for the kitties module
export default {
    // custom types
    types: {
        Address: "AccountId",
        LookupSource: "AccountId",
        KittyIndex: "u32",
        Kitty: "[u8; 16]"
    },
    // custom rpc : api.rpc.kitties.getKittyPrice
    rpc: {
        getKittyPrice:{
            description: 'Get Kitty price',
            params: [
                {
                    name: 'at',
                    type: 'BlockHash',
                    isHistoric: true,
                    isOptional: false
                },
                {
                    name: 'kittyIndex',
                    type: 'KittyIndex',
                    isOptional: false
                }
            ],
            type: 'Balance'
        }
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

# Packages

  • In the package.json file, make sure to add @polkadot/typegen as a development dependency and @polkadot/api as a regular dependency (ideally the same version). We also need ts-node as a development dependency to help us run the scripts.
  • We add scripts to run both types; generate:defs and metadata generate:meta generators (in that order, so metadata can use the types).

Here is a simplified version of package.json. Make sure in the scripts section the package name is correct and the directories are valid.

{
  "name": "kitty-birthinfo",
  "scripts": {
    "generate:defs": "ts-node --skip-project node_modules/.bin/polkadot-types-from-defs --package kitty-birthinfo/api-interfaces --input ./src/api-interfaces",
    "generate:meta": "ts-node --skip-project node_modules/.bin/polkadot-types-from-chain --package kitty-birthinfo/api-interfaces --endpoint ./src/api-interfaces/kitty.json --output ./src/api-interfaces --strict"
  },
  "dependencies": {
    "@polkadot/api": "^4.9.2"
  },
  "devDependencies": {
    "typescript": "^4.1.3",
    "@polkadot/typegen": "^4.9.2",
    "ts-node": "^8.6.2"
  }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# Type generation

Now that preparation is completed, we are ready to generate types and metadata. Run the commands below:

# Yarn to install new dependencies
yarn

# Generate types
yarn generate:defs
1
2
3
4
5

In each modules folder (eg /kitties), there should now be a generated types.ts that defines all interfaces from this modules' definitions, also a file index.ts that exports them all.

# Generate metadata
yarn generate:meta
1
2

This command will generate the metadata and a new api-augment for the APIs. As we don't want to use the built-in API, we will need to replace them by adding an explicit override in our tsconfig.json. After the updates, the paths in the config will look like this (without the comments):

{
  "compilerOptions": {
      // this is the package name we use (in the interface imports, --package for generators) */
      "kitty-birthinfo/*": ["src/*"],
      // here we replace the @polkadot/api augmentation with our own, generated from chain
      "@polkadot/api/augment": ["src/interfaces/augment-api.ts"],
      // replace the augmented types with our own, as generated from definitions
      "@polkadot/types/augment": ["src/interfaces/augment-types.ts"]
    }
}
1
2
3
4
5
6
7
8
9
10

# Usage

Now in the mapping function, we can show how the metadata and types actually decorate the API. The RPC endpoint will support the modules and methods we declared above. And to use custom rpc call, please see section Custom chain rpc calls

export async function kittyApiHandler(): Promise<void> {
    //return the KittyIndex type
    const nextKittyId = await api.query.kitties.nextKittyId();
    // return the Kitty type, input parameters types are AccountId and KittyIndex
    const allKitties  = await api.query.kitties.kitties('xxxxxxxxx',123)
    logger.info(`Next kitty id ${nextKittyId}`)
    //Custom rpc, set undefined to blockhash
    const kittyPrice = await api.rpc.kitties.getKittyPrice(undefined,nextKittyId);
}
1
2
3
4
5
6
7
8
9

If you wish to publish this project to our explorer, please include the generated files in src/api-interfaces.

# Custom chain rpc calls

To support customised chain RPC calls, we must manually inject RPC definitions for typesBundle, allowing per-spec configuration. You can define the typesBundle in the project.yml. And please remember only isHistoric type of calls are supported.

...
  types: {
    "KittyIndex": "u32",
    "Kitty": "[u8; 16]",
  }
  typesBundle: {
    spec: {
      chainname: {
        rpc: {
          kitties: {
            getKittyPrice:{
                description: string,
                params: [
                  {
                    name: 'at',
                    type: 'BlockHash',
                    isHistoric: true,
                    isOptional: false
                  },
                  {
                    name: 'kittyIndex',
                    type: 'KittyIndex',
                    isOptional: false
                  }
                ],
                type: "Balance",
            }
          }
        }
      }
    }
  }

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
Last update: November 24, 2021 02:27