├── .gitignore ├── README.md └── template ├── .env ├── .gitignore ├── abi └── README.md ├── commands.json ├── docker-compose.yml ├── package-lock.json ├── package.json ├── schema.graphql ├── squid.yaml ├── src ├── main.ts ├── model │ ├── generated │ │ ├── entity.model.ts │ │ ├── index.ts │ │ └── marshal.ts │ └── index.ts └── processor.ts └── tsconfig.json /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

2 | 3 | 4 | Subsquid Logo 5 | 6 |

7 | 8 | [![docs.rs](https://docs.rs/leptos/badge.svg)](https://docs.subsquid.io/) 9 | [![Discord](https://img.shields.io/discord/1031524867910148188?color=%237289DA&label=discord)](https://discord.gg/subsquid) 10 | 11 | [Website](https://subsquid.io) | [Docs](https://docs.subsquid.io/) | [Discord](https://discord.gg/subsquid) 12 | 13 | # Farcaster Subgraph migration 14 | 15 | This quest is to migrate the [Farcaster subgraph](https://github.com/Airstack-xyz/farcaster-subgraph) to Squid SDK. The resulting squid should match the GraphQL API of the subgraph as close as possible, by migrating `schema.graphql`. The judges reserve the right to request improvements afther the initial review of the submission. Reach out to the [Discord Channel]( https://discord.com/channels/857105545135390731/1155812879770058783) for any tech questions regarding this quest. Use ```template``` squid as a starter. 16 | 17 | # Quest Info 18 | 19 | | Category | Skill Level | Time required (hours) | Reward | Status | 20 | | ---------------- | ------------------------------------- | --------------------- | ------------------------------------- | ------ | 21 | | Squid Deployment | $\textcolor{green}{\textsf{Intermediate}}$ | ~10 | $\textcolor{red}{\textsf{1250tSQD}}$ | open | 22 | 23 | # Acceptance critera 24 | 25 | Ultimately, the solutions are accepted at the discretion of judges following a manual review. This sections is a rough guide that is in no way binding on our side. 26 | 27 | Some of the reasons why the solution will not be accepted include: 28 | - squid does not start 29 | - squid fails to sync fully due to internal errors 30 | - [batch handler filters](https://docs.subsquid.io/evm-indexing/configuration/caveats/) are not set up correctly (leads to a late sync failure in [RPC-ingesting](https://docs.subsquid.io/evm-indexing/evm-processor/#rpc-ingestion) squids) 31 | - data returned for any query is not consistent with subgraph data 32 | 33 | You may find [this tool](https://github.com/abernatskiy/compareGraphQL) to be useful for squid to subgraph API comparisons. 34 | 35 | It is desirable that your solution: 36 | - includes a suite of test GraphQL queries that touches every [schema entity](https://docs.subsquid.io/store/postgres/schema-file/entities/) and, if used, every [custom resolver](https://docs.subsquid.io/graphql-api/custom-resolvers/) at least once, with corresponding subgraph queries (listing in README is enough) 37 | - has high code quality (readability, simplicity, comments where necessary) 38 | - uses [batch processing](https://docs.subsquid.io/basics/batch-processing/) consistently 39 | - avoids any "sleeping bugs": logic errors that accidentally happen to not break the data 40 | - follows the standard squid startup procedure: 41 | ``` 42 | git clone 43 | cd 44 | npm ci 45 | sqd up 46 | sqd process & 47 | sqd serve 48 | ``` 49 | If it does not, describe your startup procedure in the README. 50 | 51 | Please test your solutions before submitting. We do allow some corrections, but judges' time is not limitless. 52 | 53 | To submit, invite the following github accounts to your private repo : [@dariaag](https://github.com/dariaag), [@belopash](https://github.com/belopash), [@abernatskiy](https://github.com/abernatskiy) and [@dzhelezov](https://github.com/dzhelezov). 54 | 55 | # Rewards 56 | 57 | tSQD rewards will be delivered via the [quests page](https://app.subsquid.io/quests) of Subsquid Cloud. Make sure you use the same GitHub handle to make a submission and when linking to that page. 58 | 59 | Winners will be listed at the quest repository README. If you do not wish to be listed please tell us that in an issue in your submission repo. 60 | 61 | # Useful links 62 | 63 | - [Quickstart](https://docs.subsquid.io/deploy-squid/quickstart/) 64 | - [TheGraph Migration guide](https://docs.subsquid.io/migrate/migrate-subgraph/) 65 | - [Cryptopunks Subgraph source code](https://github.com/itsjerryokolo/CryptoPunks) 66 | 67 | # Setup and Common errors 68 | 69 | 1. Install Node v16.x or newer [https://nodejs.org/en/download](https://nodejs.org/en/download) 70 | 2. Install Docker [https://docs.docker.com/engine/install/](https://docs.docker.com/engine/install/) 71 | 3. Install git [https://git-scm.com/book/en/v2/Getting-Started-Installing-Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) 72 | 4. Install Squid CLI 73 | 74 | ```bash 75 | npm i -g @subsquid/cli@latest 76 | ``` 77 | 78 | ## How to run a squid: 79 | 80 | Full startup procedure for newly developed squids: 81 | 82 | 1. Install dependecies: 83 | 84 | ```bash 85 | npm ci 86 | ``` 87 | 88 | 2. Generate model 89 | 90 | ```bash 91 | sqd codegen 92 | ``` 93 | 94 | 3. Generate types 95 | 96 | ```bash 97 | sqd typegen 98 | ``` 99 | 100 | 4. Build the squid 101 | 102 | ```bash 103 | sqd build 104 | ``` 105 | 106 | 5. Open docker and run: 107 | 108 | ```bash 109 | sqd up 110 | ``` 111 | 112 | 6. Generate migrations: 113 | 114 | ```bash 115 | sqd migration:generate 116 | ``` 117 | 118 | 7. Start processing: 119 | 120 | ```bash 121 | sqd process 122 | ``` 123 | 124 | 8. Start a local GraphQL server in a separate terminal: 125 | 126 | ```bash 127 | sqd serve 128 | ``` 129 | 130 | Types (`./src/abi`), models (`./src/model`) and migrations ('./db') are typically kept within squid repos after they become stable. Then the startup procedure simplifies to 131 | ```bash 132 | npm ci 133 | sqd up 134 | sqd process & 135 | sqd serve 136 | ``` 137 | 138 | ## Possible Errors: 139 | 140 | 1. Docker not installed 141 | 142 | ```bash 143 | X db Error × query-gateway Error 144 | Error response from daemon: Get "https://registry-1.docker.jo/v2/": uri ting to 127.0.0.1:8888: dial cp 127.0.0.1:8888: connectex: No connection 145 | ``` 146 | 147 | 2. Git not installed 148 | 149 | ```bash 150 | Error: Error: spawn git ENOENT 151 | at ChildProcess._handle.onexit (node: internal/child_process: 284:19) 152 | at onErrorNT (node: internal/child_process:477:16) 153 | at process.processTicksAndRejections (node: internal/process/task_queues:82:21) 154 | ``` 155 | 156 | 3. Dependencies not installed. Run `npm ci` 157 | 158 | ```bash 159 | sqd typegen 160 | TYPEGEN 161 | Error: spawn squid-evm-typegen ENOENT 162 | Code: ENOENT 163 | ``` 164 | 165 | 4. Rate-limiting. Get a private RPC endpoint from [any node provider](https://ethereumnodes.com), then change the `rpcUrl` in `processor.ts` 166 | 167 | ```bash 168 | will pause new requests for 20000ms {"rpcUrl":"https://rpc.ankr.com/eth", 169 | "reason" : "HttpError: got 429 from https://rpc.ankr.com/eth"} 170 | ``` 171 | If necessary, [rate limit your RPC queries](https://docs.subsquid.io/evm-indexing/configuration/initialization/#set-data-source). 172 | 173 | ## Best practices: 174 | 175 | 1. Batch saving 176 | ```bash 177 | let transfers: Transfers[] = []; 178 | ... 179 | ctx.store.save(transfers); 180 | ``` 181 | 182 | 2. Using map instead of array to avoid duplicate values 183 | ```bash 184 | let transfers: Map = new Map(); 185 | ... 186 | ctx.store.upsert([...transfers.values()]); 187 | ``` 188 | 3. Verify log addresses, not only topics. 189 | ```bash 190 | if (log.topics[0] === erc721.events.Transfer.topic && log.address === CONTRACT_ADDRESS) { 191 | ... 192 | } 193 | ``` 194 | -------------------------------------------------------------------------------- /template/.env: -------------------------------------------------------------------------------- 1 | DB_NAME=squid 2 | DB_PASS=squid 3 | DB_PORT=23798 4 | PROCESSOR_PROMETHEUS_PORT=3000 5 | GQL_PORT=4350 6 | -------------------------------------------------------------------------------- /template/.gitignore: -------------------------------------------------------------------------------- 1 | /node_modules 2 | /lib 3 | 4 | # IDE files 5 | /.idea 6 | pnpm-lock.yaml 7 | **/*.log -------------------------------------------------------------------------------- /template/abi/README.md: -------------------------------------------------------------------------------- 1 | # ABI folder 2 | 3 | This is a dedicated folder for ABI files. Place you contract ABI here and generate facade classes for type-safe decoding of the event, function data and contract state queries with 4 | 5 | ```sh 6 | sqd typegen 7 | ``` 8 | 9 | This `typegen` command is defined in `commands.json`. -------------------------------------------------------------------------------- /template/commands.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://cdn.subsquid.io/schemas/commands.json", 3 | "commands": { 4 | "clean": { 5 | "description": "delete all build artifacts", 6 | "cmd": ["npx", "--yes", "rimraf", "lib"] 7 | }, 8 | "build": { 9 | "description": "Build the squid project", 10 | "deps": ["clean"], 11 | "cmd": ["tsc"] 12 | }, 13 | "up": { 14 | "description": "Start a PG database", 15 | "cmd": ["docker", "compose", "up", "-d"] 16 | }, 17 | "down": { 18 | "description": "Drop a PG database", 19 | "cmd": ["docker", "compose", "down"] 20 | }, 21 | "migration:apply": { 22 | "description": "Apply the DB migrations", 23 | "cmd": ["squid-typeorm-migration", "apply"] 24 | }, 25 | "migration:generate": { 26 | "description": "Generate a DB migration matching the TypeORM entities", 27 | "deps": ["migration:clean"], 28 | "cmd": ["squid-typeorm-migration", "generate"], 29 | }, 30 | "migration:clean": { 31 | "description": "Clean the migrations folder", 32 | "cmd": ["npx", "--yes", "rimraf", "./db/migrations/*.js"], 33 | }, 34 | "migration": { 35 | "cmd": ["squid-typeorm-migration", "generate"], 36 | "hidden": true 37 | }, 38 | "codegen": { 39 | "description": "Generate TypeORM entities from the schema file", 40 | "cmd": ["squid-typeorm-codegen"] 41 | }, 42 | "typegen": { 43 | "description": "Generate data access classes for an ABI file(s) in the ./abi folder", 44 | "cmd": ["squid-evm-typegen", "./src/abi", {"glob": "./abi/*.json"}, "--multicall"] 45 | }, 46 | "process": { 47 | "description": "Load .env and start the squid processor", 48 | "deps": ["migration:apply"], 49 | "cmd": ["node", "--require=dotenv/config", "lib/main.js"] 50 | }, 51 | "process:prod": { 52 | "description": "Start the squid processor", 53 | "cmd": ["node", "lib/main.js"], 54 | "hidden": true 55 | }, 56 | "serve": { 57 | "description": "Start the GraphQL API server", 58 | "cmd": ["squid-graphql-server"] 59 | }, 60 | "serve:prod": { 61 | "description": "Start the GraphQL API server with caching and limits", 62 | "cmd": ["squid-graphql-server", 63 | "--dumb-cache", "in-memory", 64 | "--dumb-cache-ttl", "1000", 65 | "--dumb-cache-size", "100", 66 | "--dumb-cache-max-age", "1000" ] 67 | }, 68 | "check-updates": { 69 | "cmd": ["npx", "--yes", "npm-check-updates", "--filter=/subsquid/", "--upgrade"], 70 | "hidden": true 71 | }, 72 | "bump": { 73 | "description": "Bump @subsquid packages to the latest versions", 74 | "deps": ["check-updates"], 75 | "cmd": ["npm", "i", "-f"] 76 | }, 77 | "open": { 78 | "description": "Open a local browser window", 79 | "cmd": ["npx", "--yes", "opener"] 80 | } 81 | } 82 | } 83 | -------------------------------------------------------------------------------- /template/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | db: 5 | image: postgres:15 6 | environment: 7 | POSTGRES_DB: squid 8 | POSTGRES_PASSWORD: squid 9 | ports: 10 | - "${DB_PORT}:5432" 11 | # command: ["postgres", "-c", "log_statement=all"] 12 | -------------------------------------------------------------------------------- /template/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "squid", 3 | "version": "0.0.0", 4 | "private": true, 5 | "scripts": { 6 | "build": "rm -rf lib && tsc", 7 | "update": "npx npm-check-updates --filter /subsquid/ --upgrade && npm i -f" 8 | }, 9 | "dependencies": { 10 | "@subsquid/archive-registry": "^3.2.0", 11 | "@subsquid/evm-processor": "^1.8.2", 12 | "@subsquid/evm-typegen": "^3.2.2", 13 | "@subsquid/graphql-server": "^4.3.0", 14 | "@subsquid/http-client": "^1.3.0", 15 | "@subsquid/typeorm-migration": "^1.2.1", 16 | "@subsquid/typeorm-store": "^1.2.2", 17 | "dotenv": "^16.0.3", 18 | "ethers": "^6.3.0", 19 | "pg": "^8.8.0", 20 | "typeorm": "^0.3.11" 21 | }, 22 | "devDependencies": { 23 | "@subsquid/evm-typegen": "^3.2.2", 24 | "@subsquid/typeorm-codegen": "^1.3.1", 25 | "@types/node": "^18.11.18", 26 | "typescript": "^4.9.4" 27 | } 28 | } 29 | -------------------------------------------------------------------------------- /template/schema.graphql: -------------------------------------------------------------------------------- 1 | type Entity @entity { 2 | id: ID! 3 | } -------------------------------------------------------------------------------- /template/squid.yaml: -------------------------------------------------------------------------------- 1 | manifestVersion: subsquid.io/v0.1 2 | name: transactions-example 3 | version: 1 4 | description: |- 5 | Basic example of processing transactions 6 | build: 7 | 8 | deploy: 9 | addons: 10 | postgres: 11 | processor: 12 | cmd: [ "node", "lib/main" ] 13 | api: 14 | cmd: [ "npx", "squid-graphql-server", "--dumb-cache", "in-memory", "--dumb-cache-ttl", "1000", "--dumb-cache-size", "100", "--dumb-cache-max-age", "1000" ] -------------------------------------------------------------------------------- /template/src/main.ts: -------------------------------------------------------------------------------- 1 | import {TypeormDatabase} from '@subsquid/typeorm-store' 2 | import {processor} from './processor' 3 | 4 | processor.run(new TypeormDatabase({supportHotBlocks: true}), async (ctx) => { 5 | throw new Error(`Batch handler is not implemented`) 6 | }) 7 | -------------------------------------------------------------------------------- /template/src/model/generated/entity.model.ts: -------------------------------------------------------------------------------- 1 | import {Entity as Entity_, Column as Column_, PrimaryColumn as PrimaryColumn_} from "typeorm" 2 | 3 | @Entity_() 4 | export class Entity { 5 | constructor(props?: Partial) { 6 | Object.assign(this, props) 7 | } 8 | 9 | @PrimaryColumn_() 10 | id!: string 11 | } 12 | -------------------------------------------------------------------------------- /template/src/model/generated/index.ts: -------------------------------------------------------------------------------- 1 | export * from "./entity.model" 2 | -------------------------------------------------------------------------------- /template/src/model/generated/marshal.ts: -------------------------------------------------------------------------------- 1 | import assert from 'assert' 2 | 3 | 4 | export interface Marshal { 5 | fromJSON(value: unknown): T 6 | toJSON(value: T): S 7 | } 8 | 9 | 10 | export const string: Marshal = { 11 | fromJSON(value: unknown): string { 12 | assert(typeof value === 'string', 'invalid String') 13 | return value 14 | }, 15 | toJSON(value) { 16 | return value 17 | } 18 | } 19 | 20 | 21 | export const id = string 22 | 23 | 24 | export const int: Marshal = { 25 | fromJSON(value: unknown): number { 26 | assert(Number.isInteger(value), 'invalid Int') 27 | return value as number 28 | }, 29 | toJSON(value) { 30 | return value 31 | } 32 | } 33 | 34 | 35 | export const float: Marshal = { 36 | fromJSON(value: unknown): number { 37 | assert(typeof value === 'number', 'invalid Float') 38 | return value as number 39 | }, 40 | toJSON(value) { 41 | return value 42 | } 43 | } 44 | 45 | 46 | export const boolean: Marshal = { 47 | fromJSON(value: unknown): boolean { 48 | assert(typeof value === 'boolean', 'invalid Boolean') 49 | return value 50 | }, 51 | toJSON(value: boolean): boolean { 52 | return value 53 | } 54 | } 55 | 56 | 57 | export const bigint: Marshal = { 58 | fromJSON(value: unknown): bigint { 59 | assert(typeof value === 'string', 'invalid BigInt') 60 | return BigInt(value) 61 | }, 62 | toJSON(value: bigint): string { 63 | return value.toString() 64 | } 65 | } 66 | 67 | 68 | export const bigdecimal: Marshal = { 69 | fromJSON(value: unknown): bigint { 70 | assert(typeof value === 'string', 'invalid BigDecimal') 71 | return decimal.BigDecimal(value) 72 | }, 73 | toJSON(value: any): string { 74 | return value.toString() 75 | } 76 | } 77 | 78 | 79 | // credit - https://github.com/Urigo/graphql-scalars/blob/91b4ea8df891be8af7904cf84751930cc0c6613d/src/scalars/iso-date/validator.ts#L122 80 | const RFC_3339_REGEX = 81 | /^(\d{4}-(0[1-9]|1[012])-(0[1-9]|[12][0-9]|3[01])T([01][0-9]|2[0-3]):([0-5][0-9]):([0-5][0-9]|60))(\.\d{1,})?([Z])$/ 82 | 83 | 84 | function isIsoDateTimeString(s: string): boolean { 85 | return RFC_3339_REGEX.test(s) 86 | } 87 | 88 | 89 | export const datetime: Marshal = { 90 | fromJSON(value: unknown): Date { 91 | assert(typeof value === 'string', 'invalid DateTime') 92 | assert(isIsoDateTimeString(value), 'invalid DateTime') 93 | return new Date(value) 94 | }, 95 | toJSON(value: Date): string { 96 | return value.toISOString() 97 | } 98 | } 99 | 100 | 101 | export const bytes: Marshal = { 102 | fromJSON(value: unknown): Buffer { 103 | assert(typeof value === 'string', 'invalid Bytes') 104 | assert(value.length % 2 === 0, 'invalid Bytes') 105 | assert(/^0x[0-9a-f]+$/i.test(value), 'invalid Bytes') 106 | return Buffer.from(value.slice(2), 'hex') 107 | }, 108 | toJSON(value: Uint8Array): string { 109 | if (Buffer.isBuffer(value)) { 110 | return '0x' + value.toString('hex') 111 | } else { 112 | return '0x' + Buffer.from(value.buffer, value.byteOffset, value.byteLength).toString('hex') 113 | } 114 | } 115 | } 116 | 117 | 118 | export function fromList(list: unknown, f: (val: unknown) => T): T[] { 119 | assert(Array.isArray(list)) 120 | return list.map((val) => f(val)) 121 | } 122 | 123 | 124 | export function nonNull(val: T | undefined | null): T { 125 | assert(val != null, 'non-nullable value is null') 126 | return val 127 | } 128 | 129 | 130 | export const bigintTransformer = { 131 | to(x?: bigint) { 132 | return x?.toString() 133 | }, 134 | from(s?: string): bigint | undefined { 135 | return s == null ? undefined : BigInt(s) 136 | } 137 | } 138 | 139 | 140 | export const floatTransformer = { 141 | to(x?: number) { 142 | return x?.toString() 143 | }, 144 | from(s?: string): number | undefined { 145 | return s == null ? undefined : Number(s) 146 | } 147 | } 148 | 149 | 150 | export const bigdecimalTransformer = { 151 | to(x?: any) { 152 | return x?.toString() 153 | }, 154 | from(s?: any): any | undefined { 155 | return s == null ? undefined : decimal.BigDecimal(s) 156 | } 157 | } 158 | 159 | 160 | export function enumFromJson(json: unknown, enumObject: E): E[keyof E] { 161 | assert(typeof json == 'string', 'invalid enum value') 162 | let val = (enumObject as any)[json] 163 | assert(typeof val == 'string', `invalid enum value`) 164 | return val as any 165 | } 166 | 167 | 168 | const decimal = { 169 | get BigDecimal(): any { 170 | throw new Error('Package `@subsquid/big-decimal` is not installed') 171 | } 172 | } 173 | 174 | 175 | try { 176 | Object.defineProperty(decimal, "BigDecimal", { 177 | value: require('@subsquid/big-decimal').BigDecimal 178 | }) 179 | } catch (e) {} 180 | -------------------------------------------------------------------------------- /template/src/model/index.ts: -------------------------------------------------------------------------------- 1 | export * from "./generated" 2 | -------------------------------------------------------------------------------- /template/src/processor.ts: -------------------------------------------------------------------------------- 1 | import {EvmBatchProcessor} from '@subsquid/evm-processor' 2 | import {lookupArchive} from '@subsquid/archive-registry' 3 | 4 | export const processor = new EvmBatchProcessor() 5 | .setDataSource({ 6 | archive: lookupArchive('eth-mainnet'), 7 | chain: 'https://my.eth-mainnet.rpc', 8 | }) 9 | .setFinalityConfirmation(10) 10 | .setFields({ 11 | // specify field selection here 12 | }) 13 | .addLog({ 14 | address: [/* set of requested addresses */] 15 | }) 16 | -------------------------------------------------------------------------------- /template/tsconfig.json: -------------------------------------------------------------------------------- 1 | { 2 | "compilerOptions": { 3 | "module": "commonjs", 4 | "target": "es2020", 5 | "outDir": "lib", 6 | "rootDir": "src", 7 | "strict": true, 8 | "resolveJsonModule": true, 9 | "declaration": false, 10 | "sourceMap": true, 11 | "esModuleInterop": true, 12 | "experimentalDecorators": true, 13 | "emitDecoratorMetadata": true, 14 | "skipLibCheck": true 15 | }, 16 | "include": ["src"], 17 | "exclude": [ 18 | "node_modules" 19 | ] 20 | } 21 | --------------------------------------------------------------------------------