└── README.md /README.md: -------------------------------------------------------------------------------- 1 | ## Building an NFT API with filtering, sorting, relationships, and full text search with The Graph. 2 | 3 | In this workshop we'll build a subgraph for querying NTF data from the [Cryptocoven](https://etherscan.io/address/0x5180db8F5c931aaE63c74266b211F580155ecac8#code) smart contract, implementing queries for fetching NFTs as well as their owners, building relationships between them, full text search, sorting, and filtering. 4 | 5 | > The codebase for this project is located [here](https://github.com/dabit3/cryptocoven-api) 6 | 7 | ### Prerequisites 8 | 9 | To be successful in this tutorial, you should have [Node.js](https://nodejs.org/en/) installed on your machine. These days, I recommend using either [nvm](https://github.com/nvm-sh/nvm) or [fnm](https://github.com/Schniz/fnm/blob/master/docs/commands.md) to manage Node.js versions. 10 | 11 | ### Creating the Graph project in the Graph Explorer 12 | 13 | To get started, open [The Graph Hosted Service](https://thegraph.com/hosted-service/) and either sign in or create a new account. 14 | 15 | Next, go to the [dashboard](https://thegraph.com/hosted-service/dashboard) and click on __Add Subgraph__ to create a new subgraph. 16 | 17 | Configure your subgraph with the following properties: 18 | 19 | - Subgraph Name - __Cryptocovenapi__ 20 | - Subtitle - __A subgraph for querying Cryptocoven NFTs__ 21 | - Optional - Fill the description and GITHUB URL properties 22 | 23 | Once the subgraph is created, we will initialize the subgraph locally using the Graph CLI. 24 | 25 | ### Initializing a new subgraph using the Graph CLI 26 | 27 | Next, install the Graph CLI: 28 | 29 | ```sh 30 | $ npm install -g @graphprotocol/graph-cli 31 | 32 | # or 33 | 34 | $ yarn global add @graphprotocol/graph-cli 35 | ``` 36 | 37 | Once the Graph CLI has been installed you can initialize a new subgraph with the Graph CLI `init` command. 38 | 39 | There are two ways to initialize a new subgraph: 40 | 41 | 1 - From an example subgraph 42 | 43 | ```sh 44 | $ graph init --from-example / [] 45 | ``` 46 | 47 | 2 - From an existing smart contract 48 | 49 | If you already have a smart contract deployed to Ethereum mainnet or one of the testnets, initializing a new subgraph from this contract is an easy way to get up and running. 50 | 51 | ```sh 52 | $ graph init --from-contract \ 53 | [--network ] \ 54 | [--abi ] \ 55 | / [] 56 | ``` 57 | 58 | In our case we'll be starting with the [Cryptocoven NFT contract](https://etherscan.io/address/0x5180db8F5c931aaE63c74266b211F580155ecac8#code) so we can initialize from that contract address by passing in the contract address using the `--from-contract` flag: 59 | 60 | ```sh 61 | $ graph init --from-contract 0x5180db8F5c931aaE63c74266b211F580155ecac8 --protocol ethereum \ 62 | --network mainnet --contract-name Token --index-events 63 | 64 | ? Product for which to initialize › hosted-service 65 | ? Subgraph name › your-username/cryptocovenapi 66 | ? Directory to create the subgraph in › cryptocovenapi 67 | ? Ethereum network › Mainnet 68 | ? Contract address › 0x5180db8F5c931aaE63c74266b211F580155ecac8 69 | ? Contract Name · Token 70 | ``` 71 | 72 | This command will generate a basic subgraph based off of the contract address passed in as the argument to `--from-contract`. By using this contract address, the CLI will initialize a few things in your project to get you started (including fetching the `abis` and saving them in the __abis__ directory). 73 | 74 | > By passing in `--index-events` the CLI will automatically populate some code for us both in __schema.graphql__ as well as __src/mapping.ts__ based on the events emitted from the contract. 75 | 76 | The main configuration and definition for the subgraph lives in the __subgraph.yaml__ file. The subgraph codebase consists of a few files: 77 | 78 | - __subgraph.yaml__: a YAML file containing the subgraph manifest 79 | - __schema.graphql__: a GraphQL schema that defines what data is stored for your subgraph, and how to query it via GraphQL 80 | - __AssemblyScript Mappings__: AssemblyScript code that translates from the event data in Ethereum to the entities defined in your schema (e.g. mapping.ts in this tutorial) 81 | 82 | The entries in __subgraph.yaml__ that we will be working with are: 83 | 84 | - `description` (optional): a human-readable description of what the subgraph is. This description is displayed by the Graph Explorer when the subgraph is deployed to the Hosted Service. 85 | - `repository` (optional): the URL of the repository where the subgraph manifest can be found. This is also displayed by the Graph Explorer. 86 | - `dataSources.source`: the address of the smart contract the subgraph sources, and the abi of the smart contract to use. The address is optional; omitting it allows to index matching events from all contracts. 87 | - `dataSources.source.startBlock` (optional): the number of the block that the data source starts indexing from. In most cases we suggest using the block in which the contract was created. 88 | - `dataSources.mapping.entities` : the entities that the data source writes to the store. The schema for each entity is defined in the the schema.graphql file. 89 | - `dataSources.mapping.abis`: one or more named ABI files for the source contract as well as any other smart contracts that you interact with from within the mappings. 90 | - `dataSources.mapping.eventHandlers`: lists the smart contract events this subgraph reacts to and the handlers in the mapping — __./src/mapping.ts__ in the example — that transform these events into entities in the store. 91 | 92 | ### Defining the entities 93 | 94 | With The Graph, you define entity types in __schema.graphql__, and Graph Node will generate top level fields for querying single instances and collections of that entity type. Each type that should be an entity is required to be annotated with an `@entity` directive. 95 | 96 | The entities / data we will be indexing are the `Token` and `User`. 97 | 98 | To populate the entities, we'll be fetching metadata from IPFS using the token ID to get information like the token name, image, and description. 99 | 100 | For example, the base URI for cryptocoven is: 101 | 102 | ``` 103 | ipfs://QmSr3vdMuP2fSxWD7S26KzzBWcAN1eNhm4hk1qaR3x3vmj 104 | ``` 105 | 106 | Therefore, we can visit a URI like: 107 | 108 | ``` 109 | https://ipfs.io/ipfs/QmSr3vdMuP2fSxWD7S26KzzBWcAN1eNhm4hk1qaR3x3vmj/234.json 110 | ``` 111 | 112 | and see all of the metadata for the token. 113 | 114 | Using this IPFS URI, we'll be able to call out to IPFS to retreive this data based on the token ID. 115 | 116 | Update __schema.graphql__ with the following code: 117 | 118 | ```graphql 119 | type Token @entity { 120 | id: ID! 121 | tokenID: BigInt! 122 | tokenURI: String! 123 | externalURL: String! 124 | ipfsURI: String! 125 | image: String! 126 | name: String! 127 | description: String! 128 | type: String! 129 | sun: String! 130 | moon: String! 131 | rising: String! 132 | updatedAtTimestamp: BigInt! 133 | owner: User! 134 | } 135 | 136 | type User @entity { 137 | id: ID! 138 | tokens: [Token!]! @derivedFrom(field: "owner") 139 | } 140 | 141 | type _Schema_ 142 | @fulltext( 143 | name: "covenSearch" 144 | language: en 145 | algorithm: rank 146 | include: [{entity: "Token", fields: 147 | [{ name: "name" }, { name: "description" }, { name: "type" }, { name: "sun" }, { name: "moon" }, { name: "rising" } 148 | ] }] 149 | ) 150 | ``` 151 | 152 | ### On Relationships via `@derivedFrom` (from the docs): 153 | 154 | Reverse lookups can be defined on an entity through the `@derivedFrom` field. This creates a virtual field on the entity that may be queried but cannot be set manually through the mappings API. Rather, it is derived from the relationship defined on the other entity. For such relationships, it rarely makes sense to store both sides of the relationship, and both indexing and query performance will be better when only one side is stored and the other is derived. 155 | 156 | For one-to-many relationships, the relationship should always be stored on the 'one' side, and the 'many' side should always be derived. Storing the relationship this way, rather than storing an array of entities on the 'many' side, will result in dramatically better performance for both indexing and querying the subgraph. In general, storing arrays of entities should be avoided as much as is practical. 157 | 158 | Now that we have created the GraphQL schema for our app, we can generate the entities locally to start using in the `mappings` created by the CLI: 159 | 160 | ```sh 161 | graph codegen 162 | ``` 163 | 164 | In order to make working smart contracts, events and entities easy and type-safe, the Graph CLI generates AssemblyScript types from a combination of the subgraph's GraphQL schema and the contract ABIs included in the data sources. 165 | 166 | ## Updating the subgraph with the entities and mappings 167 | 168 | Now we can configure the __subgraph.yaml__ to use the entities that we have just created and configure their mappings. 169 | 170 | To do so, first update the `dataSources.mapping.entities` field with the `User` and `Token` entities: 171 | 172 | ```yaml 173 | entities: 174 | - Token 175 | - User 176 | ``` 177 | 178 | Next, update the `dataSources.mapping.eventHandlers` to include only the following three event handlers: 179 | 180 | ```yaml 181 | - event: Transfer(indexed address,indexed address,indexed uint256) 182 | handler: handleTransfer 183 | ``` 184 | 185 | Finally, update the configuration to add the `startBlock`: 186 | 187 | ```yaml 188 | source: 189 | address: "0x5180db8F5c931aaE63c74266b211F580155ecac8" 190 | abi: Token 191 | startBlock: 13547115 192 | ``` 193 | 194 | ## Assemblyscript mappings 195 | 196 | Next, open __src/mappings.ts__ to write the mappings that we defined in our subgraph subgraph `eventHandlers`. 197 | 198 | The hosted service supports an [IPFS API](https://thegraph.com/docs/developer/assemblyscript-api#api-reference) that allows you to interact with data stored on the IPFS network. 199 | 200 | This is especially useful for NFT metadata where you might have additional information about the NFT stored in JSON format which is the case with our API. 201 | 202 | Update the file with the following code: 203 | 204 | ```typescript 205 | import { ipfs, json } from '@graphprotocol/graph-ts' 206 | import { 207 | Transfer as TransferEvent, 208 | Token as TokenContract 209 | } from '../generated/Token/Token' 210 | import { 211 | Token, User 212 | } from '../generated/schema' 213 | 214 | const ipfshash = "QmSr3vdMuP2fSxWD7S26KzzBWcAN1eNhm4hk1qaR3x3vmj" 215 | 216 | export function handleTransfer(event: TransferEvent): void { 217 | /* load the token from the existing Graph Node */ 218 | let token = Token.load(event.params.tokenId.toString()) 219 | if (!token) { 220 | /* if the token does not yet exist, create it */ 221 | token = new Token(event.params.tokenId.toString()) 222 | token.tokenID = event.params.tokenId 223 | 224 | token.tokenURI = "/" + event.params.tokenId.toString() + ".json" 225 | 226 | /* combine the ipfs hash and the token ID to fetch the token metadata from IPFS */ 227 | let metadata = ipfs.cat(ipfshash + token.tokenURI) 228 | if (metadata) { 229 | const value = json.fromBytes(metadata).toObject() 230 | if (value) { 231 | /* using the metatadata from IPFS, update the token object with the values */ 232 | const image = value.get('image') 233 | const name = value.get('name') 234 | const description = value.get('description') 235 | const externalURL = value.get('external_url') 236 | 237 | if (name && image && description && externalURL) { 238 | token.name = name.toString() 239 | token.image = image.toString() 240 | token.externalURL = externalURL.toString() 241 | token.description = description.toString() 242 | token.ipfsURI = 'ipfs.io/ipfs/' + ipfshash + token.tokenURI 243 | } 244 | 245 | const coven = value.get('coven') 246 | if (coven) { 247 | let covenData = coven.toObject() 248 | const type = covenData.get('type') 249 | if (type) { 250 | token.type = type.toString() 251 | } 252 | 253 | const birthChart = covenData.get('birthChart') 254 | if (birthChart) { 255 | const birthChartData = birthChart.toObject() 256 | const sun = birthChartData.get('sun') 257 | const moon = birthChartData.get('moon') 258 | const rising = birthChartData.get('rising') 259 | if (sun && moon && rising) { 260 | token.sun = sun.toString() 261 | token.moon = moon.toString() 262 | token.rising = rising.toString() 263 | } 264 | } 265 | } 266 | 267 | } 268 | } 269 | } 270 | token.updatedAtTimestamp = event.block.timestamp 271 | 272 | /* set or update the owner field and save the token to the Graph Node */ 273 | token.owner = event.params.to.toHexString() 274 | token.save() 275 | 276 | /* if the user does not yet exist, create them */ 277 | let user = User.load(event.params.to.toHexString()) 278 | if (!user) { 279 | user = new User(event.params.to.toHexString()) 280 | user.save() 281 | } 282 | } 283 | ``` 284 | 285 | These mappings will handle events for when a new token is created or transferred. When this event fires, the mappings will save the data into the subgraph. 286 | 287 | ### Running a build 288 | 289 | Next, let's run a build to make sure that everything is configured properly. To do so, run the `build` command: 290 | 291 | ```sh 292 | $ graph build 293 | ``` 294 | 295 | If the build is successful, you should see a new __build__ folder generated in your root directory. 296 | 297 | ## Deploying the subgraph 298 | 299 | To deploy, we can run the `deploy` command using the Graph CLI. To deploy, you will first need to copy the __Access token__ for your account, available in [The Graph dashboard](https://thegraph.com/hosted-service/dashboard): 300 | 301 | ![Graph Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/820lwqh8yo3iyu7fsbhj.jpg) 302 | 303 | Next, run the following command: 304 | 305 | ```sh 306 | $ graph auth 307 | ✔ Product for which to initialize · hosted-service 308 | ✔ Deploy key · ******************************** 309 | 310 | $ yarn deploy 311 | ``` 312 | 313 | Once the subgraph is deployed, you should see it show up in your dashboard: 314 | 315 | ![Graph Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/19no8vdd1nlce9vl5c1j.jpg) 316 | 317 | When you click on the subgraph, it should open the Graph explorer: 318 | 319 | ![The Foundation Subgraph](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v5o0xae2ccg6w8i6ozn5.png) 320 | 321 | ## Querying for data 322 | 323 | Now that we are in the dashboard, we should be able to start querying for data. Run the following query to get a list of tokens and their metadata: 324 | 325 | ```graphql 326 | { 327 | tokens(first: 5) { 328 | id 329 | tokenID 330 | tokenURI 331 | externalURL 332 | image 333 | name 334 | description 335 | type 336 | sun 337 | moon 338 | rising 339 | updatedAtTimestamp 340 | owner { 341 | id 342 | } 343 | } 344 | } 345 | ``` 346 | 347 | We can also configure the order direction: 348 | 349 | ```graphql 350 | { 351 | tokens( 352 | first: 5 353 | orderDirection: desc 354 | orderBy: updatedAtTimestamp 355 | ) { 356 | id 357 | tokenID 358 | tokenURI 359 | externalURL 360 | image 361 | name 362 | description 363 | type 364 | sun 365 | moon 366 | rising 367 | updatedAtTimestamp 368 | owner { 369 | id 370 | } 371 | } 372 | } 373 | ``` 374 | 375 | Or choose to skip forward a certain number of results to implement some basic pagination: 376 | 377 | ```graphql 378 | { 379 | tokens( 380 | skip: 100 381 | orderDirection: desc 382 | orderBy: updatedAtTimestamp 383 | ) { 384 | id 385 | tokenID 386 | tokenURI 387 | externalURL 388 | image 389 | name 390 | description 391 | type 392 | sun 393 | moon 394 | rising 395 | updatedAtTimestamp 396 | owner { 397 | id 398 | } 399 | } 400 | } 401 | ``` 402 | 403 | Or query with filtering: 404 | 405 | ```graphql 406 | { 407 | tokens( 408 | where: { 409 | sun_contains: "capricorn" 410 | } 411 | ) { 412 | sun 413 | name 414 | } 415 | } 416 | ``` 417 | 418 | And full text search 419 | 420 | ```graphql 421 | { 422 | covenSearch( 423 | text: "'CRUSH PEARLS IN YOUR FISTS'" 424 | ) { 425 | id 426 | name 427 | description 428 | } 429 | } 430 | ``` 431 | 432 | > The codebase for this project is located [here](https://github.com/dabit3/cryptocoven-api) 433 | 434 | ## Next steps 435 | 436 | If you are interested in learning more about Web3, building Dapps, or building subgraphs, check out the following resources: 437 | 438 | The Graph on Twitter - [@graphprotocol](https://twitter.com/graphprotocol) 439 | 440 | [The Complete Guide to Full Stack Web3 Development](https://dev.to/dabit3/the-complete-guide-to-full-stack-ethereum-development-3j13) 441 | 442 | [The Graph Discord](thegraph.com/discord) 443 | 444 | [Solidity Docs](https://docs.soliditylang.org/) 445 | 446 | [Ethereum Developer Documentation](https://ethereum.org/en/developers/docs/) 447 | 448 | Austin Griffith on Twitter [@austingriffith](https://twitter.com/austingriffith) & [Scaffold Eth](https://github.com/austintgriffith/scaffold-eth) 449 | 450 | [Crypto Zombies](https://cryptozombies.io/) 451 | --------------------------------------------------------------------------------