├── .gitignore ├── .prettierrc ├── Get Started - Quick Intro ├── README.md ├── configuration.md ├── deployment.md ├── kpl-tokens.md ├── networking.md ├── secrets.md ├── security.md ├── storage.md └── testing.md ├── Lesson 1 ├── PartI.md ├── PartII.md ├── PartIII.md ├── PartIV.md ├── PartV.md ├── README.md ├── ez-testing-task │ ├── 1-task.js │ ├── 2-submission.js │ ├── 3-audit.js │ └── 4-distribution.js └── imgs │ ├── add-task-advanced.png │ ├── gradual-consensus.png │ ├── main-log.png │ ├── my-node-open-logs.png │ ├── node-public-key.png │ ├── open-main-log.png │ ├── public-key.png │ ├── secrets-example.png │ ├── stacking-rounds.png │ ├── system-wallet.png │ ├── task-list.png │ └── task-log.png ├── Lesson 2 ├── PartI.md ├── PartII.md ├── PartIII.md ├── README.md ├── file-sharing │ ├── config-task.yml │ └── task │ │ ├── 1-task.js │ │ ├── 2-submission.js │ │ ├── 3-audit.js │ │ └── fileUtils.js ├── imgs │ ├── enable-upnp.png │ └── require-internet.png └── upnp-basics │ ├── config-task.yml │ └── src │ └── task │ ├── 1-task.js │ ├── 2-submission.js │ ├── 3-audit.js │ ├── 5-routes.js │ └── upnpUtils.js ├── Lesson 3 ├── PartI.md ├── PartII.md ├── PartIII.md ├── README.md ├── debugger.js ├── imgs │ ├── secrets-example.png │ └── simple-crawler.png ├── prod-debug.js └── simple-crawler │ ├── config-task.yml │ └── task │ ├── 1-task.js │ ├── 2-submission.js │ ├── 3-audit.js │ ├── crawler.js │ └── fileUtils.js ├── Lesson 4 ├── PartI.md ├── PartII-new.md ├── PartII.md ├── PartIII.md ├── README.md └── caesar-task │ └── task │ ├── 1-task.js │ ├── 3-audit.js │ ├── 4-distribution.js │ └── cipher.js ├── Lesson 5 └── README.md ├── Lesson 6 ├── PartI.md ├── PartII.md ├── README.md └── imgs │ └── mint-address.png ├── Lesson 7 ├── PartI.md ├── PartII.md ├── README.md └── imgs │ └── task-id.png ├── Lesson 8 └── README.md ├── README.md └── package.json /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | test-ledger 3 | .env 4 | .env.local 5 | node_modules 6 | package-lock.json 7 | yarn.lock 8 | -------------------------------------------------------------------------------- /.prettierrc: -------------------------------------------------------------------------------- 1 | { 2 | "useTabs": false, 3 | "tabWidth": 2 4 | } 5 | -------------------------------------------------------------------------------- /Get Started - Quick Intro/README.md: -------------------------------------------------------------------------------- 1 | i# Run Your First Task in 2 Min 2 | 3 | 1. Install and set up the [desktop node](https://www.koii.network/node). 4 | 5 | 2. Add the EZ testing task ID (`CRzLkJvYnta2BnaSMMVkew7FRrZtZoNe7wGFogq7Hj5D`) to your node using the Advanced option in the Add Task tab. 6 | 7 | ![Add EZ Testing Task](../Lesson%201/imgs/add-task-advanced.png) 8 | 9 | 3. Clone the [task template repository](https://github.com/koii-network/task-template) 10 | 11 | 12 | 4. Run 13 | 14 | ```sh 15 | yarn 16 | ``` 17 | 18 | then 19 | 20 | ```sh 21 | yarn prod-debug 22 | ``` 23 | 24 | to run the task in your node. 25 | 26 | That's it! you can make changes to your task and it will update live in your node. You can add `process.env.TEST_KEYWORD` to your console logs to see them in your console, or you can view the entire task output in the task log file. 27 | 28 | ![View task log](../Lesson%201/imgs/my-node-open-logs.png) 29 | 30 | ## Going Further 31 | 32 | Check out our quick reference guides: 33 | 34 | - [Networking](./networking.md) 35 | - [Storage](./storage.md) 36 | - [Secrets](./secrets.md) 37 | - [Security](./security.md) 38 | - [KPL Tokens](./kpl-tokens.md) 39 | - [Testing](./testing.md) 40 | 41 | - [Deployment](./deployment.md) 42 | -------------------------------------------------------------------------------- /Get Started - Quick Intro/configuration.md: -------------------------------------------------------------------------------- 1 | # Configuration Quick Reference 2 | 3 | ## Task Metadata 4 | 5 | The `task_name`, `author`, `description`, `repositoryUrl` and `imageUrl` will be displayed in the desktop node. 6 | 7 | 8 | 9 | ## Task Details 10 | 11 | `task_executable_network`? 12 | `task_audit_program`? 13 | `round_time`, `audit_window`, `submission_window` - How do people know what to set for these? 14 | `minimum_stake_amount` - any guidelines? 15 | `total_bounty_amount`, `bounty_amount_per_round` 16 | `allowed_failed_distributions` 17 | `space` - what should this be? 18 | 19 | ## Requirements Tags 20 | 21 | 22 | 23 | ## Updating a Task 24 | 25 | `task_id`, `migrationDescription` 26 | 27 | 28 | -------------------------------------------------------------------------------- /Get Started - Quick Intro/deployment.md: -------------------------------------------------------------------------------- 1 | # Deployment Quick Reference 2 | 3 | Before deploying your task, you can apply for our [grant program](https://www.koii.network/founders) that can help you out with initial funding. 4 | 5 | > [!WARNING] 6 | > 7 | > In order to deploy a task, you **MUST** be using at least Node.js v18. 8 | 9 | 1. Make sure all dependencies are installed: 10 | 11 | ```sh 12 | yarn 13 | ``` 14 | 15 | 2. Build the executable: 16 | 17 | ```sh 18 | yarn webpack 19 | ``` 20 | 21 | 3. Run the create task CLI: 22 | 23 | ```sh 24 | npx @_koii/create-task-cli@latest 25 | ``` 26 | 27 | 4. Choose "Deploy a New Task". 28 | 29 | 5. At this point you will be asked how you'd like to configure your task, via the CLI or the config YML. We recommend using the YML file. 30 | 31 | 6. Enter the path to your staking wallet. If you have installed the desktop node, visit `/KOII-Desktop-Node/namespace/` and you should see a file with the name `stakingWallet.json`. Enter the full path to this file (`/KOII-Desktop-Node/namespace/stakingWallet.json`). 32 | 33 | The OS-specific paths are as follows: 34 | 35 | **Windows**: `/Users//AppData/Roaming` 36 | 37 | **Mac**: `/Users//Library/Application Support` 38 | 39 | **Linux**: `/home//.config` 40 | 41 | 7. Press 'y' to confirm you would like to deploy the task. 42 | 43 | Once your task is deployed, you will have to apply to have it [whitelisted](https://docs.koii.network/develop/write-a-koii-task/task-development-guide/task-development-flow/whitelist-task) if you would like your task to be listed in the desktop node's Add Task list. 44 | -------------------------------------------------------------------------------- /Get Started - Quick Intro/kpl-tokens.md: -------------------------------------------------------------------------------- 1 | # KPL Tokens Quick Reference 2 | 3 | ## Creating KPL Tokens 4 | 5 | Visit [this site](https://dev-blue-spl-token-creator.vercel.app/) to create a KPL token. 6 | 7 | You will be asked to enter the following information: 8 | 9 | 1. **Wallet.json** 10 | If you have deployed your own tasks in previous lessons, you should already know how to obtain your `wallet.json`. 11 | 12 | > [!NOTE] 13 | > 14 | > `Wallet.json` is handled through the front-end JavaScript, so rest assured, your `wallet.json` file will never be uploaded anywhere and will always remain on your local machine. 15 | > 16 | 17 | 2. **PNG/ICO/JPEG** 18 | This is your icon. It is best if it is square; otherwise, it may be cropped. 19 | 20 | 3. **Token Name/Token Symbol/Token Description** 21 | These are the details of your token. The symbol represents the unit, such as FIRE and KOII. 22 | 23 | > [!IMPORTANT] 24 | > 25 | > Please remember your token's Token Public Key, as it will be needed in the next step for minting. 26 | > 27 | 28 | ## Minting Tokens 29 | 30 | Enter the amount of tokens you'd like to mint, your `wallet.json` file, and your Token Public Key. 31 | 32 | ## Checking Token Balance 33 | 34 | You can check the token balance by choosing "Connect Finnie" to link your [Finnie Wallet](https://docs.koii.network/concepts/finnie-wallet/introduction), and then clicking "Fetch Tokens". 35 | -------------------------------------------------------------------------------- /Get Started - Quick Intro/networking.md: -------------------------------------------------------------------------------- 1 | # Networking Quick Reference 2 | 3 | ## Configuration 4 | 5 | Koii nodes use UPnP for node-to-node communication. This is disabled by default and must be activated by the user, so fewer nodes will be available for these tasks. In order to identify users that can perform these tasks, add `REQUIRE_INTERNET` to your `requirementTags` in `config-task.yml`: 6 | 7 | ```yml 8 | requirementsTags: 9 | - type: ADDON 10 | value: 'REQUIRE_INTERNET' 11 | ``` 12 | 13 | ## Serving Endpoints 14 | 15 | Endpoint routes can be added to index.js. Two are already defined in the task template: 16 | 17 | ```js 18 | if (app) { 19 | // Write your Express Endpoints here. 20 | // Ex. app.post('/accept-cid', async (req, res) => {}) 21 | 22 | // Sample API that return your task state 23 | app.get('/taskState', async (req, res) => { 24 | const state = await namespaceWrapper.getTaskState({ 25 | is_stake_list_required: true, 26 | }); 27 | console.log('TASK STATE', state); 28 | res.status(200).json({ taskState: state }); 29 | }); 30 | 31 | // Sample API that return the value stored in NeDB 32 | app.get('/value', async (req, res) => { 33 | const value = await namespaceWrapper.storeGet('value'); 34 | console.log('value', value); 35 | res.status(200).json({ value: value }); 36 | }); 37 | } 38 | ``` 39 | 40 | You can access endpoints like so when you are running `prod-debug`: `http://localhost:30017/task//` 41 | 42 | ## Accessing Endpoints from Other Nodes 43 | 44 | To access other nodes' endpoints, you can access the list of IP addresses currently running the task, then choose which nodes you will access. In this example, we're selecting a single random node: 45 | 46 | ```javascript 47 | async function getAddressArray() { 48 | try { 49 | // get the task state from K2 (the Koii blockchain) 50 | const taskState = await namespaceWrapper.getTaskState(); 51 | 52 | // get the list of available IP addresses from the task state 53 | // nodeList is an object with key-value pairs in the form stakingKey: ipAddress 54 | const nodeList = taskState.ip_address_list; 55 | 56 | // return just the IP addresses 57 | return Object.values(IPAddressObj); 58 | } catch (e) { 59 | console.log('ERROR GETTING TASK STATE', e); 60 | } 61 | } 62 | 63 | async function getRandomNodeEndpoint(IPAddressArray) { 64 | // choose a random index 65 | const randomIndex = Math.floor(Math.random() * IPAddressArray.length); 66 | // Return the IP address stored at the random index position 67 | return IPAddressArray[randomIndex]; 68 | } 69 | ``` 70 | 71 | You can access the endpoint(s) you've chosen during the task or audit phases 72 | 73 | ```javascript 74 | // Get a random node from the list 75 | const IPAddressObj = getAddressArray(); 76 | const randomNode = getRandomNodeEndpoint(IPAddressObj); 77 | 78 | // Fetch the data from the node 79 | const response = await axios.get(`${randomNode}/task/${TASK_ID}/`); 80 | 81 | if (response.status === 200) { 82 | const data = response.data; 83 | } else { 84 | return null; 85 | } 86 | ``` 87 | -------------------------------------------------------------------------------- /Get Started - Quick Intro/secrets.md: -------------------------------------------------------------------------------- 1 | # Secrets Quick Reference 2 | 3 | Secrets (referred to as task extensions in the desktop node) are environment variables stored by node users. These can be accessed just as you would any other environment variable, using `process.env`. During development, these should be added to your .env file. 4 | 5 | You can specify which environment variables a user must define in the `requirementsTags` section of your `config-task.yml` file, using the `TASK_VARIABLE` type. 6 | 7 | Example: 8 | 9 | ```yml 10 | requirementsTags: 11 | - type: TASK_VARIABLE 12 | value: 'TWITTER_USERNAME' 13 | description: 'The username of your volunteer Twitter account.' 14 | - type: TASK_VARIABLE 15 | value: 'TWITTER_PASSWORD' 16 | description: 'The password of your volunteer Twitter account.' 17 | - type: TASK_VARIABLE 18 | value: 'TWITTER_PHONE' 19 | description: 'If verification is required, will use your phone number to login.' 20 | ``` 21 | -------------------------------------------------------------------------------- /Get Started - Quick Intro/security.md: -------------------------------------------------------------------------------- 1 | # Security Quick Reference 2 | 3 | A message can be signed like so, using `namespaceWrapper.payloadSigning()`: 4 | 5 | ```javascript 6 | const message = "Hello World!"; 7 | 8 | // Sign payload 9 | const signedMessage = await namespaceWrapper.payloadSigning(message); 10 | ``` 11 | 12 | The `payloadSigning()` function will take care of accessing the node's private key for the signing. 13 | 14 | When you want to send this signature to another node, you'll also need to include the current node's public key. You can get this from `namespaceWrapper.getSubmitterAccount()`: 15 | 16 | ```javascript 17 | // get current node's keypair 18 | const keypair = await namespaceWrapper.getSubmitterAccount(message); 19 | 20 | // get the public key from the keypair 21 | const publicKey = keypair.publicKey; 22 | ``` 23 | 24 | You can then send the signed message and the public key to another node to be verified and decoded. This can be done with `namespaceWrapper.verifySignature()`: 25 | 26 | ```javascript 27 | // Assuming you've already retrieved the signed message and the public key 28 | const message = namespaceWrapper.verifySignature(signedMessage, publicKey); 29 | ``` 30 | -------------------------------------------------------------------------------- /Get Started - Quick Intro/storage.md: -------------------------------------------------------------------------------- 1 | # Storage Quick Reference 2 | 3 | ## Local Database 4 | 5 | Koii tasks use a local key-value store that can be used to pass data between functions. 6 | 7 | ### Setting a value 8 | 9 | ```js 10 | const value = await namespaceWrapper.storeSet('value', ''); 11 | ``` 12 | 13 | ### Getting a value 14 | 15 | ```js 16 | const value = await namespaceWrapper.storeGet('value'); 17 | ``` 18 | 19 | ## IPFS 20 | 21 | You can use IPFS in a Koii task to pass files between nodes. 22 | 23 | ### Uploading a file 24 | 25 | ```js 26 | const { KoiiStorageClient } = require('@_koii/storage-task-sdk'); 27 | const fs = require('fs'); 28 | 29 | async function storeFile(data, filename = 'value.json') { 30 | try { 31 | // Create a new instance of the Koii Storage Client 32 | const client = new KoiiStorageClient(); 33 | const basePath = await namespaceWrapper.getBasePath(); 34 | 35 | // Write the data to a temp file 36 | fs.writeFileSync(`${basePath}/${filename}`, JSON.stringify(data)); 37 | 38 | // Get the user staking account, to be used for signing the upload request 39 | const userStaking = await namespaceWrapper.getSubmitterAccount(); 40 | 41 | // Upload the file to IPFS and get the CID 42 | const { cid } = await client.uploadFile(`${basePath}/${filename}`, userStaking); 43 | 44 | // Delete the temp file 45 | fs.unlinkSync(`${basePath}/${filename}`); 46 | 47 | return cid; 48 | } catch (error) { 49 | console.error('Failed to upload file to IPFS:', error); 50 | fs.unlinkSync(`${basePath}/${filename}`); 51 | throw error; 52 | } 53 | } 54 | ``` 55 | 56 | The CID can then be sent as part of the submission to be used by other nodes. 57 | 58 | ## Retrieving a file 59 | 60 | ```js 61 | const { KoiiStorageClient } = require('@_koii/storage-task-sdk'); 62 | 63 | // Create a new instance of the Koii Storage Client 64 | const client = new KoiiStorageClient(); 65 | 66 | try { 67 | // get file from IPFS 68 | const fileBlob = await client.getFile(cid, filename); 69 | if (!fileBlob) return false; 70 | 71 | // if it's a text file, you can read the contents 72 | const fileContent = await fileBlob.text(); 73 | 74 | } catch (error) { 75 | console.error('Failed to download or validate file from IPFS:', error); 76 | throw error; 77 | } 78 | ``` 79 | -------------------------------------------------------------------------------- /Get Started - Quick Intro/testing.md: -------------------------------------------------------------------------------- 1 | # Testing Your Task Quick Reference 2 | 3 | ## unitTest.js 4 | 5 | We've provided a simple testing script that will allow you to simulate rounds and test your task and audit functionality. The script is available in `testing/unitTesting.js` and you can run it via 6 | 7 | ```sh 8 | yarn test 9 | ``` 10 | 11 | ## prod-debug.js 12 | 13 | If you would like to test integration with the desktop node, you can add the task to the desktop node using the 'Advanced' option under the Add Task list (if you have not yet deployed your task, you can use the EZ Testing Task ID supplied in the task template) and then run 14 | 15 | ```sh 16 | yarn prod-debug 17 | ``` 18 | 19 | You will then be able to run your task in your desktop node and log debugging information to your console by adding `process.env.TEST_KEYWORD` to your console logs. 20 | 21 | ## Deploying Locally with Docker 22 | 23 | Configure the Koii CLI to use a local validator: 24 | 25 | ```sh 26 | koii config set --url http://127.0.0.1:8899 27 | ``` 28 | 29 | You can confirm this was successful with 30 | 31 | ```sh 32 | koii config get 33 | ``` 34 | 35 | which should show that the RPC and Websocket URLs are using localhost: 36 | 37 | ```sh 38 | Config File: C:\Users\test\.config\koii\cli\config.yml 39 | RPC URL: http://localhost:8899 40 | WebSocket URL: ws://localhost:8900/ (computed) 41 | Keypair Path: ~/.config/koii/id.json 42 | Commitment: confirmed 43 | ``` 44 | 45 | Because this is a local environment, the tokens are not real. You can easily add to your balance to gain tokens for test deployments with 46 | 47 | ```sh 48 | koii airdrop 49 | ``` 50 | 51 | Run 52 | 53 | ```sh 54 | npx @_koii/create-task-cli@latest 55 | ``` 56 | 57 | to begin the [deployment](./deployment.md) process. 58 | 59 | You will receive a task ID and an executable CID. Note these down and update your `.env.local` with your new task ID. 60 | 61 | If you've previously deployed your task, you'll need to navigate to the `dist` folder and rename `main.js` to `.js` 62 | 63 | Update your wallet location in `docker-compose.yaml` if necessary. 64 | 65 | In one terminal window run 66 | 67 | ```sh 68 | koii-test-validator 69 | ``` 70 | 71 | and in a second run 72 | 73 | ```sh 74 | docker compose up 75 | ``` 76 | 77 | You now have a Dockerized instance of your task running locally and your terminal will display all the logging information you would normally find in the task log file. 78 | -------------------------------------------------------------------------------- /Lesson 1/PartI.md: -------------------------------------------------------------------------------- 1 | # Lesson 1: Introduction to Koii Tasks 2 | 3 | > [!TIP] 4 | > 5 | > Want to skip the explanations and get started quickly? 6 | > 7 | > [Get a task up and running in 2 minutes](../Get%20Started%20-%20Quick%20Intro/README.md). 8 | 9 | ## Part 1: Running an Existing Task 10 | 11 | Welcome to Koii! We're building an open decentralized computing network, and this is your guide for building tasks to run on the network. 12 | 13 | ### Why Koii? 14 | 15 | - The 1st Layer 1 dedicated to merging AI and DePIN ecosystem development. 16 | - Intuitive tools for building DePIN apps (AI inference, bandwidth, data scraping, etc). 17 | - Tens of thousands of compute devices and best-in-class unit economics. 18 | - A robust Web3 infra for projects. 19 | - Supports existing altcoins and stablecoins, in addition to $KOII. 20 | 21 | ### What is a Koii Task? 22 | 23 | A Koii Task is a decentralized computing job that runs across our network of nodes. We'll get into exactly how it works shortly, but for now let's run one and see it in action. 24 | 25 | Tasks run on Koii nodes, so the first thing we're going to do is set up a node. 26 | 27 | ### Install the Node 28 | 29 | To get started, you'll need to download and install the [Desktop Node](https://www.koii.network/node). Complete instruction are available [here](https://docs.koii.network/run-a-node/task-nodes/how-to-run-a-koii-node). As part of this process, a KOII wallet will be created for you. You can find the public key for your Node wallet on the sidebar or under Settings/Accounts: 30 | 31 | ![public key location in desktop node](./imgs/public-key.png) 32 | 33 | > [!TIP] 34 | > 35 | > Our docs have more information on [wallets and public keys](https://docs.koii.network/run-a-node/task-nodes/concepts/tokens-and-wallets). 36 | 37 | ### Get Some Tokens 38 | 39 | In order to run a task, you'll need to have a few tokens for staking. During the node setup process, you were directed to the [faucet](https://faucet.koii.network/) to get some free tokens, so you should be all set! 40 | 41 | > [!TIP] 42 | > 43 | > **Why do you need tokens to run a task?** 44 | > 45 | > Every task requires a stake in tokens. Staking keeps your tasks safe from bad actors: if a task runner acts maliciously, they will be penalized and lose some or all of this stake. 46 | 47 | ### Run the Task 48 | 49 | If you'd like to earn some extra KOII, you can run any of the tasks from the `Add Task` list in your node. 50 | 51 | ![View add task list](./imgs/task-list.png) 52 | 53 | However, we want to run the EZ Testing task. It is not a public task, so it needs to be added manually. In the `Add Task` tab, click on the "Advanced" link at the bottom left. Paste in the EZ Testing Task ID (`CRzLkJvYnta2BnaSMMVkew7FRrZtZoNe7wGFogq7Hj5D`) and set your stake to 1.9 KOII. Wait for the metadata to download and then start the task. Move to the `My Node` tab and you should see the task running. 54 | 55 | ![Add an unlisted task](./imgs/add-task-advanced.png) 56 | 57 | > [!TIP] 58 | > 59 | > **What is a Task ID?** 60 | > 61 | > Tasks are deployed to the Koii blockchain. In order to connect to the task on the blockchain, we use a unique identifier for each task, called a Task ID. In a later lesson we'll cover deployment and how to get your own Task ID. 62 | 63 | To get a better idea of what the task is doing, we're going to take a look at the log files. This is optional, feel free to skip to the [next step](./PartIII.md) if you'd rather get straight to working with the code. 64 | 65 | ### View the Task Log 66 | 67 | To view the log for a specific task, click any of the tasks in your Node and select 'Output Logs' as shown below: 68 | 69 | ![Open the log file](./imgs/my-node-open-logs.png) 70 | 71 | This will open a folder containing the `task.log` file: 72 | 73 | ![Task log](./imgs/task-log.png) 74 | 75 | ### View the Main Log 76 | 77 | In addition to task-specific logs, there's a main log for information about the node. You can access it from the Settings menu: 78 | 79 | ![Open main log](imgs/open-main-log.png) 80 | 81 | ![Main log](imgs/main-log.png) 82 | 83 | ### Real-time Log Updates 84 | 85 | #### MacOS and Linux 86 | 87 | Try navigating to the logs directory in your terminal: 88 | 89 | ```sh 90 | # MacOS 91 | cd /Users//Library/Application Support/KOII-Desktop-Node/logs/ 92 | ``` 93 | 94 | ```sh 95 | # Linux 96 | cd /home//.config/KOII-Desktop-Node/logs/ 97 | ``` 98 | 99 | You can then run the following command to watch your logs in real time: 100 | 101 | ```sh 102 | tail -f main.log 103 | ``` 104 | 105 | #### Windows 106 | 107 | In Windows, there is no `tail` command, but you can run the following command in PowerShell: 108 | 109 | ```sh 110 | Get-Content -Path "/Users//AppData/Roaming/KOII-Desktop-Node/logs/main.log" -Wait 111 | ``` 112 | 113 | Congratulations! Now that you've set up a node and run your first task, let's try some debugging. [Part II](./PartII.md) 114 | -------------------------------------------------------------------------------- /Lesson 1/PartII.md: -------------------------------------------------------------------------------- 1 | # Lesson 1: Introduction to Koii Tasks 2 | 3 | ## Part II: Debugging a Task 4 | 5 | ### Prerequisites 6 | 7 | - [Part I](./README.md) complete 8 | - [Node.js](https://nodejs.org/en/download/package-manager) installed 9 | 10 | ### The Task Executable 11 | 12 | When you add a task to your node, a Task Executable will be downloaded. This is a JavaScript file that contains all the code needed to run the task. When you deploy a task to the Koii network, you will build a Task Executable and receive a task ID. We'll walk through that process at the end of this lesson, but to get started quickly, we're going to use an existing task that has already been deployed for us. This is the `EZ Testing Task`, which you should have added to your Node in [part I](./README.md#run-the-task). Because we have already added it to the Node, its executable has been downloaded. 13 | 14 | > [!TIP] 15 | > 16 | > **Why EZ Testing Task?** 17 | > 18 | > We have deployed EZ Testing Task and provided its task ID to get you up and running quicker. In later lessons, we'll talk about how to deploy your own task; if you'd prefer to jump ahead and do that now, you can find the instructions in [Part III](./PartIII.md). 19 | 20 | ### Install Yarn 21 | 22 | While you can use NPM or another package manager of your choice, we have found NPM causes issues for some people, so we recommend [Yarn](https://classic.yarnpkg.com/lang/en/docs/install/), and use it in all the EZSandbox instructions. Installing it is simple: 23 | 24 | ```sh 25 | npm install --global yarn 26 | ``` 27 | 28 | ### Debugging Your First Task 29 | 30 | We will overwrite the task executable with our local copy, so we can change the code and see the results. This is simple with the `prod-debug` tool that is provided. 31 | 32 | 1. Use create-task-cli to create a new task. This repo contains the code needed to build a task executable. 33 | 34 | ```sh 35 | npx @_koii/create-task-cli@latest 36 | ``` 37 | 38 | Select create a Local Repository and select TypeScript or JavaScript. 39 | 40 | 2. Copy the `.env.example` file and rename it to `.env`. It has already been set up with the environment variables you need. 41 | 42 | ```sh 43 | cp .env.example .env 44 | ``` 45 | 46 | Run 47 | 48 | ```sh 49 | yarn 50 | ``` 51 | 52 | to install the necessary dependencies, then run 53 | 54 | ```sh 55 | yarn prod-debug 56 | ``` 57 | 58 | to start the live debugger. 59 | 60 | ### Adding Debug Logs 61 | 62 | Open the `src/task` folder and we'll start hacking through some files. To see the task flow in action you'll want to add some log statements to each of the recurring functions that run each round. 63 | 64 | In each case, navigate to the correct file then paste the code lines that have been supplied into each function. If you run into any difficulties, the completed code is available in `ez-testing-task`. 65 | 66 | We have pre-configured the `TEST_KEYWORD` environment variable to "TEST". Change this to whatever you'd like. 67 | 68 | > [!TIP] 69 | > 70 | > You may need to wait a minute while the task starts up before you see any logging. 71 | 72 | a. Task: 73 | 74 | - File Name: `1-task.js` 75 | - Code: `console.log("Started Task", new Date(), "TEST")` 76 | 77 | b. Submission: 78 | 79 | - File Name: `2-submission.js` 80 | - Code: `console.log("Started Submission", new Date(), "EZ TESTING")` 81 | 82 | c. Audit: 83 | 84 | - File Name: `3-audit.js` 85 | - Code: `console.log("Started Audit", new Date(), "EZ TESTING")` 86 | 87 | d. Distribution: 88 | 89 | - File Name: `4-distribution.js` 90 | - Code: `console.log("Started Distribution", new Date(), "TEST")` 91 | 92 | As you save each file, the task executable will be rebuilt and you should see the debugger restart. If you check your desktop node, you'll also see a message that the task has changed. 93 | 94 | Once all changes have been made, locate the EZ Testing Task in your node and press the play button to restart the task with the new executable file. 95 | 96 | Now, wait and watch the logs to see the console logs you just added. All the output for the task will be shown in your terminal, and any lines containing a watched keyword will be colored magenta to make them easier to spot. 97 | 98 | ### Accessing Your Node 99 | 100 | By using UPnP (Universal Plug and Play), each node can expose Express.js endpoints in the `src/routes.js` file. We will cover UPnP and how to make and access endpoints within tasks in the next lesson, but we have defined a couple already, so you can see them on your node. 101 | 102 | Any endpoints running on your node are located at `http://localhost:30017/task/{taskID}/{endpoint}`. 103 | 104 | You can find the task ID for the EZ Testing Task in your `.env` file. With the EZ Testing Task running, visit the `value` endpoint. You should see 105 | 106 | ```json 107 | { "value": "Hello, World!" } 108 | ``` 109 | 110 | You can also visit the `taskState` endpoint to see information about the task. 111 | 112 | Now that we've run a task and done some debugging, let's learn a bit more about what it's doing. [Part III](./PartIII.md) 113 | -------------------------------------------------------------------------------- /Lesson 1/PartIII.md: -------------------------------------------------------------------------------- 1 | # Lesson 1: Introduction to Koii Tasks 2 | 3 | ## Part III: How Do Koii Tasks Work? 4 | 5 | ### The Basics 6 | 7 | In the previous lesson, we added 4 console logs for Task, Submission, Audit, and Distribution. These are the four main parts of a Koii task: 8 | 9 | 1. Do the work (task) 10 | 2. Submit the results (submission) 11 | 3. Verify the work (audit) 12 | 4. Distribute rewards and penalties (distribution) 13 | 14 | This ensures that each node operator has an incentive to act honestly and perform the task correctly. Their work will be checked, and if they don't perform the task correctly, they will not only miss out on rewards, they could lose some or all of their staked KOII. 15 | 16 | When writing a task, you're able to customize each step: what work you want done, what should be submitted as proof, how that proof should be checked, and what rewards and penalties you want to set. 17 | 18 | ![Runtime flow](./imgs/gradual-consensus.png) 19 | 20 | > [!TIP] 21 | > 22 | > For a more in-depth explanation of how Koii Tasks run, see our docs on [runtime flow and gradual consensus](https://docs.koii.network/concepts/what-are-tasks/what-are-tasks/gradual-consensus). 23 | 24 | Now let's see how runtime flow works in the EZ Testing example. 25 | 26 | ### Example 27 | 28 | #### Do the Work (task) 29 | 30 | **File:** `src/task/1-task.js` 31 | **Action:** This task simply saves the string "Hello, World!" to the local database 32 | 33 | #### Submit Proofs (submission) 34 | 35 | **File:** `src/task/2-submission.js` 36 | **Action:** In this case, we are fetching the string from the local database and submitting it on-chain as our proof. Note that in a real task, we would usually have more complex proofs that cannot be submitted directly. We will discuss how to deal with that in later lessons. 37 | 38 | #### Review Proofs (audit) 39 | 40 | **File:** `src/task/3-audit.js` 41 | **Action:** Here we are verifying whether the submission is the string "Hello, World!" 42 | 43 | #### Distribute Rewards 44 | 45 | **File:** `src/task/4-distribution.js` 46 | **Action:** Rewards are distributed to each node that completed the work. Here we are penalizing incorrect submissions by removing 70% of their stake and equally distributing the bounty per round to all successful submissions. 47 | 48 | ### Alternate Testing 49 | 50 | When developing your task, you'll want to iterate quickly, and having to deploy a task or launch the desktop node can be a hassle. We've provided a simple solution in the form of a testing script that will allow you to simulate rounds and test your task functionality. The script is available in `tests/simulateTask.js` and you can run it via `yarn test`. There are three optional command line arguments you can use. The first is the number of rounds you want to run the task (default is 1); the second is the delay between rounds, in ms (default is 5000); and the third is the delay between individual steps in a round, in ms (default is 1000). 51 | 52 | Now that we've seen how tasks work, let's deploy one. [PartIV](./PartIV.md) 53 | -------------------------------------------------------------------------------- /Lesson 1/PartIV.md: -------------------------------------------------------------------------------- 1 | # Lesson 1: Introduction to Koii Tasks 2 | 3 | ## Part IV: Deploying a Task 4 | 5 | > [!WARNING] 6 | > 7 | > In order to deploy a task you must download the desktop node and **run at least one task**. This will make a staking wallet that is usable for IPFS, which is required during deployment. If you have followed the EZSandbox tutorial up to this point, you already have a staking wallet. 8 | 9 | ### Environment Variables 10 | 11 | Before we deploy a task, let's take a quick look at how to add environment variables to your task. These are called "Task Extensions" in the node, and they allow you to ask each node operator for customized data needed for the task - for example, Twitter login credentials for a task that's accessing Twitter. 12 | 13 | To get a variable in your task, add it to your `config-task.yml`, in the `requirementsTags` section: 14 | 15 | ```yml 16 | requirementsTags: 17 | - type: TASK_VARIABLE 18 | value: "VARIABLE_NAME" 19 | description: "Variable description" 20 | ``` 21 | 22 | The value and description will be shown in the desktop node, so make them descriptive enough that users will understand what to enter: 23 | 24 | ![Archive Twitter Task Extensions](./imgs/secrets-example.png) 25 | 26 | Then you can use them in your task like any other Node.js environment variable, with `process.env.VARIABLE_NAME`. (While testing locally, you should define these in your .env). 27 | 28 | > [!IMPORTANT] 29 | > 30 | > In order to add new environment variables to a task so they can be configured in the desktop node, you must deploy or update it. This is because the task metadata is set at the time of deployment and can only be changed by updating the task. However, if you would like to test locally first, you can add the variables to your .env and run prod-debug. 31 | 32 | ### Deploying a Task 33 | 34 | > [!WARNING] 35 | > 36 | > In order to deploy a task, you **MUST** be using at least Node.js v18. 37 | > To check the version run: 38 | > 39 | > ```sh 40 | > node --version 41 | > ``` 42 | > 43 | > If your version is too low, you can download the latest LTS version [here](https://nodejs.org/en) 44 | 45 | #### Building 46 | 47 | The first step before deployment is to build your executable. First, makes sure you have installed all the necessary dependencies using 48 | 49 | ```sh 50 | yarn 51 | ``` 52 | 53 | Then run 54 | 55 | ```sh 56 | yarn webpack 57 | ``` 58 | 59 | in order to create the executable. 60 | 61 | #### Install Koii CLI Suite 62 | 63 | If you have not already installed the Koii CLI Suite, you can find the instructions [here](https://www.koii.network/docs/develop/command-line-tool/koii-cli/install-cli). 64 | 65 | #### Get Wallet 66 | 67 | There are two ways to get a wallet for deploying your tasks. The simplest method is to use your wallet from the desktop node, which is what we'll do here. You also have the option to [create a new wallet using the CLI](https://docs.koii.network/develop/command-line-tool/koii-cli/create-wallet), if you prefer. 68 | 69 | #### Fund Your Wallet 70 | 71 | If you're attending a live event, you will receive tokens to pay the deployment fees. If you're not attending a live event, you can earn tokens by running tasks in the desktop node. 72 | 73 | To request tokens, you'll need to provide your wallet's public key. You can find that in the desktop node: 74 | 75 | ![locating public key](./imgs/node-public-key.png) 76 | 77 | #### Create Task CLI 78 | 79 | Now it's time to deploy our executable. For this you'll need to run 80 | 81 | ```sh 82 | npx @_koii/create-task-cli@latest 83 | ``` 84 | 85 | which will show you the following menu: 86 | 87 | ```sh 88 | ? Select operation › - Use arrow-keys. Return to submit. 89 | ❯ Create a New Local Repository 90 | Deploy a New Task 91 | Update Existing Task 92 | Activate/Deactivate Task 93 | Claim Reward 94 | Fund Task with More KOII 95 | Withdraw Staked Funds from Task 96 | Upload Assets to IPFS (Metadata/Local Vars) 97 | ``` 98 | 99 | Choose `Deploy a New Task`. Next, you may be asked if you want to use your Koii CLI wallet: 100 | 101 | ```sh 102 | It looks like you have a koii cli installed. Would you like to use your koii cli key (/home/laura/.config/koii/id.json) to deploy this task? › (y/N) 103 | ``` 104 | 105 | If you're not using a Koii CLI wallet, be sure to choose `no` at this point. Next you may be asked a similar question about your desktop node wallet: 106 | 107 | ```sh 108 | It looks like you have a desktop node installed. Would you like to use your desktop node key (/home/laura/.config/KOII-Desktop-Node/wallets/Laura_mainSystemWallet.json) to deploy this task? › (y/N) 109 | ``` 110 | 111 | In most cases, you should choose `yes` at this point. If you choose `no` for this as well, or if the CLI can't automatically detect the location of your wallet, you will be asked to manually enter the path to your wallet: 112 | 113 | ```sh 114 | ? Enter the path to your wallet › 115 | ``` 116 | 117 | In the case of your desktop node wallet, it should be located at `/KOII-Desktop-Node/wallets/_mainSystemWallet.json`. 118 | 119 | The OS-specific paths are as follows: 120 | 121 | **Windows**: `/Users//AppData/Roaming` 122 | 123 | **Mac**: `/Users//Library/Application Support` 124 | 125 | **Linux**: `/home//.config` (This path contains a dot folder that may be hidden by default. You can show hidden folders by pressing Ctrl-H) 126 | 127 | In the example below, the wallet is located at `home/laura/.config/KOII-Desktop-Node/wallets/Laura_mainSystemWallet.json` 128 | 129 | ![system wallet location](./imgs/system-wallet.png) 130 | 131 | After you've entered the path to your wallet, you'll be asked how you want to configure your task: 132 | 133 | ```sh 134 | ? Select operation › - Use arrow-keys. Return to submit. 135 | ❯ using CLI 136 | using config YML 137 | ``` 138 | 139 | Choose "Using config YML". 140 | 141 | #### Staking Wallet 142 | 143 | > [!NOTE] 144 | > 145 | > **Why do I need two wallets? What's the difference?** 146 | > 147 | > The wallet you use for deploying your task is the one that needs to be funded, as it will be used for paying deployment fees. However, your task executable must be uploaded to IPFS so it can be distributed to the desktop nodes. In order to ensure the security of uploads, all file uploads must be signed. This signing process requires the use of a wallet called a Staking Wallet, which has a special role in running Koii tasks. This special role allows it to be used for signing uploads. This wallet does not need to have a balance. 148 | 149 | Next, you'll be asked how you would like to upload your metadata: 150 | 151 | ```sh 152 | ? Select operation › - Use arrow-keys. Return to submit. 153 | ❯ Using KOII Storage SDK 154 | Manually Input IPFS 155 | ``` 156 | 157 | Choose `Using KOII Storage SDK`. 158 | 159 | ```sh 160 | ? It looks like you have a desktop node installed. Would you like to use your desktop node staking key (/home/laura/.config/KOII-Desktop-Node/namespace/Laura_stakingWallet.json) to sign this upload to IPFS? › (y/N) 161 | ``` 162 | 163 | If you choose no, or if your staking wallet's location cannot be found automatically, you will be asked to manually enter the path: 164 | 165 | ```sh 166 | ? Enter the path to your staking wallet › 167 | ``` 168 | 169 | When you installed the desktop node, a staking wallet was created for you automatically. This can be found in `/KOII-Desktop-Node/namespace/_stakingWallet.json`. 170 | 171 | > [!IMPORTANT] 172 | > 173 | > In order for your staking wallet to be usable by the CLI, you must run at least one task in the desktop node. 174 | 175 | #### Confirm 176 | 177 | You will be then be prompted to confirm that you want to pay the rent and bounty, type 'y' to confirm: 178 | 179 | ```sh 180 | Your account will be deducted XX KOII for creating the task, which includes the rent exemption(XX KOII) and bounty amount fees (XX KOII) › (y/N) 181 | ``` 182 | 183 | > [!NOTE] 184 | > 185 | > If you are using the default values in config-task.yml and deploying a Koii Task, your total deployment fee should be about 17 KOII, which is the minimum possible amount. If you do not have enough KOII, you can get some by running tasks in the desktop node. If you're at a Koii-sponsored event, we will provide you with the necessary tokens. 186 | 187 | Your task should now be deployed successfully and you should see a response similar to this: 188 | 189 | ```sh 190 | Calling Create Task 191 | Task Id: 9oDEkeHwyGJVect8iEF1hHPKYdkqbtRToarbi8KQtgNS 192 | Stake Pot Account Pubkey: stakepotaccountp39zkKbCKoiLp3wZ66TuUu5LtS9d 193 | Note: Task Id is basically the public key of taskStateInfoKeypair.json 194 | Success 195 | ``` 196 | 197 | > [!IMPORTANT] 198 | > 199 | > Make sure you save your task ID every time you deploy or update! Not only do you need it to run your task in the desktop node, it's required when updating your task. 200 | 201 | Congratulations, you've deployed a task! 202 | 203 | In this lesson, you've learned how to run, debug, and deploy a task. If you are interested in deploying a task in a Docker container, continue to [Part 5](PartV.md). 204 | 205 | Otherwise, let's get into the specifics of writing a task. [Lesson 2](../Lesson%202/README.md) 206 | -------------------------------------------------------------------------------- /Lesson 1/PartV.md: -------------------------------------------------------------------------------- 1 | # Lesson 1: Introduction to Koii Tasks 2 | 3 | ## Part V: Containerized ORCA Tasks (optional) 4 | 5 | Although many Koii tasks can be written exclusively in Javascript, there are many cases where it may be preferable to use a different language, or you may need advanced configurations with additional services. For these cases, we provide integration with Orca, allowing you to run your task in a Docker container while still interacting with the Koii blockchain. 6 | 7 | ### Step 1: Get the task template 8 | 9 | Orca tasks require a specialized task template that handles the interaction between your container and the blockchain. Clone [this template](https://github.com/koii-network/orca-task-template) to start developing an Orca task. The README covers the specific of how to integrate your container. 10 | 11 | ### Step 2: Prepare your container 12 | 13 | If you're starting from scratch, we provide a simple example container with a minimal Flask server, demonstrating how to set up the necessary endpoints. 14 | 15 | If you are modifying an existing container, you will need to ensure that it is running an HTTP server on port 8080 with the following endpoints: 16 | 17 | - `/healthz`: To verify your container is running, Orca requires an endpoint at that accepts a post request and returns a 200 response. The content of the response is unimportant. 18 | - `/task/:roundNumber`. This endpoint should kick off the task each round, and store the result of the task (your proofs) with the round number, so it can be retrieved by `submission`. This will require a database solution of your choice, managed from within the container. 19 | - `/submission/:roundNumber` Retrieves the stored submission data. 20 | - `/audit`: Check the submission (using whatever method makes sense for your task) and return a boolean representing whether or not the submission was correct. 21 | 22 | ### Upload your container 23 | 24 | You will need to store your container image in a publicly-accessible container repository like Docker Hub. 25 | 26 | ### Update the settings file 27 | 28 | Once uploaded, update your image url in `OrcaSettings.js`. If you have a more complex configuration, you can define a Kubernetes pod spec. See the example provided; your primary container name _must_ be as shown, and the basePath must be included if adding additional volumes. 29 | 30 | ### Deploy 31 | 32 | Deploy a KOII or KPL task just as you normally would! 33 | -------------------------------------------------------------------------------- /Lesson 1/README.md: -------------------------------------------------------------------------------- 1 | # Lesson 1: Introduction to Koii Tasks 2 | 3 | - Part 1: [Running an Existing Task](./PartI.md) 4 | 5 | - Part 2: [Debugging a Task](./PartII.md) 6 | 7 | - Part 3: [How Do Koii Tasks Work?](./PartIII.md) 8 | 9 | - Part 4: [Deploying a Task](./PartIV.md) 10 | 11 | - Part 5: [Containerized Tasks](./PartV.md) 12 | -------------------------------------------------------------------------------- /Lesson 1/ez-testing-task/1-task.js: -------------------------------------------------------------------------------- 1 | import { namespaceWrapper } from "@_koii/task-manager/namespace-wrapper"; 2 | 3 | export async function task(roundNumber) { 4 | // Run your task and store the proofs to be submitted for auditing 5 | // The submission of the proofs is done in the submission function 6 | try { 7 | console.log(`EXECUTE TASK FOR ROUND ${roundNumber}`); 8 | console.log("Started Task", new Date(), "TEST"); 9 | // you can optionally return this value to be used in debugging 10 | await namespaceWrapper.storeSet("value", "Hello, World!"); 11 | } catch (error) { 12 | console.error("EXECUTE TASK ERROR:", error); 13 | } 14 | } 15 | -------------------------------------------------------------------------------- /Lesson 1/ez-testing-task/2-submission.js: -------------------------------------------------------------------------------- 1 | import { namespaceWrapper } from "@_koii/task-manager/namespace-wrapper"; 2 | 3 | export async function submission(roundNumber) { 4 | /** 5 | * Submit the task proofs for auditing 6 | * Must return a string of max 512 bytes to be submitted on chain 7 | */ 8 | try { 9 | console.log(`MAKE SUBMISSION FOR ROUND ${roundNumber}`); 10 | console.log("Started Submission", new Date(), "EZ TESTING"); 11 | return await namespaceWrapper.storeGet("value"); 12 | } catch (error) { 13 | console.error("MAKE SUBMISSION ERROR:", error); 14 | } 15 | } 16 | -------------------------------------------------------------------------------- /Lesson 1/ez-testing-task/3-audit.js: -------------------------------------------------------------------------------- 1 | export async function audit(submission, roundNumber, submitterKey) { 2 | /** 3 | * Audit a submission 4 | * This function should return true if the submission is correct, false otherwise 5 | */ 6 | console.log(`AUDIT SUBMISSION FOR ROUND ${roundNumber} from ${submitterKey}`); 7 | console.log("Started Audit", new Date(), "EZ TESTING"); 8 | return submission === "Hello, World!"; 9 | } 10 | -------------------------------------------------------------------------------- /Lesson 1/ez-testing-task/4-distribution.js: -------------------------------------------------------------------------------- 1 | // Define the percentage by which to slash the stake of submitters who submitted incorrect values 2 | // 0.7 = 70% 3 | const SLASH_PERCENT = 0.7; 4 | 5 | export async function distribution(submitters, bounty, roundNumber) { 6 | /** 7 | * Generate the reward list for a given round 8 | * This function should return an object with the public keys of the submitters as keys 9 | * and the reward amount as values 10 | * 11 | * IMPORTANT: If the slashedStake or reward is not an integer, the distribution list will be rejected 12 | * Values are in ROE, or the KPL equivalent (1 Token = 10^9 ROE) 13 | * 14 | */ 15 | console.log(`MAKE DISTRIBUTION LIST FOR ROUND ${roundNumber}`); 16 | 17 | // Initialize an empty object to store the final distribution list 18 | const distributionList = {}; 19 | 20 | // Initialize an empty array to store the public keys of submitters with correct values 21 | const approvedSubmitters = []; 22 | 23 | // Iterate through the list of submitters and handle each one 24 | for (const submitter of submitters) { 25 | // If the submitter's votes are 0, they do not get any reward 26 | if (submitter.votes === 0) { 27 | distributionList[submitter.publicKey] = 0; 28 | 29 | // If the submitter's votes are negative (submitted incorrect values), slash their stake 30 | } else if (submitter.votes < 0) { 31 | // Slash the submitter's stake by the defined percentage 32 | const slashedStake = Math.floor(submitter.stake * SLASH_PERCENT); 33 | // Add the slashed amount to the distribution list 34 | // since the stake is positive, we use a negative value to indicate a slash 35 | distributionList[submitter.publicKey] = -slashedStake; 36 | 37 | // Log that the submitter's stake has been slashed 38 | console.log("CANDIDATE STAKE SLASHED", submitter.publicKey, slashedStake); 39 | 40 | // If the submitter's votes are positive, add their public key to the approved submitters list 41 | } else { 42 | approvedSubmitters.push(submitter.publicKey); 43 | } 44 | } 45 | 46 | // If no submitters submitted correct values, return the current distribution list 47 | if (approvedSubmitters.length === 0) { 48 | console.log("NO NODES TO REWARD"); 49 | return distributionList; 50 | } 51 | 52 | // Calculate the reward for each approved submitter by dividing the bounty per round equally among them 53 | const reward = Math.floor(bounty / approvedSubmitters.length); 54 | 55 | console.log("REWARD PER NODE", reward); 56 | 57 | // Assign the calculated reward to each approved submitter 58 | approvedSubmitters.forEach((candidate) => { 59 | distributionList[candidate] = reward; 60 | }); 61 | 62 | // Return the final distribution list 63 | return distributionList; 64 | } 65 | -------------------------------------------------------------------------------- /Lesson 1/imgs/add-task-advanced.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 1/imgs/add-task-advanced.png -------------------------------------------------------------------------------- /Lesson 1/imgs/gradual-consensus.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 1/imgs/gradual-consensus.png -------------------------------------------------------------------------------- /Lesson 1/imgs/main-log.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 1/imgs/main-log.png -------------------------------------------------------------------------------- /Lesson 1/imgs/my-node-open-logs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 1/imgs/my-node-open-logs.png -------------------------------------------------------------------------------- /Lesson 1/imgs/node-public-key.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 1/imgs/node-public-key.png -------------------------------------------------------------------------------- /Lesson 1/imgs/open-main-log.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 1/imgs/open-main-log.png -------------------------------------------------------------------------------- /Lesson 1/imgs/public-key.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 1/imgs/public-key.png -------------------------------------------------------------------------------- /Lesson 1/imgs/secrets-example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 1/imgs/secrets-example.png -------------------------------------------------------------------------------- /Lesson 1/imgs/stacking-rounds.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 1/imgs/stacking-rounds.png -------------------------------------------------------------------------------- /Lesson 1/imgs/system-wallet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 1/imgs/system-wallet.png -------------------------------------------------------------------------------- /Lesson 1/imgs/task-list.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 1/imgs/task-list.png -------------------------------------------------------------------------------- /Lesson 1/imgs/task-log.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 1/imgs/task-log.png -------------------------------------------------------------------------------- /Lesson 2/PartI.md: -------------------------------------------------------------------------------- 1 | # Lesson 2: Networking and Storage Task 2 | 3 | ## Part I: Node to Node Communication 4 | 5 | So far, we've seen how to navigate the node and run a simple task. In this lesson, we'll get an introduction to writing our own tasks by looking at node-to-node communication. First, we'll enable networking on our node. 6 | 7 | ![Lesson_2_updated__1_](https://github.com/koii-network/ezsandbox/assets/66934242/a6ba91f1-7148-47f7-9d4d-b87a1bc28f11) 8 | 9 | ### UPnP 10 | 11 | Everything we've done until now has only really involved a single node, with a task that was independent of other nodes. This may not always be the case however; having a way to collaborate with other nodes can be extremely useful for writing tasks used for decentralized social applications, Web3 gaming, and many other tasks such as mathematical computations. 12 | 13 | Koii uses Universal Plug and Play (UPnP) to accomplish this. UPnP is most commonly used by devices like gaming consoles and smart devices which makes connecting to these devices very straightforward, but that's not its only application. UPnP is a set of networking protocols that allows a network of devices to seamlessly connect to each other, making it perfect for allowing Koii nodes to communicate. 14 | 15 | ### UPnP Security on Koii 16 | 17 | As mentioned above, UPnP is awesome because it greatly reduces the complexity required for having interconnected devices. However, the tradeoff is that this allows direct access to your device, leading to possible security concerns because UPnP enables automatic port forwarding. 18 | 19 | At Koii, we recognize this issue, which is why UPnP is NOT required for you to use the Node and is disabled by default. You are free to enable or disable it as you see fit. 20 | 21 | UPnP allows Koii tasks to utilize our vast network of over 60,000 Nodes to communicate effectively which is why it is such a powerful tool. Any tasks whitelisted by Koii are publicly available to inspect so you can be assured that UPnP is being used for productivity, not malice. 22 | 23 | Additionally, if you would like to use UPnP with a more secure setup, you can manually use tunneling by toggling the `Enforce Tunneling` option as shown in the screenshot above. Tunneling allows your device to set up a proxy before connecting with other devices, which enhances security since your device won't be completely exposed to another, however it does require additional resources. 24 | 25 | ### Enabling UPnP in the Node 26 | 27 | If you want to enable UPnP on your Node, head to Settings -> Networking -> Enable Networking Features and simply toggle on the `Enable Networking` feature. 28 | 29 | Additionally, check the advanced tab to download the required UPnP binaries if you don't have them. 30 | 31 | ![Enable UPnP](./imgs/enable-upnp.png) 32 | 33 | In the next lesson, we'll learn how we can write our own task that makes use of UPnP for networking. 34 | 35 | [Click here to start PartII. Writing a Networking Task](./PartII.md) 36 | -------------------------------------------------------------------------------- /Lesson 2/PartII.md: -------------------------------------------------------------------------------- 1 | # Lesson 2: Networking and Storage Task 2 | 3 | ## Part II: Writing the Task 4 | 5 | Now that you've got a basic understanding of how networking is done in the Node, it's time for deploy your own task that utilizes REST APIs! 6 | 7 | Prerequisites: 8 | 9 | - Understanding of UPnP 10 | - Basic understanding of task template 11 | - Knowledge on task deployment 12 | 13 | ### Initial Setup 14 | 15 | Clone the [task template repo](https://github.com/koii-network/task-template) to get started. We'll walk you through how to edit the template files for this task. 16 | 17 | In order to enable node to node communication, we must edit the `config-task.yml`. We will discuss this file further in [Lesson 3](../Lesson%203/README.md) but for now, just add the `REQUIRE_INTERNET` addon and a `SECRET` variable as shown below: 18 | 19 | ```yml 20 | requirementsTags: 21 | - type: TASK_VARIABLE 22 | value: "SECRET" 23 | description: "The secret you want to tell other nodes!" 24 | - type: ADDON 25 | value: "REQUIRE_INTERNET" 26 | ``` 27 | 28 | See [`upnp-basics/config-task.yml`](./upnp-basics/config-task.yml) if you're stuck. 29 | 30 | ### Create Routes: Your Node as a Server 31 | 32 | Head to the `src/task/5-routes.js` file in the task template. You should see a `routes()` function with an example endpoint: 33 | 34 | ```javascript 35 | export function routes() { 36 | app.get("/value", async (_req, res) => { 37 | const value = await namespaceWrapper.storeGet("value"); 38 | console.log("value", value); 39 | res.status(200).json({ value: value }); 40 | }); 41 | } 42 | ``` 43 | 44 | This endpoint is accessible from other Nodes, allowing you to share the information for use by other nodes. 45 | 46 | There are two ways you can access this endpoint when the task is running on your node: 47 | 48 | 1. Visit `http://localhost:30017/task//value` 49 | 2. Check the task logs. When the task starts, there will be an entry in the logs noting which port the task is running on. You can then view the endpoint by visiting `http://localhost:/value`. Note that this port changes periodically; it's better to use the first option. 50 | 51 | The endpoints you define in `routes.js` will dictate what kinds of communications can occur between your device and another. Let's create a simple mechanism for passing basic data between two nodes. 52 | 53 | Try one yourself! Change the endpoint to `/secret`. Save a secret word or phrase of your choice (maximum 512 bytes) in your `.env` file with the key `secret` and return a JSON response `{ secret: }`. ([Answer here](./upnp-basics/src/task/5-routes.js)) 54 | 55 | ### Access Endpoints: Your Node as a Client 56 | 57 | With our own endpoint set up, we now need to make calls to other Nodes' endpoints and access their secrets. 58 | 59 | We've provided a file called [`upnpUtils.js`](./upnp-basics/src/task/upnpUtils.js); you can use these utility functions to help you work with other nodes. 60 | 61 | `getAddressArray()` grabs the list of IP addresses for all the nodes running this task. `getRandomNodeEndpoint()` takes the list of IP addresses and chooses one at random, return that node's value. 62 | 63 | In your task function, get a random node and retrieve its secret, then save it to the local DB with the key `secret`. ([Answer here](./upnp-basics/src/task/1-task.js)) 64 | 65 | ### Submitting 66 | 67 | In your submission function, get the secret from the local DB and return it. ([Answer here](./upnp-basics/src/task/2-submission.js)) 68 | 69 | ### Auditing 70 | 71 | The last thing we'll need to do is audit the results. Because the value can vary from node to node, we won't be able to check a specific value. Instead, we'll just confirm that the value is a string. 72 | 73 | In your audit function, return a boolean value: `true` if the value is a non-empty string, `false` otherwise. This is your node's vote; it will be combined with all other votes on this submission to decide if it's valid or not. ([Answer here](./upnp-basics/src/task/3-audit.js)) 74 | 75 | > [!WARNING] 76 | > 77 | > This is not a very good audit! A node could easily pass audit without accessing another node by just submitting any non-empty string. We'll look at ways to write better audits in a later lesson. 78 | 79 | > [!IMPORTANT] 80 | > In order to test this task, you must set `SECRET` in your .env file 81 | 82 | With this simple setup, every Node can provide server endpoints to be reached, and call other Node's endpoints to fetch data, resulting in Node to Node communication! 83 | 84 | In the next section, we'll learn how to store and share files. [Part III. File Storage & Sharing](./PartIII.md) 85 | -------------------------------------------------------------------------------- /Lesson 2/PartIII.md: -------------------------------------------------------------------------------- 1 | # Lesson 2: Networking and Storage Task 2 | 3 | ## Part III: File Storage & Sharing 4 | 5 | Now that we've seen how communication works between nodes, let's try something a little different: working with files. 6 | 7 | ### IPFS 8 | 9 | Koii uses IPFS (the InterPlanetary File System) to store data outside of its blockchain. This helps keep transactions fast and cost-effective. We have developed the [Koii Storage SDK](https://www.npmjs.com/package/@_koii/storage-task-sdk) so you can easily upload and retrieve files, and this lesson will go over how to use it. 10 | 11 | ### Uploading a File 12 | 13 | To begin, clone the [task template repo](https://github.com/koii-network/task-template). Since we're going to be using the Koii Storage SDK for IPFS, we'll need to add this to the package.json: 14 | 15 | ```sh 16 | yarn add @_koii/storage-task-sdk 17 | ``` 18 | 19 | Also, this task will use the SECRET .env variable we defined in the previous UPnP task. Be sure to set it if you haven't yet. 20 | 21 | This time, instead of having an endpoint that directly shares a value, we'll add the value to a file and upload it to IPFS. When the upload is successful, you'll receive a content identifier (CID) that can be used to retrieve the file later. 22 | 23 | We've provided [some useful utility functions](./file-sharing/task/fileUtils.js) for working with IPFS. 24 | 25 | In your task function, get a value from your `.env` file (don't forget to add the variable to your `config-task.yml` if you're going to deploy the task). and save it to a file on IPFS, then save the CID to your local DB. ([Answer here](./file-sharing/task/1-task.js)) 26 | 27 | ### Submitting 28 | 29 | This is very similar to the previous submissions you've done; see if you can work out what you need to do! ([Answer here](./file-sharing/task/2-submission.js)) 30 | 31 | ### Retrieving a file 32 | 33 | We've provided the necessary functions to get the file content, make use of them to validate the file content. Just like in the UPnP lesson, check if the value is a string and is not empty. ([Answer here](./file-sharing/task/3-audit.js)) 34 | 35 | > [!WARNING] 36 | > 37 | > Like the UPnP task, this is a poor audit. In particular, if a user uploads a file that isn't text, or if they provide an invalid CID, we simply let an error be thrown and don't vote on their submission, but it would be more appropriate to vote false. 38 | > However, voting false to every error isn't an ideal solution either - if, for example, a node has network issues and can't retrieve the file from IPFS, it could end up voting false when it shouldn't. 39 | 40 | And that's it! You've successfully written a task that uses IPFS to store data. 41 | 42 | > [!IMPORTANT] 43 | > In order to test this task, you must set `STAKING_WALLET_PATH` and `SECRET` in your .env file 44 | 45 | Next up, we'll take a look at secrets and the config options in [Lesson 3](../Lesson%203/README.md). 46 | -------------------------------------------------------------------------------- /Lesson 2/README.md: -------------------------------------------------------------------------------- 1 | # Lesson 2: Networking and Storage Task 2 | 3 | - Lesson 1: [Node to Node Communication](./PartI.md) 4 | 5 | - Lesson 2: [Writing the Task](./PartII.md) 6 | 7 | - Lesson 3: [File Storage & Sharing](./PartIII.md) 8 | -------------------------------------------------------------------------------- /Lesson 2/file-sharing/config-task.yml: -------------------------------------------------------------------------------- 1 | # See the config-task.yml file in the task template repository for the full configuration 2 | 3 | requirementsTags: 4 | - type: CPU 5 | value: "4-core" 6 | - type: RAM 7 | value: "5 GB" 8 | - type: STORAGE 9 | value: "5 GB" 10 | - type: TASK_VARIABLE 11 | value: "SECRET" 12 | description: "The secret you want to tell other nodes!" 13 | -------------------------------------------------------------------------------- /Lesson 2/file-sharing/task/1-task.js: -------------------------------------------------------------------------------- 1 | import { namespaceWrapper } from "@_koii/namespace-wrapper"; 2 | import { storeFile } from "./fileUtils.js"; 3 | 4 | export async function task(roundNumber) { 5 | // Run your task and store the proofs to be submitted for auditing 6 | // The submission of the proofs is done in the submission function 7 | console.log(`EXECUTE TASK FOR ROUND ${roundNumber}`); 8 | try { 9 | const cid = await storeFile(process.env.SECRET); 10 | await namespaceWrapper.storeSet("cid", cid); 11 | } catch (error) { 12 | console.error("EXECUTE TASK ERROR:", error); 13 | } 14 | } 15 | -------------------------------------------------------------------------------- /Lesson 2/file-sharing/task/2-submission.js: -------------------------------------------------------------------------------- 1 | import { namespaceWrapper } from "@_koii/namespace-wrapper"; 2 | 3 | export async function submission(roundNumber) { 4 | /** 5 | * Submit the task proofs for auditing 6 | * Must return a string of max 512 bytes to be submitted on chain 7 | */ 8 | try { 9 | console.log(`MAKE SUBMISSION FOR ROUND ${roundNumber}`); 10 | return await namespaceWrapper.storeGet("cid"); 11 | } catch (error) { 12 | console.error("MAKE SUBMISSION ERROR:", error); 13 | } 14 | } 15 | -------------------------------------------------------------------------------- /Lesson 2/file-sharing/task/3-audit.js: -------------------------------------------------------------------------------- 1 | import { getFileText } from "./fileUtils.js"; 2 | 3 | export async function audit(submission, roundNumber, submitterKey) { 4 | /** 5 | * Audit a submission 6 | * This function should return true if the submission is correct, false otherwise 7 | */ 8 | console.log(`AUDIT SUBMISSION FOR ROUND ${roundNumber} from ${submitterKey}`); 9 | const fileContent = await getFileText(submission); 10 | return typeof fileContent === "string" && fileContent.length > 0; 11 | } 12 | -------------------------------------------------------------------------------- /Lesson 2/file-sharing/task/fileUtils.js: -------------------------------------------------------------------------------- 1 | import { KoiiStorageClient } from "@_koii/storage-task-sdk"; 2 | import { namespaceWrapper } from "@_koii/namespace-wrapper"; 3 | import fs from "fs"; 4 | 5 | export async function storeFile(data, filename = "value.json") { 6 | // Create a new instance of the Koii Storage Client 7 | const client = KoiiStorageClient.getInstance({}); 8 | const basePath = await namespaceWrapper.getBasePath(); 9 | try { 10 | // Write the data to a temp file 11 | fs.writeFileSync(`${basePath}/${filename}`, JSON.stringify(data)); 12 | 13 | // Get the user staking account, to be used for signing the upload request 14 | const userStaking = await namespaceWrapper.getSubmitterAccount(); 15 | 16 | // Upload the file to IPFS and get the CID 17 | const { cid } = await client.uploadFile( 18 | `${basePath}/${filename}`, 19 | userStaking 20 | ); 21 | 22 | console.log(`Stored file CID: ${cid}`); 23 | // Delete the temp file 24 | fs.unlinkSync(`${basePath}/${filename}`); 25 | 26 | return cid; 27 | } catch (error) { 28 | console.error("Failed to upload file to IPFS:", error); 29 | fs.unlinkSync(`${basePath}/${filename}`); 30 | throw error; 31 | } 32 | } 33 | 34 | export async function getFileBlob(cid, filename = "value.json") { 35 | const client = new KoiiStorageClient(); 36 | return await client.getFile(cid, filename); 37 | } 38 | 39 | export async function getFileText(cid, filename = "value.json") { 40 | const fileBlob = await getFileBlob(cid, filename); 41 | if (!fileBlob) return null; 42 | return await fileBlob.text(); 43 | } 44 | -------------------------------------------------------------------------------- /Lesson 2/imgs/enable-upnp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 2/imgs/enable-upnp.png -------------------------------------------------------------------------------- /Lesson 2/imgs/require-internet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 2/imgs/require-internet.png -------------------------------------------------------------------------------- /Lesson 2/upnp-basics/config-task.yml: -------------------------------------------------------------------------------- 1 | # See the config-task.yml file in the task template repository for the full configuration 2 | 3 | requirementsTags: 4 | - type: CPU 5 | value: "4-core" 6 | - type: RAM 7 | value: "5 GB" 8 | - type: STORAGE 9 | value: "5 GB" 10 | - type: TASK_VARIABLE 11 | value: "SECRET" 12 | description: "The secret you want to tell other nodes!" 13 | - type: ADDON 14 | value: "REQUIRE_INTERNET" 15 | -------------------------------------------------------------------------------- /Lesson 2/upnp-basics/src/task/1-task.js: -------------------------------------------------------------------------------- 1 | import { namespaceWrapper, TASK_ID } from "@_koii/namespace-wrapper"; 2 | import axios from "axios"; 3 | 4 | import { getAddressArray, getRandomNodeEndpoint } from "./upnpUtils.js"; 5 | 6 | export async function task(roundNumber) { 7 | // Run your task and store the proofs to be submitted for auditing 8 | // The submission of the proofs is done in the submission function 9 | console.log(`EXECUTE TASK FOR ROUND ${roundNumber}`); 10 | try { 11 | // Get a list of the available IP addresses 12 | const IPAddressArray = await getAddressArray(); 13 | 14 | // Get a random node from the list 15 | const randomNode = await getRandomNodeEndpoint(IPAddressArray); 16 | console.log("RANDOM NODE", randomNode); 17 | 18 | // Fetch the value from the random node 19 | const response = await axios.get(`${randomNode}/task/${TASK_ID}/secret`); 20 | const secret = response.data.secret; 21 | console.log("SECRET", secret); 22 | 23 | // Store the result in the local database 24 | await namespaceWrapper.storeSet("secret", secret); 25 | } catch (error) { 26 | console.error("EXECUTE TASK ERROR:", error); 27 | } 28 | } 29 | -------------------------------------------------------------------------------- /Lesson 2/upnp-basics/src/task/2-submission.js: -------------------------------------------------------------------------------- 1 | import { namespaceWrapper } from "@_koii/namespace-wrapper"; 2 | 3 | export async function submission(roundNumber) { 4 | /** 5 | * Submit the task proofs for auditing 6 | * Must return a string of max 512 bytes to be submitted on chain 7 | */ 8 | try { 9 | console.log(`MAKE SUBMISSION FOR ROUND ${roundNumber}`); 10 | // Fetch and return the secret from the local database 11 | return await namespaceWrapper.storeGet("secret"); 12 | } catch (error) { 13 | console.error("MAKE SUBMISSION ERROR:", error); 14 | } 15 | } 16 | -------------------------------------------------------------------------------- /Lesson 2/upnp-basics/src/task/3-audit.js: -------------------------------------------------------------------------------- 1 | export async function audit(submission, roundNumber, submitterKey) { 2 | /** 3 | * Audit a submission 4 | * This function should return true if the submission is correct, false otherwise 5 | */ 6 | console.log(`AUDIT SUBMISSION FOR ROUND ${roundNumber} from ${submitterKey}`); 7 | // confirm that the submission is a string and is not empty 8 | return typeof submission === "string" && submission.length > 0; 9 | } 10 | -------------------------------------------------------------------------------- /Lesson 2/upnp-basics/src/task/5-routes.js: -------------------------------------------------------------------------------- 1 | import "dotenv/config"; 2 | 3 | export function setupRoutes(app) { 4 | if (app) { 5 | app.get("/secret", async (_req, res) => { 6 | res.status(200).json({ secret: process.env.SECRET }); 7 | }); 8 | } 9 | } 10 | -------------------------------------------------------------------------------- /Lesson 2/upnp-basics/src/task/upnpUtils.js: -------------------------------------------------------------------------------- 1 | import {namespaceWrapper} from "@_koii/namespace-wrapper"; 2 | 3 | export async function getAddressArray() { 4 | try { 5 | // get the task state from K2 (the Koii blockchain) 6 | const taskState = await namespaceWrapper.getTaskState(); 7 | console.log(taskState); 8 | 9 | // get the list of available IP addresses from the task state 10 | // nodeList is an object with key-value pairs in the form stakingKey: ipAddress 11 | const nodeList = taskState.ip_address_list ?? {}; 12 | 13 | // return just the IP addresses 14 | return Object.values(nodeList); 15 | } catch (e) { 16 | console.log("ERROR GETTING TASK STATE", e); 17 | } 18 | } 19 | 20 | export async function getRandomNodeEndpoint(IPAddressArray) { 21 | if (!IPAddressArray || IPAddressArray.length === 0) { 22 | throw new Error("No IP addresses available"); 23 | } 24 | // Choose a random index 25 | const randomIndex = Math.floor(Math.random() * IPAddressArray.length); 26 | // Return the IP address stored at the random index position 27 | return IPAddressArray[randomIndex]; 28 | } 29 | -------------------------------------------------------------------------------- /Lesson 3/PartI.md: -------------------------------------------------------------------------------- 1 | # Lesson 3: Secrets & Config 2 | 3 | _**DISCLAIMER**: This should only be used for educational and legal purposes such as publicly archiving data. Koii does not condone using tasks to steal data or infringe on personal privacy_ 4 | 5 | ## Part I. Secrets 6 | 7 | One of the best use cases for Koii tasks is web crawling. While there are many webpages that allow users to endlessly scroll without having an account, most content-based websites only give a small snippet of content without logging in. An autonomous web crawler has limited usefulness without a means to login. 8 | 9 | This is where secrets become very handy! 10 | 11 | ### Secrets in Your Node 12 | 13 | If you have explored the Node enough, you may have noticed that some tasks require **Task Extensions** before you're able to run them. These task extensions are secrets, similar to environment variable in a .env file. 14 | 15 | ![Archive Twitter Task Extensions](./imgs/secrets-example.png) 16 | 17 | Secrets allow a user to save private information so that we can use it without ever having access to the actual values. With the help of secrets, your crawler task can easily utilize a user's account to login, get full access to a webpage's content, and start archiving information. 18 | 19 | ### Secrets in Task Development 20 | 21 | In [Lesson 2](../Lesson%202/README.md), we added something to our `task-config.yml` requirements section to make UPnP configuration work. Now, we can see that a task developer can also use the requirements section to specify any number of secrets they may need for a task to function! For example, this is `Archive Twitter's` requirements section: 22 | 23 | ```yml 24 | requirementsTags: 25 | - type: TASK_VARIABLE 26 | value: 'TWITTER_USERNAME' 27 | description: 'The username of your volunteer Twitter account.' 28 | - type: TASK_VARIABLE 29 | value: 'TWITTER_PASSWORD' 30 | description: 'The password of your volunteer Twitter account.' 31 | - type: TASK_VARIABLE 32 | value: 'TWITTER_PHONE' 33 | description: 'If verification is required, will use your phone number to login.' 34 | - type: CPU 35 | value: '4-core' 36 | - type: RAM 37 | value: '5 GB' 38 | - type: STORAGE 39 | value: '5 GB' 40 | ``` 41 | 42 | (You can look at the `Twitter Archive` task more in depth [here](https://github.com/koii-network/task-X).) 43 | 44 | As you can see, each task extension has a corresponding value and description, which will be used by the Node. These task extensions will be automatically linked as environment variables for your use and can be accessed just like an entry in your .env, e.g. `process.env.TWITTER_USERNAME` 45 | 46 | One final thing to note is that during local development, you must specify these secrets in your .env file. For example, if you have a secret called `FIRST_NAME` in your config-task.yml, you would need to have a `FIRST_NAME` entry in your local .env file. 47 | 48 | Effectively, adding requirements allows you to build a .env on the user's computer, which can be used when they run the task code. 49 | 50 | Let's take a deeper look at the option in the `config-task.yml` file. [Part II. The Task Config File](./PartII.md) 51 | -------------------------------------------------------------------------------- /Lesson 3/PartII.md: -------------------------------------------------------------------------------- 1 | # Lesson 3: Secrets & Config 2 | 3 | ## Part II. The Task Config File 4 | 5 | This is a good point to take a closer look at the `config-task.yml` file and understand what the various options mean. 6 | 7 | ```yaml 8 | # Task Metadata - this will be displayed in the user's node in the task listing 9 | task_name: 'task name' # Maximum 24 characters 10 | author: 'Koii' 11 | description: 'description' 12 | repositoryUrl: 'https://github.com/koii-network/task-template' # Replace with your own repo URL 13 | imageUrl: 'imageUrl' 14 | infoUrl: 'infoUrl' 15 | 16 | # This is the method used to manage the distribution of the task executable to the users' nodes. You will not usually need to change this 17 | # Possible values: DEVELOPMENT, ARWEAVE or IPFS 18 | # IPFS is the default value, as the CLI automatically manages the upload process via the Koii Storage SDK. 19 | task_executable_network: 'IPFS' 20 | 21 | # Path to your executable webpack if the selected network is IPFS 22 | # If DEVELOPMENT name it `main` 23 | task_audit_program: 'dist/main.js' 24 | 25 | # The total duration of your task, measured in slots (with each slot approximately equal to 408ms). 26 | # There are three stages (rounds). The task stage uses the entire round time. 27 | # The submission stage submits task proofs during the submission window and audits them during the audit window. The remaining round time is a delay where no work is done. 28 | # The distribution stage prepares the distribution list for rewards during the submission window and audits the distribution list during the distribution window. The remaining round time is used to distribute rewards. 29 | round_time: 1500 30 | 31 | # The length in slots of the window for submitting task proofs/ditribution list. The submission window should be at least 1/3 of the round time. 32 | submission_window: 600 33 | 34 | # The length in slots of the window for auditing submissions/distribution list each round. The audit window should be at least 1/3 of the round time. 35 | audit_window: 600 36 | 37 | # Minimum stake amount: The minimum amount of KOII (or KPL token if it's a KPL task) that a user must stake in order to participate in the task. 38 | # Staked tokens can be recovered when the user is finished running the task 39 | # Staked tokens can be lost if the user's submission fails an audit 40 | minimum_stake_amount: 0.1 41 | 42 | # Task Bounty Type: Can be KOII or KPL 43 | task_type: 'KOII' 44 | 45 | # ONLY if task type is KPL, otherwise ignored 46 | # Token Mint Address. Fire Token address is given as an example here. 47 | token_type: "FJG2aEPtertCXoedgteCCMmgngSZo1Zd715oNBzR7xpR" 48 | 49 | # The total bounty amount that will be distributed to the task 50 | # This cannot be changed when updating a task 51 | total_bounty_amount: 10 52 | # The maximum amount that can be distributed each round 53 | # The actual amount distributed can be smaller than this value, but not larger 54 | bounty_amount_per_round: 0.1 55 | 56 | # Number of times the distribution list will be re-submitted if it fails audit. 57 | # It's also the number of rounds of submission data that will be kept on chain. 58 | allowed_failed_distributions: 3 59 | 60 | # Space: Space in MBs for the account size, that holds the task data. 61 | # For testing tasks this can be set to 0.1, but for production it should be set to at least 1. 62 | space: 0.1 63 | 64 | # Note that the value field in RequirementTag is optional, so it is up to you to include it or not based on your use case. 65 | # To add more global variables and task variables, please refer to the type, value, description format shown below. 66 | 67 | requirementsTags: 68 | - type: CPU 69 | value: '4-core' 70 | - type: RAM 71 | value: '5 GB' 72 | - type: STORAGE 73 | value: '5 GB' 74 | 75 | # ONLY for updating a task, otherwise ignored 76 | # Previous task ID. Every time you update the task, you should use the task ID from the previous update 77 | task_id: '' 78 | 79 | # Migration description: Provide the description for changes made in the new version of the task. 80 | migrationDescription: '' 81 | ``` 82 | 83 | 84 | 85 | Now that we've discussed the config options, let's move on to writing our crawler task. [Part III. Building a Crawler](./PartIII.md) 86 | -------------------------------------------------------------------------------- /Lesson 3/PartIII.md: -------------------------------------------------------------------------------- 1 | # Lesson 3: Secrets & Config 2 | 3 | ## Part III. Building a Crawler 4 | 5 | ![Lesson 3](https://github.com/koii-network/ezsandbox/assets/66934242/5cc14e75-0c0a-4625-b809-dc12af7d49a1) 6 | 7 | Before we start writing our own crawler code, it's helpful to understand how to structure a web crawler task on Koii. We've [previously covered IPFS](../Lesson%202/PartIII.md), so this should be fairly straightforward. 8 | 9 | Prerequisites: 10 | 11 | - General understanding of IPFS, see [Lesson 2, Part 3](../Lesson%202/PartIII.md) if you need a refresher 12 | - Understanding of [Puppeteer](https://pptr.dev/) 13 | 14 | ### Headless Mode 15 | 16 | Puppeteer is a browser than can be controlled programmatically. Because it is designed to be automated, it has a headless mode which allows it to run silently, without running a visible browser window. We have set `headless` to `false` so you'll be able to see the browser automation working, but if you don't need to see it, you can set `headless` to `true` instead. You should always set `headless: true` if you are deploying the task. 17 | 18 | ### Install packages 19 | 20 | As usual, make sure you clone the [task template](https://github.com/koii-network/task-template) as your first step. We'll also need to install a couple of packages for IPFS and Puppeteer: 21 | 22 | ```sh 23 | yarn add @_koii/storage-task-sdk puppeteer-chromium-resolver 24 | ``` 25 | 26 | ### Environment variables 27 | 28 | The task we're building here doesn't require any login info, but we'll be asking the user for a keyword they want to search for. Add the following to the `config-task.yml` in the requirementsTags: 29 | 30 | ```yaml 31 | - type: TASK_VARIABLE 32 | value: "SEARCH_TERM" 33 | description: "keyword to search for" 34 | ``` 35 | 36 | **NOTE**: If you're testing locally, make sure to also add `SEARCH_TERM` to your .env. 37 | 38 | ### Building the Crawler 39 | 40 | We've provided a [crawl function](./simple-crawler/task/crawler.js) you can use to scrape [redflagdeals](https://forums.redflagdeals.com/hot-deals-f9/`). If you examine this code, you'll notice it's only useful for the site we're scraping; scraping code must always be customized for your specific use case. 41 | 42 | ### Using the Crawler 43 | 44 | Use the crawler in your task function to search for the keyword you defined in your `.env` file and save the results to IPFS. You may want to use the [file utilities](./task/fileUtils.js) from the previous lesson. ([Answer here](./simple-crawler/task/1-task.js)) 45 | 46 | ### Submitting 47 | 48 | Submit your CID. ([Answer here](./simple-crawler/task/2-submission.js)) 49 | 50 | ### Auditing 51 | 52 | Let's do a little more with our audit this time: 53 | 54 | 1. Retrieve the file from IPFS and verify it exists. 55 | 2. Check that the file content is a non-empty string, then parse it as JSON. 56 | 3. Check that the content of the file is an array with at least one entry. 57 | 4. Check that each value within the array is a string. 58 | 59 | > [!WARNING] 60 | > 61 | > Our audits are getting better but they could still be improved significantly. See if you can spot some of the possible problems with this audit - how could someone exploit it? 62 | 63 | [Answer here](./simple-crawler/task/3-audit.js) 64 | 65 | > [!IMPORTANT] 66 | > In order to test this task, you must set `STAKING_WALLET_PATH` and `SEARCH_TERM` in your .env file 67 | 68 | Just like that, you've successfully created your very own web crawler! This template is very customizable and relatively simple. As you encounter more dynamic webpages, you may find it more difficult to web crawl. If you're concerned about websites with logging in, cookies, or dynamic content, we recommend checking out our [Twitter Archiver!](https://github.com/koii-network/task-X) 69 | 70 | [Click here to start Lesson 4](../Lesson%204//README.md) 71 | -------------------------------------------------------------------------------- /Lesson 3/README.md: -------------------------------------------------------------------------------- 1 | # Lesson 3: Secrets & Config 2 | 3 | - Part I: [Secrets](./PartI.md) 4 | 5 | - Part II: [The Task Config File](./PartII.md) 6 | 7 | - Part III: [Building a Crawler](./PartIII.md) 8 | -------------------------------------------------------------------------------- /Lesson 3/debugger.js: -------------------------------------------------------------------------------- 1 | require('dotenv').config; 2 | const os = require('os'); 3 | const path = require('path'); 4 | const { Connection, PublicKey } = require('@_koii/web3.js'); 5 | 6 | class Debugger { 7 | /* 8 | Create .env file with following variables or direclty input values to be used in live-debugging mode. 9 | */ 10 | static taskID = 11 | process.env.TASK_ID || 'CRzLkJvYnta2BnaSMMVkew7FRrZtZoNe7wGFogq7Hj5D'; 12 | static webpackedFilePath = process.env.WEBPACKED_FILE_PATH || 'dist/main.js'; 13 | static keywords = [process.env.TEST_KEYWORD] || ['']; 14 | static nodeDir = process.env.NODE_DIR || ''; 15 | 16 | static async getConfig() { 17 | Debugger.nodeDir = await this.getNodeDirectory(); 18 | 19 | let destinationPath = 20 | 'executables/' + (await this.gettask_audit_program()) + '.js'; 21 | let logPath = 'namespace/' + Debugger.taskID + '/task.log'; 22 | 23 | console.log('Debugger.nodeDir', Debugger.nodeDir); 24 | 25 | return { 26 | webpackedFilePath: Debugger.webpackedFilePath, 27 | destinationPath: destinationPath, 28 | keywords: Debugger.keywords, 29 | logPath: logPath, 30 | nodeDir: Debugger.nodeDir, 31 | taskID: Debugger.taskID, 32 | }; 33 | } 34 | 35 | static async getNodeDirectory() { 36 | if (Debugger.nodeDir) { 37 | return Debugger.nodeDir; 38 | } 39 | const homeDirectory = os.homedir(); 40 | let nodeDirectory; 41 | 42 | switch (os.platform()) { 43 | case 'linux': 44 | nodeDirectory = path.join( 45 | homeDirectory, 46 | '.config', 47 | 'KOII-Desktop-Node', 48 | ); 49 | break; 50 | case 'darwin': 51 | nodeDirectory = path.join( 52 | homeDirectory, 53 | 'Library', 54 | 'Application Support', 55 | 'KOII-Desktop-Node', 56 | ); 57 | break; 58 | case 'win32': 59 | // For Windows, construct the path explicitly as specified 60 | nodeDirectory = path.join( 61 | homeDirectory, 62 | 'AppData', 63 | 'Roaming', 64 | 'KOII-Desktop-Node', 65 | ); 66 | break; 67 | default: 68 | nodeDirectory = path.join( 69 | homeDirectory, 70 | 'AppData', 71 | 'Roaming', 72 | 'KOII-Desktop-Node', 73 | ); 74 | } 75 | 76 | return nodeDirectory; 77 | } 78 | 79 | static async gettask_audit_program() { 80 | const connection = new Connection('https://mainnet.koii.network'); 81 | const taskId = Debugger.taskID; 82 | const accountInfo = await connection.getAccountInfo(new PublicKey(taskId)); 83 | if (!accountInfo) { 84 | console.log(`${taskId} doesn't contain any distribution list data`); 85 | return null; 86 | } 87 | 88 | const data = JSON.parse(accountInfo.data.toString()); 89 | console.log('data.task_audit_program', data.task_audit_program); 90 | return data.task_audit_program; 91 | } 92 | } 93 | 94 | module.exports = Debugger; 95 | -------------------------------------------------------------------------------- /Lesson 3/imgs/secrets-example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 3/imgs/secrets-example.png -------------------------------------------------------------------------------- /Lesson 3/imgs/simple-crawler.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 3/imgs/simple-crawler.png -------------------------------------------------------------------------------- /Lesson 3/prod-debug.js: -------------------------------------------------------------------------------- 1 | const { spawn } = require('cross-spawn'); 2 | const fs = require('fs'); 3 | require('dotenv').config(); 4 | const Debugger = require('./debugger'); 5 | const Tail = require('tail').Tail; 6 | 7 | const startWatching = async () => { 8 | console.log('Watching for file changes...'); 9 | // watch and trigger builds 10 | await build(); 11 | }; 12 | 13 | /* build and webpack the task */ 14 | const build = async () => { 15 | console.log('Building...'); 16 | const child = await spawn('npm', ['run', 'webpack:test'], { stdio: 'inherit' }); 17 | 18 | await child.on('close', code => { 19 | if (code !== 0) { 20 | console.error('Build failed'); 21 | } else { 22 | console.log('Build successful'); 23 | copyWebpackedFile(); 24 | } 25 | return; 26 | }); 27 | }; 28 | 29 | /* copy the task to the Desktop Node runtime folder */ 30 | const copyWebpackedFile = async () => { 31 | const debugConfig = await Debugger.getConfig(); 32 | console.log('debugConfig', debugConfig); 33 | const nodeDIR = debugConfig.nodeDir; 34 | const sourcePath = __dirname + '/' + debugConfig.webpackedFilePath; 35 | const desktopNodeExecutablePath = nodeDIR + '/' + debugConfig.destinationPath; 36 | const desktopNodeLogPath = nodeDIR + '/' + debugConfig.logPath; 37 | const keywords = debugConfig.keywords; 38 | const taskID = debugConfig.taskID; 39 | 40 | if (!sourcePath || !desktopNodeExecutablePath) { 41 | console.error('Source path or destination path not specified in .env'); 42 | return; 43 | } 44 | 45 | console.log( 46 | `Copying webpacked file from ${sourcePath} to ${desktopNodeExecutablePath}...`, 47 | ); 48 | 49 | fs.copyFile(sourcePath, desktopNodeExecutablePath, async err => { 50 | if (err) { 51 | console.error('Error copying file:', err); 52 | } else { 53 | console.log('File copied successfully'); 54 | tailLogs(desktopNodeLogPath, keywords, taskID); 55 | } 56 | }); 57 | }; 58 | 59 | /* tail logs */ 60 | const tailLogs = async (desktopNodeLogPath, keywords, taskID) => { 61 | console.log('Watchings logs for messages containing ', keywords); 62 | 63 | // Ensure the log file exists, or create it if it doesn't 64 | try { 65 | await fs.promises.access(desktopNodeLogPath, fs.constants.F_OK); 66 | } catch (err) { 67 | console.log(`Log file not found, creating ${desktopNodeLogPath}`); 68 | await fs.promises.writeFile(desktopNodeLogPath, '', { flag: 'a' }); // 'a' flag ensures the file is created if it doesn't exist and not overwritten if it exists 69 | } 70 | 71 | let tail = new Tail(desktopNodeLogPath, '\n', {}, true); 72 | 73 | console.warn( 74 | 'Now watching logs for messages containing ', 75 | keywords, 76 | 'Please start the Task', 77 | taskID, 78 | ' and keep it running on the Desktop Node.', 79 | ); 80 | 81 | tail.on('line', function (data) { 82 | // console.log(data); 83 | if (keywords.some(keyword => data.includes(keyword))) { 84 | console.log(`PROD$ ${data}`); 85 | } 86 | }); 87 | 88 | tail.on('error', function (error) { 89 | console.log('ERROR: ', error); 90 | }); 91 | }; 92 | 93 | startWatching(); 94 | -------------------------------------------------------------------------------- /Lesson 3/simple-crawler/config-task.yml: -------------------------------------------------------------------------------- 1 | # See the config-task.yml file in the task template repository for the full configuration 2 | 3 | requirementsTags: 4 | - type: CPU 5 | value: "4-core" 6 | - type: RAM 7 | value: "5 GB" 8 | - type: STORAGE 9 | value: "5 GB" 10 | - type: TASK_VARIABLE 11 | value: "SEARCH_TERM" 12 | description: "The search term you want to use for the crawler" 13 | -------------------------------------------------------------------------------- /Lesson 3/simple-crawler/task/1-task.js: -------------------------------------------------------------------------------- 1 | import { namespaceWrapper } from "@_koii/namespace-wrapper"; 2 | import { storeFile } from "./fileUtils.js"; 3 | import { crawl } from "./crawler.js"; 4 | 5 | export async function task(roundNumber) { 6 | // Run your task and store the proofs to be submitted for auditing 7 | // The submission of the proofs is done in the submission function 8 | console.log(`EXECUTE TASK FOR ROUND ${roundNumber}`); 9 | try { 10 | const postTitles = await crawl(process.env.SEARCH_TERM); 11 | const cid = await storeFile(postTitles); 12 | await namespaceWrapper.storeSet("cid", cid); 13 | } catch (error) { 14 | console.error("EXECUTE TASK ERROR:", error); 15 | } 16 | } 17 | -------------------------------------------------------------------------------- /Lesson 3/simple-crawler/task/2-submission.js: -------------------------------------------------------------------------------- 1 | import { namespaceWrapper } from "@_koii/namespace-wrapper"; 2 | 3 | export async function submission(roundNumber) { 4 | /** 5 | * Submit the task proofs for auditing 6 | * Must return a string of max 512 bytes to be submitted on chain 7 | */ 8 | try { 9 | console.log(`MAKE SUBMISSION FOR ROUND ${roundNumber}`); 10 | return await namespaceWrapper.storeGet("cid"); 11 | } catch (error) { 12 | console.error("MAKE SUBMISSION ERROR:", error); 13 | } 14 | } 15 | -------------------------------------------------------------------------------- /Lesson 3/simple-crawler/task/3-audit.js: -------------------------------------------------------------------------------- 1 | import { getFileBlob } from "../../../Lesson 2/file-sharing/task/fileUtils.js"; 2 | 3 | export async function audit(submission, roundNumber, submitterKey) { 4 | /** 5 | * Audit a submission 6 | * This function should return true if the submission is correct, false otherwise 7 | */ 8 | console.log(`AUDIT SUBMISSION FOR ROUND ${roundNumber} from ${submitterKey}`); 9 | 10 | // Verify the upload exists 11 | const upload = await getFileBlob(submission); 12 | if (!upload) { 13 | return false; 14 | } 15 | 16 | // Verify the file content is a non-empty string 17 | const fileContent = await upload.text(); 18 | if ( 19 | !fileContent || 20 | fileContent.length === 0 || 21 | typeof fileContent !== "string" 22 | ) { 23 | return false; 24 | } 25 | const postTitles = JSON.parse(fileContent); 26 | // Verify the file content is a non-empty array 27 | if (!Array.isArray(postTitles) || postTitles.length === 0) { 28 | return false; 29 | } 30 | // Verify each post title is a non-empty string 31 | postTitles.forEach((title) => { 32 | if (typeof title !== "string" || title.length === 0) { 33 | return false; 34 | } 35 | }); 36 | return true; 37 | } 38 | -------------------------------------------------------------------------------- /Lesson 3/simple-crawler/task/crawler.js: -------------------------------------------------------------------------------- 1 | import PCR from "puppeteer-chromium-resolver"; 2 | 3 | export async function crawl(searchTerm) { 4 | const options = {}; 5 | const stats = await PCR(options); 6 | console.log(`Chrome Path: ${stats.executablePath}`); 7 | 8 | // Set up puppeteer 9 | // set headless = false for visualization debugging, set headless = true for production 10 | const browser = await stats.puppeteer.launch({ 11 | headless: false, 12 | executablePath: stats.executablePath, 13 | }); 14 | 15 | // Open a new page 16 | const page = await browser.newPage(); 17 | // Set the user agent to a common browser user agent 18 | await page.setUserAgent( 19 | "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36" 20 | ); 21 | // The URL to scrape 22 | const url = `https://forums.redflagdeals.com/hot-deals-f9/`; 23 | 24 | // `documentloaded` means the loading icon on the left of the tab is resolved 25 | await page.goto(url, { waitUntil: "domcontentloaded" }); 26 | 27 | // Wait for the search bar to load 28 | await page.waitForSelector("#search_keywords"); 29 | 30 | // type the search term into the search bar 31 | await page.type("#search_keywords", searchTerm); 32 | 33 | // submit the search term 34 | await page.keyboard.press("Enter"); 35 | 36 | // Wait for the links to load after the search term is submitted 37 | await page.waitForSelector("h2.topictitle"); 38 | 39 | // Get the titles of the links 40 | let titles = null; 41 | try { 42 | titles = await page.$$eval("h2.topictitle a", (links) => 43 | links.map((link) => link.textContent.trim()) 44 | ); 45 | } catch (error) { 46 | console.log("Error:", error); 47 | } 48 | 49 | // close puppeteer 50 | await browser.close(); 51 | return titles; 52 | } 53 | -------------------------------------------------------------------------------- /Lesson 3/simple-crawler/task/fileUtils.js: -------------------------------------------------------------------------------- 1 | import { KoiiStorageClient } from "@_koii/storage-task-sdk"; 2 | import { namespaceWrapper } from "@_koii/namespace-wrapper"; 3 | import fs from "fs"; 4 | 5 | export async function storeFile(data, filename = "value.json") { 6 | // Create a new instance of the Koii Storage Client 7 | const client = KoiiStorageClient.getInstance({}); 8 | const basePath = await namespaceWrapper.getBasePath(); 9 | try { 10 | // Write the data to a temp file 11 | fs.writeFileSync(`${basePath}/${filename}`, JSON.stringify(data)); 12 | 13 | // Get the user staking account, to be used for signing the upload request 14 | const userStaking = await namespaceWrapper.getSubmitterAccount(); 15 | 16 | // Upload the file to IPFS and get the CID 17 | const { cid } = await client.uploadFile( 18 | `${basePath}/${filename}`, 19 | userStaking 20 | ); 21 | 22 | console.log(`Stored file CID: ${cid}`); 23 | // Delete the temp file 24 | fs.unlinkSync(`${basePath}/${filename}`); 25 | 26 | return cid; 27 | } catch (error) { 28 | console.error("Failed to upload file to IPFS:", error); 29 | fs.unlinkSync(`${basePath}/${filename}`); 30 | throw error; 31 | } 32 | } 33 | 34 | export async function getFileBlob(cid, filename = "value.json") { 35 | const client = new KoiiStorageClient(); 36 | return await client.getFile(cid, filename); 37 | } 38 | 39 | export async function getFileText(cid, filename = "value.json") { 40 | const fileBlob = await getFileBlob(cid, filename); 41 | if (!fileBlob) return null; 42 | return await fileBlob.text(); 43 | } 44 | -------------------------------------------------------------------------------- /Lesson 4/PartI.md: -------------------------------------------------------------------------------- 1 | # Lesson 4: Auditing & Distribution 2 | 3 | ## Part I: Auditing Concepts 4 | 5 | ![Lesson 4](https://github.com/koii-network/ezsandbox/assets/66934242/dce7f56b-02e9-4e75-8fef-1e4b8ecf0f95) 6 | 7 | If you recall from [Lesson 1](../Lesson%201/PartIII.md), we discussed how tasks are able to do work, verify each other's work, then distribute rewards based on those verified results. We've touched briefly on auditing, but for the most part we've focused on building out the task side of things i.e, we've only built the "work" part of a Koii task. 8 | 9 | Auditing is the next big step for us as it helps us ensure that our tasks are being run properly and it prevents bad actors from exploiting us for free rewards. You can typically find a task's audit logic in `task/audit.js`. 10 | 11 | Prerequisites: 12 | 13 | - General understanding of [gradual consensus and task flow](https://docs.koii.network/concepts/what-are-tasks/what-are-tasks/gradual-consensus) 14 | - Lessons 1-3 completed 15 | 16 | ### The Purpose of Audits 17 | 18 | One of the potential concerns with using other people's machines as a decentralized solution for your compute resources is that you can't necessarily trust them to do it. Without having a way to verify their work, you're essentially hiring an employee and trusting them to do all the work assigned without any sort of reviews. 19 | 20 | The solution is to allow other nodes to check each other's work... but this introduces another problem. Think about asking someone to calculate a math sum. How do you verify their answer is correct? You do the calculation yourself. If there were two nodes in the network, each one would only be able to do half as much work, because 50% of their time would be spent repeating the calculations of the other node. The solution to this problem is _controlled redundancy_. 21 | 22 | ### Controlled Redundancy 23 | 24 | When you create your task's audit mechanism, you can decide exactly how much you need to review work. If you have a mission critical task, then you may require 20 nodes to verify a result before you're able to confidently claim that it's correct. For a different task, such as a simple addition, maybe you're confident that only 2 nodes need to verify as result. Koii gives you the freedom to scrutinize work with a customizable level of control, enabling you to make those decisions precisely for your use case. 25 | 26 | You could even start off with a highly redundant system and slowly reduce the amount of audits that are done based on a node's reputation. [Learn more about reputation here](https://docs.koii.network/concepts/what-are-tasks/designing-tasks/using-reputation#definition-of-carp) 27 | 28 | ### Audit Operations 29 | 30 | While writing your own audit might seem difficult, you can think of it as a combination of basic operations. 31 | 32 | Different types of tasks can be grouped based on their mechanism. For example, every web crawler task will have a similar audit system. With more complicated tasks, you can break down your requirements into more basic steps to better conceptualize the audit mechanism. 33 | 34 | We've mentioned some of the audit mechanisms in the previous lessons, but let's go over them again: 35 | 36 | #### 0. Trivial Example 37 | 38 | ```javascript 39 | return true; 40 | ``` 41 | 42 | The simplest possible audit is no audit: we're simply returning true for the vote no matter what. This is useful for tasks that don't require an audit, but if you're distributing rewards, it's a bad idea. 43 | 44 | #### 1. Lesson 1: [EZ Testing](../Lesson%201/ez-testing-task/3-audit.js) 45 | 46 | ```javascript 47 | return submission === "Hello, World!"; 48 | ``` 49 | 50 | This task's job is to simply submit a hardcoded value, `Hello, World!`. This makes our audit logic rather simple, if the submitted value is anything other than `Hello, World!` then we know that something has gone wrong and we can vote against this submission. 51 | 52 | While this is a great starter example for learning how a Koii task works, you'll almost always want your tasks to return dynamic data as a submission. 53 | 54 | #### 2. Lesson 2: [UPnP Basics](../Lesson%202/upnp-basics/src/task/3-audit.js) 55 | 56 | ```javascript 57 | return typeof submission === "string" && submission.length > 0; 58 | ``` 59 | 60 | #### 3. Lesson 2: [File Sharing](../Lesson%202/file-sharing/task/3-audit.js) 61 | 62 | ```javascript 63 | const fileContent = await getFileText(submission); 64 | return typeof fileContent === "string" && fileContent.length > 0; 65 | ``` 66 | 67 | In these two tasks, we allowed nodes to return a secret of their choice. This means there isn't a single consistent value to be checked, which is closer to how a real task would work. Here we check the type of the data rather than its value; while this is often a useful step in your audit, it's not enough on its own. 68 | 69 | #### 4. Simple Crawler Task - [Lesson 3](../Lesson%203/simple-crawler/task/3-audit.js) 70 | 71 | ```javascript 72 | // Verify the upload exists 73 | const upload = await getFileBlob(submission); 74 | if (!upload) { 75 | return false; 76 | } 77 | 78 | // Verify the file content is a non-empty string 79 | const fileContent = await upload.text(); 80 | if ( 81 | !fileContent || 82 | fileContent.length === 0 || 83 | typeof fileContent !== "string" 84 | ) { 85 | return false; 86 | } 87 | const postTitles = JSON.parse(fileContent); 88 | // Verify the file content is a non-empty array 89 | if (!Array.isArray(postTitles) || postTitles.length === 0) { 90 | return false; 91 | } 92 | // Verify each post title is a non-empty string 93 | postTitles.forEach((title) => { 94 | if (typeof title !== "string" || title.length === 0) { 95 | return false; 96 | } 97 | }); 98 | return true; 99 | ``` 100 | 101 | This is a common pattern for scraped data - you want to validate not just the type but also the shape of the data. However, like the audits before it, there are two serious issues: 102 | 103 | 1. There is no way to know if the data you're getting is the data you want. 104 | 2. Any node can make a submission and get a reward, whether they've performed the task or not. 105 | 106 | 107 | 108 | Now let's take a look at distribution concepts in [Part II](./PartII.md) 109 | -------------------------------------------------------------------------------- /Lesson 4/PartII-new.md: -------------------------------------------------------------------------------- 1 | # Lesson 4: Auditing & Distribution 2 | 3 | ## Part II: Advanced Auditing Techniques 4 | 5 | Now that we've covered some of the basics of writing an audit, let's talk about how to write a good one. 6 | 7 | Prerequisites: 8 | 9 | - General understanding of [gradual consensus and task flow](https://docs.koii.network/concepts/what-are-tasks/what-are-tasks/gradual-consensus) 10 | - Understanding of [audit mechanisms](./PartI.md) 11 | 12 | To understand why audits are so important, you need to understand this key fact: *Your task executable can be changed by anyone*. In fact, we used `prod-debug` in Lesson 1 to do just that! And it gets even worse: anyone can make a submission to your task directly on the blockchain, without ever having *seen* your task code. 13 | 14 | ### Zero Trust 15 | 16 | In cybersecurity, there is a principle called zero trust. Its fundamental principle is "never trust, always verify", and that's what we have to do when we write a Koii task. This can seem like an overwhelming task, so let's talk about some of the common problems you could encounter and how to deal with them. 17 | 18 | ### Fake Data 19 | 20 | In all of the audits we've discussed so far, a node could pass audit by simply supplying data of the correct shape. In the simple crawler example, we have no way of knowing if the post titles we get back are from redflagdeals or not. The strategy to deal with this is simple but resource heavy: auditing nodes perform the same work and confirm whether it matches the submitted proofs or not. You can use sampling and check only a portion of the work if necessary. If there's a possibility the data has changed between submission and audit, you can add some tolerance (e.g. prices can by 5% higher or lower, 80% of results should match). It's important here to balance between ensuring your data is correct and avoiding unfair penalties for honest nodes. 21 | 22 | This still may not be enough, however. It is possible for nodes to submit data from the right source, but that doesn't guarantee it's the data you want. Going back to our redflagdeals example, you could check that the post titles belong to actual posts on the forum, but that won't tell you whether or not they showed up for the search query - you don't even know what keyword they chose! A user could simply grab titles from random pages. The solution to this is to make users submit more information in their proofs. In this case, a pretty good options would be to get the search query, the links to the pages that were found, and the titles of those pages. Because search pages keep updating as new content is added, you can check if the pages they provided actually show up in the search results for that query. 23 | 24 | ### Duplicate Data 25 | 26 | Another way for a node to make a valid submission is to copy an existing valid submission. There are a few possibilities here: 27 | 28 | 1. A node resubmits its own valid submission as a new one. 29 | 2. One node provides a valid submission to a group of nodes and all of them make the same submission. 30 | 3. A node takes another node's submission from the blockchain and resubmits it as their own before the submission window closes. 31 | 32 | #### Resubmitting a valid submission from a previous round 33 | 34 | Let's continue with the redflagdeals example - what if the search term they choose is uncommon and the results don't change very often? Then they could keep submitting the same post titles over and over without performing the search again. This could go one for days, the node simply collecting rewards without doing anything to earn them. There are a couple of ways you can deal with this: 35 | 36 | - If the data you're collecting is timestamped in some way, you can exclude results that are too old. 37 | - Make sure the task is different each round. For example, you might want to assign a different keyword each round. 38 | 39 | #### Passing a valid submission to other nodes 40 | 41 | Because many nodes are making the same submission in the same round, the data would be valid for all of them during that round. The solution here is to make sure you're not assigning the exact same task to all nodes, so that they are not able to duplicate their submissions. This requires you to keep track of which work each node should be doing. 42 | 43 | #### Stealing a valid submission 44 | 45 | The solution here involves 46 | -------------------------------------------------------------------------------- /Lesson 4/PartII.md: -------------------------------------------------------------------------------- 1 | # Lesson 4: Auditing & Distribution 2 | 3 | ## Part III: Distribution Concepts 4 | 5 | At this point, you should be familiar with how a task is developed, and how you can verify as task's work. The final piece of the puzzle is looking at how rewards are distributed. 6 | 7 | Prerequisites: 8 | 9 | - General understanding of [gradual consensus and task flow](https://docs.koii.network/concepts/what-are-tasks/what-are-tasks/gradual-consensus) 10 | - Understanding of [audit mechanisms](./PartI.md) 11 | 12 | ### Distributing Rewards 13 | 14 | Now that we've figured out different ways to verify if a node is doing honest work, we should also figure out how we want to reward that honesty. For the first time, we're going to look at [the distribution function](../Lesson%201/ez-testing-task/4-distribution.js). We take care of fetching the audit submissions and counting the votes, all you need to do is decide on your compensation logic. 15 | 16 | Distributions mechanisms follow this general format: 17 | 18 | 1. Decide how much you want to reward 19 | 2. Decide how much you want to penalize 20 | 3. Mark each node as valid (accepted submission) or slashed (rejected submission) 21 | 4. Distribute! 22 | 23 | ### Default Distribution Logic 24 | 25 | We've provided some default distribution logic that will work for many tasks. We: 26 | 27 | 1. Distribute the round bounty equally between all valid submissions 28 | 2. Take 70% of the stake for invalid submissions 29 | 30 | The full distribution logic is available [here](../Lesson%201/ez-testing-task/4-distribution.js) 31 | 32 | This function must return an object with submitter's public keys as the keys and their reward amount (negative if a penalty) as the value. 33 | 34 | #### Walkthrough 35 | 36 | We start out with an empty list and an array to store the public keys of the submitters who made valid submissions: 37 | 38 | ```js 39 | const distributionList = {}; 40 | const approvedSubmitters = []; 41 | ``` 42 | 43 | Next, we iterate through the list of submitters. If they have no votes, they receive neither a reward nor a penalty. If their votes are negative, we issue a penalty, which is a percentage of the submitter's stake (in this case, we've set SLASH_PERCENT to 0.7). If their votes are positive, we add them to the list of submitters who will receive a reward. 44 | 45 | ```js 46 | for (const submitter of submitters) { 47 | if (submitter.votes === 0) { 48 | distributionList[submitter.publicKey] = 0; 49 | } else if (submitter.votes < 0) { 50 | const slashedStake = submitter.stake * SLASH_PERCENT; 51 | distributionList[submitter.publicKey] = -slashedStake; 52 | console.log("CANDIDATE STAKE SLASHED", submitter.publicKey, slashedStake); 53 | } else { 54 | approvedSubmitters.push(submitter.publicKey); 55 | } 56 | } 57 | ``` 58 | 59 | Finally, we divide the bounty per round by the number of people who will receive a reward and we distribute the reward equally among them. We finish off by returning the completed distribution list. 60 | 61 | ```js 62 | const reward = Math.floor(bounty / approvedSubmitters.length); 63 | console.log("REWARD PER NODE", reward); 64 | approvedSubmitters.forEach((candidate) => { 65 | distributionList[candidate] = reward; 66 | }); 67 | return distributionList; 68 | ``` 69 | 70 | ### Shared Data 71 | 72 | One thing to note about generating distribution lists is that only one node per round is chosen to generate the list. Once this list is generated, it needs to be shared and audited by other nodes, similar to regular audit logic. So remember, there are two audits: 73 | 74 | 1. Auditing a node's task work 75 | 2. Auditing the round's distribution list 76 | 77 | Data can be shared to all nodes in the network by fetching a list of all the other nodes, and comparing their stored data with yours. Using timestamps, you can verify who has the most recent data and use the latest information, similar to [Link State routing](https://en.wikipedia.org/wiki/Link-state_routing_protocol). 78 | 79 | If you want to learn more about data sharing across a task, [click here](https://docs.koii.network/develop/linktree/data-sharing). 80 | 81 | Just like that we've successfully gone through both audit and distribution mechanisms and we're now ready to write our own. 82 | 83 | 84 | 85 | Now let's try building our own audit and distribution mechanisms in [Part III](./PartIII.md) 86 | -------------------------------------------------------------------------------- /Lesson 4/PartIII.md: -------------------------------------------------------------------------------- 1 | # Lesson 4: Auditing & Distribution 2 | 3 | ## Part IV: Building Audit and Distribution Mechanisms 4 | 5 | The previous sections were conceptually heavy topics, so lets do something more fun and create our own audit and distribution systems! 6 | 7 | Prerequisites: 8 | 9 | 10 | 11 | - Understanding of [Audit](./PartI.md) and [Distribution](./PartII.md) Mechanisms 12 | - Basic knowledge of a [Caesar Cipher](https://en.wikipedia.org/wiki/Caesar_cipher) 13 | 14 | ### Caesar Cipher Task 15 | 16 | If you navigate to the attached task for this lesson, you'll see that it simulates a mini Caesar cipher. For this task, the submission logic is completed for you but it's up to you to _decipher_ the audit logic! Let's take a look at what's going on in the `task()` function from `submissions.js` 17 | 18 | ```javascript 19 | const originalMsg = "koii rocks!"; 20 | const randomShift = CaesarCipher.getRandomShiftNum(); 21 | const encryptedMsg = CaesarCipher.encrypt(originalMsg, randomShift); 22 | const value = randomShift + encryptedMsg; 23 | ``` 24 | 25 | 1. We're first specifying an original message that will be encrypted and submitted. It looks like you decided with the message, "koii rocks!". We agree! 26 | 27 | 2. Next, we get a random number from 1-5 to shift our Caesar cipher by. This is specifically done to make your life a little harder! We don't want your audit logic to be as simple as checking if the value equals `"koii rocks!"`, that'd be no fun. But, for simplicity sake, we decided to keep the shift between 1 and 5 letters. 28 | 29 | 3. We encrypt the message with a standard Caesar cipher using the random number we want to shift by. If you're curious about the code used to do this, feel free to investigate `caesar-cipher/caesar-cipher.js`. 30 | 31 | 4. We finally store the encrypted message along with the shift number prepended to the start. This is will come in handy when writing our audit logic! 32 | 33 | ### Auditing The Caesar Task 34 | 35 | Now, if we navigate to the `validateNode()` function in `audit.js`, we'll see that it's left blank for you to fill in! Go ahead and give it a go right now and if you can't figure it out, come back here for the answer. Hint: Remember the value we prepended at the start of our submission? That might come in handy! 36 | 37 | If you're having a little trouble figuring it out, no worries. Here's a step-by-step guide to one solution: 38 | 39 | 1. We first want to grab the shift value that is prepended before submission. This will help us decrypt the rest of the message! 40 | 41 | 2. We can then slice off the first character to get our encrypted message 42 | 43 | 3. With our encrypted message and our shift value, we can simply use the decrypt function from our CaesarCipher class to get our original string back! 44 | 45 | 4. We compare our result with the answer that we expect, `'koii rocks!'`, and adjust our vote accordingly. 46 | 47 | If you're still having trouble, take a look at our solution in the [`after` folder](./caesar-task/after/task/audit.js#L16). 48 | 49 | ### Adjusting Distribution Mechanisms 50 | 51 | Now let's try changing the distribution rewards. Try to make these three changes: 52 | 53 | 1. When there are no votes, slash by 50%. 54 | 2. When there are more negative than positive votes, slash by 100%. 55 | 3. Instead of distributing the bounty equally, give each successful submission 0.25 KOII. Note that rewards are distributed in Roe, not KOII. 56 | 57 | > [!NOTE] 58 | > 59 | > Roe is the minimum unit of KOII. 1 KOII = 1,000,000,000 Roe. 60 | 61 | Again, if you run into difficulties, you can see our answer in the [`after` folder](./caesar-task/after/task/distribution.js#L105). 62 | 63 | You've reached the end of this lesson which means you're now familiar with audit and distribution mechanisms. The next lesson will discuss security and hardening. 64 | 65 | Now we'll talk about security and hardening in [Lesson 5](../Lesson%205/README.md) 66 | -------------------------------------------------------------------------------- /Lesson 4/README.md: -------------------------------------------------------------------------------- 1 | # Lesson 4: Auditing & Distribution 2 | 3 | - Part I: [Auditing Concepts](./PartI.md) 4 | 5 | 10 | 11 | - Part II: [Distribution Concepts](./PartII.md) 12 | 13 | - Part III: [Building Audit and Distribution Mechanisms](./PartIII.md) 14 | -------------------------------------------------------------------------------- /Lesson 4/caesar-task/task/1-task.js: -------------------------------------------------------------------------------- 1 | import { namespaceWrapper } from "@_koii/namespace-wrapper"; 2 | import { encrypt, getRandomShiftNum } from "./cipher.js"; 3 | 4 | export async function task(roundNumber) { 5 | // Run your task and store the proofs to be submitted for auditing 6 | // The submission of the proofs is done in the submission function 7 | try { 8 | console.log(`EXECUTE TASK FOR ROUND ${roundNumber}`); 9 | const originalMsg = "Koii rocks!"; 10 | const randomShift = getRandomShiftNum(); 11 | const encryptedMsg = encrypt(originalMsg, randomShift); 12 | const value = `{"shift": ${randomShift}, "message": "${encryptedMsg}"}`; 13 | await namespaceWrapper.storeSet("value", value); 14 | } catch (error) { 15 | console.error("EXECUTE TASK ERROR:", error); 16 | } 17 | } 18 | -------------------------------------------------------------------------------- /Lesson 4/caesar-task/task/3-audit.js: -------------------------------------------------------------------------------- 1 | import { decrypt } from "./cipher.js"; 2 | 3 | export async function audit(submission, roundNumber, submitterKey) { 4 | /** 5 | * Audit a submission 6 | * This function should return true if the submission is correct, false otherwise 7 | */ 8 | console.log(`AUDIT SUBMISSION FOR ROUND ${roundNumber} from ${submitterKey}`); 9 | const submissionObj = JSON.parse(submission); 10 | const shift = submissionObj.shift; 11 | const encryptedMsg = submissionObj.message; 12 | const message = decrypt(encryptedMsg, shift); 13 | return message === "Koii rocks!"; 14 | } 15 | -------------------------------------------------------------------------------- /Lesson 4/caesar-task/task/4-distribution.js: -------------------------------------------------------------------------------- 1 | const SLASH_PERCENT = 0.7; 2 | 3 | export function distribution(submitters, bounty, roundNumber) { 4 | /** 5 | * Generate the reward list for a given round 6 | * This function should return an object with the public keys of the submitters as keys 7 | * and the reward amount as values 8 | */ 9 | console.log(`MAKE DISTRIBUTION LIST FOR ROUND ${roundNumber}`); 10 | const distributionList = {}; 11 | const approvedSubmitters = []; 12 | // Slash the stake of submitters who submitted incorrect values 13 | // and make a list of submitters who submitted correct values 14 | for (const submitter of submitters) { 15 | if (submitter.votes === 0) { 16 | distributionList[submitter.publicKey] = 0; 17 | } else if (submitter.votes < 0) { 18 | const slashedStake = submitter.stake * SLASH_PERCENT; 19 | distributionList[submitter.publicKey] = -slashedStake; 20 | console.log("CANDIDATE STAKE SLASHED", submitter.publicKey, slashedStake); 21 | } else { 22 | approvedSubmitters.push(submitter.publicKey); 23 | } 24 | } 25 | // reward the submitters who submitted correct values 26 | const reward = Math.floor(bounty / approvedSubmitters.length); 27 | console.log("REWARD PER NODE", reward); 28 | approvedSubmitters.forEach((candidate) => { 29 | distributionList[candidate] = reward; 30 | }); 31 | return distributionList; 32 | } 33 | -------------------------------------------------------------------------------- /Lesson 4/caesar-task/task/cipher.js: -------------------------------------------------------------------------------- 1 | export function getRandomShiftNum() { 2 | return Math.floor(Math.random() * 5) + 1; 3 | } 4 | 5 | export function encrypt(text, shift) { 6 | let encodedText = ""; 7 | for (let i = 0; i < text.length; i++) { 8 | const char = text[i]; 9 | if (char.match(/[a-z]/i)) { 10 | // Checks if the character is a letter, case insensitive 11 | const base = 12 | char === char.toLowerCase() ? "a".charCodeAt(0) : "A".charCodeAt(0); 13 | const code = ((char.charCodeAt(0) - base + shift + 26) % 26) + base; 14 | encodedText += String.fromCharCode(code); 15 | } else { 16 | encodedText += char; 17 | } 18 | } 19 | return encodedText; 20 | } 21 | 22 | export function decrypt(text, shift) { 23 | return encrypt(text, -shift); 24 | } 25 | -------------------------------------------------------------------------------- /Lesson 5/README.md: -------------------------------------------------------------------------------- 1 | # Lesson 5: Security & Hardening 2 | 3 | In this lesson, we are only discussing namespaceWrapper functions, so we don't have a full task. We'll be looking at how to sign and verify messages sent from node to node. 4 | 5 | Prerequisites: 6 | 7 | - Completion of lessons 1-4 8 | 9 | ## Using Signatures 10 | 11 | To improve security, you might want to verify that the data you're receiving is coming from the source you expect. This can be done by verifying messages with signatures. 12 | 13 | A message can be signed like so, using [`namespaceWrapper.payloadSigning()`](./koiiNode.js#L222): 14 | 15 | ```javascript 16 | const message = "Hello World!"; 17 | 18 | // Sign payload 19 | const signedMessage = await namespaceWrapper.payloadSigning(message); 20 | ``` 21 | 22 | The `payloadSigning()` function will take care of accessing the node's private key for the signing. 23 | 24 | When you want to send this signature to another node, you'll also need to include the current node's public key. You can get this from `namespaceWrapper.getMainAccountPubkey()`: 25 | 26 | ```javascript 27 | const publicKey = namespaceWrapper.getMainAccountPubkey(); 28 | ``` 29 | 30 | You can then send the signed message and the public key to another node to be verified and decoded. The decoding can be done with `namespaceWrapper.verifySignature()`: 31 | 32 | ```javascript 33 | // Assuming you've already retrieved the signed message and the public key 34 | const message = namespaceWrapper.verifySignature(signedMessage, publicKey); 35 | ``` 36 | 37 | Next up, we'll look at using custom tokens in [Lesson 6](../Lesson%206/README.md). 38 | -------------------------------------------------------------------------------- /Lesson 6/PartI.md: -------------------------------------------------------------------------------- 1 | # Lesson 6: KPL Tokens 2 | 3 | ## Part 1: Creating a KPL Token 4 | 5 | KPL stands for Koii Program Library. It supports users in creating various fungible tokens, which can be used as rewards for tasks (currently in progress) and like other tokens, they are interchangeable and divisible. 6 | 7 | Visit [this site](http://kpl.koii.network/) to create KPL Tokens. 8 | 9 | ### Create Token 10 | 11 | First, upload your JSON wallet. This will be the same wallet you used to deploy a task in Lesson 1. 12 | 13 | If you're using your main wallet from the desktop node, it will be located at `/KOII-Desktop-Node/wallets/_mainSystemWallet.json`. 14 | 15 | The OS-specific paths are as follows: 16 | 17 | **Windows**: `/Users//AppData/Roaming` 18 | 19 | **Mac**: `/Users//Library/Application Support` 20 | 21 | **Linux**: `/home//.config` (This path contains a dot folder that will be hidden by default. You can show hidden folders by pressing Ctrl-H) 22 | 23 | If you created your wallet with `koii-keygen`, the default location is `/.config/koii/id.json`. The `.config` folder may be hidden in your file explorer, you can press `Cmd-Shift-.` on Mac or `Ctrl-H` on Linux. On Windows, there is no keyboard shortcut but in the file explorer you can select `View > Show > Hidden Files`. Make sure your wallet has a balance of at least 0.1 KOII to cover the cost of creating the mint and minting your tokens. 24 | 25 | > [!NOTE] 26 | > 27 | > Your JSON wallet is handled through front-end JavaScript only. Your wallet file will never be uploaded anywhere and will always remain on your local machine. 28 | 29 | Next, you'll need an image for your token. It can be a .png, .ico, or .jpg. It should be a square image; if it's rectangular, you may not like how it's cropped. 30 | 31 | Finally enter the token details: the name, the symbol (for example, FIRE or KOII), and the description. 32 | 33 | > [!TIP] 34 | > 35 | > When you create the token, you'll be given a public key - this is your mint address. Hold onto this, you'll need it when minting tokens and when deploying a KPL Task. 36 | 37 | ### Mint Tokens 38 | 39 | Minting your tokens is simple: you just need to enter the number of tokens, your JSON wallet file (the same one you used when creating the tokens), and your mint address. 40 | 41 | ### Tokens Owned 42 | 43 | Afterwards, you can check the tokens you own by clicking "Connect Finnie" to link your Finnie Wallet, and then clicking "Fetch Tokens". 44 | 45 | Now that you've created your token, you can use it to [deploy a task](./PartII.md)! 46 | -------------------------------------------------------------------------------- /Lesson 6/PartII.md: -------------------------------------------------------------------------------- 1 | # Lesson 6: KPL Tokens 2 | 3 | ## Deploying a KPL Task 4 | 5 | 1. Make sure you have enough token in your account to cover the cost of the bounty, and enough KOII to cover the cost of the deployment fees. The deployment fee is determined by the `space` value in `config-task.yml`. If you are testing a task, you can set the space to 0.1, which will cost about 7 KOII. If you are deploying into production, the space should be at least 1, which would cost about 70 KOII. 6 | 7 | 2. In `config-task.yml`, set the values you want for `minimum_stake`, `total_bounty_amount`, and `bounty_amount_per_round`. These will all be in your token, not in KOII. 8 | 9 | 3. In `config-task.yml`, change `task_type` to `KPL` and enter the token mint address in `token_type`. 10 | 11 | > [!NOTE] 12 | > 13 | > You can deploy a task with *any* KPL token as long as you have sufficient balance and you know the mint address; it doesn't have to be a token you created. 14 | 15 | Follow the instructions from [Lesson 1](../Lesson%201/PartIV.md#deploying-a-task) for deploying a task! 16 | -------------------------------------------------------------------------------- /Lesson 6/README.md: -------------------------------------------------------------------------------- 1 | # Lesson 6: KPL Tokens 2 | 3 | - Lesson 1: [Creating a KPL Token](./PartI.md) 4 | 5 | - Lesson 2: [Deploying a KPL Task](./PartII.md) 6 | -------------------------------------------------------------------------------- /Lesson 6/imgs/mint-address.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 6/imgs/mint-address.png -------------------------------------------------------------------------------- /Lesson 7/PartI.md: -------------------------------------------------------------------------------- 1 | # Lesson 7: Deployment and Updates 2 | 3 | ## Part I: Mainnet and Desktop Node 4 | 5 | ### Deploying to Mainnet 6 | 7 | > [!WARNING] 8 | > 9 | > In order to deploy or update a task, you **MUST** be using at least Node.js v18. 10 | 11 | Deploying a Koii task is covered in [Lesson 1](../Lesson%201/PartIV.md#deploying-a-task) and deploying a KPL task is covered in [Lesson 6](../Lesson%206/PartII.md). 12 | 13 | ### Updating a Task 14 | 15 | Updating a task is very similar to deployment, and just requires a couple of changes to your `config-task.yml`. At the bottom of the file, you'll see this section: 16 | 17 | ```yml 18 | # OPTIONAL variables variables for creating task / REQUIRED variables for update task 19 | 20 | # ONLY provide the task_id and migrationDescription if you are updating the task otherwise leave blank 21 | # task_id: Previous task id 22 | task_id: '' 23 | 24 | # migrationDescription: Provide the description for changes made in new version of task 25 | migrationDescription: '' 26 | ``` 27 | 28 | Both of these must be filled in when updating. The `task_id` should be the last task ID you used when updating or deploying your task. If you are running your task in the desktop node you can find it here: 29 | 30 | ![task id](./imgs/task-id.png) 31 | 32 | Now, when running 33 | 34 | ```sh 35 | npx @_koii/create-task-cli@latest 36 | ``` 37 | 38 | Choose `Update Existing Task` and follow the prompts, which you should already be familiar with from deploying a task. If you're unsure of any steps, refer to the [Lesson 1 deployment instructions](../Lesson%201/PartIV.md#deploying-a-task). 39 | -------------------------------------------------------------------------------- /Lesson 7/PartII.md: -------------------------------------------------------------------------------- 1 | # Lesson 7: Deployment and Updates 2 | 3 | ## Part II: Local Validator and Docker 4 | 5 | Testing tasks locally through Docker is a process that requires a little set up. Essentially, we have to create a local version of the backend that's used to verify a task's work, which is known as K2. After creating our local K2, we'll deploy the task there for testing! 6 | 7 | ## Environment Setup 8 | 9 | 1. Before starting, please ensure your Koii Node app is **NOT** running! To get a local instance of K2, run `koii-test-validator` and leave the terminal running. This will serve as the backend you deploy the task to. 10 | 11 | 2. Next, we should set your Koii CLI to deploy tasks locally. We can do this using: 12 | 13 | ```sh 14 | koii config set --url http://127.0.0.1:8899 15 | ``` 16 | 17 | Verify you're using the correct settings with `koii config get`, which should show something like this: 18 | 19 | ```sh 20 | Config File: C:\Users\test\.config\koii\cli\config.yml 21 | RPC URL: http://127.0.0.1:8899 22 | WebSocket URL: ws://127.0.0.1:8900/ (computed) 23 | Keypair Path: ~/.config/koii/id.json 24 | Commitment: confirmed 25 | ``` 26 | 27 | 3. With the backend taken care of, lets make sure our task wants to deploy to the right environment. Within `env.local`, ensure you change `K2_NODE_URL` to point to the right endpoint: 28 | 29 | ```javascript 30 | K2_NODE_URL = "http://127.0.0.1:8899"; // Linux 31 | 32 | K2_NODE_URL = "http://host.docker.internal:8899"; // Windows & Mac 33 | ``` 34 | 35 | 4. With that setup, we can deploy the task using `npx @_koii/create-task-cli@latest`. If you forgot how, [head back to lesson 1](../Lesson%201/PartIV.md#deploying-a-task). 36 | 37 | **NOTE**: This is a local environment that doesn't reflect your actual wallet, so there's no need to worry about cost. In fact, if you use the command `koii airdrop 10000` you can give yourself plenty of tokens to test with! 38 | 39 | 5. Now that your task is live, note down the task ID and the executable CID. When you deployed your task, some `TASK DATA` was logged to the console. Look for `task_audit_program_id`; this is your executable CID. Place your taskID in your `.env.local` as shown below: 40 | 41 | ```dotenv 42 | TASKS="ETHVehVJbepd4RZjUqoR2iveYTAsauLpd4kiCRnPUE7Y" //Your task's ID 43 | ``` 44 | 45 | 6. Navigate to the `dist` folder and rename `main.js` to `.js`, using the executable CID you just saved. 46 | 47 | ## Using Docker 48 | 49 | Once the task is deployed, we can finally use Docker! Navigate to `docker-compose.yaml` and verify that `~/.config/koii:/app/config` accurately represents your wallet location. 50 | 51 | Finally, in a separate terminal from the one running `koii-test-validator`, run `docker compose up` and you'll have a Dockerized instance of your task spun up. 52 | 53 | This terminal will now showcase all the logs in real time that you could expect from deploying your task live on the Node. You can observe rounds, submissions, errors, warnings, or anything else you find interesting. 54 | 55 | If you ran into any issues or want to understand this topic more in depth, learn more about Dockerized Tasks [here](https://docs.koii.network/develop/write-a-koii-task/task-development-kit-tdk/test/docker-test). 56 | 57 |
58 |
59 | 60 | You've reached the end of this lesson which means you're now familiar with building tasks that can network. Additionally, you now know how to test them locally with Docker! The next lesson will introduce using secrets and building our very own web crawler. 61 | -------------------------------------------------------------------------------- /Lesson 7/README.md: -------------------------------------------------------------------------------- 1 | # Lesson 7: Deployment and Updates 2 | 3 | - Lesson 1: [Mainnet and Desktop Node](./PartI.md) 4 | 5 | - Lesson 2: [Local Validator and Docker](./PartII.md) 6 | -------------------------------------------------------------------------------- /Lesson 7/imgs/task-id.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/koii-network/ezsandbox/8d10cbcfe7c4d8a6df4f3f6122757e533647e68a/Lesson 7/imgs/task-id.png -------------------------------------------------------------------------------- /Lesson 8/README.md: -------------------------------------------------------------------------------- 1 | Coming soon 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![banner_with_icon](https://github.com/koii-network/ezsandbox/assets/113378734/40046741-843f-47f4-9bf8-76a57198cc81) 2 | 3 | Powered by [Koii](https://koii.network) - Over 60,000 community devices at your fingertips 4 | 5 | Curious to see the source code? [Learn More](#open-source-roadmap) 6 | 7 | # Welcome to EZSandbox 8 | 9 | In this series of workshops, we'll get you up and running to build your first community-hosted application in no time. 10 | 11 | This sandbox will take you through a few phases of development to try using Koii Tasks at all levels. 12 | 13 | 1. Deploy Locally on your Koii Task Node to Debug and Iterate Rapidly 14 | 15 | 2. Deploy to Docker to test audits and incentive mechanisms 16 | 17 | 3. Launch on the Community Cloud 18 | 19 | # Lessons and Code Samples 20 | 21 | In this project, we'll start by demonstrating the key features of the Node compute environment. After some local testing, we'll harden our incentive mechanism and deploy it to the Koii cloud. 22 | 23 | Koii is a network of people, using their nodes to support a diverse ecosystem of products and services, all operated by community members like you. 24 | 25 | Decentralized Applications on the Koii Cloud run in modules called 'Tasks' and anyone can join by running a Koii Node, a program users install which manages and runs Tasks. 26 | 27 | At the end of these tutorials, you'll be ready to build your first Koii Application that other community members can then run on their Node. 28 | 29 |
30 |
31 | 32 | ![Koii tasks](https://github.com/koii-network/ezsandbox/assets/113378734/04edd56a-04e8-4a9f-9b89-752ba046b3ad) 33 | 34 | ## Lesson 1: Introduction to Koii Tasks 35 | 36 | In the first lesson, we'll set up a Koii Node and start debugging an existing Task. 37 | 38 | This lesson will teach you: 39 | 40 | - How to debug tasks live with your Node 41 | - How Tasks run in the node 42 | - How to connect to your node 43 | 44 | [Start Here](./Lesson%201/README.md) 45 | 46 |
47 |
48 | 49 | ![Networking and storage](https://github.com/koii-network/ezsandbox/assets/5794319/14abeb3f-3cb3-4c08-b553-2aa5e2839828) 50 | 51 | ## Lesson 2: Networking and Storage Task 52 | 53 | Once we've got the basics down, we can move on to writing a task of our own. We'll learn how to use networking and storage with the example of a simple file server. We'll also see how to deploy our app on a Dockerized node and test it out locally. 54 | 55 | [Start Here](./Lesson%202/README.md) 56 | 57 |
58 |
59 | 60 | ![Secrets & Config](https://github.com/koii-network/ezsandbox/assets/113378734/2d6c43e6-d51b-4eca-80ce-2365ebafa881) 61 | 62 | ## Lesson 3: Secrets & Config 63 | 64 | One of the best use cases for Koii nodes is to gather data from the web. In this tutorial, we'll show you how to use local secrets on your node, take a closer look at the config options, and learn how to build out a full web crawler that runs on any participating Task Nodes. 65 | 66 | [Start Here](./Lesson%203/README.md) 67 | 68 |
69 |
70 | 71 | ![Auditing & Distribution](https://github.com/koii-network/ezsandbox/assets/113378734/d9ecac0d-7c89-4f8e-8038-5f0d77425c63) 72 | 73 | ## Lesson 4: Auditing & Distribution 74 | 75 | We can now start to add audit and distribution mechanisms, learning more about how to verify work and define incentives. We'll also learn how data can be shared between nodes by looking at an example. 76 | 77 | [Start Here](./Lesson%204/README.md) 78 | 79 |
80 |
81 | 82 | ![Security & Hardening](https://github.com/koii-network/ezsandbox/assets/113378734/a2c81c09-a108-483c-80f3-d1271cb2d339) 83 | 84 | ## Lesson 5: Security & Hardening 85 | 86 | Now that you've seen several different types of tasks, this lesson will cover how to add authorized accounts, verify signatures, and manage general authentication and data authority issues. 87 | 88 | [Start Here](./Lesson%205/README.md) 89 | 90 |
91 |
92 | 93 | ![Custom Tokens](https://github.com/koii-network/ezsandbox/assets/113378734/3b3c5c4b-ab28-4a49-9462-de7753586bdf) 94 | 95 | ## Lesson 6: Using Custom Tokens for Tasks 96 | 97 | This lesson will teach you how to deploy your own custom KPL token on Koii. 98 | 99 | [Start Here](./Lesson%206/README.md) 100 | 101 |
102 |
103 | 104 | ![Deployment](https://github.com/koii-network/ezsandbox/assets/113378734/e8bccde8-f815-41fc-9467-26cf982157e0) 105 | 106 | ## Lesson 7: Deploying your Task 107 | 108 | Once everything is tightened down, it's time to get your community and start running nodes. We'll get you a small grant in KOII to fund your task bounty, deploy the task, and run it on your node. 109 | 110 | [Start Here](./Lesson%207/README.md) 111 | 112 |
113 |
114 | 115 | ![Performance Improvements](https://github.com/koii-network/ezsandbox/assets/113378734/65327ccd-8abd-41d4-8719-c1b4f3ed9da4) 116 | 117 | ## Lesson 8: Performance Improvements 118 | 119 | After your task is live, it's time to consider improving your work. In this final lesson, we'll cover some tips on debugging, multi-node simulations, and how to publish an update to your Task. 120 | 121 | [Start Here](./Lesson%208/README.md) 122 | 123 |
124 |
125 | 126 | # Open Source Roadmap 127 | Koii is committed to being an open source project, but we are a small team and focused on improving developer and user experience at the moment. 128 | - K2 has been audited by Halborn, a leading security firm and the original auditor of the Solana codebase which it was forked from. K2 is planned to be open sourced in mid July 2024. [Click here for the Full Audit Report](https://twitter.com/HalbornSecurity/status/1784862949581938785) 129 | - The Task Node is currently being audited, and we plan to open source that codebase as soon as it has been fully audited. This decision was made to protect the community of node operators from any critical vulnerabilities, but we do offer the source code to community members, and you can ask to receive access by contacting us on [discord](discord.gg/koii-network). Usually these requests are resolved within 48 hrs. 130 | 131 | ## We are sorry that we can't do this sooner. 132 | This code base has been rapidly iterated, with now over 70 versions, so it was only recently possible to begin audits and we are keen to open source the codebase right away. 133 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | {} 2 | --------------------------------------------------------------------------------