├── .github └── ISSUE_TEMPLATE │ ├── FIP_review_request.yaml │ ├── Idea_review_request.yaml │ ├── open-problem-issue-template.md │ └── starmap_template.md ├── .gitignore ├── FIP0036 ├── Public_Data │ ├── FIL_Poll_DummyData.csv │ ├── README.md │ ├── all_votes.csv │ ├── raw_votes.csv │ ├── tallied_votes.csv │ └── tallied_votesa.csv ├── README.md └── post_process │ ├── README.md │ ├── __init__.py │ ├── counting.py │ ├── dataHandler.py │ ├── datasets │ ├── longShort.csv │ └── msigs.json │ ├── datautils.py │ ├── groups.py │ ├── main.py │ ├── preprocess.py │ ├── requirements.txt │ ├── sentinel.py │ └── votes.py ├── HC gas simulation.ipynb ├── IPC cross net fees.ipynb ├── LICENSE-APACHE.md ├── LICENSE-MIT.md ├── README.md ├── notebooks ├── FIP-0032 │ └── FIP-0032-backtest-cel-v1.2.ipynb ├── Self preserving SP's simulation.ipynb ├── baseline_crossing │ ├── power_analysis.ipynb │ ├── power_forecast.ipynb │ ├── power_forecast_v2.ipynb │ ├── power_model.py │ ├── power_model_v2.py │ ├── readme.md │ └── sector_updates_trends.ipynb └── shortfall │ ├── shortfall_dynamics.pdf │ ├── shortfall_dynamics_v0.nb │ └── shortfall_dynamics_v0.pdf ├── open_problems ├── README.md ├── analytics │ ├── collusion_community_detection.md │ ├── filecoin_mining_concentration.md │ └── filecoin_onboarding_forecast.md ├── econ_modelling │ ├── filecoin_ml_general.md │ ├── network_stability_in_rare_events.md │ └── optimal_control_for_blockchains.md └── incentive_design │ ├── incentivized_network_formation.md │ ├── truthful_games.md │ └── value_attribution.md └── tools ├── MpoolUtils ├── README.md ├── mpoolUtils.py ├── requirements.txt └── test.py ├── README.md └── minerUtils └── minerExplore.py /.github/ISSUE_TEMPLATE/FIP_review_request.yaml: -------------------------------------------------------------------------------- 1 | name: CEL FIP Review 2 | description: Request for CryptoEconLab to do FIP-related Modeling, Analysis, Review, and/or Comment Work 3 | title: "[FIP Review]: " 4 | labels: ["FIP Review"] 5 | assignees: 6 | - octocat 7 | body: 8 | - type: markdown 9 | attributes: 10 | value: | 11 | See associated [submission process and SLA](https://github.com/protocol/CryptoEconLab/blob/main/README.md#work-request-hammer_and_wrench) for expectations on timelines and level of effort. Work products can range from very high-level and expedient to very sophisticated and labor-intensive, depending on the requirements. 12 | 13 | - type: textarea 14 | id: request_summary 15 | attributes: 16 | label: Request Summary 17 | description: Succinctly summarize the request that you are asking us to explore in a paragraph or two 18 | placeholder: Short summary of the request / issue / question here... 19 | validations: 20 | required: true 21 | 22 | - type: dropdown 23 | id: audience 24 | attributes: 25 | label: Audience / Consumer for this Review 26 | description: Select one or more of 27 | options: 28 | - Core Developers 29 | - Ecosystem Developers 30 | - Storage Providers 31 | - Token Holders 32 | multiple: true 33 | validations: 34 | required: true 35 | 36 | - type: dropdown 37 | id: timeframe 38 | attributes: 39 | label: Timeframe 40 | description: Select one or more of 41 | options: 42 | - No Rush 43 | - Needed in 6 months 44 | - Needed in 3 months 45 | - Needed in 1 month 46 | validations: 47 | required: true 48 | 49 | - type: textarea 50 | id: rationale 51 | attributes: 52 | label: Rationale 53 | description: Short description of why this analysis is needed 54 | validations: 55 | required: false 56 | 57 | - type: textarea 58 | id: desired_deliverables 59 | attributes: 60 | label: Desired Deliverables 61 | description: It would be helpful to describe any specific deliverables you might have in mind to give insight into the request. For example “A graph of quantities X and Y plotted over time for the next 12 months” 62 | validations: 63 | required: false 64 | 65 | - type: textarea 66 | id: related_info 67 | attributes: 68 | label: Relevant Links (Including FIPs & FIP discussions) 69 | description: Additional resources, documentation, and links that may be helpful. **PLEASE INCLUDE LINKS TO FIPS OR FIP DISCUSSIONS.** 70 | validations: 71 | required: false 72 | 73 | - type: input 74 | id: contact 75 | attributes: 76 | label: Contact Details (if Private Response Requested) 77 | description: Optional. We will default to public responses on the created issue, but if you request an alternate communication method, how should we get in touch with you? 78 | placeholder: ex. email@example.com 79 | validations: 80 | required: false 81 | 82 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/Idea_review_request.yaml: -------------------------------------------------------------------------------- 1 | name: CEL Idea Review 2 | description: Request for CryptoEconLab to do Modeling, Analysis, Review, and/or Comment Work on an idea that is not yet a formal FIP 3 | title: "[Idea Review]: " 4 | labels: ["Idea Review"] 5 | assignees: 6 | - octocat 7 | body: 8 | - type: markdown 9 | attributes: 10 | value: | 11 | See associated [submission process and SLA](https://github.com/protocol/CryptoEconLab/blob/main/README.md#work-request-hammer_and_wrench) for expectations on timelines and level of effort. Work products can range from very high-level and expedient to very sophisticated and labor-intensive, depending on the requirements. 12 | 13 | - type: textarea 14 | id: request_summary 15 | attributes: 16 | label: Request Summary 17 | description: Succinctly summarize the request that you are asking us to explore in a paragraph or two 18 | placeholder: Short summary of the request / issue / question here... 19 | validations: 20 | required: true 21 | 22 | - type: dropdown 23 | id: audience 24 | attributes: 25 | label: Audience / Consumer for this Review 26 | description: Select one or more of 27 | options: 28 | - Core Developers 29 | - Ecosystem Developers 30 | - Storage Providers 31 | - Token Holders 32 | multiple: true 33 | validations: 34 | required: true 35 | 36 | - type: dropdown 37 | id: timeframe 38 | attributes: 39 | label: Timeframe 40 | description: Select one or more of 41 | options: 42 | - No Rush 43 | - Needed in 6 months 44 | - Needed in 3 months 45 | - Needed in 1 month 46 | validations: 47 | required: true 48 | 49 | - type: textarea 50 | id: rationale 51 | attributes: 52 | label: Rationale 53 | description: Short description of why this analysis is needed 54 | validations: 55 | required: false 56 | 57 | - type: textarea 58 | id: desired_deliverables 59 | attributes: 60 | label: Desired Deliverables 61 | description: It would be helpful to describe any specific deliverables you might have in mind to give insight into the request. For example “A graph of quantities X and Y plotted over time for the next 12 months” 62 | validations: 63 | required: false 64 | 65 | - type: textarea 66 | id: related_info 67 | attributes: 68 | label: Additional Information 69 | description: Additional resources, documentation, and links that may be helpful 70 | validations: 71 | required: false 72 | 73 | - type: input 74 | id: contact 75 | attributes: 76 | label: Contact Details (if Private Response Requested) 77 | description: Optional. We will default to public responses on the created issue, but if you request an alternate communication method, how should we get in touch with you? 78 | placeholder: ex. email@example.com 79 | validations: 80 | required: false 81 | 82 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/open-problem-issue-template.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Open Problem issue template 3 | about: for posing new Open Problems for discussion 4 | title: "[Open Problem] -" 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | _**Notes on creating the open problem statement**: The purpose of the open problem statement is twofold. Firstly, it should convince the reader that the problem you are presenting is worth working on. Secondly, it should provide enough background and understanding of the problem that all design decisions and requirements are comprehensively described and motivated. Feel free to deviate from the following template if you prefer, or answer the following questions as succinctly as possible for an easy open problem statement._ 11 | 12 | _While this template was made to support the RFP program, the open problem statements themselves are purely for the benefit of the community, and there is no obligation to make or request an RFP for each open problem._ 13 | 14 | # Title 15 | 16 | ## Description 17 | 18 | ## State of the Art 19 | 20 | *This survey on the State of the Art does not have to be exhaustive but it should provide a good entry point to existing work on the topic. Later contributors can augment the survey via PR.* 21 | 22 | ### Current approaches within the CryptoEconLab network 23 | > Existing attempts and strategies to solve the problem within the context of the relevant PL project 24 | 25 | 26 | ### Current approaches within the broader research ecosystem 27 | > How do people try to solve this problem more generally? Does it resemble any other recognized research problems? 28 | 29 | 30 | ### Known shortcomings of existing solutions 31 | > What are the limitations of current solutions within the project ecosystem and elsewhere? 32 | 33 | ## Solving this Open Problem 34 | 35 | ### Estimated impact 36 | > How would we and/or the world benefit from solving this problem? 37 | 38 | ### Proposed solution definition 39 | > What defines a complete solution? What hard constraints should it obey? Are there additional "soft" constraints that a solution would ideally obey? 40 | 41 | ## Supplementary Material 42 | 43 | ### Relevant reading 44 | 45 | ### Existing conversations/threads 46 | > Link to prior discussions relevant to the problem 47 | 48 | ### Extra notes 49 | > anything else deemed relevant to the problem 50 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/starmap_template.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: StarMap Issue Template 3 | about: Create an Issue that is tracked for StarMap roadmapping 4 | title: "StarMap Issue Template" 5 | labels: ["starmaps"] 6 | assignees: '' 7 | --- 8 | eta: 2022-12-31 9 | 10 | children: 11 | - https://github.com/protocol/CryptoEconLab/issues/33 12 | - https://github.com/protocol/CryptoEconLab/issues/34 13 | - https://github.com/protocol/CryptoEconLab/issues/35 14 | 15 | parent: 16 | - https://github.com/protocol/CryptoEconLab/issues/55 17 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | share/python-wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | MANIFEST 28 | 29 | # PyInstaller 30 | # Usually these files are written by a python script from a template 31 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 32 | *.manifest 33 | *.spec 34 | 35 | # Installer logs 36 | pip-log.txt 37 | pip-delete-this-directory.txt 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .nox/ 43 | .coverage 44 | .coverage.* 45 | .cache 46 | nosetests.xml 47 | coverage.xml 48 | *.cover 49 | *.py,cover 50 | .hypothesis/ 51 | .pytest_cache/ 52 | cover/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | .pybuilder/ 76 | target/ 77 | 78 | # Jupyter Notebook 79 | .ipynb_checkpoints 80 | 81 | # IPython 82 | profile_default/ 83 | ipython_config.py 84 | 85 | # pyenv 86 | # For a library or package, you might want to ignore these files since the code is 87 | # intended to run in multiple environments; otherwise, check them in: 88 | .python-version 89 | 90 | # pipenv 91 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 92 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 93 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 94 | # install all needed dependencies. 95 | Pipfile.lock 96 | 97 | # poetry 98 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 99 | # This is especially recommended for binary packages to ensure reproducibility, and is more 100 | # commonly ignored for libraries. 101 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 102 | #poetry.lock 103 | 104 | # pdm 105 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 106 | #pdm.lock 107 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 108 | # in version control. 109 | # https://pdm.fming.dev/#use-with-ide 110 | .pdm.toml 111 | 112 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 113 | __pypackages__/ 114 | 115 | # Celery stuff 116 | celerybeat-schedule 117 | celerybeat.pid 118 | 119 | # SageMath parsed files 120 | *.sage.py 121 | 122 | # Environments 123 | .env 124 | .venv 125 | env/ 126 | venv/ 127 | ENV/ 128 | env.bak/ 129 | venv.bak/ 130 | 131 | # Spyder project settings 132 | .spyderproject 133 | .spyproject 134 | 135 | # Rope project settings 136 | .ropeproject 137 | 138 | # mkdocs documentation 139 | /site 140 | 141 | # mypy 142 | .mypy_cache/ 143 | .dmypy.json 144 | dmypy.json 145 | 146 | # Pyre type checker 147 | .pyre/ 148 | 149 | # pytype static type analyzer 150 | .pytype/ 151 | 152 | # Cython debug symbols 153 | cython_debug/ 154 | 155 | # PyCharm 156 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 157 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 158 | # and can be added to the global gitignore or merged into this file. For a more nuclear 159 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 160 | .idea/ 161 | 162 | 163 | # MacOS files 164 | .DS_Store 165 | 166 | # Language tool 167 | ltex* 168 | 169 | # Misc 170 | SecretString.txt -------------------------------------------------------------------------------- /FIP0036/Public_Data/FIL_Poll_DummyData.csv: -------------------------------------------------------------------------------- 1 | Category ,Approve,Reject,Timestamp 2 | SP_Deals,50%,50%,9/16/22 3 | SP_Capacity,50%,50%,9/16/22 4 | Storage_Clients,50%,50%,9/16/22 5 | Token_Holders,50%,50%,9/16/22 6 | Core_Devs,50%,50%,9/16/22 -------------------------------------------------------------------------------- /FIP0036/Public_Data/README.md: -------------------------------------------------------------------------------- 1 | # Public Data Sources for FIP-0036 2 | -------------------------------------------------------------------------------- /FIP0036/Public_Data/tallied_votes.csv: -------------------------------------------------------------------------------- 1 | ,Approve,Reject,id 2 | 0,5,8,core_dev 3 | 1,2735751929913627153510365,2184929923559288090581933,token_holder 4 | 2,15714131043418112,63684310644352000,clients 5 | 3,4.662438547452592e+18,2.537502343885226e+18,capacity 6 | 4,30504554830222336,130048626444320768,deals 7 | -------------------------------------------------------------------------------- /FIP0036/Public_Data/tallied_votesa.csv: -------------------------------------------------------------------------------- 1 | ,Approve,Reject,id 2 | 0,0,0,core_dev 3 | 1,2073491717409743066936071,1930463683772856338957938,token_holder 4 | 2,15714122185048064,21118149956527104,clients 5 | 3,3.29744619501781e+18,2.3492326929832346e+18,capacity 6 | 4,22422964241683456,117402654143791104,deals 7 | -------------------------------------------------------------------------------- /FIP0036/README.md: -------------------------------------------------------------------------------- 1 | # Public Resources for FIP-0036 2 | 3 | 4 | The post_process folder contains all post-processing scripts for the FIP-0036 FIL Poll conducted Q3 2022. 5 | 6 | -------------------------------------------------------------------------------- /FIP0036/post_process/README.md: -------------------------------------------------------------------------------- 1 | # Post-processing scripts 2 | 3 | 4 | The main file is main.py. This file, essentially, processes each vote by taking the following steps: 5 | 6 | 1.Receive a signed vote from X 7 | 2.Validate the signature 8 | 3.Confirm that X exists on chain 9 | 3.1 (if it does) Add X's balance to the Token Holder vote (Group 4)) 10 | 4. Iterate over all deals, adding up the bytes of deals where X is the proposer 11 | 4.1 Add these bytes to the client vote (Group 3) 12 | 5. Iterate over all SPs, checking if X is the owner / worker of an SP Y 13 | 5.1. If it is, AND the SP hasn't already voted: 14 | 5.1.1 Add Y's raw bytes to the SP capacity vote (Group 2) 15 | 5.1.2 Iterate over all deals, adding up the bytes of deals where Y is the provider, add these bytes to Deal Storage vote (Group 1) 16 | 17 | This process is contained in the `counting.py` module. The rest of the files are: 18 | 19 | `sentinel.py`: module to connect to the sentinel database. IT REQUIRES A VALID CONNECTION STRING STORED AS `SecretString.txt` IN THIS FOLDER. Alternatively, you can download the required datasets from here https://drive.google.com/drive/folders/1HFdZClf2KjtiUbTBy6VkvTlw6dVTJtEJ?usp=sharing 20 | and pass an empty connection string. 21 | 22 | `datautils.py`: module to query the sentinel database and perform several table-lookup operations 23 | 24 | `votes.py`: module with vote data. It connects to the FIlPoll API and queries the up-to-date list of votes 25 | 26 | `counting.py`: vpte class.performs the algorithm above for each vote 27 | 28 | `groups.py`: group class. 29 | 30 | `preprocess.py`: preprocess chain data. 31 | 32 | 33 | -------------------------------------------------------------------------------- /FIP0036/post_process/__init__.py: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /FIP0036/post_process/counting.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Created on Sun Sep 18 23:27:17 2022 5 | 6 | @author: juan 7 | """ 8 | 9 | import json 10 | import datautils as utils 11 | 12 | 13 | 14 | def findMsigs(signature,ms2): 15 | 16 | '''returns list of multisigs that have ::signature:: as a signer ''' 17 | 18 | 19 | signer=signature['short'] 20 | 21 | 22 | msig_ids=[] 23 | 24 | for i in range(len(ms2)): 25 | vv=ms2['Signer'].iloc[i] 26 | if signer in vv: 27 | print('count') 28 | 29 | msig_ids.append(ms2['ID'].iloc[i]) 30 | ms2['count'].iloc[i]+=1 31 | 32 | 33 | 34 | if ms2['yes'].iloc[i] < ms2['Threshold'].iloc[i] and ms2['no'].iloc[i] < ms2['Threshold'].iloc[i]: 35 | 36 | if signature['optionName']=='Approve': 37 | ms2['yes'].iloc[i]+=1 38 | elif signature['optionName']=='Reject': 39 | ms2['no'].iloc[i]+=1 40 | 41 | 42 | return ms2,msig_ids 43 | 44 | 45 | 46 | 47 | def countVote(vote,groups_of_voters,datasets,signatures,ms2): 48 | 49 | list_of_deals=datasets['deals'] 50 | list_of_miners=datasets['miners'] 51 | list_of_addresses_and_ids=datasets['addresses'] 52 | list_of_powers=datasets['powers'] 53 | list_long_short=datasets['longShort'] 54 | list_of_balances=datasets['balances'] 55 | 56 | 57 | 58 | 59 | #-------------------------------------------------------------------------- 60 | # 61 | # Prepares the vote: gets some info related to the vote 62 | # 63 | #-------------------------------------------------------------------------- 64 | signature=json.loads(vote["signature"]) 65 | 66 | # This is a fix originated from having manually addded core dev votes into the filPOll 67 | 68 | if signature=={}: 69 | signature['signer']=vote['signerAddress'] 70 | signature['address']=vote['address'] 71 | 72 | signature['constituentGroupId']=vote['constituentGroupId'] 73 | signature['balance']=0 74 | signature['power']=0 75 | if vote['optionId']==50: 76 | signature['optionName']='Reject' 77 | elif vote['optionId']==49: 78 | signature['optionName']='Approve' 79 | 80 | 81 | # adds short id to the vote. This is necesary for the table lookup 82 | signature=utils.addShortAndLongId(signature=signature, 83 | list_of_addresses_and_ids=list_of_addresses_and_ids, 84 | longShort=list_long_short) 85 | 86 | 87 | 88 | 89 | is_msig= signature['short'] in list(ms2['ID']) 90 | 91 | 92 | ms2,msig_ids=findMsigs(signature,ms2) 93 | signature['msigs']=msig_ids 94 | 95 | if is_msig: 96 | signature['is_msig']=True 97 | else: 98 | signature['is_msig']=False 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | #checks if it's a core dev by chcecking if constituent group =6. 107 | # this value was hardcoded by filpoll 108 | is_core_dev =vote['constituentGroupId']==6 109 | 110 | # check if its an owner or worker address 111 | result,otherID=utils.is_worker_or_owner(Id=signature['short'], 112 | list_of_miners=list_of_miners) 113 | 114 | 115 | # see which SPs have signature as owner or worker 116 | SPs= utils.get_owners_and_workers(list_of_miners,signature['short']) 117 | 118 | #-------------------------------------------------------------------------- 119 | # 120 | # votes as core dev 121 | # 122 | #-------------------------------------------------------------------------- 123 | 124 | if is_core_dev: 125 | groups_of_voters["core"].validateAndAddVote(signature) 126 | 127 | #-------------------------------------------------------------------------- 128 | # 129 | # votes as token holder 130 | # 131 | #-------------------------------------------------------------------------- 132 | if vote['balance']>0: 133 | groups_of_voters["token"].validateAndAddVote(signature) 134 | #-------------------------------------------------------------------------- 135 | # 136 | # votes as client; Iterate over deals, adds bytes where X is the proposer 137 | # 138 | #-------------------------------------------------------------------------- 139 | 140 | dealsForX=utils.get_market_deals_from_id(list_of_deals, 141 | user_id=signature['short'], 142 | side='client_id') 143 | totalBytes = dealsForX["padded_piece_size"].sum() 144 | if totalBytes > 0: 145 | groups_of_voters["client"].validateAndAddVote(signature, 146 | amount=totalBytes) 147 | 148 | #-------------------------------------------------------------------------- 149 | # 150 | # checks for overriding SP votes 151 | # 152 | #-------------------------------------------------------------------------- 153 | result,otherID=utils.is_worker_or_owner(Id=signature['short'], 154 | list_of_miners=list_of_miners) 155 | 156 | 157 | if otherID!=signature['short']: 158 | 159 | otherID_long=utils.longFromShort(otherID, list_of_addresses_and_ids) 160 | has_cap_voted=groups_of_voters['capacity'].has_voted(otherID_long) 161 | has_deal_voted= groups_of_voters['deal'].has_voted(otherID_long) 162 | has_other_voted=has_cap_voted or has_deal_voted 163 | 164 | #if it's an owner, override the entries from the worker, if they exist 165 | if result=='owner' and has_other_voted: 166 | print(' ') 167 | print('overwritting '+otherID_long) 168 | for gr in ['capacity','deal']: 169 | if has_other_voted and (otherID_long!=signature['signer']): 170 | groups_of_voters[gr].removeVote(otherID_long) 171 | 172 | # and if its a worker and owner has already votes, skip 173 | elif result=='worker' and has_other_voted: 174 | print(' ') 175 | print('{} had already voted, skipping'.format(otherID_long)) 176 | signatures.append(signature) 177 | return groups_of_voters,signatures,ms2 178 | 179 | #-------------------------------------------------------------------------- 180 | # 181 | # Adds up SP votes (commited capacity and deal SPs) 182 | # 183 | #-------------------------------------------------------------------------- 184 | 185 | total_power_SPs=0 186 | totalBytesY=0 187 | other_info={'miner_ids':[]} 188 | 189 | for Y_i in SPs: 190 | power_Y_i=utils.get_power(Y_i, list_of_powers) 191 | total_power_SPs+=power_Y_i 192 | #-------------------------------------------------------------------------- 193 | # 194 | # Iterate over all deals, adding up the bytes of deals 195 | # where Y is the provider, add these bytes to Deal Storage vote (Group 1) 196 | # 197 | #-------------------------------------------------------------------------- 198 | dealsByY=utils.get_market_deals_from_id(list_of_deals, 199 | user_id=Y_i, 200 | side='provider_id') 201 | #adds each SPs RBP to the total bytes counter 202 | totalBytesY += dealsByY["padded_piece_size"].sum() 203 | #-------------------------------------------------------------------------- 204 | # 205 | # from Add Y's raw bytes to the SP capacity vote (Group 2) 206 | # 207 | #-------------------------------------------------------------------------- 208 | other_info['miner_ids'].append(Y_i) 209 | # 210 | # Adds miner Available balance (balance-locked-pledged), if any 211 | # 212 | balance_yi=list_of_balances[list_of_balances['minerId']==Y_i] 213 | if len(balance_yi)>0: 214 | av_balance_yi=int(balance_yi['balance'].values[0]) 215 | av_balance_yi+=-int(balance_yi['lockedBalance'].values[0]) 216 | av_balance_yi+=-int(balance_yi['initial_pledge'].values[0]) 217 | groups_of_voters["token"].validateAndAddVote(signature, 218 | amount=av_balance_yi, 219 | validate=False) 220 | 221 | 222 | 223 | 224 | 225 | groups_of_voters["capacity"].validateAndAddVote(signature, 226 | amount=total_power_SPs, 227 | other_info=other_info) 228 | groups_of_voters["deal"].validateAndAddVote(signature,amount=totalBytesY) 229 | 230 | 231 | #updates signature tracker 232 | signature['client_bytes']=totalBytes 233 | signature['capcity_bytes']=total_power_SPs 234 | signature['deal_bytes']=totalBytesY 235 | vote_out=totalBytes+total_power_SPs+totalBytesY 236 | 237 | signature['diff']=int(signature['power'])-vote_out 238 | signatures.append(signature) 239 | 240 | 241 | return groups_of_voters,signatures,ms2 242 | -------------------------------------------------------------------------------- /FIP0036/post_process/dataHandler.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | This is the datahabndler class. This class is used to interact with an SQL database 5 | @author: juan 6 | """ 7 | 8 | -------------------------------------------------------------------------------- /FIP0036/post_process/datautils.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Created on Mon Sep 12 09:19:15 2022 5 | 6 | @author: juan 7 | """ 8 | # imports required libraries 9 | import numpy as np 10 | import matplotlib.pyplot as plt 11 | import pandas as pd 12 | from sentinel import sentinel 13 | import os 14 | 15 | 16 | 17 | def longFromShort(Id:str,list_of_addresses_and_ids:list): 18 | ''' 19 | Gets long Id (address) from short ID 20 | 21 | 22 | Parameters 23 | ---------- 24 | Id : str 25 | short id. 26 | list_of_addresses_and_ids : list 27 | a list with addresses and ids 28 | 29 | Returns 30 | ------- 31 | long : str 32 | address associated with Id 33 | 34 | ''' 35 | 36 | try: 37 | long=list_of_addresses_and_ids[ 38 | list_of_addresses_and_ids['id']==Id]['address'].values[0] 39 | except: 40 | long=None 41 | return long 42 | 43 | def getShortId(signature:dict,list_of_addresses_and_ids:list,longShort:list): 44 | ''' 45 | gets short id from long-format address 46 | 47 | Parameters 48 | ---------- 49 | signature : dict 50 | dictionary with the signatures we want to get id for 51 | list_of_addresses_and_ids : list 52 | table with addresses and Ids 53 | 54 | Returns 55 | ------- 56 | None. 57 | 58 | ''' 59 | 60 | A=signature['address'] 61 | X=signature['signer'] 62 | if A[:2]=='f0': # if it start with f0 63 | signature['short']=A 64 | 65 | else: 66 | 67 | try: 68 | signature['short']=longShort[longShort['long']==X]['short'].values[0] 69 | 70 | except: 71 | 72 | try: 73 | signature['short']=list_of_addresses_and_ids[ 74 | list_of_addresses_and_ids['address']==X 75 | ]['id'].values[0] 76 | except: 77 | #finds it onchain from lotus 78 | try: 79 | 80 | cmd='/usr/local/bin/lotus state lookup '+X 81 | signature['short']=os.popen(cmd).read()[:-1] 82 | except: 83 | print('error pinging from lotus. Make sure that you have initialized lotus daemon and that you are using the right path for lotus') 84 | 85 | 86 | return signature 87 | 88 | 89 | 90 | 91 | 92 | def addShortAndLongId(signature:dict,list_of_addresses_and_ids,longShort): 93 | 94 | 95 | 96 | signature['long']=signature['signer'] 97 | signature=getShortId(signature, list_of_addresses_and_ids,longShort) 98 | 99 | return signature 100 | 101 | 102 | 103 | 104 | 105 | 106 | def is_in_list(x:str,list_:list): 107 | ''' 108 | checks if an address x is in the list_ 109 | 110 | used to check if x is in list_ 111 | 112 | Parameters 113 | ---------- 114 | x : str 115 | address. 116 | list_ : list 117 | list to be checked against 118 | 119 | Returns 120 | ------- 121 | is_it : boolean 122 | True if it is, false otherwise 123 | ''' 124 | 125 | 126 | is_it=x in list_ 127 | return is_it 128 | 129 | 130 | 131 | def connect_to_sentinel(secret_string: str): 132 | """ 133 | creates a connection coursor to the sentinel database 134 | 135 | Parameters 136 | ---------- 137 | secret_string : str 138 | connection string to connect to sentinel. it should have the form: 139 | postgres://readonly:j@read.lilium.sh:13573/mainnet?sslmode=require 140 | 141 | Returns 142 | ------- 143 | db : a sentinel object which is essenyially an sql aclhemy cuorsor connected to sentiel 144 | 145 | """ 146 | f = open(secret_string, "r") 147 | NAME_DB = f.read() 148 | 149 | # initializes the class 150 | db = sentinel(NAME_DB) 151 | return db 152 | def get_miner_locked_funds(database: sentinel or None, height: int): 153 | 154 | 155 | try: 156 | miner_locked=pd.read_csv('datasets/miner_locked.csv') 157 | except: 158 | print('miner locked balances not found, querrying sentinel ...') 159 | 160 | 161 | QUERY = """ 162 | SELECT DISTINCT "miner_id", "height", "pre_commit_deposits", "initial_pledge", "locked_funds" 163 | FROM "visor"."miner_locked_funds" 164 | WHERE "height"<='{}' 165 | ORDER BY "height" DESC 166 | """.format(height) 167 | miner_locked = database.customQuery(QUERY) 168 | miner_locked.to_csv('datasets/miner_locked_funds.csv') 169 | return miner_locked 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | def get_miner_sector_deals(database: sentinel or None, miner_id: str, height: int): 179 | """ 180 | Returns a pandas dataframe with information regarding sector deals for 181 | miner with `miner_id` updated until `height` 182 | 183 | Parameters 184 | ---------- 185 | database : sentinel 186 | db : a sentinel object which is essenyially an sql aclhemy cuorsor connected to sentiel 187 | miner_id : str 188 | id of the miner we want to get this info from 189 | height : int 190 | cutoff height. 191 | 192 | Returns 193 | ------- 194 | sector_deals : pandas.Dataframe containing relevant data; 195 | "miner_id","sector_id","activation_epoch","expiration_epoch", 196 | "deal_weight","verified_deal_weight","height" 197 | 198 | """ 199 | 200 | QUERY = """ 201 | SELECT "miner_id","sector_id","activation_epoch","expiration_epoch", 202 | "deal_weight","verified_deal_weight","height" 203 | FROM "visor"."miner_sector_infos" 204 | WHERE "height"<='{}', miner_id='{}' 205 | ORDER BY "height" DESC 206 | """.format(height, miner_id 207 | ) 208 | sector_deals = database.customQuery(QUERY) 209 | return sector_deals 210 | 211 | 212 | def get_owned_SPs(database: sentinel or None, height: int): 213 | """ 214 | Get SP addresses owned by or with worker id `owner_id` up until `height` 215 | 216 | Parameters 217 | ---------- 218 | database : sentinel 219 | db : a sentinel object which is essenyially an sql aclhemy cuorsor connected to sentiel 220 | owner_id : str 221 | owner or worker id we want to do this with 222 | height : int 223 | cutoff height. 224 | 225 | Returns 226 | ------- 227 | miner_infos : pandas.Dataframe containing relevant data; 228 | "miner_id", "multi_addresses", "height" 229 | """ 230 | 231 | try: 232 | miner_infos=pd.read_csv('datasets/miner_infos.csv') 233 | except: 234 | print('miner info (owner/worker) not found, querrying sentinel ...') 235 | 236 | 237 | QUERY = """ 238 | SELECT "miner_id", "owner_id", "worker_id", "height" 239 | FROM "visor"."miner_infos" 240 | WHERE "height"<='{}' 241 | ORDER BY "height" DESC 242 | """.format(height) 243 | miner_infos = database.customQuery(QUERY) 244 | miner_infos=miner_infos.sort_values(by='height', ascending=False) 245 | miner_infos=miner_infos.groupby(by='miner_id').head(1) 246 | 247 | miner_infos.to_csv('datasets/miner_infos.csv') 248 | return miner_infos 249 | 250 | 251 | def is_worker_or_owner(Id,list_of_miners): 252 | ''' checks if the Id corresponds to a worker or an owner, by checking if 253 | such an Id can be found in the given column''' 254 | 255 | worker=Id in list(list_of_miners['worker_id']) 256 | owner=Id in list(list_of_miners['owner_id']) 257 | 258 | if owner==False and worker==True: 259 | result='worker' 260 | otherID=list_of_miners[list_of_miners['worker_id']==Id]['owner_id'].values[0] 261 | if owner==True: 262 | result='owner' 263 | otherID=list_of_miners[list_of_miners['owner_id']==Id]['worker_id'].values[0] 264 | if owner==False and worker==False: 265 | result='Neither' 266 | otherID=None 267 | return result,otherID 268 | 269 | def get_owners_and_workers(data:pd.core.frame.DataFrame, 270 | address: str): 271 | 272 | 273 | ''' 274 | returns a list of miner_id that have owner or worker =address 275 | ''' 276 | SPs=[] 277 | 278 | spo=data[data['owner_id']==address]['miner_id'] 279 | spw=data[data['worker_id']==address]['miner_id'] 280 | 281 | sps=pd.concat([spo,spw]).unique() 282 | SPs=list(sps) 283 | 284 | 285 | 286 | 287 | 288 | return SPs 289 | 290 | 291 | 292 | 293 | 294 | def get_all_power_actors(database: sentinel or None, height: int): 295 | """ 296 | generates a list of all miners with their power QAP and RBP 297 | 298 | Parameters 299 | ---------- 300 | database : sentinel 301 | db : a sentinel object which is essenyially an sql aclhemy cuorsor connected to sentiel 302 | 303 | height : int 304 | cutoff height. 305 | 306 | Returns 307 | ------- 308 | power_actors : pandas dataframer containing a list of all miners with their power QAP and RBP 309 | 310 | """ 311 | QUERY = """ 312 | SELECT DISTINCT "miner_id", "height", "raw_byte_power", "quality_adj_power" 313 | FROM "visor"."power_actor_claims" 314 | WHERE "height"<={} 315 | ORDER BY "height" DESC 316 | """.format(height) 317 | power_actors = database.customQuery(QUERY) 318 | return power_actors 319 | 320 | 321 | def get_addresses(database: sentinel or None,height:int): 322 | """ 323 | Parameters 324 | ---------- 325 | database : sentinel 326 | db : a sentinel object which is essenyially an sql aclhemy cuorsor connected to sentiel 327 | 328 | Returns 329 | ------- 330 | actors : pandas dataframe with list of all different actors 331 | DESCRIPTION. 332 | 333 | """ 334 | 335 | try: 336 | 337 | actors=pd.read_csv('datasets/list_of_addresses.csv') 338 | except: 339 | print('list of addresses not found, querying...') 340 | QUERY = """SELECT "id", "address" FROM "visor"."id_addresses" WHERE height<='{}' 341 | """.format(height) 342 | actors = database.customQuery(QUERY) 343 | actors.to_csv('datasets/list_of_addresses.csv') 344 | return actors 345 | 346 | 347 | def get_active_power_actors(database: sentinel or None, height: int): 348 | """ 349 | generates a list of all active (with deals expiring after max_height) miners with their power QAP and RBP 350 | 351 | Parameters 352 | ---------- 353 | database : sentinel 354 | db : a sentinel object which is essenyially an sql aclhemy cuorsor connected to sentiel 355 | 356 | height : int 357 | cutoff height. 358 | 359 | Returns 360 | ------- 361 | power_actors : pandas dataframer containing a list of all active miners with their power QAP and RBP 362 | 363 | """ 364 | try: 365 | active_powers = pd.read_csv("datasets/power_actors.csv") 366 | except: 367 | power_actors = get_all_power_actors(database=database, height=height) 368 | positive_powers = power_actors[power_actors["quality_adj_power"] > 0] 369 | active_powers = positive_powers.sort_values( 370 | "height", ascending=False 371 | ).drop_duplicates("miner_id") 372 | active_powers.to_csv("datasets/power_actors.csv") 373 | return active_powers 374 | 375 | 376 | 377 | def get_power(miner_id:str, list_of_powers:pd.core.frame.DataFrame): 378 | try: 379 | power=list_of_powers[list_of_powers['miner_id']==miner_id]['raw_byte_power'].values[0] 380 | 381 | except: 382 | power=0 383 | return power 384 | 385 | 386 | 387 | def get_market_deals_from_id(market_deals:pd.core.frame.DataFrame, 388 | user_id: str, side:str): 389 | ''' 390 | return the market deals that involve 'user_id' as eihter a client or a provider 391 | 392 | Parameters 393 | ---------- 394 | market_deals : pd.core.frame.DataFrame 395 | DESCRIPTION. 396 | user_id : str 397 | DESCRIPTION. 398 | side : str 399 | 'client_id' or 'provider_id', determines which side we are looking at 400 | 401 | Returns 402 | ------- 403 | df : TYPE 404 | DESCRIPTION. 405 | 406 | ''' 407 | 408 | 409 | df=market_deals[market_deals[side]==user_id] 410 | 411 | return df 412 | 413 | 414 | def get_market_deals(database: sentinel or None, height: int): 415 | 416 | 417 | 418 | try: 419 | deals=pd.read_csv('datasets/market_deals.csv') 420 | except: 421 | 422 | 423 | 424 | 425 | print('getting list of market deal proposals...') 426 | 427 | 428 | 429 | query='''SELECT DISTINCT "piece_cid", "padded_piece_size", 430 | "client_id", "provider_id","height" 431 | FROM "visor"."market_deal_proposals" 432 | WHERE "height"<={} AND "end_epoch">={} AND "start_epoch"<={}'''.format(height,height,height) 433 | 434 | deals= database.customQuery(query) 435 | deals.to_csv('datasets/market_deals.csv') 436 | 437 | return deals 438 | 439 | 440 | def toObs(group,name:str): 441 | 442 | tally=group.tally 443 | 444 | total=tally['Approve']+tally['Reject'] 445 | 446 | 447 | Approve={"group":name, 448 | "Approve":tally['Approve'], 449 | "percent":tally['Approve']/total} 450 | 451 | Reject={"group":name, 452 | "Approve":tally['Reject'], 453 | "percent":tally['Reject']/total} 454 | 455 | return Approve,Reject 456 | 457 | 458 | 459 | 460 | def get_balances(database: sentinel or None, height: int): 461 | ''' 462 | 463 | return a list of balances up-untila centain height 464 | 465 | 466 | Parameters 467 | ---------- 468 | database : sentinel 469 | DESCRIPTION. 470 | height : int 471 | DESCRIPTION. 472 | 473 | Returns 474 | ------- 475 | balances : TYPE 476 | DESCRIPTION. 477 | 478 | ''' 479 | 480 | 481 | try: 482 | balances=pd.read_csv('datasets/balances.csv') 483 | except: 484 | 485 | 486 | print('getting list of balances proposals...') 487 | 488 | query='''SELECT DISTINCT "id", "balance", "height" 489 | FROM "visor"."actors" 490 | WHERE "height"<={}'''.format(height) 491 | 492 | balances= database.customQuery(query) 493 | balances.to_csv('datasets/balances.csv') 494 | 495 | return balances 496 | 497 | def get_msigs(database: sentinel or None, height: int): 498 | 499 | ''' returns list of msigs with threshold >1''' 500 | 501 | 502 | try: 503 | msigGM=pd.read_csv('datasets/msigs.csv') 504 | except: 505 | 506 | 507 | print('getting list of msigs...') 508 | 509 | query='''SELECT * FROM "visor"."multisig_approvals" 510 | WHERE "height"<={}'''.format(height) 511 | 512 | msig= database.customQuery(query) 513 | msigs=msig.drop(columns=['state_root','message','method','gas_used','transaction_id']) 514 | msigG=msigs.sort_values(by='height',ascending=False).groupby(by='multisig_id').head(1) 515 | msigGM=msigG[msigG['threshold']>1] 516 | msigGM.to_csv('datasets/msigs.csv') 517 | 518 | return msigGM 519 | 520 | 521 | def get_short_addresses_on_chain(long:list): 522 | from tqdm import tqdm 523 | import os 524 | ''' 525 | queries lotus directly to get short Ids from long ids 526 | 527 | 528 | REQUIRES A LOTUS DAEMON ON THE BACKGROUND 529 | 530 | Parameters 531 | ---------- 532 | long : list 533 | a list of addresses in long format 534 | 535 | Returns 536 | ------- 537 | results : dict 538 | a ldict of short ids associated to each long address entry 539 | 540 | ''' 541 | 542 | 543 | results={'long':long, 'short':[]} 544 | 545 | 546 | for a in tqdm(long): 547 | 548 | cmd='/usr/local/bin/lotus state lookup '+a 549 | results['short'].append(os.popen(cmd).read()) 550 | 551 | 552 | return results 553 | 554 | def get_locked(database: sentinel or None, height: int): 555 | 556 | query='''SELECT "initial_pledge","miner_id","height" FROM "visor"."miner_locked_funds" 557 | WHERE "height"<={}'''.format(height) 558 | lockedbalances= database.customQuery(query) 559 | 560 | 561 | lockedbalances.sort_values(by='height',acending=False) 562 | lockedbalances.groupby('miner_id').head(1) 563 | return lockedbalances 564 | 565 | 566 | if __name__ == "__main__": 567 | # minerId = "f01740934" 568 | 569 | HEIGHT = 2162760 570 | 571 | db = connect_to_sentinel(secret_string="SecretString.txt") 572 | # msigs=get_msigs(db,HEIGHT) 573 | # #locked=get_miner_locked_funds(database=db, height=HEIGHT) 574 | 575 | # # msd=get_miner_sector_deals(database=db,miner_id=minerId,height=HEIGHT) 576 | -------------------------------------------------------------------------------- /FIP0036/post_process/groups.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Created on Mon Sep 12 17:03:00 2022 5 | defines classes for each of the groups 6 | 7 | Group 1: Deal Storage (weighted by Deal Bytes) 8 | SPs sign the voting message from their owner address or worker address 9 | Group 2: Capacity (weighted by Raw Bytes) 10 | SPs sign the voting message from their owner address or worker address 11 | Group 3: Clients (weighted by Deal Bytes) 12 | Client sign the voting message from their deal making account (the account used to propose deals) 13 | Group 4: Token Holders (weighted by FIL balance, for both regular wallet and multisig accounts) 14 | Sign the voting message using their wallet account 15 | A multisig wallet vote is valid when the number of signatures received from the token holders(that are signers of the multisig) matches the threshold 16 | I.e: if a multisig has 5 signers with threshold as 3, then 3 out of 5 signers must sign and submit the voting message. 17 | 18 | 19 | @author: JP. 20 | juan.madrigalcianci@protocol.ai 21 | """ 22 | import pandas as pd 23 | from dataclasses import dataclass 24 | 25 | @dataclass 26 | class Vote: 27 | signer:str 28 | vote:str 29 | quantity:str 30 | other:dict 31 | groupID:int 32 | index:int 33 | 34 | 35 | def to_dict(self): 36 | 37 | aux={'signer':self.signer, 38 | 'vote':self.vote, 39 | 'quantity':self.quantity, 40 | 'other':self.other, 41 | 'groupID':self.groupID, 42 | 'index':self.index 43 | } 44 | return aux 45 | 46 | 47 | 48 | class groups: 49 | ''' 50 | This is a generic group object. it takes 51 | 52 | * `groupid`, where 53 | 54 | 1. deal storage 55 | 2. capacity 56 | 3. clients 57 | 4. token holders 58 | 5. core dev 59 | 60 | * `votes` is an array counting the votes (''Aprove, "Reject", "Abstain") 61 | * `address` address of the person or multisig that voted 62 | * `quantity` displays the quantity of each vote, it could be, e.g., raw bytes, head count, etc 63 | * `idType` miner_id or worker_id if groupdid=1 , regular or multisig if groupid=4, else None 64 | 65 | 66 | ''' 67 | def __init__(self,groupID): 68 | self.groupID=groupID 69 | self.votedMoreThanOnce=[] 70 | self.listVotes=[] 71 | 72 | names={'1':'deals', 73 | '2':'capacity', 74 | '3': 'clients', 75 | '4':'token_holder', 76 | '5':'core_dev'} 77 | 78 | self.groupName=names[str(self.groupID)] 79 | 80 | 81 | def validateAndAddVote(self,signature:dict,amount=None,other_info:dict=[],validate=True): 82 | ''' 83 | validates and adds a vote (as a dict) as a new vote on this group. 84 | Notice that an address might be able to vote for two different groups 85 | but not fo the same group more than once 86 | 87 | Parameters 88 | ---------- 89 | signature : dict 90 | a dictionary with the signature information 91 | 92 | ''' 93 | 94 | 95 | # gets right vote quantity 96 | 97 | if self.groupID==1 or self.groupID==2 or self.groupID==3: 98 | quantity=int(signature['power']) 99 | elif self.groupID==4: 100 | 101 | quantity=int(signature['balance']) 102 | 103 | 104 | elif self.groupID==5: 105 | quantity=1 106 | else: 107 | print('wrong group ID!') 108 | 109 | 110 | 111 | if validate and self.is_ellegible(signature["address"]): 112 | 113 | 114 | if amount is not None: 115 | quantity=amount 116 | 117 | thisVote=Vote(signer=signature["address"], 118 | vote=signature["optionName"], 119 | quantity=quantity, 120 | other=other_info, 121 | groupID=self.groupID, 122 | index=len(self.listVotes)) 123 | 124 | 125 | self.listVotes.append(thisVote) 126 | 127 | 128 | else: 129 | self.votedMoreThanOnce.append(signature["address"]) 130 | 131 | 132 | 133 | def list_to_df(self): 134 | 135 | df=[] 136 | for ll in self.listVotes: 137 | df.append(ll.to_dict() ) 138 | return pd.DataFrame(df) 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | def removeVote(self,address): 150 | ''' 151 | removes a vote with address \addresss\. This is useful when overriding stuff 152 | 153 | Parameters 154 | ---------- 155 | address : str 156 | signer address of the vote to remove 157 | ''' 158 | for vv in self.listVotes: 159 | if vv.signer==address: 160 | self.listVotes.remove(vv) 161 | 162 | 163 | 164 | 165 | def has_voted(self,address:str): 166 | has_voted=False 167 | for vv in self.listVotes: 168 | if address==vv.signer: 169 | has_voted=True 170 | return has_voted 171 | 172 | 173 | def is_ellegible(self,address:str): 174 | ''' 175 | checks whether an address `address` is ellegible (i.e., hasnt it already voted) 176 | 177 | Parameters 178 | ---------- 179 | adress:: str 180 | address 181 | 182 | Returns 183 | ------- 184 | is_it : boolean 185 | `is_it` ellegible? True or False, 186 | 187 | ''' 188 | 189 | is_it=True 190 | for vv in self.listVotes: 191 | if address==vv.signer: 192 | is_it=False 193 | 194 | return is_it 195 | 196 | def getUnits(self): 197 | ''' 198 | gets the units for each voting category; can be bytes, people or tokens 199 | 200 | Returns 201 | ------- 202 | units : str 203 | units for the groupID 204 | 205 | ''' 206 | Id=self.groupID 207 | if Id==1 or Id==2 or Id==3: 208 | units='PiB' 209 | elif Id==4: 210 | units=' FIL' 211 | elif Id==5: 212 | units=' devs' 213 | else: 214 | print('wrong group ID!') 215 | return units 216 | 217 | 218 | 219 | def count(self): 220 | ''' 221 | tallies the votes stored in the group. 222 | 223 | Returns 224 | ------- 225 | 226 | 227 | ''' 228 | list_of_votes=pd.DataFrame(self.listVotes) 229 | total_votes=list_of_votes['quantity'].sum() 230 | print('-----------') 231 | 232 | if self.groupID==1 or self.groupID==2 or self.groupID==3: 233 | divisor=2**50 234 | elif self.groupID==4: 235 | divisor=int(1e18) 236 | else: 237 | divisor=1 238 | 239 | self.tally = list_of_votes.groupby("vote")["quantity"].sum().to_dict() 240 | total_votes = sum(self.tally.values())/divisor 241 | for op, voted_for_op in self.tally.items(): 242 | voted_for_op=voted_for_op/divisor 243 | 244 | 245 | 246 | 247 | print('option: '+str(op)) 248 | print('There were a total of '+str(voted_for_op)+' '+self.getUnits()+' in favor of '+op) 249 | print('this represents {}% of the vote'.format(round(100*voted_for_op/(total_votes),6) )) 250 | print('-----------') 251 | 252 | 253 | 254 | # if __name__=='__main__': 255 | 256 | # OUTCOMES=['Approve','Reject','Abstain'] 257 | # GROUPS=['deals','capacity','clients','tokens','devs'] 258 | # N_VOTES=100 259 | # deals=groups(1) 260 | # capacity=groups(2) 261 | # clients=groups(3) 262 | # tokens=groups(4) 263 | # devs=groups(5) 264 | 265 | # voters={ 266 | # "deals":deals, 267 | # "capacity":capacity, 268 | # "clients":clients, 269 | # "tokens":tokens, 270 | # "devs":devs} 271 | 272 | 273 | 274 | # for i in range(N_VOTES): 275 | # for gr in GROUPS: 276 | # vote=np.random.choice(OUTCOMES) 277 | # address='f2_sample_'+str(i)+'_'+gr 278 | # if gr=='tokens': 279 | # quantity=np.random.randint(int(1e18)) 280 | # elif gr=='deals' or gr=='clients' or gr=='capacity': 281 | # quantity=np.random.randint(int(1e15),int(1e17)) 282 | # else: 283 | # quantity=1 284 | 285 | # voters[gr].votes.append(vote) 286 | # voters[gr].address.append(address) 287 | # voters[gr].quantity.append(quantity) 288 | 289 | # #Tests the tallying of the votes# 290 | # for gr in GROUPS: 291 | # print('###################') 292 | # print('Counting '+str(gr)) 293 | # print('###################') 294 | # print('') 295 | # print('') 296 | 297 | # voters[gr].count() 298 | 299 | print('') 300 | 301 | 302 | 303 | 304 | 305 | 306 | 307 | 308 | 309 | 310 | 311 | 312 | 313 | 314 | 315 | 316 | -------------------------------------------------------------------------------- /FIP0036/post_process/main.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | from groups import groups 5 | from tqdm import tqdm 6 | from preprocess import dataPreprocess 7 | from counting import countVote 8 | import pandas as pd 9 | import numpy as np 10 | 11 | export_path='../Public_Data' 12 | 13 | 14 | ''' 15 | runs the post-processing code. results get exported to export_path 16 | 17 | Parameters 18 | ---------- 19 | export_path : str 20 | where to export the results to 21 | 22 | Returns 23 | ------- 24 | None, but stores csvs with raw votes, processed votes and tallied votes 25 | in export_path 26 | 27 | 28 | ''' 29 | # 30 | ADD_MSIGS=False 31 | HEIGHT = 2162760 32 | ##################################################################################### 33 | # 34 | # gets relevant datasets 35 | # 36 | #################################################################################### 37 | datasets=dataPreprocess(height=HEIGHT,secretString='SecretString.txt') 38 | list_of_votes=datasets['votes'] 39 | N_votes=len(list_of_votes) 40 | signatures=[] 41 | deal=groups(1) 42 | capacity=groups(2) 43 | client=groups(3) 44 | token=groups(4) 45 | core=groups(5) 46 | groups_of_voters={ 47 | 'core':core, 48 | 'token':token, 49 | 'client':client, 50 | 'capacity':capacity, 51 | 'deal':deal 52 | } 53 | print('') 54 | print('begin counting...') 55 | print('') 56 | print('there are {} different signatures'.format(N_votes)) 57 | 58 | 59 | 60 | list_of_addresses_and_ids=datasets['addresses'] 61 | list_long_short=datasets['longShort'] 62 | 63 | ms2=pd.read_json('datasets/ms.json') 64 | ms2['yes']=np.zeros(len(ms2)) 65 | ms2['count']=np.zeros(len(ms2)) 66 | ms2['no']=np.zeros(len(ms2)) 67 | is_msig=[] 68 | 69 | 70 | 71 | 72 | 73 | ##################################################################################### 74 | # 75 | # main loop 76 | # 77 | #################################################################################### 78 | 79 | #%% 80 | for ii in tqdm(range(N_votes)): 81 | 82 | vote = list_of_votes.iloc[ii] 83 | groups_of_voters,signatures,ms2=countVote(vote,groups_of_voters, 84 | datasets,signatures,ms2) 85 | 86 | 87 | ##################################################################################### 88 | # 89 | # displays tallied votes 90 | # 91 | #################################################################################### 92 | GROUPS=['deal','capacity','client','token','core'] 93 | 94 | for gr in GROUPS: 95 | print('###################') 96 | print('Counting '+str(gr)) 97 | print('###################') 98 | print('') 99 | print('') 100 | try: 101 | groups_of_voters[gr].count() 102 | except: 103 | pass 104 | 105 | print('') 106 | 107 | ##################################################################################### 108 | # 109 | # exports 110 | # 111 | #################################################################################### 112 | 113 | def to_dict(vote): 114 | 115 | aux={'signer':vote.signer, 116 | 'vote':vote.vote, 117 | 'quantity':vote.quantity, 118 | 'other':vote.other, 119 | 'groupID':vote.groupID, 120 | 'index':vote.index 121 | } 122 | return aux 123 | 124 | 125 | all_votes=[] 126 | tallied=[] 127 | 128 | 129 | for gr in groups_of_voters: 130 | all_votes.append(groups_of_voters[gr].listVotes) 131 | 132 | try: 133 | tt=groups_of_voters[gr].tally 134 | tt['id']=groups_of_voters[gr].groupName 135 | except: 136 | tt={'Approve': 0, 137 | 'Reject':0, 138 | 'id': groups_of_voters[gr].groupName} 139 | 140 | 141 | tallied.append(tt) 142 | 143 | all_votes_df=pd.DataFrame(all_votes) 144 | tallied=pd.DataFrame(tallied) 145 | 146 | all_votes_df.to_csv(export_path+'/all_votes.csv') 147 | list_of_votes.to_csv(export_path+'/raw_votes.csv') 148 | tallied.to_csv(export_path+'/tallied_votes.csv') 149 | 150 | 151 | 152 | 153 | all_votes_l=[] 154 | for aa in all_votes: 155 | for ii in aa: 156 | all_votes_l.append(ii) 157 | 158 | dv=[] 159 | 160 | for dd in all_votes_l: 161 | dv.append(dd.to_dict()) 162 | dv=pd.DataFrame(dv) 163 | dv=dv[dv['quantity']>0] 164 | summary=dv.groupby(by=['groupID','vote'])['quantity'].sum() 165 | 166 | 167 | dg=dv[dv['groupID']!=4] 168 | dg['quantity']=dg['quantity'] 169 | dg['groupID']=dg['groupID'].apply(lambda x: str(x)) 170 | dg['quantity']=dg['quantity'].apply(lambda x: float(x)) 171 | dg.to_csv('dg.csv') 172 | 173 | dt=dv[dv['groupID']==4] 174 | dt['quantity']=dt['quantity'].apply(lambda x: float(x)) 175 | dt.to_csv('dt.csv') 176 | 177 | 178 | ############################################################################### 179 | # 180 | # in case we are counting msigs 181 | # 182 | ############################################################################### 183 | 184 | if ADD_MSIGS: 185 | 186 | dts=dt 187 | yesM=ms2[ms2['yes']>=ms2['Threshold']] 188 | noM=ms2[ms2['no']>=ms2['Threshold']] 189 | th=groups_of_voters['token'] 190 | i_=len(dt) 191 | for i in range(len(yesM)): 192 | 193 | vote={'signer':yesM['ID'].iloc[i], 194 | 'vote':'Approve', 195 | 'quantity':yesM['Balance'].iloc[i], 196 | 'other':'', 197 | 'groupID':4} 198 | 199 | aa=pd.DataFrame(vote,index=[0]) 200 | dtY=pd.concat([dt,aa]) 201 | 202 | 203 | for i in range(len(noM)): 204 | 205 | vote={'signer':noM['ID'].iloc[i], 206 | 'vote':'Reject', 207 | 'quantity':noM['Balance'].iloc[i], 208 | 'other':'', 209 | 'groupID':4} 210 | 211 | aa=pd.DataFrame(vote,index=[]) 212 | dtN=pd.concat([dt,aa]) 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | -------------------------------------------------------------------------------- /FIP0036/post_process/preprocess.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Preprocess data, this implies either downloading it or loading the following datasets: 5 | 6 | votes 7 | market deals 8 | owners 9 | core devs 10 | 11 | 12 | @author: JP 13 | """ 14 | 15 | import datautils as utils 16 | from votes import Votes 17 | import pandas as pd 18 | 19 | #%% 20 | def dataPreprocess(height:int,secretString:str or None): 21 | 22 | 23 | 24 | if secretString is None: 25 | db=None 26 | #connects to sentinel 27 | else: 28 | print('connecting to sentinel..') 29 | db = utils.connect_to_sentinel(secret_string=secretString) 30 | 31 | 32 | #gets market deals 33 | print('getting list of deals...') 34 | listDeals=utils.get_market_deals(database=db, height=height) 35 | #gets miner info 36 | print('getting miner info...') 37 | 38 | miner_info=utils.get_owned_SPs(database=db, height=height) 39 | #lists addresses 40 | print('getting miner and address info...') 41 | list_addresses=utils.get_addresses(database=db, height=height) 42 | #lists get miner power 43 | print('getting miner power info...') 44 | list_powers=utils.get_active_power_actors(database=db, height=height) 45 | 46 | 47 | print('getting list of core devs...') 48 | # list_core_devs=list(pd.read_csv('datasets/listOfCoreDevs.csv')) 49 | list_core_devs = [] 50 | 51 | 52 | 53 | #gets list of votes 54 | 55 | print('getting list of votes...') 56 | try: 57 | 58 | votes=Votes() ; votes.update() 59 | listVotes=votes.votes 60 | 61 | 62 | listVotes=pd.DataFrame(listVotes) 63 | listVotes.to_csv('datasets/listOfVotes.csv') 64 | 65 | except: 66 | print('cou;dnt connect to vote db, reading stored') 67 | listVotes=pd.read_csv('datasets/listOfVotes.csv') 68 | 69 | 70 | try: 71 | print('getting list of mappings...') 72 | 73 | list_of_mappings=pd.read_csv('datasets/longShort.csv') 74 | except: 75 | import os 76 | from tqdm import tqdm 77 | 78 | print('building list of mappings...') 79 | 80 | 81 | voters=list(listVotes['signerAddress']) 82 | list_of_mappings=[] 83 | 84 | for i in tqdm(range(len(voters))): 85 | # import pdb 86 | # pdb.set_trace() 87 | cmd='/usr/local/bin/lotus state lookup '+voters[i] 88 | id_short=os.popen(cmd).read()[:-1] 89 | list_of_mappings.append(id_short) 90 | list_of_mappings=pd.DataFrame({'long':voters, 91 | 'short':list_of_mappings}) 92 | 93 | 94 | list_of_mappings.to_csv('datasets/longShort.csv') 95 | 96 | print('getting miner balances...') 97 | list_of_balances=pd.read_csv('datasets/miner_balances.csv') 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | results={'deals':listDeals, 107 | 'miners':miner_info, 108 | 'addresses':list_addresses, 109 | 'votes':listVotes, 110 | 'core':list_core_devs, 111 | 'powers':list_powers, 112 | 'longShort':list_of_mappings, 113 | 'balances':list_of_balances 114 | } 115 | 116 | return results 117 | # gets list of miner, owner, worker 118 | 119 | 120 | 121 | if __name__=='__main__': 122 | HEIGHT=2162760 123 | results=dataPreprocess(height=HEIGHT,sectreString='SecretString.txt') 124 | deals=results['deals'] 125 | votes=results['votes'] 126 | miners=results['miners'] 127 | addresses=results['addresses'] 128 | core_devs=results['core'] 129 | 130 | -------------------------------------------------------------------------------- /FIP0036/post_process/requirements.txt: -------------------------------------------------------------------------------- 1 | tqdm 2 | pandas 3 | SQLAlchemy 4 | numpy 5 | SQLAlchemy==1.3.7 6 | -------------------------------------------------------------------------------- /FIP0036/post_process/sentinel.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Created on Tue Aug 2 11:17:06 2022 5 | 6 | This module is used to get the relevant data from the 7 | sentinel database. 8 | 9 | Needs: SQLAlchemy and pandas 10 | pip install SQLAlchemy==1.3.7 11 | It also needs a connection string to the sentinel DB, which im storing locally 12 | as a txt file. You'll need to provide this. 13 | 14 | @author: JP, juan.madrigalcianci@protocol.ai 15 | """ 16 | 17 | import sqlalchemy as sqa 18 | import pandas as pd 19 | 20 | 21 | class sentinel: 22 | """ 23 | class to connect to the sentinel database. 24 | """ 25 | 26 | def __init__(self, connString): 27 | try: 28 | self.connection = sqa.create_engine(connString).connect() 29 | print("connected to sentinel") 30 | except: 31 | print("Error while connecting") 32 | 33 | 34 | 35 | def customQuery(self, SQL): 36 | # print('performing custom query...') 37 | df = pd.read_sql(SQL, self.connection) 38 | # print('done!') 39 | return df 40 | 41 | 42 | 43 | 44 | if __name__ == "__main__": 45 | import time 46 | 47 | # reads the connection string stored as a text file in 48 | # SecretString.txt 49 | f = open("SecretString.txt", "r") 50 | NAME_DB = f.read() 51 | 52 | # initializes the class 53 | db = sentinel(NAME_DB) 54 | 55 | # gets derived gas output dataframe 56 | MIN_BLOCK = 2044500 57 | t0 = time.time() 58 | # df=db.getGasfromBlock(minBlock=MIN_BLOCK,unique=False) 59 | tf = time.time() - t0 60 | print("request time " + str(tf) + "seconds") 61 | -------------------------------------------------------------------------------- /FIP0036/post_process/votes.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | This script pings the real-live voter data and retrieves votes 5 | 6 | 7 | @author: JP, juan.madrigalcianci@protocol.ai""" 8 | 9 | 10 | import json 11 | import pandas as pd 12 | from urllib.request import urlopen 13 | from datetime import datetime 14 | pd.options.mode.chained_assignment = None # default='warn' 15 | 16 | class Votes: 17 | ''' 18 | class to ping the votes 19 | ''' 20 | 21 | def __init__(self): 22 | self.URL='https://api.filpoll.io/api/polls/16/view-votes' 23 | self.response=urlopen(self.URL) 24 | 25 | def getVotes(self): 26 | ''' 27 | Retrieves an updated list of votes as a pandas dataframe on 28 | self.votes 29 | ''' 30 | 31 | data_json = json.loads(self.response.read()) 32 | df=pd.DataFrame(data_json) 33 | N_votes=len(df) 34 | for n in range(N_votes): 35 | # # puts votes in the right format, as database returns all strings 36 | df['power'].iloc[n]=int(df['power'].iloc[n]) 37 | df['balance'].iloc[n]=int(df['balance'].iloc[n]) 38 | df['lockedBalance'].iloc[n]=int(df['lockedBalance'].iloc[n]) 39 | df['createdAt']=df['createdAt'].apply(lambda x:datetime.strptime(x, '%Y-%m-%dT%H:%M:%S.%fZ')) 40 | df['updatedAt']=df['updatedAt'].apply(lambda x:datetime.strptime(x, '%Y-%m-%dT%H:%M:%S.%fZ')) 41 | 42 | 43 | self.votes=df 44 | 45 | def update(self): 46 | ''' 47 | prints what was the latest vote together with the number of votes casted 48 | ''' 49 | self.getVotes() 50 | df=self.votes 51 | last=df['updatedAt'].max() 52 | print(' last vote was updated at ') 53 | print(last) 54 | print(' as of right now, there have been {} recorded votes'.format(len(df))) 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | if __name__=='__main__': 68 | votes=Votes() 69 | 70 | votes.update() 71 | vv=votes.votes 72 | 73 | 74 | 75 | -------------------------------------------------------------------------------- /LICENSE-APACHE.md: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /LICENSE-MIT.md: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 Protocol Labs 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |

2 | 3 | 4 | 5 | 6 |

7 | 8 | # CryptoEconLab 9 | 10 | **Welcome to the CryptoEconLab public repository. Here you will be able to learn about and contribute to our Open Problems, RFPs, and Research Projects, as well as keep tabs on what we're planning for the future.** You can learn more about our lab in our [website](https://cryptoeconlab.io/) and the [Protocol Labs Research webpage](https://cryptoeconlab.io/). You can also follow us on [Twitter](https://mobile.twitter.com/cryptoeconlab) and [LinkedIn](https://www.linkedin.com/company/cryptoeconlab/) to keep taps on relevant discussions and information. 11 | 12 | ## Table of Contents 13 | 14 | - [CryptoEconLab](#cryptoeconlab) 15 | - [Table of Contents](#table-of-contents) 16 | - [Mission & Vision 💭](#mission--vision-) 17 | - [Research 📚](#research-) 18 | - [Open Problems](#open-problems) 19 | - [Research Grants](#research-grants) 20 | - [Our work](#our-work) 21 | - [Collaborations](#collaborations) 22 | - [Work Request :hammer_and_wrench:](#work-request-hammer_and_wrench) 23 | - [How to make a Request](#how-to-make-a-request) 24 | - [SLA / What to expect from CEL](#sla--what-to-expect-from-cel) 25 | - [Contact 📞](#contact-) 26 | 27 | ## Mission & Vision 💭 28 | 29 | Cryptoeconomics is an emerging field of incentive and mechanism design in cryptographically secured peer-to-peer networks. This multidisciplinary endeavor is at the core of every Web3 project, pulling in knowledge from computer science, network science, statistics, economics, and systems engineering. 30 | 31 | CryptoEconLab is Protocol Lab's hub for research on economic incentives, coordination game theory, and novel marketplaces. We aim to develop capacity to design, validate, deploy, and govern large-scale economic systems. CryptoEconLab strives to empower projects in the ecosystem through novel incentives and advance humanity’s understanding of multiagent systems and algorithmic steering of economic networks. 32 | 33 | We are applying our learnings to grow and maintain the Filecoin ecosystem, where we are active drivers and participants. Filecoin is a layer1 blockchain that orchestrates a decentralized data storage network designed to store humanity’s most important information - and Filecoin’s unique cryptoeconomic system is central to its design. 34 | 35 | ## Research 📚 36 | 37 | CryptoEconLab’s current focus areas are: 38 | 39 | - Incentive & token design 40 | - Optimal pricing and resource allocation in distributed networks 41 | - Real-world experience & business impact 42 | - Network analytics & data-driven monitoring 43 | - Formation, diffusion, and learning in networks 44 | - Modeling & simulation 45 | - Value attribution and graph-based algorithms 46 | - Evolutionary game theory, population games, state-based potential games 47 | - Prediction markets, automated market makers, reputation systems 48 | - Governance process & principles 49 | 50 | ### Open Problems 51 | 52 | We welcome discussion of our [current Open Problems](https://github.com/protocol/CryptoEconLab/tree/main/open_problems) on our [GitHub discussion page](https://github.com/protocol/CryptoEconLab/discussions/categories/ideas-open-problems-and-proposals). Please join us in exploring the future of cryptoeconomics by contributing to the solution of current problems and posing new ones! 53 | 54 | The open problems' folder include research questions that can be used to develop an M.Sc. thesis or a Ph.D. industry-experience project for students in computer science, statistics, data science, complexity, economics or related areas. These questions can also serve as the basis for independent postdoctoral research. 55 | 56 | These project ideas can be discussed on the [CryptoEconLab Discussion Board](https://github.com/protocol/CryptoEconLab/discussions); inquiries about opportunities for [grant-supported research](https://grants.protocol.ai/) can be directed to [research-grants@protocol.ai](mailto:research@protocol.ai). 57 | 58 | ### Research Grants 59 | 60 | Protocol Labs has a grant program targeted at supporting research within the scope and mission of the company. It includes topic-specific proposals (the Requests for Proposals or RFPs) and open research awards (which are intended to support researchers pursuing topics of interest to Protocol Labs). 61 | 62 | The full details and application process can be found in the dedicated [GitHub repo](https://github.com/protocol/research-RFPs). We encourage the community to read the currently [open RPFs](https://github.com/protocol/research-grants#requests-for-proposals-rfps) and to [apply with topics](https://github.com/protocol/research-grants#open-research-grants) related to our lab's research scope. 63 | 64 | ### Our work 65 | 66 | This repository also includes code and analysis our lab has been working on. In the `tools` folder, you will find the open source tools we built. The folder `analysis` contains Jupyter notebooks and other auxiliary code that reproduces analysis and research developed by our lab. 67 | 68 | 69 | ### Collaborations 70 | 71 | We are very interested in forming collaborations with researchers and engineers working in our fields of interest, and we offer several grants and research fellowships to support these working relationships. Please check out the [PL Research website](https://research.protocol.ai/outreach/) for further details and application instructions. 72 | 73 | 74 | ## Work Request :hammer_and_wrench: 75 | 76 | ### How to make a Request 77 | 78 | We maintain a transparent work-request process using Github Issues to best serve our customers and stakeholders. To create a request, please fill out the appropriate [Work Request Issue template](https://github.com/protocol/CryptoEconLab/issues/new/choose). 79 | 80 | ### SLA / What to expect from CEL 81 | 82 | Resulting modeling, analysis, or review work can range from very high-level and expedient to very sophisticated and labor-intensive, depending on the requirements, so we use the following **service level agreement (SLA)** to set expectations for timelines and level of effort when an issue or request is generated for CryptoEconLab: 83 | 84 | CEL will review each submitted Github Issue within a maximum of 2 weeks and assess whether we have capacity and ability to engage. If so, we will respond in the comments with the following information: 85 | - The issue should contain all of the information requested in the form. 86 | - Questions or clarifications on the submitted request 87 | - Possible paths forward, which may contain different levels of support depending on staff availability and priority 88 | - We will then work with the requestor to develop an appropriate work plan and communication plan that is mutually agreeable. At a minimum, this would be commenting on outcomes in the Github issue; but could also expand to include modeling or analysis workstreams and/or discussions, documents, reports, or meetings. 89 | 90 | In general, we will work to support the following timelines after a request has completed the above steps (submission and scope agreement): 91 | - 2 weeks for initial review of FIPs or FIP discussions 92 | - 2 weeks to interpret new questions about existing CryptoEconLab models and analysis 93 | - 2+ weeks to scope and plan for new CryptoEconLab models and analysis (this does not include the actual modeling and analysis, which cannot be estimated before understanding all the previous steps for a given request) 94 | 95 | If questions or clarifications about the original request take more time to resolve, all subsequent time frames would extend by this amount. 96 | 97 | ## Contact 📞 98 | 99 | * Main website: [cryptoeconlab.io](https://cryptoeconlab.io/) 100 | * Twitter: [@cryptoeconlab](https://mobile.twitter.com/cryptoeconlab) 101 | * LinkedIn: [cryptoeconlab](https://www.linkedin.com/company/cryptoeconlab/) 102 | * Filecoin Slack Channel: [#cryptoeconomics](https://filecoinproject.slack.com/archives/C047LKMDRLM) 103 | -------------------------------------------------------------------------------- /notebooks/baseline_crossing/power_model.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import pandas as pd 3 | 4 | 5 | def forecast_qa_onboardings( 6 | rb_onboard_power: float, fil_plus_rate: float, forecast_lenght: int, 7 | qap_multiplier: float) -> np.array: 8 | qa_onboard_power = (1 + (qap_multiplier-1) * fil_plus_rate) * rb_onboard_power 9 | qa_onboarded_power_vec = np.ones(forecast_lenght) * qa_onboard_power 10 | qa_onboarded_power_vec[0] = 0.0 11 | return qa_onboarded_power_vec 12 | 13 | 14 | def forecast_rb_onboardings(rb_onboard_power: float, forecast_lenght: int) -> np.array: 15 | rb_onboarded_power_vec = np.ones(forecast_lenght) * rb_onboard_power 16 | rb_onboarded_power_vec[0] = 0.0 17 | return rb_onboarded_power_vec 18 | 19 | 20 | def forecast_power_stats( 21 | power_zero: float, 22 | known_scheduled_expire_vec: np.array, 23 | forecast_onboarded_power_vec: np.array, 24 | renewal_rate: float, 25 | onboard_length: int, 26 | renewal_length: int, 27 | forecast_lenght: int, 28 | ) -> pd.DataFrame: 29 | forecast_steps_vec = np.arange(forecast_lenght) 30 | onboarded_power_cumsum_vec = forecast_onboarded_power_vec.cumsum() 31 | forecast_scheduled_expirations_vec = forecast_power_scheduled_expirations( 32 | forecast_onboarded_power_vec, 33 | known_scheduled_expire_vec, 34 | renewal_rate, 35 | onboard_length, 36 | renewal_length, 37 | forecast_lenght, 38 | ) 39 | expired_power_vec = (1 - renewal_rate) * forecast_scheduled_expirations_vec 40 | expired_power_cumsum_vec = expired_power_vec.cumsum() 41 | power_zero_vec = np.ones(forecast_lenght) * power_zero 42 | total_power = power_zero_vec + onboarded_power_cumsum_vec - expired_power_cumsum_vec 43 | rb_df = pd.DataFrame( 44 | { 45 | "forecasting_step": forecast_steps_vec, 46 | "onboarded_power": forecast_onboarded_power_vec, 47 | "cum_onboarded_power": onboarded_power_cumsum_vec, 48 | "expired_power": expired_power_vec, 49 | "cum_expired_power": expired_power_cumsum_vec, 50 | "total_power": total_power, 51 | } 52 | ) 53 | return rb_df 54 | 55 | 56 | def forecast_power_scheduled_expirations( 57 | forecast_onboarded_power_vec: np.array, 58 | known_scheduled_expire_vec: np.array, 59 | renewal_rate: float, 60 | onboard_length: int, 61 | renewal_length: int, 62 | forecast_lenght: int, 63 | ) -> np.array: 64 | # Scheduled expirations from known sectors 65 | known_scheduled_expire_vec_pad = pad_power_scheduled_expirations_from_known_power( 66 | known_scheduled_expire_vec, forecast_lenght 67 | ) 68 | # Scheduled expirations from new onboardings 69 | scheduled_expire_onboard_vec = ( 70 | forecast_power_scheduled_expirations_from_onboarded_power( 71 | forecast_onboarded_power_vec, onboard_length, forecast_lenght 72 | ) 73 | ) 74 | # Scheduled expirations from renewals 75 | scheduled_expire_renew_vec = ( 76 | forecast_power_scheduled_expirations_from_renewed_power( 77 | forecast_onboarded_power_vec, 78 | known_scheduled_expire_vec_pad, 79 | renewal_rate, 80 | onboard_length, 81 | renewal_length, 82 | forecast_lenght, 83 | ) 84 | ) 85 | # Total scheduled expirations 86 | total_scheduled_expire_vec = ( 87 | known_scheduled_expire_vec_pad 88 | + scheduled_expire_onboard_vec 89 | + scheduled_expire_renew_vec 90 | ) 91 | return total_scheduled_expire_vec 92 | 93 | 94 | def pad_power_scheduled_expirations_from_known_power( 95 | known_scheduled_expire_vec: np.array, forecast_lenght: int 96 | ) -> np.array: 97 | pad_size = forecast_lenght - len(known_scheduled_expire_vec) 98 | known_scheduled_expire_vec_pad = np.concatenate( 99 | [known_scheduled_expire_vec, np.zeros(pad_size)] 100 | ) 101 | return known_scheduled_expire_vec_pad 102 | 103 | 104 | def forecast_power_scheduled_expirations_from_onboarded_power( 105 | onboard_power_vec: np.array, onboard_length: int, forecast_lenght: int 106 | ) -> np.array: 107 | schedule_expire_onboard_full_vec = np.concatenate( 108 | [np.zeros(onboard_length), onboard_power_vec] 109 | ) 110 | schedule_expire_onboard_vec = schedule_expire_onboard_full_vec[:forecast_lenght] 111 | return schedule_expire_onboard_vec 112 | 113 | 114 | def forecast_power_scheduled_expirations_from_renewed_power( 115 | onboard_power_vec: np.array, 116 | schedule_expire_known_vec_pad: np.array, 117 | renewal_rate: float, 118 | onboard_length: int, 119 | renewal_length: int, 120 | forecast_lenght: int, 121 | ) -> np.array: 122 | schedule_expire_renewed_known_full_vec = renewal_rate * np.concatenate( 123 | [np.zeros(renewal_length), schedule_expire_known_vec_pad] 124 | ) 125 | schedule_expire_renewed_known_vec = schedule_expire_renewed_known_full_vec[ 126 | :forecast_lenght 127 | ] 128 | schedule_expire_renewed_onboards_full_vec = renewal_rate * np.concatenate( 129 | [np.zeros(onboard_length + renewal_length), onboard_power_vec] 130 | ) 131 | schedule_expire_renewed_onboards_vec = schedule_expire_renewed_onboards_full_vec[ 132 | :forecast_lenght 133 | ] 134 | schedule_expire_renew_vec = ( 135 | schedule_expire_renewed_known_vec + schedule_expire_renewed_onboards_vec 136 | ) 137 | return schedule_expire_renew_vec 138 | -------------------------------------------------------------------------------- /notebooks/baseline_crossing/power_model_v2.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import pandas as pd 3 | from typing import Callable, Tuple 4 | 5 | # -------------------------------------------------------------------------------------- 6 | # QA Multiplier functions 7 | # -------------------------------------------------------------------------------------- 8 | def compute_qa_factor( 9 | fil_plus_rate: float, 10 | fil_plus_m: float = 10.0, 11 | duration_m: Callable = None, 12 | duration: int = None, 13 | ) -> float: 14 | fil_plus_multipler = 1.0 + (fil_plus_m - 1) * fil_plus_rate 15 | if duration_m is None: 16 | return fil_plus_multipler 17 | else: 18 | return duration_m(duration) * fil_plus_multipler 19 | 20 | 21 | # -------------------------------------------------------------------------------------- 22 | # Onboardings 23 | # -------------------------------------------------------------------------------------- 24 | def forecast_rb_daily_onboardings( 25 | rb_onboard_power: float, forecast_lenght: int 26 | ) -> np.array: 27 | rb_onboarded_power_vec = np.ones(forecast_lenght) * rb_onboard_power 28 | return rb_onboarded_power_vec 29 | 30 | 31 | def forecast_qa_daily_onboardings( 32 | rb_onboard_power: float, 33 | fil_plus_rate: float, 34 | forecast_lenght: int, 35 | fil_plus_m: float = 10.0, 36 | duration_m: Callable = None, 37 | duration: int = None, 38 | ) -> np.array: 39 | # If duration_m is not provided, qa_factor = 1.0 + 9.0 * fil_plus_rate 40 | qa_factor = compute_qa_factor(fil_plus_rate, fil_plus_m, duration_m, duration) 41 | qa_onboard_power = qa_factor * rb_onboard_power 42 | qa_onboarded_power_vec = np.ones(forecast_lenght) * qa_onboard_power 43 | return qa_onboarded_power_vec 44 | 45 | 46 | # -------------------------------------------------------------------------------------- 47 | # Renewals 48 | # -------------------------------------------------------------------------------------- 49 | def compute_day_rb_renewed_power( 50 | day_i: int, 51 | day_scheduled_expire_power_vec: np.array, 52 | renewal_rate: float, 53 | ): 54 | day_renewed_power = renewal_rate * day_scheduled_expire_power_vec[day_i] 55 | return day_renewed_power 56 | 57 | 58 | def compute_day_qa_renewed_power( 59 | day_i: int, 60 | day_rb_scheduled_expire_power_vec: np.array, 61 | renewal_rate: float, 62 | fil_plus_rate: float, 63 | fil_plus_m: float = 10.0, 64 | duration_m: Callable = None, 65 | duration: int = None, 66 | ): 67 | qa_factor = compute_qa_factor(fil_plus_rate, fil_plus_m, duration_m, duration) 68 | day_renewed_power = ( 69 | qa_factor * renewal_rate * day_rb_scheduled_expire_power_vec[day_i] 70 | ) 71 | return day_renewed_power 72 | 73 | 74 | # -------------------------------------------------------------------------------------- 75 | # Scheduled expirations 76 | # -------------------------------------------------------------------------------------- 77 | def compute_day_se_power( 78 | day_i: int, 79 | known_scheduled_expire_vec: np.array, 80 | day_onboard_vec: np.array, 81 | day_renewed_vec: np.array, 82 | duration: int, 83 | ): 84 | # Scheduled expirations coming from known active sectors 85 | if day_i > len(known_scheduled_expire_vec) - 1: 86 | known_day_se_power = 0.0 87 | else: 88 | known_day_se_power = known_scheduled_expire_vec[day_i] 89 | # Scheduled expirations coming from modeled sectors 90 | if day_i - duration >= 0: 91 | model_day_se_power = ( 92 | day_onboard_vec[day_i - duration] + day_renewed_vec[day_i - duration] 93 | ) 94 | else: 95 | model_day_se_power = 0.0 96 | # Total scheduled expirations 97 | day_se_power = known_day_se_power + model_day_se_power 98 | return day_se_power 99 | 100 | 101 | # -------------------------------------------------------------------------------------- 102 | # Power stats 103 | # -------------------------------------------------------------------------------------- 104 | def forecast_power_stats( 105 | rb_power_zero: float, 106 | qa_power_zero: float, 107 | rb_onboard_power: float, 108 | rb_known_scheduled_expire_vec: np.array, 109 | qa_known_scheduled_expire_vec: np.array, 110 | renewal_rate: float, 111 | fil_plus_rate: float, 112 | duration: int, 113 | forecast_lenght: int, 114 | fil_plus_m: float = 10.0, 115 | duration_m: Callable = None, 116 | ) -> Tuple[pd.DataFrame, pd.DataFrame]: 117 | # Forecast onboards 118 | day_rb_onboarded_power = forecast_rb_daily_onboardings( 119 | rb_onboard_power, forecast_lenght 120 | ) 121 | total_rb_onboarded_power = day_rb_onboarded_power.cumsum() 122 | day_qa_onboarded_power = forecast_qa_daily_onboardings( 123 | rb_onboard_power, 124 | fil_plus_rate, 125 | forecast_lenght, 126 | fil_plus_m, 127 | duration_m, 128 | duration, 129 | ) 130 | total_qa_onboarded_power = day_qa_onboarded_power.cumsum() 131 | # Initialize scheduled expirations and renewals 132 | day_rb_scheduled_expire_power = np.zeros(forecast_lenght) 133 | day_rb_renewed_power = np.zeros(forecast_lenght) 134 | day_qa_scheduled_expire_power = np.zeros(forecast_lenght) 135 | day_qa_renewed_power = np.zeros(forecast_lenght) 136 | # Run loop to forecast daily scheduled expirations and renewals 137 | for day_i in range(forecast_lenght): 138 | # Raw-power stats 139 | day_rb_scheduled_expire_power[day_i] = compute_day_se_power( 140 | day_i, 141 | rb_known_scheduled_expire_vec, 142 | day_rb_onboarded_power, 143 | day_rb_renewed_power, 144 | duration, 145 | ) 146 | day_rb_renewed_power[day_i] = compute_day_rb_renewed_power( 147 | day_i, day_rb_scheduled_expire_power, renewal_rate 148 | ) 149 | # Quality-adjusted stats 150 | day_qa_scheduled_expire_power[day_i] = compute_day_se_power( 151 | day_i, 152 | qa_known_scheduled_expire_vec, 153 | day_qa_onboarded_power, 154 | day_qa_renewed_power, 155 | duration, 156 | ) 157 | day_qa_renewed_power[day_i] = compute_day_qa_renewed_power( 158 | day_i, 159 | day_rb_scheduled_expire_power, 160 | renewal_rate, 161 | fil_plus_rate, 162 | fil_plus_m, 163 | duration_m, 164 | duration, 165 | ) 166 | # Compute total scheduled expirations and renewals 167 | total_rb_scheduled_expire_power = day_rb_scheduled_expire_power.cumsum() 168 | total_rb_renewed_power = day_rb_renewed_power.cumsum() 169 | total_qa_scheduled_expire_power = day_qa_scheduled_expire_power.cumsum() 170 | total_qa_renewed_power = day_qa_renewed_power.cumsum() 171 | # Total RB power 172 | rb_power_zero_vec = np.ones(forecast_lenght) * rb_power_zero 173 | rb_total_power = ( 174 | rb_power_zero_vec 175 | + total_rb_onboarded_power 176 | - total_rb_scheduled_expire_power 177 | + total_rb_renewed_power 178 | ) 179 | # Total QA power 180 | qa_power_zero_vec = np.ones(forecast_lenght) * qa_power_zero 181 | qa_total_power = ( 182 | qa_power_zero_vec 183 | + total_qa_onboarded_power 184 | - total_qa_scheduled_expire_power 185 | + total_qa_renewed_power 186 | ) 187 | # Build DataFrames 188 | rb_df = pd.DataFrame( 189 | { 190 | "forecasting_step": np.arange(forecast_lenght), 191 | "onboarded_power": day_rb_onboarded_power, 192 | "cum_onboarded_power": total_rb_onboarded_power, 193 | "expire_scheduled_power": day_rb_scheduled_expire_power, 194 | "cum_expire_scheduled_power": total_rb_scheduled_expire_power, 195 | "renewed_power": day_rb_renewed_power, 196 | "cum_renewed_power": total_rb_renewed_power, 197 | "total_power": rb_total_power, 198 | } 199 | ) 200 | rb_df["power_type"] = "raw-byte" 201 | qa_df = pd.DataFrame( 202 | { 203 | "forecasting_step": np.arange(forecast_lenght), 204 | "onboarded_power": day_qa_onboarded_power, 205 | "cum_onboarded_power": total_qa_onboarded_power, 206 | "expire_scheduled_power": day_qa_scheduled_expire_power, 207 | "cum_expire_scheduled_power": total_qa_scheduled_expire_power, 208 | "renewed_power": day_qa_renewed_power, 209 | "cum_renewed_power": total_qa_renewed_power, 210 | "total_power": qa_total_power, 211 | } 212 | ) 213 | qa_df["power_type"] = "quality-adjusted" 214 | return rb_df, qa_df 215 | -------------------------------------------------------------------------------- /notebooks/baseline_crossing/readme.md: -------------------------------------------------------------------------------- 1 | # Baseline crossing analysis 2 | 3 | ### June 2022 4 | 5 | ## Overview 6 | 7 | Filecoin uses a hybrid model for mining that incorporates two minting mechanisms - Simple minting and Baseline minting. 8 | Simple minting is the most common minting we see in other blockchains, where newly minted tokens follow a simple exponential decay model. This type of minting aims to encourage early adoption since the number of new tokens awarded to miners gets exponentially smaller as time goes on. 9 | 10 | In order to encourage long-term storage and to align incentives with the growth of the network’s utility, Filecoin introduced Baseline minting. The team defined a baseline function that determines the expected level of growth for the network. The tokens reserved for Baseline minting are only distributed if the network as a whole achieves or goes beyond that predefined growth goal. Thus, if storage providers don’t onboard storage at the expected rate, the network will not mint at full speed and the total rewards distributed will be lowered. When the network achieves the growth goals, tokens are minted at full capacity and follow a “normal” exponential decay. 11 | 12 | In April 2021, the network crossed the baseline function and started to grow faster than the initial goal. This was a major milestone for Filecoin (more info here). Since then, the network has been minting tokens at full speed since the network has managed to maintain a high growth rate. However, current market conditions (e.g. China’s crackdown, crypto market crash, etc.) are putting pressure on the Storage Providers (SPs) and their growth capability. 13 | 14 | The goal of this project is to investigate this problem and understand the potential consequences of the network experiencing a downward baseline crossing. In particular, we will estimate the likelihood of observing a baseline crossing assuming current market conditions, we will model the consequences of that event for both SPs and the network as a whole, and we will look into potential mitigation strategies. 15 | 16 | The work is divided into 4 main milestones: 17 | 18 | 1. BlockScience work review *[skipped due to time constrains]* 19 | 2. Baseline Crossing prediction *[completed]*: 20 | - [Power model spec v1](https://hackmd.io/@msilvaPL/H1uuNlItq) 21 | - [Power model spec v2](https://hackmd.io/@msilvaPL/SkapZkrdc) 22 | - [Power scenarios analysis](https://hackmd.io/@msilvaPL/SJHCpzBuc) 23 | 3. Side effects modeling *[theoretical document started, but not completed]*: 24 | - [Impact on minting and rewards](https://hackmd.io/@msilvaPL/ry6ZDtNK9) 25 | 4. Mitigation mechanisms and information sharing *[not started]* 26 | 27 | ## Inside this folder 28 | 29 | This folder contains all the code and analysis done for the Baseline crossing analysis. Currently, it only includes the work done for Milestone 2: 30 | 31 | - `power_analysis.ipynb`: EDA about historical network power statistics 32 | - `power_forecast.ipynb`: v1 analysis of four possible scenarios for network power evolution. This notebook uses the old model coded in `power_model.py` 33 | - `power_forecast_v2.ipynb`: v2 analysis of four possible scenarios for network power evolution. This notebook uses the newer v2 model coded in `power_model_v2.py` 34 | - `power_model.py`: code of the old model to forecast power statistics 35 | - `power_model_v2.py`: code of the version 2 model to forecast power statistics 36 | - `sector_updates_trends.ipynb`: EDA about sectors expiration and renewals 37 | 38 | ## Requirements 39 | 40 | ``` 41 | altair==4.2.0 42 | jupyterlab==3.4.2 43 | numpy==1.22.4 44 | pandas==1.4.2 45 | requests==2.27.1 46 | ``` 47 | 48 | ## Additional documentation 49 | 50 | - [Project docs](https://hackmd.io/@cryptoecon/Bkit3d6ej/%2F96OArWoLQvu1HtSnfwgrnQ) -------------------------------------------------------------------------------- /notebooks/shortfall/shortfall_dynamics.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/protocol/CryptoEconLab/799e02ae37c6cb08580d9fc289ef22af5013075b/notebooks/shortfall/shortfall_dynamics.pdf -------------------------------------------------------------------------------- /notebooks/shortfall/shortfall_dynamics_v0.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/protocol/CryptoEconLab/799e02ae37c6cb08580d9fc289ef22af5013075b/notebooks/shortfall/shortfall_dynamics_v0.pdf -------------------------------------------------------------------------------- /open_problems/README.md: -------------------------------------------------------------------------------- 1 | 🥕 2 | This folder contains the Open Problems currently under consideration in CryptoEconLab. 3 | -------------------------------------------------------------------------------- /open_problems/analytics/collusion_community_detection.md: -------------------------------------------------------------------------------- 1 | # Detection of collusion communities in the FIL+ program 2 | 3 | ## Introduction to the problem 4 | 5 | Filecoin is a decentralized storage network. Here, Storage Providers (SPs) contribute storage to the network and, in turn, receive a proportional claim on network consensus and block rewards. Filecoin is then built on the concept of "available storage". However, available storage is not useful is clients are not using that storage to store real data. With this in mind, the Filecoin community introduced the Filecoin Plus program. 6 | 7 | Filecoin Plus aims to maximize the amount of useful storage on Filecoin by adding a layer of social trust to the Network. Clients can apply to Notaries to receive DataCap, which can be used to incentivize Storage Providers to take storage deals. Storage Providers who take deals that are compensated with DataCap receive a 10x multiplier on their block rewards. Filecoin Plus puts power in the hands of Clients and incentivizes SPs to support real use case on the Network. 8 | 9 | The job of the Notaries is to access whether a Client applying with a given dataset is a legitimate entity and has, in fact, real data to be stored. If there were no checks in place, a Storage Provider could pretend to be a Client and submit "trash" data just to get a higher share of rewards. Even though we have these checks in place, because there is a level of human interaction between Notaries, Clients and Storage Providers, there is space for abuse. 10 | 11 | If we want to scale this process and serve ever more Clients, we need to have automated processes in place to detect collusion and abuse. This is whether this project fits in. We are interested in analysis previous applications (and the corresponding deals) and to find automated ways to detect collusion in the application process and to prevent future abuse taking place. 12 | 13 | ### Research questions 14 | 15 | 1. What collusion typologies can we find in previous FIL+ applications? 16 | 2. Can we use automated methods to detect abuse in previous FIL+ applications? 17 | 3. Can we develop a reputation scope for the application participants based on the collusion detection methods in order to inform future application requests? 18 | 19 | ## Solving this problem 20 | 21 | Clients, Notaries and Storage Providers form a graph that can be analyzed. Each participant is a node and each application forms a set of edges connecting the participants involved in the application process. 22 | 23 | With this in mind, one can apply an exploratory data analysis on this graph in order to specific cases or graph topologies that could be indicative of collusion between the parties. Possible methods to explore include: 24 | - Community detection 25 | - Clustering coefficient analysis 26 | - Network motifs 27 | 28 | After this exploratory phase, we want to apply the most promising methods to score the entire set of participants and to develop a collusion score for each participant. This can then be used in future application processes to prevent abuse. 29 | 30 | ### Estimated impact 31 | 32 | - Understand how collusion happens in the FIL+ application process 33 | - Having a score for Clients, Notaries and Storage Providers that summarizes their collusion activities in past applications can be a good tool to improve and automate the application process. 34 | 35 | ## Additional reading 36 | 37 | - [DataCap application](https://github.com/filecoin-project/filecoin-plus-client-onboarding) 38 | - [FIL+ dashboard](https://filplus.d.interplanetary.one/) 39 | - [DataCap application funnel analysis](https://observablehq.com/@starboard/filplus-ldn-application-funnel) 40 | - [Heterogeneous Network Motifs](https://arxiv.org/abs/1901.10026) 41 | - [Community detection in graphs](https://arxiv.org/abs/0906.0612) 42 | - [Clustering coefficient from Wiki](https://en.wikipedia.org/wiki/Clustering_coefficient) -------------------------------------------------------------------------------- /open_problems/analytics/filecoin_mining_concentration.md: -------------------------------------------------------------------------------- 1 | # Concentration in Filecoin storage mining and network attacks 2 | 3 | ## Introduction to the problem 4 | 5 | Filecoin is a decentralized storage network. Here, Storage Providers (SPs) contribute storage to the network and, in turn, receive a proportional claim on network consensus and block rewards. This means that the Storage Provider with the most storage capacity is also the participant with the largest consensus stake. Thus, the most direct way of measure concentration of "power" on Filecoin is to assess concentration of storage capacity. 6 | 7 | Why is this important? The more concentrated the network consensus is, the more fragile it is to takeovers by a small set of individuals, which can be disastrous. By taking control of the network, an attacker can meddle with the underlying transaction infrastructure and extract value from the network. This [scenario happened to Bitcoin Gold](https://qz.com/1287701/bitcoin-golds-51-attack-is-every-cryptocurrencys-nightmare-scenario), a fork of Bitcoin with smaller market cap and hashrate). In 2018, an attacker took control of more than half of the network consensus power (which, in this case, is the hashrate) and was capable of defrauding a few exchanges. These 51% attacks happened as well to [Verge](https://thenextweb.com/news/hackers-verge-blockchain-steal-1-7m), [Monacoin](https://www.ccn.com/japanese-cryptocurrency-monacoin-hit-by-selfish-mining-attack/) and [Electroneum](https://thenextweb.com/news/hackers-verge-blockchain-steal-1-7m). 8 | 9 | Going back to Filecoin, the protocol has a few features that make it harder for attacks willing to take over the network. Firstly, in order to gain storage capacity in the network, operators need to do some "physical" work" (i.e. buy and maintain storage hardware and do some heavy computation to process the storage when it is first being added to the network). In addition, operators need to submit collateral proportional to the storage capacity they are adding to the network. 10 | 11 | The combination of "physical" work" and staking leads to the cost of onboarding a certain amount of storage and, thus, these onboarding costs per unit of storage directly impact the security of Filecoin against takeover attacks. Another factor that impacts Filecoin's security is the current capacity in the network. The larger the capacity, the harder it is to gain a large share of the consensus power. Thus, understanding these costs is key to understand how safe the network is against such attacks. Note that the costs vary with time, with both the total storage capacity and the onboarding costs per unit of storage varying historically. 12 | 13 | Another important aspect of analyzing the security of the network is assessing how concentrated the network currently is. In order words, how much would it cost if the Storage Provider with the most consensus power wanted to take over the network? And what if the top two or top five Storage Providers colluded to take over the network? 14 | 15 | To answer these questions, one needs to estimate the onboarding costs per unit of storage, similarly to analysis for a new attacker. However, there is an additional step is not straightforward. The main way of identifying Storage Providers within the Filecoin network is through their miner ID. However, because the network is pseudoanonymous, there is no registration stopping providers to create multiple miner IDs. Thus, there needs to be some work around clustering the miner IDs into their ultimate Storage Provider. 16 | 17 | Finally, it is important to understand what is the consensus power share threshold where an attacker could start ti cause harm to the network. The team at ConsensusLab found an [attack that only requires a 20%](https://github.com/filecoin-project/FIPs/discussions/501). At the same time, they propose a fix that would increase the share to 44%. Thus, for the scope of this analysis, we are interested in investigating both cutoffs (20% and 44%). 18 | 19 | ### Research questions 20 | 21 | 1. What is the cost for an attacker without any previous storage capacity in Filecoin to achieve a 20% or 44% share of the storage power? How has the cost evolved since the launch of the network? 22 | 2. Can we cluster miner IDs into their ultimate Storage Provider? 23 | 3. How concentrated is the Filecoin network, both in terms of miner IDs and Storage Providers? 24 | 4. What is the cost for the top Storage Provider to achieve a take a 20% or 44% share of the storage power? How has these cost evolved since the launch of the network? 25 | 5. How many Storage providers need to collude to take over the network? 26 | 27 | 28 | ## Solving this problem 29 | 30 | Solving this problem will involve three main stages: 31 | 32 | 1. Estimate the current and historical cost for a 20% takeover and a 44% takeover. 33 | 1. Understand the costs associated with onboarding a single unit of storage in the Filecoin network. This [dashboard](https://dashboard.starboard.ventures/capacity-services) and this [calculator](https://observablehq.com/@starboard/sproi) can be good starting points. 34 | 2. Extract how much power needs to be added to reach the desired power share by a newcomer. The dashboard linked above is another good source of data. 35 | 2. Group miner IDs into the ultimate Storage Provider. This is the task with the most room for exploration and innovation. Some possible approaches include: 36 | 1. Analyzing the blockchain activity to find ownership links and financial links. The [Lily dataset](https://lilium.sh/data/) is a good source for this task. 37 | 2. Analyzing information from off-chain sources such as [FIL+ applications](https://github.com/filecoin-project/filecoin-plus-client-onboarding) or the auction market [BigDataExchange](https://www.bigdataexchange.io/) 38 | 3. Estimate the current and historical cost for a takeover of the top Storage Providers and how much collusion would be needed. This stage mostly combines the previous two stages. 39 | 40 | ### Estimated impact 41 | 42 | 1. Have a clearer view on the current safety of the Filecoin network 43 | 2. Guide future discussions about changes in economic parameters such as collateral. 44 | 45 | ## Additional reading 46 | 47 | - [Understanding Filecoin Circulating Supply](https://filecoin.io/blog/filecoin-circulating-supply/) 48 | - [Filecoin's cryptoeconomic constructions](https://filecoin.io/blog/posts/filecoin-s-cryptoeconomic-constructions/) 49 | - [Filecoin collateral spec](https://spec.filecoin.io/#section-systems.filecoin_mining.miner_collaterals) 50 | - [Filecoin consensus spec](https://spec.filecoin.io/algorithms/expected_consensus/) -------------------------------------------------------------------------------- /open_problems/analytics/filecoin_onboarding_forecast.md: -------------------------------------------------------------------------------- 1 | # Forecasting storage onboarding in the Filecoin Network 2 | 3 | ## Introduction to the problem 4 | 5 | Filecoin is a decentralized storage network. Here, Storage Providers (SPs) contribute storage to the network and, in turn, receive a proportional claim on network consensus and block rewards. Therefore, storage capacity is the single most important metric of growth for Filecoin. 6 | 7 | Every day, we see some new storage being onboarded by both existing and new providers. On the other end, every day, some storage is terminated or expired, and the respective capacity exits the network. The net effect of these two time-series (onboardings and exits) results in the [daily changes observed in the total storage capacity of Filecoin](https://observablehq.com/@starboard/chart-network-storage-capacity). In this project, we are interested in the new storage entering the network every day (i.e., the onboardings). 8 | 9 | Another important factor to storage onboarding is the committed duration. Every day, Storage Providers have two decisions: 10 | 1. How many sectors are they adding? *Note that a sector in Filecoin is a unit of storage*. 11 | 2. For how long each sector will be committed. 12 | 13 | Currently, Storage Providers can commit sector for periods between 6 months and 1.5 years. So, besides forecasting how many sectors are onboarded, it is also key to forecast the duration of those sectors. 14 | 15 | 16 | ### Research questions 17 | 18 | 1. Does time-series forecasting methods work for predicting storage onboarding in Filecoin? 19 | 2. Does splitting the forecast by miner ID improve the model performance? 20 | 3. Can we categorize miner IDs into similar onboarding profiles? Does this split help with forecasting? 21 | 22 | ## Solving this problem 23 | 24 | The ultimate goal of this project is to design good forecasting models for Filecoin storage onboarding. The base model would be to forecast directly on the aggregate storage being onboarded by the network each day. Here, one can test multiple models, including classical time-series models such as ARMA or ARIMA, and more recent model such as LSTMs. 25 | 26 | After this first model, we then want to split the forecasting task by miner ID or by groups of miner IDs. More concretely, the idea is to split the historical storage onboarding time-series into a set of miner-level time series and then apply forecasting to each time-series independently. To get the total onboarding forecast, one would add the individual time-series. The underlying assumption is that the past onboarding rates of individual Storage Providers are a good predictor of their short-term behavior. 27 | 28 | ### Estimated impact 29 | 30 | Being able to forecast the storage onboarding capacity of the Filecoin network will be extremely helpful to the community (Filecoin participants can use it to plan their operations) and to the CryptoEconLab team (our lab can take advantage of the models to guide our planing and design of economic policies). 31 | 32 | ## Additional reading 33 | 34 | - [Historical time-series of daily changes in Filecoin's storage capacity](https://observablehq.com/@starboard/chart-network-storage-capacity) 35 | - [Sector onboarding time-series](https://observablehq.com/@starboard/chart-prove-commit-32-64-gib-splits) 36 | - [Lily dataset](https://lilium.sh/data/) -------------------------------------------------------------------------------- /open_problems/econ_modelling/filecoin_ml_general.md: -------------------------------------------------------------------------------- 1 | # Using machine learning to model the Filecoin network 2 | 3 | The Filecoin network has a near-perfect record of storage provider state, with sector events across the network reported at 30-second intervals. Despite this wealth of data, our understanding is still restricted by the difficulty of interpreting time-changing interactions between pairs of nodes. Yet understanding mutualism and reciprocity, rent-seeking, community structure, sparsity and degree heterogeneity on the network are critical to building a deep understanding of cryptoeconomic dynamics. 4 | 5 | We seek to employ machine learning approaches to model this data-rich system. We are particularly interested in approaches involving models furnished with the appropriate inductive biases, e.g. mutually self-exciting temporal models [1] and deep graphical neural nets [2]. We believe that machine learning may be particularly useful in modeling the Filecoin [sector lifecycle](https://spec.filecoin.io/systems/filecoin_mining/sector/lifecycle/) and develop optimal responses to network fault events. 6 | 7 | ##### Related resources 8 | 1. [Modelling sparsity, heterogeneity, reciprocity, and community structure in temporal interaction data](https://proceedings.neurips.cc/paper/2018/file/160c88652d47d0be60bfbfed25111412-Paper.pdf) 9 | 10 | 2. [GNNs: A review of methods and applications](https://arxiv.org/ftp/arxiv/papers/1812/1812.08434.pdf) -------------------------------------------------------------------------------- /open_problems/econ_modelling/network_stability_in_rare_events.md: -------------------------------------------------------------------------------- 1 | # Scaling structure and network stability 2 | 3 | Self-organized criticality provides a framework to reason about relaxation events with burst-like scale invariant power laws[1]. It’s the common thread across complex systems that links the dynamics of the natural world, from rainfall (think relaxations in the sky!) and earthquakes, to social network tweet storms[2], and rare events in volatile financial markets[3]. 4 | 5 | The Filecoin decentralized network generates millions of unique events per day. We'd like to understand the emergent scaling structure of event cascades, and how this structure can inform the economic impact of rare events on future network stability. 6 | 7 | These questions might be explored in the context of modeling [Filecoin circulating supply](https://filecoin.io/blog/filecoin-circulating-supply/), understanding the dynamic adjustment of base fees, and examining relationships between storage deal state transitions and other protocol activities. For example, Filecoin uses EIP1559-style[4] gas fees, but how effective are they in practice at regulating volatility[5] in the network, and can a more effective structure be designed? 8 | 9 | ##### Related resources 10 | 11 | 1. [Rain: Relaxations in the Sky](https://arxiv.org/pdf/cond-mat/0204109.pdf) 12 | 13 | 2. [A tutorial on Hawkes Processes for events in social media](https://arxiv.org/pdf/1708.06401.pdf) 14 | 15 | 3. [State dependent Hawkes processes and their application to limit order book modelling](https://arxiv.org/abs/1809.08060) 16 | 17 | 4. [ETH EIP1559](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1559.md) 18 | 19 | 5. [EIP1559 empirical analysis](https://arxiv.org/pdf/2201.05574.pdf) -------------------------------------------------------------------------------- /open_problems/econ_modelling/optimal_control_for_blockchains.md: -------------------------------------------------------------------------------- 1 | # Implementing optimal control in a blockchain system 2 | 3 | This problem is about using online optimal control to update tokenomic parameters systematically. Filecoin mainnet has been live for over a year. The decentralized network relies on key parameters to ensure stable dynamics and agent alignment through incentivized behaviors. One challenge is online optimization of parameters that define the mechanistic structure of the protocol. On a technical level, this is akin to tuning the engine of a 747 while it’s off the ground, and leads to a classic high-stakes exploration/exploitation trade-off[1].  4 | 5 | Work on this project may benefit from considering how the lessons of the optimal control literature can be applied to blockchains. We seek to develop an understanding of what optimal search on a live mainnet looks like, and how decisions from a computational protocol might be combined with a human governance layer. This project is about developing an optimal theoretical framework, but potentially has a strong practical element – deciding when and by how much to update blockchain network parameters is becoming a bigger issue[2,3,4] as chains mature. 6 | 7 | ##### Related resources 8 | 1. [A tutorial on Thompson Sampling](https://web.stanford.edu/~bvr/pubs/TS_Tutorial.pdf) 9 | 10 | 2. [EIP-1559](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1559.md) 11 | 12 | 3. ["Tokenomics is back (thanks to CRV)"](https://doseofdefi.substack.com/p/tokenomics-is-back-thanks-to-crv?utm_source=url) 13 | 14 | 4. ["Yearn Finance Changes Up its Tokenomics and YFI Soars 85%"](https://thedefiant.io/yearn-tokenomics-change/) -------------------------------------------------------------------------------- /open_problems/incentive_design/incentivized_network_formation.md: -------------------------------------------------------------------------------- 1 | # Incentivized Network Formation 2 | 3 | ## Introduction to the Problem 4 | 5 | Traditional network formation literature has focused on coming up with models to explain how networks are formed: random graphs, the [Erdős–Rényi model](https://www.renyi.hu/~p_erdos/1959-11.pdf) for the evolution of random networks, the [Watts–Strogatz model](http://worrydream.com/refs/Watts-CollectiveDynamicsOfSmallWorldNetworks.pdf) for the generation of random graphs with small world properties, the [Price](https://www.science.org/doi/10.1126/science.149.3683.510) and [Barabási–Albert models](https://arxiv.org/abs/cond-mat/0106096) introducing a preferential attachment mechanism, and the [Expander Graphs](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.393.1430&rep=rep1&type=pdf) described by Bassalygo and Pinsker. 6 | 7 | Cryptoeconomic networks -- like many other economic and behavioral networks -- present the opportunity to define microeconomic structures that can enable specific targerted macro network structures to emerge. Being able to incentivize the formation of specific network topologies and influence network structural parameters is a powerful tool in token economic design with applications in geographic incentivization and market and payment network design. 8 | 9 | This Open Problem seeks novel methods for the discovery and incentivization of optimal network structures under a variety of conditions. 10 | 11 | ## State of the Art 12 | 13 | ### Network Formation Games 14 | 15 | Network formation games describe the principles of interaction that determine network structure, which in turn affects economic outcomes -- including whether or not coordination will occur ([0](https://authors.library.caltech.edu/79723/1/sswp1160.pdf), [1](https://dir.ilam.ac.ir/mozafar/gt/s15/Nisan_Non-printable.pdf#page=508), [2](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.454.9588&rep=rep1&type=pdf)). Understanding the processes that produce different network configurations is therefore important to understanding how and where value is generated in a network. 16 | 17 | Network interaction dynamics can lead self-interested entities to form large and efficient networks([1](https://dir.ilam.ac.ir/mozafar/gt/s15/Nisan_Non-printable.pdf#page=508)), but path dependencies during network formation can produce inefficient structures ([2](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.454.9588&rep=rep1&type=pdf)). Tradeoffs between efficiency and stability in determining network quality can be quantitatively modeled in a game-theoretical framework ([3](https://web.stanford.edu/~jacksonm/netsurv.pdf)), and we can evaluate network quality using measures asessing various aspects of distance/connectivity, congestion, and cost. 18 | 19 | Research on swarm formation and cooperation (e.g. [4](http://www.ppgia.pucpr.br/~alceu/mestrado/aula3/PSO_2.pdf), [5](https://www.sciencedirect.com/science/article/pii/S0925231219316558), [6](http://utpedia.utp.edu.my/16515/),[7](https://ieeexplore.ieee.org/abstract/document/1655446)) has provided insights into optimizing the search for condition-appropriate topologies and the dynamics of cooperative behaviours in a spatial network that may be relevant to this problem. Swarm research also provides a useful analogy to the problem of creating incentives for individual agents to produce a desired emergent global behaviour, as well as methods for evaluating system performance across a network of numerous agents. The swarm literature also models approaches to the general problem of emergent intelligent collective behaviour under the the important constraints of *scalability*, *robustness*, and *decentralization*. 20 | 21 | Additionally, results from game theory and biology have demonstrated the significance of spatial structure in determining interaction dynamics and their resulting equilibria, as well as evidence for the co-evolution of network structure and strategy ([8](https://link.springer.com/chapter/10.1007/978-3-642-01284-6_11)). These results have interesting implications for dynamic networks in which geography can be considred a salient variable. 22 | 23 | ### Known shortcomings of existing solutions 24 | 25 | Much of the literature on this topic focuses on observing, describing, and modeling the dynamics of network topology emergence. We hope to stimulate further work on *methods of influencing the development of specific network topologies*, particularly in a decentralized or distributed system, which is something that we are now empowered to do with digital tokens. 26 | 27 | While the discussion above is largely framed in terms of network topology, and network topology is an important factor in determining network behavior, CryptoEconLab believes that there may be other parameters relevant to the function of cryptoeconomic networks that are not completely captured by models focused on topological features. We invite the community to use a broad definition of *"network structure"* in conducting its explorations of this question. 28 | 29 | ## Solving this Open Problem 30 | 31 | We would like to use the principles of interaction gleaned from observations of network formation games to understand the connection between agent (node) behavior and network structure. 32 | 33 | We seek solutions explicating the tradeoffs between cost and benefits of (decentralized) network participation for various agents under different conditions and/or optimizing the social cost of a (decentralized) network under various conditions. 34 | 35 | These solutions may be modeled using simulated data or via observations of a live network deployment. 36 | 37 | We invite solutions investigating both topological and non-topological properties relevant to the function of cryptoeconomic networks. 38 | 39 | ### What does a useful contribution to this problem look like? 40 | 41 | A good response to this Open Problem would be the creation of a general framework that can be contextualized for different use-cases, allowing us to explore *a network's reaction to changing incentives*. We are looking for description of the [leverage points](http://www.donellameadows.org/wp-content/userfiles/Leverage_Points.pdf) in a network with predictive power: what are the parameters we can tune to accomplish different objectives? 42 | 43 | Your contribution might resemble [ZX's work](https://research.protocol.ai/publications/on-modeling-blockchain-enabled-economic-networks-as-stochastic-dynamical-systems/2020zhang.pdf) using state space representations to model blockchain-enabled networks or [Axel's exploration](https://hackmd.io/@R02mDHrYQ3C4PFmNaxF5bw/B1A_BSztt) of the incentives produced by different block reward structures. 44 | 45 | ### Estimated impact 46 | 47 | Improved network control via decentralized incentive mechanisms will enable new innovations in many systems modeled as a network, including but not limited to: 48 | - Filecoin storage and retrieval market performance improvements 49 | - improved logistics networks 50 | - more efficient transportation networks and two-sided markets 51 | - improvements to CDN and indexing networks 52 | 53 | ## Related Reading 54 | 55 | These papers illustrate interesting approaches to related problems from the broader research ecosystem, and we've referenced them in internal discussions about this open problem: 56 | 57 | 0. Jackson, M.O. 2003. [Allocation rules for network games](https://authors.library.caltech.edu/79723/1/sswp1160.pdf) 58 | 1. Tardos, E., and Wexler, T., in Nisan _et al_ 2007. [Algorithmic Game Theory Ch19: Network Formation Games and the Potential Function Method](https://dir.ilam.ac.ir/mozafar/gt/s15/Nisan_Non-printable.pdf#page=508) 59 | 2. Watts, A. 1999. [A Dynamic Model of Network Formation](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.454.9588&rep=rep1&type=pdf) 60 | 3. Jackson, M.O. 2003. [A Survey of Models of Network Formation: Stability and Efficiency](https://web.stanford.edu/~jacksonm/netsurv.pdf) 61 | 4. Eberhardt, R., and Kennedy, J. 1995. [A New Optimizer Using Particle Swarm Theory](http://www.ppgia.pucpr.br/~alceu/mestrado/aula3/PSO_2.pdf) 62 | 5. Xiao, H., _et al._ 2020. [Two-level structure swarm formation system with self-organized topology network](https://www.sciencedirect.com/science/article/pii/S0925231219316558) 63 | 6. Dennis, L.K.H. 2015. [Cooperative Network Formation between Swarm Robots](http://utpedia.utp.edu.my/16515/) 64 | 7. Freeman, R.A., _et al._ 2006. [Distributed Estimation and Control of Swarm Formation Statistics](https://ieeexplore.ieee.org/abstract/document/1655446) 65 | 8. Skyrms, B., and Pemantle, R. 2009. [A Dynamic Model of Social Network Formation](https://link.springer.com/chapter/10.1007/978-3-642-01284-6_11) 66 | 9. Derks,J., _et al._ 2008. [Local Dynamics in Network Formation](https://dke.maastrichtuniversity.nl/f.thuijsman/local%20dynamics.pdf) 67 | 10. Bala, V., and Goyal, S. 2003. [A Noncooperative Model of Network Formation](https://onlinelibrary.wiley.com/doi/abs/10.1111/1468-0262.00155) 68 | 11. Hoory, S., _et al._ 2006. [Expander graphs and their applications](https://www.ams.org/journals/bull/2006-43-04/S0273-0979-06-01126-8/S0273-0979-06-01126-8.pdf) 69 | 12. Nielsen, M.A. 2005. [Introduction to expander graphs](https://michaelnielsen.org/blog/archive/notes/expander_graphs.pdf) 70 | 13. Even-Dar, E. _et al._ [A Network Formation Game for Bipartite Exchange Economies](https://www.cis.upenn.edu/~mkearns/papers/econform.pdf) 71 | 72 | ### Notes on existing conversations 73 | 74 | [Github discussion](https://github.com/protocol/CryptoEconLab/discussions/5#discussioncomment-1942506) about the Bridges of Koenigsberg Problem and cryptoeconomic networks 75 | 76 | 77 | -------------------------------------------------------------------------------- /open_problems/incentive_design/truthful_games.md: -------------------------------------------------------------------------------- 1 | 2 | # Truthful Games for Incentives under Unreliable Signals 3 | 4 | ## Introduction to the problem 5 | 6 | Some of the large distributed systems that we are most interested in are best modeled as games of [*incomplete information*](moz-extension://5a397cd6-7082-4c36-a528-b9b0137c872f/pdfjs/viewer.html?file=https://web.stanford.edu/~jdlevin/Econ%20203/Bayesian.pdf), where players don't have common knowledge of various crucial features of the game being played, like possible moves and outcomes, potential payoffs, or even the existence of other players. 7 | 8 | [*Signaling games*](http://econ.ucsd.edu/~jsobel/Paris_Lectures/20070527_Signal_encyc_Sobel.pdf) are an important type of dynamic model with incomplete information. Signaling games are strategic interactions in which players can use the actions of their opponents to make inferences about hidden information: a player with private information sends a signal to an uniformed player (or players) who acts contingent on the signal. These games have been examined in the context of understanding the [job market](https://viterbi-web.usc.edu/~shaddin/cs590fa13/papers/jobmarketsignaling.pdf), [product pricing and advertising](https://cowles.yale.edu/sites/default/files/files/pub/d07/d0709.pdf), [insurance](https://www.nber.org/system/files/working_papers/w23556/w23556.pdf), [product warranties](https://faculty.fuqua.duke.edu/~qc2/BA532/1981%20JLE%20grossman.pdf), [bargaining strategies](https://www1.cmc.edu/pages/faculty/MONeill/math188/papers/rubinstein5.pdf), and even [animal behaviour](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1073.4125&rep=rep1&type=pdf), among numerous other applications. 9 | 10 | Much of cryptoeconomics today relies on “provable” signals to construct incentives and mechanisms. However, not all signals are verifiable and many are subject to [“wash trading”](https://www.investopedia.com/terms/w/washtrading.asp), wherein players engage in market activity in order to mislead other players in the market. 11 | 12 | In the large distributed networks CryptoEconLab is studying, it is difficult to verify the truthfulness of the signals players send. When protocols can’t rely on cryptographic primitives to restrain undesirable behaviours, it is natural to look into economic games that can align participants' incentives with that of the system. We therefore wish to design games with payoff structures encouraging truth-telling as the optimal action for rational players: designs in which truth-telling by all players forms a Bayesian Nash equilibrium. We are also interested in exploring deviations from this equilibrium related to the rationality assumption. 13 | 14 | There is a parallel between this framing and that of DSIC games where truth-telling is a dominant strategy. 15 | 16 | This Open Problem seeks game models describing mechanisms for incentiving truth telling in distributed multiplayer signalling game. We are particularly interested in discovering whether there exists a subclass of actions that can be incentivized without being proven. 17 | 18 | ## State of the Art 19 | 20 | ### Current approaches within CryptoEconLab and the broader research ecosystem 21 | 22 | This Open Problem stems from our current work in the field of [algorithmic mechanism design](http://www.cs.cmu.edu/~sandholm/cs15-892F07/Algorithmic%20mechanism%20design.pdf), particularly in the area of [retrieval mining](https://retrieval.market/). 23 | 24 | We see parallels between this Open Problem and explorations of **truthful mechanism design** in [facility location](https://www.ifaamas.org/Proceedings/aamas2019/pdfs/p1470.pdf), [congestion games](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.768.8075&rep=rep1&type=pdf), and [combinatorial auctions](https://www.aaai.org/Papers/AAAI/2002/AAAI02-058.pdf). 25 | 26 | These examples illustrate approaches to related problems under additional constraints, such as polynomial mechanism runtime and feasibility within a global budget. 27 | 28 | The design of the [CacheCash decentralized CDN network](https://academiccommons.columbia.edu/doi/10.7916/d8-kmv2-7n57) includes a defense against cache accounting attacks that rests partially upon making honesty more profitable than cheating. Game designs that make dishonest behaviour costly are a potential approach to this problem. 29 | 30 | ### Known shortcomings of existing solutions 31 | 32 | Some of the above games rely on or assume a central authority (e.g. the goverment planning to locate a [facility](https://www.ifaamas.org/Proceedings/aamas2019/pdfs/p1470.pdf)) and will need to be modified to better fit a decentralized context with explicit financial incentives where it is difficult or impossible to prove that any particular action was performed. 33 | 34 | [Vickrey](http://www.cs.princeton.edu/courses/archive/spring09/cos444/papers/vickrey61.pdf)-[Clarke](https://www.jstor.org/stable/30022651)-[Groves](http://www.eecs.harvard.edu/cs286r/courses/spring02/papers/groves73.pdf) (VCG) mechanisms are truthful, and maximize social welfare, but have high computational complexity and may require the implementation of [additional functions](http://robotics.stanford.edu/~amirr/vcgbased.pdf) to be made feasible. 35 | 36 | These are minor shortcomings relative to the major problem: the difficulty of eliciting private information that is technically unverifiable. 37 | 38 | 39 | ## Solving this Open Problem 40 | 41 | ### What does a useful contribution to this problem look like? 42 | 43 | Game models should take place in a decentralized system. 44 | 45 | Any proposed mechanisms should be **truthful** -- the optimal strategy for each player is to send an honest signal/report accurate information -- and **individually rational** -- all participants benefit from the game. When appropriate, solutions should include an analysis of the mechanism's performance against a selected **objective function**. 46 | 47 | We are particularly interested in discovering whether there exists a subclass of actions that can be incentivized without being proven. 48 | 49 | We believe that useful contributions can be derived from modeling this question as a general problem with private belief: establishing general rules such that the private belief in question remains unknown, but it is incentive-compatible to reveal it. We seek to identify the bounds constraining the revelation of the private belief. 50 | 51 | ### Estimated impact 52 | 53 | It is expected that solutions to this Open Problem will contribute to the design of new truthful mechanisms, and that this will enable improvements in networked [data storage](https://filecoin.io/), computation, and [retrieval markets](https://retrieval.market/) and internet routing- related protocols, among other uses. 54 | 55 | ## Related Reading 56 | 57 | 1. Hörner, J., _et al._ 2015. [Truthful Equilibria in Dynamic Bayesian Games](https://elischolar.library.yale.edu/cgi/viewcontent.cgi?article=3330&context=cowles-discussion-paper-series) 58 | 59 | 2. Chen, X., _et al._ 2019. [Truthful Mechanisms for Location Games of Dual-Role Facilities](https://www.ifaamas.org/Proceedings/aamas2019/pdfs/p1470.pdf) 60 | 61 | 3. Rogers, R., and Roth, A. 2013. [Asymptotically truthful equilibrium selection in large congestion games ](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.768.8075&rep=rep1&type=pdf) 62 | 63 | 4. Kao, M-Y., _et al._ 2005. [Towards truthful mechanisms for binary demand games](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.83.9640&rep=rep1&type=pdf) 64 | 65 | 5. Mu’alem, A., and Nisan, N. 2002. [Truthful approximation mechanisms for restricted combinatorial auctions](https://www.aaai.org/Papers/AAAI/2002/AAAI02-058.pdf) 66 | 67 | 6. Almashaqbeh, G. 2019. [CacheCash: A Cryptocurrency-based Decentralized Content Delivery Network](https://academiccommons.columbia.edu/doi/10.7916/d8-kmv2-7n57) ([related powerpoint](https://ghadaalmashaqbeh.github.io/slides/abc-cryblock-2019.pdf)) 68 | 69 | 7. Almashaqbeh, G., _et al._ 2019. [CAPnet: A Defense Against Cache Accounting Attacks on Content Distribution Networks](https://ssl.engineering.nyu.edu/papers/almashaqbeh_capnet_cns19.pdf) 70 | 71 | 8. Feldman, M., _et al._ 2021.[Distributed Signaling Games](https://arxiv.org/pdf/1404.2861.pdf) 72 | -------------------------------------------------------------------------------- /open_problems/incentive_design/value_attribution.md: -------------------------------------------------------------------------------- 1 | # Value Attribution in a Network of Contributions 2 | 3 | ## Introduction to the problem 4 | 5 | As we continue working towards an open and decentralized future, new incentives, attribution structures, and algorithms will play an increasingly important role. This is not just in the context of _project contributions_ such as labor, capital, technology or insights (in the context of SourceCred and network organizations like Protocol Labs) but also in the context of _digital creation and attribution_ (in the context of the Metaverse and leveraging the unique content addressable network that the Protocol Labs technology stack enables). 6 | 7 | The ability to create fair, transparent, and strong incentives for different groups to work towards a common goal will be a powerful tool. Subjective questions such as how impact is measured, evaluated, benchmarked -- and the mechanism and governance process through which this is administered -- can be expected to have an impact on how the effectiveness of these programs. 8 | 9 | Similarly, as teams work towards an open and permissionless Metaverse, there are also interesting value-attribution questions raised as people compose digital assets to construct new ones. 10 | 11 | ## State of the Art 12 | 13 | ### Current approaches within the CryptoEconLab and broader research ecosystem 14 | 15 | This Open Problem refers to a general class of **impact evaluation** applications, of which e.g. [SourceCred](https://sourcecred.io/) and are Protocol Lab's approach to [Networked OKRs & Milestone Bounties](https://youtu.be/pJqdkuOMe98) and [Impact Evaluators](https://youtu.be/dpLtrugjfMc) specific implementations. This class of applications is particularly relevant to the problem of [retroactive public goods funding](https://medium.com/ethereum-optimism/retroactive-public-goods-funding-33c9b7d00f0c), and these tools make use of the observation that "it's easier to agree on what _was_ useful than on what _will be_ useful ([1](https://medium.com/ethereum-optimism/retroactive-public-goods-funding-33c9b7d00f0c))." 16 | 17 | We see parallels between this problem and the question of *marketing attribution*: the process of determining the value of marketing communications in customer acquisition and behavoiur and assigning it to interactions with different nodes in the customer journey, estimating the incremental value of each contact and the spillove effects along the path to purchase (e.g. [1](https://www.sciencedirect.com/science/article/abs/pii/S0268401220314523?via%3Dihub),[2](https://pure.rug.nl/ws/files/81649379/The_path_to_purchase_and_attribution_modeling_Introduction_to_special_section.pdf), [3](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2343077)). Research into crowdsourcing has provided insights into the appropriate forms of organization for different tasks conducted in a peer production or open innovation environment (e.g. [4](https://journals.sagepub.com/doi/10.1177/0165551514550140)). 18 | 19 | Approaches to credit attribution from the field of [bibliometrics](https://garfield.library.upenn.edu/papers/pricenetworks1965.pdf), where academic citations are conceptualized as a network, may also prove useful. The [*colored directed acyclic graphs*](https://gateway.pinata.cloud/ipfs/Qmed7oNVNfMQftA8zvWxNS7h2JLwkwcGCuBEsAqovEqjPy) proposed by Stephenson and Zargham builds on bibliometric insights to suggest a structure and mechanism for describing and incentivizing information flow across networks and may be extended to represent other directed flows. 20 | 21 | ### Known shortcomings of existing solutions 22 | 23 | Impact evaluation mechanisms in dynamic, active systems require robust mechanisms for prospectively incentivizing needed work within a network. Additionally, there remain opportunities to extend the impact evaluation model to encompass new and different types of value networks. 24 | 25 | ## Solving this Open Problem 26 | 27 | We seek solutions to the general questions *"how is value created and disseminated within a network?"* and *"how can we incentivize collaborative value creation in a distributed system?"* 28 | 29 | Ideally, these solutions should be demonstrated in a context where public data is available, and which have appreciable relevance to scientific, open source, and Web3 value creation networks. 30 | 31 | ### Estimated impact 32 | 33 | We expect that contributions to this Open Problem will enable more powerful incentivization and more targeted reward of project contributors and digital value creators in a variety of areas, from scientific research to open-source software to digital art. 34 | 35 | Progress in this area will also help us to model value creation and flows within Web3 networks, including the IPFS network of CIDS and the Filecoin network of service providers. 36 | 37 | 38 | ## Related Resources 39 | 40 | These references illustrate some of the areas we think provide useful insights to this problem: 41 | 42 | 1. Buhalis, D., and Volchek, K. 2021. [Bridging marketing theory and big data analytics: The taxonomy of marketing attribution](https://www.sciencedirect.com/science/article/abs/pii/S0268401220314523?via%3Dihub) 43 | 2. Kannan, P., _et al._ 2016. [The path to purchase and attribution modeling: Introduction to special section](https://pure.rug.nl/ws/files/81649379/The_path_to_purchase_and_attribution_modeling_Introduction_to_special_section.pdf) 44 | 3. Anderl, E., _et al._ 2013. [Mapping the Customer Journey: A Graph-Based Framework for Online Attribution Modeling](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2343077) 45 | 4. Nakatsu, R., _et al._ 2014. [A taxonomy of crowdsourcing based on task complexity](https://journals.sagepub.com/doi/10.1177/0165551514550140) 46 | 5. Stephenson, M., and Zargham, M. [Colored directed acyclic graphs for multi-class attribution networks](https://gateway.pinata.cloud/ipfs/Qmed7oNVNfMQftA8zvWxNS7h2JLwkwcGCuBEsAqovEqjPy) 47 | 6. de Solla Price, D. 1965. [Statistical Studies of networks of scientific papers](https://garfield.library.upenn.edu/papers/pricenetworks1965.pdf) 48 | 49 | ### Notes on existing conversations 50 | 51 | Recently we've been modeling this problem in a NFT system, observing changes in the value of (potentially) correlated NFTs where the value of one NFT shifts. 52 | 53 | 54 | For more on retroactive public goods funding schemes, see the presentations in the [_Funding the Commons_](https://fundingthecommons.io/) workshop. 55 | 56 | -------------------------------------------------------------------------------- /tools/MpoolUtils/README.md: -------------------------------------------------------------------------------- 1 | # README 2 | 3 | ###### author: Juan P. Madrigal-Cianci, CEL. 2022. 4 | ###### juan.madrigalcianci@protocol.ai 5 | 6 | --- 7 | 8 | This is a tool to get info from the Mpool using glif. It contains the file 9 | `MpoolUtils.py` which contains two functions: 10 | 11 | * `MpoolQuerry`: queries the Mpool and saves/attaches to a json file 12 | 13 | * `listenMpool`: keeps running MpoolQuerry every "freq" seconds until 14 | it quits. i.e., it keeps gathering data 15 | 16 | See relevant docstrings in the script. The file `test.py` provides a minimally working example of how to use the `MpoolQuerry` method to grab gas data from the Mpool and plots it. 17 | 18 | 19 | This is in the hopes that it might be useful to estimate demand, as well as 20 | for visualizing messages that do not necessarilly get included on the chain. 21 | 22 | 23 | 24 | 25 | -------------------------------------------------------------------------------- /tools/MpoolUtils/mpoolUtils.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Created on Thu Oct 13 16:10:48 2022 5 | 6 | 7 | This is a tool to get info from the Mpool using glif. 8 | It contains two functions: 9 | 10 | MpoolQuerry: queries the Mpool and saves/attaches to a json file 11 | 12 | listenMpool: keeps running MpoolQuerry every "freq" seconds until 13 | it quits. i.e., it keeps gathering data 14 | 15 | This is in the hopes that it might be useful to estimate demand, as well as 16 | for visualizing messages that do not necessarilly get included on the chain. 17 | 18 | 19 | @author: Juan P. Madrigal-Cianci. CEL. 20 | juan.madrigalcianci@protocol.ai 21 | 22 | """ 23 | 24 | import requests 25 | import json 26 | import pandas as pd 27 | import time 28 | import math 29 | 30 | def listenMpool(filename:str=None,freq:int=10): 31 | ''' 32 | listens to the Mpool every ::freq:: seconds by calling 'MpoolQuerry' with 33 | that frequency 34 | 35 | Parameters 36 | ---------- 37 | filename : str or None 38 | location where you want to load/save the json file. If None, defaults 39 | to ./mpool.json 40 | 41 | freq : int, default 10 42 | how often to check. 43 | 44 | Returns 45 | ------- 46 | None 47 | 48 | ''' 49 | 50 | print("running") 51 | while True: 52 | try: 53 | MpoolQuerry(filename=filename) 54 | print('waiting...') 55 | time.sleep(freq) 56 | continue 57 | except: 58 | print('there was an error!') 59 | quit 60 | 61 | 62 | def getHeight(): 63 | ''' 64 | gets chain height based on the current time. 65 | 66 | Returns 67 | ------- 68 | height: int. 69 | height of the chain at the time this function is called 70 | 71 | ''' 72 | 73 | 74 | FILECOIN_GENESIS_UNIX_EPOCH = 1598306400 75 | unixEpoch=time.time() 76 | return math.floor((unixEpoch - FILECOIN_GENESIS_UNIX_EPOCH) / 30) 77 | 78 | 79 | 80 | 81 | def MpoolQuerry(filename:str=None)->dict: 82 | ''' 83 | queries the Mpool and saves/attaches to a json file located in filename.json 84 | 85 | Parameters 86 | ---------- 87 | filename : str or None 88 | location where you want to load/save the json file. If None, defaults 89 | to ./mpool.json 90 | 91 | Returns 92 | ------- 93 | results : dict 94 | dictionary containing a list of messages in the Mpool. 95 | Each message has keys: 96 | * gasLimit=is measured in units of gas and set by the message sender. 97 | It imposes a hard limit on the amount of gas 98 | (i.e., number of units of gas) that a message’s execution 99 | should be allowed to consume on chain 100 | * gasFeeCap= is the maximum price that the message sender is willing 101 | to pay per unit of gas (measured in attoFIL/gas unit) 102 | * gasPremium= is the price per unit of gas (measured in attoFIL/gas) 103 | that the message sender is willing to pay 104 | (on top of the BaseFee) to “tip” the miner that 105 | will include this message in a block 106 | * CID= the CID of the message 107 | * height= The height of the chain at the time the Mpool was queried 108 | 109 | 110 | see https://spec.filecoin.io/systems/filecoin_vm/gas_fee/ 111 | 112 | ''' 113 | #------------------------------------------------------------------------ 114 | # connects to the glif api 115 | #------------------------------------------------------------------------ 116 | 117 | url = "https://api.node.glif.io" 118 | 119 | payload = "{\n\"jsonrpc\": \"2.0\",\n\"method\": \"Filecoin.MpoolPending\",\n\"id\": 1,\n\"params\": [null]\n}\n" 120 | headers = { 121 | 'Authorization': 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.7J3Bh0YHYlHVMdfjxDs_PUotZ3OQ7r4jQnfYG0m8isk', 122 | 'Content-Type': 'application/json' 123 | } 124 | 125 | response = requests.request("POST", url, headers=headers, data=payload) 126 | trxArray=json.loads(response.text)['result'] 127 | 128 | 129 | #------------------------------------------------------------------------ 130 | # checks whether the file exists. If it does, it loads it, otherwise, 131 | # preallicates 132 | #------------------------------------------------------------------------ 133 | 134 | if filename is None: 135 | filename='./Mpool.json' 136 | 137 | try: 138 | f = open(filename) 139 | 140 | results = json.load(f) 141 | print('file '+filename+' found. attaching...') 142 | gasLimit=results['gasLimit'] 143 | gasFeeCap=results['gasFeeCap'] 144 | gasPremium=results['gasPremium'] 145 | CID=results['CID'] 146 | height=results['height'] 147 | 148 | N_old_entries=len(CID) 149 | 150 | except: 151 | print('file '+filename+' not found. creating...') 152 | 153 | gasLimit=[] 154 | gasFeeCap=[] 155 | gasPremium=[] 156 | CID=[] 157 | height=[] 158 | N_old_entries=0 159 | #------------------------------------------------------------------------ 160 | # iterates over all trxs and extracts relevant info 161 | #------------------------------------------------------------------------ 162 | N=len(trxArray) 163 | 164 | currentHeight=getHeight() 165 | for i in range(N): 166 | 167 | aux=trxArray[i]['Message'] 168 | gasLimit.append(int(aux['GasLimit'])) 169 | gasFeeCap.append(int(aux['GasFeeCap'])) 170 | gasPremium.append(int(aux['GasPremium'])) 171 | CID.append(aux['CID']['/']) 172 | height.append(currentHeight) 173 | 174 | results={'gasLimit':gasLimit, 175 | 'gasFeeCap':gasFeeCap, 176 | 'gasPremium':gasPremium, 177 | 'CID':CID, 178 | 'height':height} 179 | #------------------------------------------------------------------------ 180 | # removes duplicated entries by CID 181 | #------------------------------------------------------------------------ 182 | df=pd.DataFrame(results) 183 | df=df.drop_duplicates(subset=['CID']) 184 | N_new=len(df) 185 | results=df.to_dict(orient='list') 186 | #------------------------------------------------------------------------ 187 | # exports as json file 188 | #------------------------------------------------------------------------ 189 | with open(filename, 'w') as f: 190 | json.dump(results, f) 191 | 192 | 193 | 194 | 195 | print('found {} new entries.'.format(N_new-N_old_entries)) 196 | print('total number of entries is {}'.format(N_new)) 197 | 198 | return results 199 | 200 | 201 | if __name__=='__main__': 202 | FILENAME='./Mpool.json' 203 | # import os 204 | res=MpoolQuerry(FILENAME) 205 | # os.system('rm '+FILENAME) 206 | #listenMpool(FILENAME) 207 | 208 | 209 | 210 | 211 | -------------------------------------------------------------------------------- /tools/MpoolUtils/requirements.txt: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Created on Thu Oct 13 18:07:08 2022 5 | 6 | @author: juan 7 | """ 8 | 9 | -------------------------------------------------------------------------------- /tools/MpoolUtils/test.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Created on Thu Oct 13 18:45:16 2022 5 | 6 | This is a MWE to test the MpoolUtils.py script 7 | 8 | @author: juan 9 | """ 10 | 11 | from mpoolUtils import MpoolQuerry 12 | import pandas as pd 13 | import matplotlib.pyplot as plt 14 | 15 | 16 | # gets a list of messages in the Mpool at the current time 17 | messages=MpoolQuerry(filename='test.json') 18 | # converts messages to a pandas dataframe and shows their head 19 | df=pd.DataFrame(messages) 20 | print(df.head()) 21 | 22 | 23 | plt.loglog(df['gasFeeCap'],df['gasLimit'],'.') 24 | plt.title(' log-scale plot of gasLimit Vs. gasFeeCap') 25 | plt.xlabel('log(gasFeeCap)') 26 | plt.ylabel('log(gasLimit)') 27 | plt.show() 28 | 29 | import os 30 | os.system('rm test.json') -------------------------------------------------------------------------------- /tools/README.md: -------------------------------------------------------------------------------- 1 | # Toolss Folder 2 | 3 | 4 | This is a folder dedicated to tools that can be used accross different projects. 5 | -------------------------------------------------------------------------------- /tools/minerUtils/minerExplore.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Created on Fri Oct 14 15:42:03 2022 5 | 6 | 7 | This is a tool to get miner info using glif. 8 | 9 | 10 | 11 | 12 | @author: Juan P. Madrigal-Cianci. CEL. 13 | juan.madrigalcianci@protocol.ai 14 | 15 | """ 16 | 17 | import requests 18 | import json 19 | import time 20 | import sqlalchemy as sqa 21 | import pandas as pd 22 | from tqdm import tqdm 23 | 24 | def getAllMiners(filename:str=None): 25 | ''' 26 | returns a list of all miner addresses§ 27 | 28 | Parameters 29 | ---------- 30 | savefile : str, optional 31 | DESCRIPTION. The default is None. 32 | 33 | Returns 34 | ------- 35 | minerList: a list contaiinng all miners in the network 36 | 37 | ''' 38 | url = "https://api.node.glif.io" 39 | 40 | payload = "{\n\"jsonrpc\": \"2.0\",\n\"method\": \"Filecoin.StateListMiners\",\n\"id\": 1,\n\"params\": [null]\n}\n" 41 | headers = { 42 | 'Authorization': 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.7J3Bh0YHYlHVMdfjxDs_PUotZ3OQ7r4jQnfYG0m8isk', 43 | 'Content-Type': 'application/json' 44 | } 45 | 46 | response = requests.request("POST", url, headers=headers, data=payload) 47 | minerList=json.loads(response.text)['result'] 48 | 49 | if filename is None: 50 | filename='./listOfMiners.json' 51 | 52 | with open(filename, 'w') as f: 53 | json.dump(minerList, f) 54 | return minerList 55 | 56 | 57 | 58 | def getValidMiners(secretString:str): 59 | ''' 60 | 61 | Gets a list of valid miners, defined as those who have RBP>0 62 | 63 | Parameters 64 | ---------- 65 | secretString : str 66 | connection string to sentinel 67 | 68 | Returns 69 | ------- 70 | a dictionary with keys 71 | 72 | 'miner_id': id of a given miner 73 | 'raw_byte_power': rbp of that miner 74 | 'quality_adj_power': qap of that miner 75 | 76 | 77 | 78 | ''' 79 | 80 | if secretString is None: 81 | f=open('SecretString.txt') 82 | secretString=f.read() 83 | 84 | engine=sqa.create_engine(secretString) 85 | conn=engine.connect() 86 | query=''' SELECT * FROM "visor"."power_actor_claims"''' 87 | df=pd.read_sql(sql=query, con=conn) 88 | df['raw_byte_power']=df['raw_byte_power']/2**40 # in PiBs 89 | df['quality_adj_power']=df['quality_adj_power']/2**40 # in PiBs 90 | 91 | dfg=df.groupby(by='miner_id')['raw_byte_power','quality_adj_power'].sum() 92 | dfg=dfg[dfg['raw_byte_power']>0] 93 | 94 | 95 | results={ 96 | 'miner_id':list(dfg.index), 97 | 'raw_byte_power':list(dfg['raw_byte_power']), 98 | 'quality_adj_power':list(dfg['quality_adj_power']) 99 | 100 | } 101 | return results 102 | 103 | 104 | def getMinerBalance(minerID:str): 105 | ''' 106 | Gets miner Available balance based on their miner ID. 107 | 108 | here: 109 | 110 | Available Balance = Address Balance - Initial Pledged - Rewards Locked 111 | 112 | Parameters 113 | ---------- 114 | minerID : str 115 | id of the miner we want to investigate. This is usually an f0 number 116 | 117 | Returns 118 | ------- 119 | balance: int 120 | available balance of a given miner 121 | 122 | ''' 123 | 124 | url = "https://api.node.glif.io" 125 | 126 | minerID='''"{}"'''.format(minerID) 127 | 128 | payload = "{\n\"jsonrpc\": \"2.0\",\n\"method\": \"Filecoin.StateMinerAvailableBalance\",\n\"id\": 1,\n\"params\": ["+minerID+",null]\n}\n" 129 | headers = { 130 | 'Authorization': 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.7J3Bh0YHYlHVMdfjxDs_PUotZ3OQ7r4jQnfYG0m8isk', 131 | 'Content-Type': 'application/json' 132 | } 133 | 134 | response = requests.request("POST", url, headers=headers, data=payload) 135 | try: 136 | balance=json.loads(response.text)['result'] 137 | except: 138 | balance=0 139 | 140 | 141 | return balance 142 | 143 | 144 | def getValidMinerInfo(filename:str=None, 145 | secretString:str=None): 146 | ''' 147 | 148 | 149 | returns balances for vsalid miners 150 | 151 | Parameters 152 | ---------- 153 | filename : str, optional 154 | where to store a json file. The default is None. 155 | 156 | secretString:str optional 157 | connection string to sentinel. The default is None. 158 | 159 | 160 | 161 | 162 | Returns 163 | ------- 164 | dict with miner id, balances, rbp and qap. 165 | 166 | ''' 167 | 168 | if filename is None: 169 | filename='./minerInfo.json' 170 | 171 | miners= getValidMiners(secretString) 172 | balances=[] 173 | for mm in tqdm(miners['miner_id']): 174 | balances.append(getMinerBalance(mm)) 175 | 176 | miners['balances']=balances 177 | 178 | 179 | with open(filename, 'w') as f: 180 | json.dump(miners, f) 181 | 182 | 183 | return miners 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | def getAllMinersBalance(filename:str=None,listOfMiners:list=None): 192 | ''' 193 | Gets all balances from all miners. This will probably 194 | take a huge amount of time to run 195 | 196 | Parameters 197 | ---------- 198 | filename : str, optional 199 | DESCRIPTION. The default is None. 200 | flistOfMiners: list, optional, 201 | DESCRIPTION: a list of minersIds 202 | 203 | Returns 204 | ------- 205 | a dict with keys 206 | minerId: a list of miners 207 | balance: a list with balance in attoFIL of each miner 208 | 209 | ''' 210 | 211 | 212 | #gets all miners 213 | 214 | if listOfMiners is None: 215 | 216 | print('getting list of all miners...') 217 | listOfMiners=getAllMiners() 218 | else: 219 | print('reading existing list of miners...') 220 | 221 | balances=[] 222 | 223 | print('getting available balances for each miner. Will take a while...') 224 | print('') 225 | for mm in tqdm(listOfMiners): 226 | balances.append(getMinerBalance(mm)) 227 | results={ 228 | 'minerID':listOfMiners, 229 | 'balance':balances 230 | } 231 | print('done!') 232 | if filename is None: 233 | filename='./listOfMinersWithBalance.json' 234 | 235 | with open(filename, 'w') as f: 236 | json.dump(results, f) 237 | return results 238 | 239 | if __name__=='__main__': 240 | miners=getValidMinerInfo('./validMiners.json') 241 | 242 | 243 | 244 | 245 | 246 | --------------------------------------------------------------------------------