├── README.md ├── baselines ├── README.md ├── configs_for_reproducibility │ ├── [CODET5,GNN-GSE]_v1_c.txt │ ├── [CODET5,GNN-GSE]_v1_r.txt │ ├── [CODET5,GNN-GSE]_v2_c.txt │ ├── [CODET5,GNN-GSE]_v2_r.txt │ ├── [G-CODEBERT,GNN-GSE]_v1_c.txt │ ├── [G-CODEBERT,GNN-GSE]_v1_r.txt │ ├── [G-CODEBERT,GNN-GSE]_v2_c.txt │ ├── [G-CODEBERT,GNN-GSE]_v2_r.txt │ ├── code2vec_v1_c.txt │ ├── code2vec_v1_r.txt │ ├── code2vec_v2_c.txt │ ├── code2vec_v2_r.txt │ ├── codet5_frozen_v1_c.txt │ ├── codet5_frozen_v1_r.txt │ ├── codet5_frozen_v2_c.txt │ ├── codet5_frozen_v2_r.txt │ ├── codet5_rand_v1_c.txt │ ├── codet5_rand_v1_r.txt │ ├── codet5_rand_v2_c.txt │ ├── codet5_rand_v2_r.txt │ ├── codet5_v1_c.txt │ ├── codet5_v1_r.txt │ ├── codet5_v2_c.txt │ ├── codet5_v2_r.txt │ ├── gnn_dse_2L_v1_c.txt │ ├── gnn_dse_2L_v1_r.txt │ ├── gnn_dse_2L_v2_c.txt │ ├── gnn_dse_2L_v2_r.txt │ ├── gnn_dse_v1_c.txt │ ├── gnn_dse_v1_r.txt │ ├── gnn_dse_v2_c.txt │ ├── gnn_dse_v2_r.txt │ ├── graphcodebert_L_v1_c.txt │ ├── graphcodebert_L_v1_r.txt │ ├── graphcodebert_L_v2_c.txt │ ├── graphcodebert_L_v2_r.txt │ ├── graphcodebert_v1_c.txt │ ├── graphcodebert_v1_r.txt │ ├── graphcodebert_v2_c.txt │ └── graphcodebert_v2_r.txt └── src │ ├── adapt.py │ ├── aug21autodse │ ├── __pycache__ │ │ └── result.cpython-39.pyc │ └── result.py │ ├── autodse │ ├── __pycache__ │ │ ├── result.cpython-37.pyc │ │ └── result.cpython-39.pyc │ └── result.py │ ├── check_model.py │ ├── config.py │ ├── config_ds.py │ ├── convert_db_to_pkl.py │ ├── data.py │ ├── data_analysis.py │ ├── data_code2vec.py │ ├── data_multi_modality.py │ ├── data_programl.py │ ├── data_src_code.py │ ├── database.py │ ├── design_sampling.py │ ├── early_stopping.py │ ├── graph_transfotmer.py │ ├── link_graph.py │ ├── main.py │ ├── model.py │ ├── model_code2vec.py │ ├── model_factory.py │ ├── model_multi_modality.py │ ├── model_node_token_interact.py │ ├── model_visualization.py │ ├── my_maxvol.py │ ├── nn.py │ ├── nn_att.py │ ├── our_codet5.py │ ├── our_transformer_conv.py │ ├── parameter.py │ ├── pretrained_GNN_handler.py │ ├── protonet_miniimagenet.py │ ├── result.py │ ├── rose_gen_dot.py │ ├── saver.py │ ├── test.py │ ├── test.sh │ ├── text_encoder.py │ ├── train.py │ ├── train_pairwise.py │ ├── utils.py │ └── utils_nn.py └── data └── HLSyn_data.tar.gz /README.md: -------------------------------------------------------------------------------- 1 | # Towards a Comprehensive Benchmark for High-Level Synthesis Targeted to FPGAs 2 | 3 | ## Description 4 | 5 | High-level synthesis (HLS) aims to raise the abstraction layer in hardware design, enabling the design of domain-specific accelerators (DSAs) like field-programmable gate arrays (FPGAs) using C/C++ instead of hardware description languages (HDLs). Compiler directives in the form of pragmas play a crucial role in modifying the microarchitecture within the HLS framework. However, the space of possible microarchitectures grows exponentially with the number of pragmas. Moreover, the evaluation of each candidate design using the HLS tool consumes significant time, ranging from minutes to hours, leading to a time-consuming optimization process. To accelerate this process, machine learning models have been used to predict design quality in milliseconds. However, existing open-source datasets for training such models are limited in terms of design complexity and available optimizations. In this paper, we present HLSyn, the first benchmark that addresses these limitations. It contains more complex programs with a wider range of optimization pragmas, making it a comprehensive dataset for training and evaluating design quality prediction models. The HLSyn benchmark consists of 42 unique programs/kernels, resulting in over 42,000 labeled designs. We conduct an extensive comparison of state-of-the-art baselines to assess their effectiveness in predicting design quality. As an ongoing project, we anticipate expanding the HLSyn benchmark in terms of both quantity and variety of programs to further support the development of this field. 6 | 7 | ## Citation 8 | 9 | If you use the dataset in your work, please cite our paper. 10 | ``` 11 | @article{chang2023dr, 12 | title={Towards a Comprehensive Benchmark for High-Level Synthesis Targeted to FPGAs}, 13 | author={Yunsheng Bai, Atefeh Sohrabizadeh, Zongyue Qin, Ziniu Hu, Yizhou Sun, and Jason Cong}, 14 | journal={NeurIPS}, 15 | year={2023} 16 | } 17 | ``` 18 | 19 | ## Data 20 | 21 | `data/HLSyn_data.tar.gz` is the data stored in .json, .gexf, and .c format. 22 | 23 | In addition, the format used by our baseline code can be downloaded from [here](https://zenodo.org/records/8034115) 24 | 25 | The two format has same contents. And the former one is more readable. 26 | 27 | ## Data Content and Format 28 | 29 | The source code and pragma-augmented programl graph representation of each kernel is stored in the `.c` and `.gexf` file under `sources` and `graphs` folders, respectively. 30 | 31 | Notice that in the source code and graph, the value of praga parameters is are variables. To get specific values for them and the corresponding prediction labels under those values, you need to load the json file under the `design/v18` (27,867 design points generated by AMD/Xilinx SDx 2018.3) and `design/v20` (13,729 data points generated by AMD/Xilinx Vitis 2020.2) folders, which contain design data points under different version of HLS tools. 32 | 33 | In the `design/v18` and `design/v20` folders, each json file is a dictionary where the key is the specific parameters of a design and the value is the corresponding prediction labels. 34 | 35 | The value of each design data point contains the following fields: 36 | 37 | - `points`: the dictionary stores the specific values for each pragma parameter. 38 | - `valid`: if the design is a valid design. Notice that if for a design point `valid=False` but `perf != 0`, it can still be used for regression task. 39 | - `perf`: value of the performance target 40 | - `res_util`: a dictionary of values for the resource utilization targets 41 | 42 | An example of the design data point in `v18/aes.json` is shown below 43 | 44 | ``` 45 | "__PIPE__L1-NA.__PIPE__L2-NA.__TILE__L2-1": { 46 | "perf": 6832.0, 47 | "point": { 48 | "__PIPE__L1": "", 49 | "__PIPE__L2": "", 50 | "__TILE__L2": 1 51 | }, 52 | "res_util": { 53 | "util-BRAM": 0.0, 54 | "util-DSP": 0.0, 55 | "util-LUT": 0.08, 56 | "util-FF": 0, 57 | "total-BRAM": 0.0, 58 | "total-DSP": 0.0, 59 | "total-LUT": 105187.0, 60 | "total-FF": 0 61 | }, 62 | "valid": true 63 | } 64 | ``` 65 | 66 | 67 | ## Leaderboard 68 | 69 | Coming soon 70 | 71 | 72 | ## Running Baselines 73 | 74 | Check instructions [here](https://github.com/ZongyueQin/HLSyn/tree/main/baselines) 75 | 76 | ## License 77 | 78 | [CC BY-SA 4.0 license](https://creativecommons.org/licenses/by-sa/4.0/legalcode) 79 | 80 | ## Acknowledgement 81 | 82 | This work was partially supported by NSF 2211557, NSF 1937599, NSF 2119643, NSF 2303037, NASA, SRC JUMP 2.0 Center, Okawa Foundation, Amazon Research, Cisco, Picsart, Snapchat, and CDSC industrial partners ( https://cdsc.ucla.edu/partners/). We would also like to thank AMD/Xilinx for HACC equipment donation and Marci Baun for editing the paper. 83 | -------------------------------------------------------------------------------- /baselines/README.md: -------------------------------------------------------------------------------- 1 | ## Run 2 | 3 | To run the training and inference processes, modify `config.py` and run `python main.py`. A log folder will be created under `src/logs` with hyperparameter details which are specified by `config.py`. 4 | 5 | 6 | ## Tips 7 | 8 | 1. You may need to set `force_regen` to True in `config.py` in order to pre-process the raw data. Once the preprocessed datasets are generated and stored as pickle files under `save`, you can set `force_regen` to False for experiments. If some reason the datasets need to be re-regenerated, simply set `force_regen` to True, and the code will delete any existing pickle files and re-generate/pre-process the dataset. 9 | 10 | It may seem a bit confusing at this point, but we believe this parameter can be useful in practice, e.g. when you want to introduce some new features to the dataset, and thus instead of manually deleting the earlier pre-processed data files, you can just set this parameter to True. 11 | 12 | In the released hyperparameter files under `configs_for_reproducibility`, some files set `force_regen` to True but some set to False -- It should always be set to True if you run the training/inference process for the first time on that machine. 13 | 14 | 2. When training, the `subtask` parameter in `config.py` should be set to "train". When testing (with adaptation to the six held-out kernels), set it to "inference". In addition, specify the trained model path `load_model`, e.g. "train_2023-05-30T22-10-35.899225_class_/val_model_state_dict.pth". The code will automatically load that checkpoint. 15 | 16 | If you encounter errors during inference, it is very likely due to mismatched hyperparameters and suggest you need to modify `config.py`'s hyperparameters to match the ones you used to train that model. For example, the model "train_2023-05-30T22-10-35.899225_class_/val_model_state_dict.pth" corresponds to a transformer-based model that receives the source code text as input. If `config.py` sets `multi_modality` to True, meaning you want to have a model receiving both the source code text and the assembly code graph as inputs, then there might be an error when the code tries to load that checkpoint. In such cases, manually set `multi_modality` to False along with other parameters in `config.py` to the ones you used to train that model will fix the issues. 17 | 18 | In short, when loading a trained model's checkpoint, please double-check the `config.py` file to ensure the hyperparameters you specify match the ones used to train that model. 19 | 20 | 4. If `no localdse error` when pickle loading objects, try marking `dse_database` as source directory and let PyCharm handle the addition of it to PYTHONPATH. 21 | 22 | ## Data Files 23 | 24 | Download from https://zenodo.org/record/8034115 (The data is in different format of the other link, but has same contents) and extract under the project root directory with the folder name being "dse_database". Please change the "dse_database_name" parameter in `config.py` to reflect the actual dse folder name, e.g. `dse_database_06122023`. The reason is that, we may release newer version of our dataset in future, and in that case, a different folder such as `dse_database_06122023_new` can be added and the only change to code is the "dse_database_name" parameter in `config.py`. In other words, we maintain different `dse_database`s under the project directories each corresponding to one version of our dataset. 25 | 26 | 27 | 28 | -------------------------------------------------------------------------------- /baselines/configs_for_reproducibility/code2vec_v1_c.txt: -------------------------------------------------------------------------------- 1 | model : code2vec 2 | v2_db : False 3 | use_redis : False 4 | task : class 5 | subtask : train 6 | sequence_modeling : False 7 | code2vec_data_version : neurips2023_benchmark 8 | multi_modality : False 9 | load_db : None 10 | load_model : None 11 | dataset : programl 12 | benchmarks : ['machsuite', 13 | : 'poly'] 14 | tag : wmp-d 15 | round_num : 13 16 | graph_type : 17 | graph_transformer_option : None 18 | gae_T : False 19 | gae_P : False 20 | SSL : False 21 | load_pretrained_GNN : False 22 | class_model_path : None 23 | all_kernels : True 24 | sample_finetune : False 25 | FT_extra : False 26 | new_speedup : True 27 | feature_extract : False 28 | ignore_kernels : [] 29 | test_kernels : ['stencil-3d', 30 | : 'doitgen-red', 31 | : 'gemm-ncubed', 32 | : 'fdtd-2d', 33 | : 'jacobi-2d', 34 | : 'trmm-opt'] 35 | val_ratio_in_test_kernels : 0 36 | val_ratio_in_train_kernels : 0.15 37 | test_ratio_in_train_kernels : 0.15 38 | tvt_split_by : kernels_inductive 39 | itype_mask_perc : 0 40 | gtype : programl 41 | only_pragma_nodes : False 42 | encode_full_text : None 43 | fulltext_dim : None 44 | MAML : False 45 | force_regen : False 46 | load_encoders_label : None 47 | encoder_path : None 48 | model_tag : test 49 | activation : elu 50 | outlier_removal : None 51 | no_pragma : False 52 | num_layers : 2 53 | D : 512 54 | target : ['perf', 55 | : 'util-LUT', 56 | : 'util-FF', 57 | : 'util-DSP', 58 | : 'util-BRAM'] 59 | gnn_type : transformer 60 | min_allowed_latency : 100.0 61 | encode_edge : True 62 | encode_edge_position : False 63 | ptrans : False 64 | jkn_mode : max 65 | jkn_enable : True 66 | node_attention : True 67 | node_attention_MLP : False 68 | separate_P_T : False 69 | pragma_encoder : True 70 | pragma_uniform_encoder : True 71 | epsilon : 0.001 72 | normalizer : 10000000.0 73 | util_normalizer : 1 74 | max_number : 10000000000.0 75 | norm_method : speedup-log2 76 | target_preproc : None 77 | target_convert_back : True 78 | invalid : False 79 | multi_target : True 80 | activation_type : elu 81 | margin_loss : False 82 | save_model : True 83 | save_every_epoch : 10000 84 | encode_log : False 85 | target_factor : 1 86 | target_transform : None 87 | loss_scale : {'perf': 1.0, 88 | : 'util-DSP': 1.0, 89 | : 'util-BRAM': 1.0, 90 | : 'util-LUT': 1.0, 91 | : 'util-FF': 1.0} 92 | pairwise_class : False 93 | batch_size : 64 94 | data_loader_num_workers : 0 95 | device : cuda:2 96 | epoch_num : 200 97 | debug_iter : -1 98 | ignore_testing : True 99 | ignore_validation : False 100 | shuffle : True 101 | opt_type : AdamW 102 | lr : 0.0001 103 | max_grad_norm : None 104 | weight_decay : 0.01 105 | plot_pred_points : True 106 | fix_randomness : True 107 | random_seed : 123 108 | user : xxx 109 | hostname : xxxx 110 | exp_name : 111 | ts : 2023-06-04T15-21-31.611893 112 | 113 | Code2VecNet( 114 | (loss_function): CrossEntropyLoss() 115 | (decoder): ModuleDict( 116 | (perf): MLP( 117 | (activation): ELU(alpha=1.0) 118 | (layers): ModuleList( 119 | (0): Linear(in_features=384, out_features=192, bias=True) 120 | (1): Linear(in_features=192, out_features=96, bias=True) 121 | (2): Linear(in_features=96, out_features=48, bias=True) 122 | (3): Linear(in_features=48, out_features=24, bias=True) 123 | (4): Linear(in_features=24, out_features=12, bias=True) 124 | (5): Linear(in_features=12, out_features=2, bias=True) 125 | ) 126 | ) 127 | ) 128 | ) 129 | -------------------------------------------------------------------------------- /baselines/configs_for_reproducibility/code2vec_v1_r.txt: -------------------------------------------------------------------------------- 1 | model : code2vec 2 | v2_db : False 3 | use_redis : False 4 | task : regression 5 | subtask : train 6 | sequence_modeling : False 7 | code2vec_data_version : neurips2023_benchmark 8 | multi_modality : False 9 | load_db : None 10 | load_model : None 11 | dataset : programl 12 | benchmarks : ['machsuite', 13 | : 'poly'] 14 | tag : wmp-d 15 | round_num : 13 16 | graph_type : 17 | graph_transformer_option : None 18 | gae_T : False 19 | gae_P : False 20 | SSL : False 21 | load_pretrained_GNN : False 22 | class_model_path : None 23 | all_kernels : True 24 | sample_finetune : False 25 | FT_extra : False 26 | new_speedup : True 27 | feature_extract : False 28 | ignore_kernels : [] 29 | test_kernels : ['stencil-3d', 30 | : 'doitgen-red', 31 | : 'gemm-ncubed', 32 | : 'fdtd-2d', 33 | : 'jacobi-2d', 34 | : 'trmm-opt'] 35 | val_ratio_in_test_kernels : 0 36 | val_ratio_in_train_kernels : 0.15 37 | test_ratio_in_train_kernels : 0.15 38 | tvt_split_by : kernels_inductive 39 | itype_mask_perc : 0 40 | gtype : programl 41 | only_pragma_nodes : False 42 | encode_full_text : None 43 | fulltext_dim : None 44 | MAML : False 45 | force_regen : True 46 | load_encoders_label : None 47 | encoder_path : None 48 | model_tag : test 49 | activation : elu 50 | outlier_removal : None 51 | no_pragma : False 52 | num_layers : 2 53 | D : 512 54 | target : ['perf', 55 | : 'util-LUT', 56 | : 'util-FF', 57 | : 'util-DSP', 58 | : 'util-BRAM'] 59 | gnn_type : transformer 60 | min_allowed_latency : 100.0 61 | encode_edge : True 62 | encode_edge_position : False 63 | ptrans : False 64 | jkn_mode : max 65 | jkn_enable : True 66 | node_attention : True 67 | node_attention_MLP : False 68 | separate_P_T : False 69 | pragma_encoder : True 70 | pragma_uniform_encoder : True 71 | epsilon : 0.001 72 | normalizer : 10000000.0 73 | util_normalizer : 1 74 | max_number : 10000000000.0 75 | norm_method : speedup-log2 76 | target_preproc : None 77 | target_convert_back : True 78 | invalid : False 79 | multi_target : True 80 | activation_type : elu 81 | margin_loss : False 82 | save_model : True 83 | save_every_epoch : 10000 84 | encode_log : False 85 | target_factor : 1 86 | target_transform : None 87 | loss_scale : {'perf': 1.0, 88 | : 'util-DSP': 1.0, 89 | : 'util-BRAM': 1.0, 90 | : 'util-LUT': 1.0, 91 | : 'util-FF': 1.0} 92 | pairwise_class : False 93 | batch_size : 64 94 | data_loader_num_workers : 0 95 | device : cuda:5 96 | epoch_num : 1000 97 | debug_iter : -1 98 | ignore_testing : True 99 | ignore_validation : False 100 | shuffle : True 101 | opt_type : AdamW 102 | lr : 0.0001 103 | max_grad_norm : None 104 | weight_decay : 0.01 105 | plot_pred_points : True 106 | fix_randomness : True 107 | random_seed : 123 108 | user : xxx 109 | hostname : xxxx 110 | exp_name : 111 | ts : 2023-06-04T14-45-40.194009 112 | 113 | Code2VecNet( 114 | (loss_function): MSELoss() 115 | (decoder): ModuleDict( 116 | (perf): MLP( 117 | (activation): ELU(alpha=1.0) 118 | (layers): ModuleList( 119 | (0): Linear(in_features=384, out_features=192, bias=True) 120 | (1): Linear(in_features=192, out_features=96, bias=True) 121 | (2): Linear(in_features=96, out_features=48, bias=True) 122 | (3): Linear(in_features=48, out_features=24, bias=True) 123 | (4): Linear(in_features=24, out_features=12, bias=True) 124 | (5): Linear(in_features=12, out_features=1, bias=True) 125 | ) 126 | ) 127 | (util-LUT): MLP( 128 | (activation): ELU(alpha=1.0) 129 | (layers): ModuleList( 130 | (0): Linear(in_features=384, out_features=192, bias=True) 131 | (1): Linear(in_features=192, out_features=96, bias=True) 132 | (2): Linear(in_features=96, out_features=48, bias=True) 133 | (3): Linear(in_features=48, out_features=24, bias=True) 134 | (4): Linear(in_features=24, out_features=12, bias=True) 135 | (5): Linear(in_features=12, out_features=1, bias=True) 136 | ) 137 | ) 138 | (util-FF): MLP( 139 | (activation): ELU(alpha=1.0) 140 | (layers): ModuleList( 141 | (0): Linear(in_features=384, out_features=192, bias=True) 142 | (1): Linear(in_features=192, out_features=96, bias=True) 143 | (2): Linear(in_features=96, out_features=48, bias=True) 144 | (3): Linear(in_features=48, out_features=24, bias=True) 145 | (4): Linear(in_features=24, out_features=12, bias=True) 146 | (5): Linear(in_features=12, out_features=1, bias=True) 147 | ) 148 | ) 149 | (util-DSP): MLP( 150 | (activation): ELU(alpha=1.0) 151 | (layers): ModuleList( 152 | (0): Linear(in_features=384, out_features=192, bias=True) 153 | (1): Linear(in_features=192, out_features=96, bias=True) 154 | (2): Linear(in_features=96, out_features=48, bias=True) 155 | (3): Linear(in_features=48, out_features=24, bias=True) 156 | (4): Linear(in_features=24, out_features=12, bias=True) 157 | (5): Linear(in_features=12, out_features=1, bias=True) 158 | ) 159 | ) 160 | (util-BRAM): MLP( 161 | (activation): ELU(alpha=1.0) 162 | (layers): ModuleList( 163 | (0): Linear(in_features=384, out_features=192, bias=True) 164 | (1): Linear(in_features=192, out_features=96, bias=True) 165 | (2): Linear(in_features=96, out_features=48, bias=True) 166 | (3): Linear(in_features=48, out_features=24, bias=True) 167 | (4): Linear(in_features=24, out_features=12, bias=True) 168 | (5): Linear(in_features=12, out_features=1, bias=True) 169 | ) 170 | ) 171 | ) 172 | ) 173 | -------------------------------------------------------------------------------- /baselines/configs_for_reproducibility/code2vec_v2_c.txt: -------------------------------------------------------------------------------- 1 | model : code2vec 2 | v2_db : True 3 | use_redis : False 4 | task : class 5 | subtask : train 6 | sequence_modeling : False 7 | code2vec_data_version : neurips2023_benchmark 8 | multi_modality : False 9 | load_db : None 10 | load_model : None 11 | dataset : programl 12 | benchmarks : ['machsuite', 13 | : 'poly'] 14 | tag : wmp-d 15 | round_num : 3 16 | graph_type : 17 | graph_transformer_option : None 18 | gae_T : False 19 | gae_P : False 20 | SSL : False 21 | load_pretrained_GNN : False 22 | class_model_path : None 23 | all_kernels : True 24 | sample_finetune : False 25 | FT_extra : False 26 | only_common_db : False 27 | test_extra : False 28 | only_new_points : False 29 | new_speedup : True 30 | feature_extract : False 31 | ignore_kernels : [] 32 | test_kernels : ['gemm-p-large', 33 | : 'covariance', 34 | : 'gemm-ncubed', 35 | : 'symm', 36 | : 'fdtd-2d-large', 37 | : 'trmm-opt'] 38 | val_ratio_in_test_kernels : 0 39 | val_ratio_in_train_kernels : 0.15 40 | test_ratio_in_train_kernels : 0.15 41 | tvt_split_by : kernels_inductive 42 | itype_mask_perc : 0 43 | gtype : programl 44 | only_pragma_nodes : False 45 | encode_full_text : None 46 | fulltext_dim : None 47 | MAML : False 48 | force_regen : False 49 | load_encoders_label : None 50 | encoder_path : None 51 | model_tag : test 52 | activation : elu 53 | outlier_removal : None 54 | no_pragma : False 55 | num_layers : 2 56 | D : 512 57 | target : ['perf', 58 | : 'util-LUT', 59 | : 'util-FF', 60 | : 'util-DSP', 61 | : 'util-BRAM'] 62 | gnn_type : transformer 63 | min_allowed_latency : 100.0 64 | encode_edge : True 65 | encode_edge_position : False 66 | ptrans : False 67 | jkn_mode : max 68 | jkn_enable : True 69 | node_attention : True 70 | node_attention_MLP : False 71 | separate_P_T : False 72 | pragma_encoder : True 73 | pragma_uniform_encoder : True 74 | epsilon : 0.001 75 | normalizer : 10000000.0 76 | util_normalizer : 1 77 | max_number : 10000000000.0 78 | norm_method : speedup-log2 79 | target_preproc : None 80 | target_convert_back : True 81 | invalid : False 82 | multi_target : True 83 | activation_type : elu 84 | margin_loss : False 85 | save_model : True 86 | save_every_epoch : 10000 87 | encode_log : False 88 | target_factor : 1 89 | target_transform : None 90 | loss_scale : {'perf': 1.0, 91 | : 'util-DSP': 1.0, 92 | : 'util-BRAM': 1.0, 93 | : 'util-LUT': 1.0, 94 | : 'util-FF': 1.0} 95 | pairwise_class : False 96 | batch_size : 64 97 | data_loader_num_workers : 0 98 | device : cuda:7 99 | epoch_num : 200 100 | debug_iter : -1 101 | ignore_testing : True 102 | ignore_validation : False 103 | shuffle : True 104 | opt_type : AdamW 105 | lr : 0.0001 106 | max_grad_norm : None 107 | weight_decay : 0.01 108 | plot_pred_points : True 109 | fix_randomness : True 110 | random_seed : 123 111 | user : xxx 112 | hostname : xxxx 113 | exp_name : 114 | ts : 2023-06-04T15-21-08.094629 115 | 116 | Code2VecNet( 117 | (loss_function): CrossEntropyLoss() 118 | (decoder): ModuleDict( 119 | (perf): MLP( 120 | (activation): ELU(alpha=1.0) 121 | (layers): ModuleList( 122 | (0): Linear(in_features=384, out_features=192, bias=True) 123 | (1): Linear(in_features=192, out_features=96, bias=True) 124 | (2): Linear(in_features=96, out_features=48, bias=True) 125 | (3): Linear(in_features=48, out_features=24, bias=True) 126 | (4): Linear(in_features=24, out_features=12, bias=True) 127 | (5): Linear(in_features=12, out_features=2, bias=True) 128 | ) 129 | ) 130 | ) 131 | ) 132 | -------------------------------------------------------------------------------- /baselines/configs_for_reproducibility/code2vec_v2_r.txt: -------------------------------------------------------------------------------- 1 | model : code2vec 2 | v2_db : True 3 | use_redis : False 4 | task : regression 5 | subtask : train 6 | sequence_modeling : False 7 | code2vec_data_version : neurips2023_benchmark 8 | multi_modality : False 9 | load_db : None 10 | load_model : None 11 | dataset : programl 12 | benchmarks : ['machsuite', 13 | : 'poly'] 14 | tag : wmp-d 15 | round_num : 3 16 | graph_type : 17 | graph_transformer_option : None 18 | gae_T : False 19 | gae_P : False 20 | SSL : False 21 | load_pretrained_GNN : False 22 | class_model_path : None 23 | all_kernels : True 24 | sample_finetune : False 25 | FT_extra : False 26 | only_common_db : False 27 | test_extra : False 28 | only_new_points : False 29 | new_speedup : True 30 | feature_extract : False 31 | ignore_kernels : [] 32 | test_kernels : ['gemm-p-large', 33 | : 'covariance', 34 | : 'gemm-ncubed', 35 | : 'symm', 36 | : 'fdtd-2d-large', 37 | : 'trmm-opt'] 38 | val_ratio_in_test_kernels : 0 39 | val_ratio_in_train_kernels : 0.15 40 | test_ratio_in_train_kernels : 0.15 41 | tvt_split_by : kernels_inductive 42 | itype_mask_perc : 0 43 | gtype : programl 44 | only_pragma_nodes : False 45 | encode_full_text : None 46 | fulltext_dim : None 47 | MAML : False 48 | force_regen : True 49 | load_encoders_label : None 50 | encoder_path : None 51 | model_tag : test 52 | activation : elu 53 | outlier_removal : None 54 | no_pragma : False 55 | num_layers : 2 56 | D : 512 57 | target : ['perf', 58 | : 'util-LUT', 59 | : 'util-FF', 60 | : 'util-DSP', 61 | : 'util-BRAM'] 62 | gnn_type : transformer 63 | min_allowed_latency : 100.0 64 | encode_edge : True 65 | encode_edge_position : False 66 | ptrans : False 67 | jkn_mode : max 68 | jkn_enable : True 69 | node_attention : True 70 | node_attention_MLP : False 71 | separate_P_T : False 72 | pragma_encoder : True 73 | pragma_uniform_encoder : True 74 | epsilon : 0.001 75 | normalizer : 10000000.0 76 | util_normalizer : 1 77 | max_number : 10000000000.0 78 | norm_method : speedup-log2 79 | target_preproc : None 80 | target_convert_back : True 81 | invalid : False 82 | multi_target : True 83 | activation_type : elu 84 | margin_loss : False 85 | save_model : True 86 | save_every_epoch : 10000 87 | encode_log : False 88 | target_factor : 1 89 | target_transform : None 90 | loss_scale : {'perf': 1.0, 91 | : 'util-DSP': 1.0, 92 | : 'util-BRAM': 1.0, 93 | : 'util-LUT': 1.0, 94 | : 'util-FF': 1.0} 95 | pairwise_class : False 96 | batch_size : 64 97 | data_loader_num_workers : 0 98 | device : cuda:4 99 | epoch_num : 1000 100 | debug_iter : -1 101 | ignore_testing : True 102 | ignore_validation : False 103 | shuffle : True 104 | opt_type : AdamW 105 | lr : 0.0001 106 | max_grad_norm : None 107 | weight_decay : 0.01 108 | plot_pred_points : True 109 | fix_randomness : True 110 | random_seed : 123 111 | user : xxx 112 | hostname : xxxx 113 | exp_name : 114 | ts : 2023-06-04T14-46-11.020272 115 | 116 | Code2VecNet( 117 | (loss_function): MSELoss() 118 | (decoder): ModuleDict( 119 | (perf): MLP( 120 | (activation): ELU(alpha=1.0) 121 | (layers): ModuleList( 122 | (0): Linear(in_features=384, out_features=192, bias=True) 123 | (1): Linear(in_features=192, out_features=96, bias=True) 124 | (2): Linear(in_features=96, out_features=48, bias=True) 125 | (3): Linear(in_features=48, out_features=24, bias=True) 126 | (4): Linear(in_features=24, out_features=12, bias=True) 127 | (5): Linear(in_features=12, out_features=1, bias=True) 128 | ) 129 | ) 130 | (util-LUT): MLP( 131 | (activation): ELU(alpha=1.0) 132 | (layers): ModuleList( 133 | (0): Linear(in_features=384, out_features=192, bias=True) 134 | (1): Linear(in_features=192, out_features=96, bias=True) 135 | (2): Linear(in_features=96, out_features=48, bias=True) 136 | (3): Linear(in_features=48, out_features=24, bias=True) 137 | (4): Linear(in_features=24, out_features=12, bias=True) 138 | (5): Linear(in_features=12, out_features=1, bias=True) 139 | ) 140 | ) 141 | (util-FF): MLP( 142 | (activation): ELU(alpha=1.0) 143 | (layers): ModuleList( 144 | (0): Linear(in_features=384, out_features=192, bias=True) 145 | (1): Linear(in_features=192, out_features=96, bias=True) 146 | (2): Linear(in_features=96, out_features=48, bias=True) 147 | (3): Linear(in_features=48, out_features=24, bias=True) 148 | (4): Linear(in_features=24, out_features=12, bias=True) 149 | (5): Linear(in_features=12, out_features=1, bias=True) 150 | ) 151 | ) 152 | (util-DSP): MLP( 153 | (activation): ELU(alpha=1.0) 154 | (layers): ModuleList( 155 | (0): Linear(in_features=384, out_features=192, bias=True) 156 | (1): Linear(in_features=192, out_features=96, bias=True) 157 | (2): Linear(in_features=96, out_features=48, bias=True) 158 | (3): Linear(in_features=48, out_features=24, bias=True) 159 | (4): Linear(in_features=24, out_features=12, bias=True) 160 | (5): Linear(in_features=12, out_features=1, bias=True) 161 | ) 162 | ) 163 | (util-BRAM): MLP( 164 | (activation): ELU(alpha=1.0) 165 | (layers): ModuleList( 166 | (0): Linear(in_features=384, out_features=192, bias=True) 167 | (1): Linear(in_features=192, out_features=96, bias=True) 168 | (2): Linear(in_features=96, out_features=48, bias=True) 169 | (3): Linear(in_features=48, out_features=24, bias=True) 170 | (4): Linear(in_features=24, out_features=12, bias=True) 171 | (5): Linear(in_features=12, out_features=1, bias=True) 172 | ) 173 | ) 174 | ) 175 | ) 176 | -------------------------------------------------------------------------------- /baselines/configs_for_reproducibility/codet5_frozen_v1_c.txt: -------------------------------------------------------------------------------- 1 | model : our 2 | v2_db : False 3 | use_redis : False 4 | task : class 5 | subtask : train 6 | sequence_modeling : True 7 | data_repr : penhance 8 | code_encoder : codet5 9 | chunk_emb : cls 10 | finetune_bert : False 11 | replace_with_random_weights : False 12 | add_edges_fc_graph : True 13 | chunk_offset : 16 14 | max_code_tokens_len : 64 15 | token_att_masking : True 16 | preserve_keywords : True 17 | pk_version : 2 18 | bypass_tf_layers : False 19 | apply_act_conv_first : False 20 | vis_transformer_att : False 21 | multi_modality : False 22 | load_db : None 23 | load_model : None 24 | dataset : programl 25 | benchmarks : ['machsuite', 26 | : 'poly'] 27 | tag : wmp-d 28 | round_num : 13 29 | graph_type : 30 | graph_transformer_option : None 31 | gae_T : False 32 | gae_P : False 33 | SSL : False 34 | load_pretrained_GNN : False 35 | class_model_path : None 36 | all_kernels : True 37 | sample_finetune : False 38 | FT_extra : False 39 | new_speedup : True 40 | feature_extract : False 41 | ignore_kernels : [] 42 | test_kernels : ['stencil-3d', 43 | : 'doitgen-red', 44 | : 'gemm-ncubed', 45 | : 'fdtd-2d', 46 | : 'jacobi-2d', 47 | : 'trmm-opt'] 48 | val_ratio_in_test_kernels : 0 49 | val_ratio_in_train_kernels : 0.15 50 | test_ratio_in_train_kernels : 0.15 51 | tvt_split_by : kernels_inductive 52 | itype_mask_perc : 0 53 | gtype : programl 54 | only_pragma_nodes : False 55 | encode_full_text : None 56 | fulltext_dim : None 57 | MAML : False 58 | force_regen : False 59 | load_encoders_label : None 60 | encoder_path : None 61 | model_tag : test 62 | activation : elu 63 | outlier_removal : None 64 | no_pragma : False 65 | num_layers : 8 66 | D : 512 67 | target : ['perf', 68 | : 'util-LUT', 69 | : 'util-FF', 70 | : 'util-DSP', 71 | : 'util-BRAM'] 72 | gnn_type : transformer 73 | min_allowed_latency : 100.0 74 | encode_edge : False 75 | encode_edge_position : False 76 | ptrans : False 77 | jkn_mode : max 78 | jkn_enable : True 79 | node_attention : True 80 | node_attention_MLP : False 81 | separate_P_T : False 82 | pragma_encoder : True 83 | pragma_uniform_encoder : True 84 | epsilon : 0.001 85 | normalizer : 10000000.0 86 | util_normalizer : 1 87 | max_number : 10000000000.0 88 | norm_method : speedup-log2 89 | target_preproc : None 90 | target_convert_back : True 91 | invalid : False 92 | multi_target : True 93 | activation_type : elu 94 | margin_loss : False 95 | save_model : True 96 | save_every_epoch : 10000 97 | encode_log : False 98 | target_factor : 1 99 | target_transform : None 100 | loss_scale : {'perf': 1.0, 101 | : 'util-DSP': 1.0, 102 | : 'util-BRAM': 1.0, 103 | : 'util-LUT': 1.0, 104 | : 'util-FF': 1.0} 105 | pairwise_class : False 106 | batch_size : 4 107 | data_loader_num_workers : 0 108 | device : cuda:2 109 | epoch_num : 200 110 | debug_iter : -1 111 | ignore_testing : True 112 | ignore_validation : False 113 | shuffle : True 114 | opt_type : AdamW 115 | lr : 1e-05 116 | max_grad_norm : None 117 | weight_decay : 0.01 118 | plot_pred_points : True 119 | fix_randomness : True 120 | random_seed : 123 121 | user : xxx 122 | hostname : xxxx 123 | exp_name : 124 | ts : 2023-06-02T16-10-03.110404 125 | 126 | Net( 127 | (bert_model): T5Stack( 128 | (embed_tokens): Embedding(32234, 512) 129 | (block): ModuleList( 130 | (0): T5Block( 131 | (layer): ModuleList( 132 | (0): T5LayerSelfAttention( 133 | (SelfAttention): T5Attention( 134 | (q): Linear(in_features=512, out_features=512, bias=False) 135 | (k): Linear(in_features=512, out_features=512, bias=False) 136 | (v): Linear(in_features=512, out_features=512, bias=False) 137 | (o): Linear(in_features=512, out_features=512, bias=False) 138 | (relative_attention_bias): Embedding(32, 8) 139 | ) 140 | (layer_norm): T5LayerNorm() 141 | (dropout): Dropout(p=0.1, inplace=False) 142 | ) 143 | (1): T5LayerFF( 144 | (DenseReluDense): T5DenseActDense( 145 | (wi): Linear(in_features=512, out_features=2048, bias=False) 146 | (wo): Linear(in_features=2048, out_features=512, bias=False) 147 | (dropout): Dropout(p=0.1, inplace=False) 148 | (act): ReLU() 149 | ) 150 | (layer_norm): T5LayerNorm() 151 | (dropout): Dropout(p=0.1, inplace=False) 152 | ) 153 | ) 154 | ) 155 | (1): T5Block( 156 | (layer): ModuleList( 157 | (0): T5LayerSelfAttention( 158 | (SelfAttention): T5Attention( 159 | (q): Linear(in_features=512, out_features=512, bias=False) 160 | (k): Linear(in_features=512, out_features=512, bias=False) 161 | (v): Linear(in_features=512, out_features=512, bias=False) 162 | (o): Linear(in_features=512, out_features=512, bias=False) 163 | ) 164 | (layer_norm): T5LayerNorm() 165 | (dropout): Dropout(p=0.1, inplace=False) 166 | ) 167 | (1): T5LayerFF( 168 | (DenseReluDense): T5DenseActDense( 169 | (wi): Linear(in_features=512, out_features=2048, bias=False) 170 | (wo): Linear(in_features=2048, out_features=512, bias=False) 171 | (dropout): Dropout(p=0.1, inplace=False) 172 | (act): ReLU() 173 | ) 174 | (layer_norm): T5LayerNorm() 175 | (dropout): Dropout(p=0.1, inplace=False) 176 | ) 177 | ) 178 | ) 179 | (2): T5Block( 180 | (layer): ModuleList( 181 | (0): T5LayerSelfAttention( 182 | (SelfAttention): T5Attention( 183 | (q): Linear(in_features=512, out_features=512, bias=False) 184 | (k): Linear(in_features=512, out_features=512, bias=False) 185 | (v): Linear(in_features=512, out_features=512, bias=False) 186 | (o): Linear(in_features=512, out_features=512, bias=False) 187 | ) 188 | (layer_norm): T5LayerNorm() 189 | (dropout): Dropout(p=0.1, inplace=False) 190 | ) 191 | (1): T5LayerFF( 192 | (DenseReluDense): T5DenseActDense( 193 | (wi): Linear(in_features=512, out_features=2048, bias=False) 194 | (wo): Linear(in_features=2048, out_features=512, bias=False) 195 | (dropout): Dropout(p=0.1, inplace=False) 196 | (act): ReLU() 197 | ) 198 | (layer_norm): T5LayerNorm() 199 | (dropout): Dropout(p=0.1, inplace=False) 200 | ) 201 | ) 202 | ) 203 | (3): T5Block( 204 | (layer): ModuleList( 205 | (0): T5LayerSelfAttention( 206 | (SelfAttention): T5Attention( 207 | (q): Linear(in_features=512, out_features=512, bias=False) 208 | (k): Linear(in_features=512, out_features=512, bias=False) 209 | (v): Linear(in_features=512, out_features=512, bias=False) 210 | (o): Linear(in_features=512, out_features=512, bias=False) 211 | ) 212 | (layer_norm): T5LayerNorm() 213 | (dropout): Dropout(p=0.1, inplace=False) 214 | ) 215 | (1): T5LayerFF( 216 | (DenseReluDense): T5DenseActDense( 217 | (wi): Linear(in_features=512, out_features=2048, bias=False) 218 | (wo): Linear(in_features=2048, out_features=512, bias=False) 219 | (dropout): Dropout(p=0.1, inplace=False) 220 | (act): ReLU() 221 | ) 222 | (layer_norm): T5LayerNorm() 223 | (dropout): Dropout(p=0.1, inplace=False) 224 | ) 225 | ) 226 | ) 227 | (4): T5Block( 228 | (layer): ModuleList( 229 | (0): T5LayerSelfAttention( 230 | (SelfAttention): T5Attention( 231 | (q): Linear(in_features=512, out_features=512, bias=False) 232 | (k): Linear(in_features=512, out_features=512, bias=False) 233 | (v): Linear(in_features=512, out_features=512, bias=False) 234 | (o): Linear(in_features=512, out_features=512, bias=False) 235 | ) 236 | (layer_norm): T5LayerNorm() 237 | (dropout): Dropout(p=0.1, inplace=False) 238 | ) 239 | (1): T5LayerFF( 240 | (DenseReluDense): T5DenseActDense( 241 | (wi): Linear(in_features=512, out_features=2048, bias=False) 242 | (wo): Linear(in_features=2048, out_features=512, bias=False) 243 | (dropout): Dropout(p=0.1, inplace=False) 244 | (act): ReLU() 245 | ) 246 | (layer_norm): T5LayerNorm() 247 | (dropout): Dropout(p=0.1, inplace=False) 248 | ) 249 | ) 250 | ) 251 | (5): T5Block( 252 | (layer): ModuleList( 253 | (0): T5LayerSelfAttention( 254 | (SelfAttention): T5Attention( 255 | (q): Linear(in_features=512, out_features=512, bias=False) 256 | (k): Linear(in_features=512, out_features=512, bias=False) 257 | (v): Linear(in_features=512, out_features=512, bias=False) 258 | (o): Linear(in_features=512, out_features=512, bias=False) 259 | ) 260 | (layer_norm): T5LayerNorm() 261 | (dropout): Dropout(p=0.1, inplace=False) 262 | ) 263 | (1): T5LayerFF( 264 | (DenseReluDense): T5DenseActDense( 265 | (wi): Linear(in_features=512, out_features=2048, bias=False) 266 | (wo): Linear(in_features=2048, out_features=512, bias=False) 267 | (dropout): Dropout(p=0.1, inplace=False) 268 | (act): ReLU() 269 | ) 270 | (layer_norm): T5LayerNorm() 271 | (dropout): Dropout(p=0.1, inplace=False) 272 | ) 273 | ) 274 | ) 275 | ) 276 | (final_layer_norm): T5LayerNorm() 277 | (dropout): Dropout(p=0.1, inplace=False) 278 | ) 279 | (conv_first): MLP( 280 | (activation): ELU(alpha=1.0) 281 | (layers): ModuleList( 282 | (0): Linear(in_features=512, out_features=256, bias=True) 283 | (1): Linear(in_features=256, out_features=128, bias=True) 284 | (2): Linear(in_features=128, out_features=512, bias=True) 285 | ) 286 | ) 287 | (conv_layers): ModuleList( 288 | (0): TransformerConv(512, 512, heads=1) 289 | (1): TransformerConv(512, 512, heads=1) 290 | (2): TransformerConv(512, 512, heads=1) 291 | (3): TransformerConv(512, 512, heads=1) 292 | (4): TransformerConv(512, 512, heads=1) 293 | (5): TransformerConv(512, 512, heads=1) 294 | (6): TransformerConv(512, 512, heads=1) 295 | ) 296 | (jkn): JumpingKnowledge(max) 297 | (gate_nn): Sequential( 298 | (0): Linear(in_features=512, out_features=512, bias=True) 299 | (1): ReLU() 300 | (2): Linear(in_features=512, out_features=1, bias=True) 301 | ) 302 | (glob): MyGlobalAttention(gate_nn=Sequential( 303 | (0): Linear(in_features=512, out_features=512, bias=True) 304 | (1): ReLU() 305 | (2): Linear(in_features=512, out_features=1, bias=True) 306 | ), nn=None) 307 | (loss_function): CrossEntropyLoss() 308 | (MLPs): ModuleDict( 309 | (perf): MLP( 310 | (activation): ELU(alpha=1.0) 311 | (layers): ModuleList( 312 | (0): Linear(in_features=512, out_features=256, bias=True) 313 | (1): Linear(in_features=256, out_features=128, bias=True) 314 | (2): Linear(in_features=128, out_features=64, bias=True) 315 | (3): Linear(in_features=64, out_features=32, bias=True) 316 | (4): Linear(in_features=32, out_features=16, bias=True) 317 | (5): Linear(in_features=16, out_features=2, bias=True) 318 | ) 319 | ) 320 | ) 321 | ) 322 | -------------------------------------------------------------------------------- /baselines/configs_for_reproducibility/codet5_rand_v1_c.txt: -------------------------------------------------------------------------------- 1 | model : our 2 | v2_db : False 3 | use_redis : False 4 | task : class 5 | subtask : train 6 | sequence_modeling : True 7 | data_repr : penhance 8 | code_encoder : codet5 9 | chunk_emb : cls 10 | finetune_bert : True 11 | replace_with_random_weights : True 12 | add_edges_fc_graph : True 13 | chunk_offset : 16 14 | max_code_tokens_len : 64 15 | token_att_masking : True 16 | preserve_keywords : True 17 | pk_version : 2 18 | bypass_tf_layers : False 19 | apply_act_conv_first : False 20 | vis_transformer_att : False 21 | multi_modality : False 22 | load_db : None 23 | load_model : None 24 | dataset : programl 25 | benchmarks : ['machsuite', 26 | : 'poly'] 27 | tag : wmp-d 28 | round_num : 13 29 | graph_type : 30 | graph_transformer_option : None 31 | gae_T : False 32 | gae_P : False 33 | SSL : False 34 | load_pretrained_GNN : False 35 | class_model_path : None 36 | all_kernels : True 37 | sample_finetune : False 38 | FT_extra : False 39 | new_speedup : True 40 | feature_extract : False 41 | ignore_kernels : [] 42 | test_kernels : ['stencil-3d', 43 | : 'doitgen-red', 44 | : 'gemm-ncubed', 45 | : 'fdtd-2d', 46 | : 'jacobi-2d', 47 | : 'trmm-opt'] 48 | val_ratio_in_test_kernels : 0 49 | val_ratio_in_train_kernels : 0.15 50 | test_ratio_in_train_kernels : 0.15 51 | tvt_split_by : kernels_inductive 52 | itype_mask_perc : 0 53 | gtype : programl 54 | only_pragma_nodes : False 55 | encode_full_text : None 56 | fulltext_dim : None 57 | MAML : False 58 | force_regen : False 59 | load_encoders_label : None 60 | encoder_path : None 61 | model_tag : test 62 | activation : elu 63 | outlier_removal : None 64 | no_pragma : False 65 | num_layers : 8 66 | D : 512 67 | target : ['perf', 68 | : 'util-LUT', 69 | : 'util-FF', 70 | : 'util-DSP', 71 | : 'util-BRAM'] 72 | gnn_type : transformer 73 | min_allowed_latency : 100.0 74 | encode_edge : False 75 | encode_edge_position : False 76 | ptrans : False 77 | jkn_mode : max 78 | jkn_enable : True 79 | node_attention : True 80 | node_attention_MLP : False 81 | separate_P_T : False 82 | pragma_encoder : True 83 | pragma_uniform_encoder : True 84 | epsilon : 0.001 85 | normalizer : 10000000.0 86 | util_normalizer : 1 87 | max_number : 10000000000.0 88 | norm_method : speedup-log2 89 | target_preproc : None 90 | target_convert_back : True 91 | invalid : False 92 | multi_target : True 93 | activation_type : elu 94 | margin_loss : False 95 | save_model : True 96 | save_every_epoch : 10000 97 | encode_log : False 98 | target_factor : 1 99 | target_transform : None 100 | loss_scale : {'perf': 1.0, 101 | : 'util-DSP': 1.0, 102 | : 'util-BRAM': 1.0, 103 | : 'util-LUT': 1.0, 104 | : 'util-FF': 1.0} 105 | pairwise_class : False 106 | batch_size : 4 107 | data_loader_num_workers : 0 108 | device : cuda:3 109 | epoch_num : 200 110 | debug_iter : -1 111 | ignore_testing : True 112 | ignore_validation : False 113 | shuffle : True 114 | opt_type : AdamW 115 | lr : 1e-05 116 | max_grad_norm : None 117 | weight_decay : 0.01 118 | plot_pred_points : True 119 | fix_randomness : True 120 | random_seed : 123 121 | user : xxx 122 | hostname : xxxx 123 | exp_name : 124 | ts : 2023-06-02T16-04-04.301829 125 | 126 | Net( 127 | (bert_model): T5Stack( 128 | (embed_tokens): Embedding(32234, 512) 129 | (block): ModuleList( 130 | (0): T5Block( 131 | (layer): ModuleList( 132 | (0): T5LayerSelfAttention( 133 | (SelfAttention): T5Attention( 134 | (q): Linear(in_features=512, out_features=512, bias=False) 135 | (k): Linear(in_features=512, out_features=512, bias=False) 136 | (v): Linear(in_features=512, out_features=512, bias=False) 137 | (o): Linear(in_features=512, out_features=512, bias=False) 138 | (relative_attention_bias): Embedding(32, 8) 139 | ) 140 | (layer_norm): T5LayerNorm() 141 | (dropout): Dropout(p=0.1, inplace=False) 142 | ) 143 | (1): T5LayerFF( 144 | (DenseReluDense): T5DenseActDense( 145 | (wi): Linear(in_features=512, out_features=2048, bias=False) 146 | (wo): Linear(in_features=2048, out_features=512, bias=False) 147 | (dropout): Dropout(p=0.1, inplace=False) 148 | (act): ReLU() 149 | ) 150 | (layer_norm): T5LayerNorm() 151 | (dropout): Dropout(p=0.1, inplace=False) 152 | ) 153 | ) 154 | ) 155 | (1): T5Block( 156 | (layer): ModuleList( 157 | (0): T5LayerSelfAttention( 158 | (SelfAttention): T5Attention( 159 | (q): Linear(in_features=512, out_features=512, bias=False) 160 | (k): Linear(in_features=512, out_features=512, bias=False) 161 | (v): Linear(in_features=512, out_features=512, bias=False) 162 | (o): Linear(in_features=512, out_features=512, bias=False) 163 | ) 164 | (layer_norm): T5LayerNorm() 165 | (dropout): Dropout(p=0.1, inplace=False) 166 | ) 167 | (1): T5LayerFF( 168 | (DenseReluDense): T5DenseActDense( 169 | (wi): Linear(in_features=512, out_features=2048, bias=False) 170 | (wo): Linear(in_features=2048, out_features=512, bias=False) 171 | (dropout): Dropout(p=0.1, inplace=False) 172 | (act): ReLU() 173 | ) 174 | (layer_norm): T5LayerNorm() 175 | (dropout): Dropout(p=0.1, inplace=False) 176 | ) 177 | ) 178 | ) 179 | (2): T5Block( 180 | (layer): ModuleList( 181 | (0): T5LayerSelfAttention( 182 | (SelfAttention): T5Attention( 183 | (q): Linear(in_features=512, out_features=512, bias=False) 184 | (k): Linear(in_features=512, out_features=512, bias=False) 185 | (v): Linear(in_features=512, out_features=512, bias=False) 186 | (o): Linear(in_features=512, out_features=512, bias=False) 187 | ) 188 | (layer_norm): T5LayerNorm() 189 | (dropout): Dropout(p=0.1, inplace=False) 190 | ) 191 | (1): T5LayerFF( 192 | (DenseReluDense): T5DenseActDense( 193 | (wi): Linear(in_features=512, out_features=2048, bias=False) 194 | (wo): Linear(in_features=2048, out_features=512, bias=False) 195 | (dropout): Dropout(p=0.1, inplace=False) 196 | (act): ReLU() 197 | ) 198 | (layer_norm): T5LayerNorm() 199 | (dropout): Dropout(p=0.1, inplace=False) 200 | ) 201 | ) 202 | ) 203 | (3): T5Block( 204 | (layer): ModuleList( 205 | (0): T5LayerSelfAttention( 206 | (SelfAttention): T5Attention( 207 | (q): Linear(in_features=512, out_features=512, bias=False) 208 | (k): Linear(in_features=512, out_features=512, bias=False) 209 | (v): Linear(in_features=512, out_features=512, bias=False) 210 | (o): Linear(in_features=512, out_features=512, bias=False) 211 | ) 212 | (layer_norm): T5LayerNorm() 213 | (dropout): Dropout(p=0.1, inplace=False) 214 | ) 215 | (1): T5LayerFF( 216 | (DenseReluDense): T5DenseActDense( 217 | (wi): Linear(in_features=512, out_features=2048, bias=False) 218 | (wo): Linear(in_features=2048, out_features=512, bias=False) 219 | (dropout): Dropout(p=0.1, inplace=False) 220 | (act): ReLU() 221 | ) 222 | (layer_norm): T5LayerNorm() 223 | (dropout): Dropout(p=0.1, inplace=False) 224 | ) 225 | ) 226 | ) 227 | (4): T5Block( 228 | (layer): ModuleList( 229 | (0): T5LayerSelfAttention( 230 | (SelfAttention): T5Attention( 231 | (q): Linear(in_features=512, out_features=512, bias=False) 232 | (k): Linear(in_features=512, out_features=512, bias=False) 233 | (v): Linear(in_features=512, out_features=512, bias=False) 234 | (o): Linear(in_features=512, out_features=512, bias=False) 235 | ) 236 | (layer_norm): T5LayerNorm() 237 | (dropout): Dropout(p=0.1, inplace=False) 238 | ) 239 | (1): T5LayerFF( 240 | (DenseReluDense): T5DenseActDense( 241 | (wi): Linear(in_features=512, out_features=2048, bias=False) 242 | (wo): Linear(in_features=2048, out_features=512, bias=False) 243 | (dropout): Dropout(p=0.1, inplace=False) 244 | (act): ReLU() 245 | ) 246 | (layer_norm): T5LayerNorm() 247 | (dropout): Dropout(p=0.1, inplace=False) 248 | ) 249 | ) 250 | ) 251 | (5): T5Block( 252 | (layer): ModuleList( 253 | (0): T5LayerSelfAttention( 254 | (SelfAttention): T5Attention( 255 | (q): Linear(in_features=512, out_features=512, bias=False) 256 | (k): Linear(in_features=512, out_features=512, bias=False) 257 | (v): Linear(in_features=512, out_features=512, bias=False) 258 | (o): Linear(in_features=512, out_features=512, bias=False) 259 | ) 260 | (layer_norm): T5LayerNorm() 261 | (dropout): Dropout(p=0.1, inplace=False) 262 | ) 263 | (1): T5LayerFF( 264 | (DenseReluDense): T5DenseActDense( 265 | (wi): Linear(in_features=512, out_features=2048, bias=False) 266 | (wo): Linear(in_features=2048, out_features=512, bias=False) 267 | (dropout): Dropout(p=0.1, inplace=False) 268 | (act): ReLU() 269 | ) 270 | (layer_norm): T5LayerNorm() 271 | (dropout): Dropout(p=0.1, inplace=False) 272 | ) 273 | ) 274 | ) 275 | ) 276 | (final_layer_norm): T5LayerNorm() 277 | (dropout): Dropout(p=0.1, inplace=False) 278 | ) 279 | (conv_first): MLP( 280 | (activation): ELU(alpha=1.0) 281 | (layers): ModuleList( 282 | (0): Linear(in_features=512, out_features=256, bias=True) 283 | (1): Linear(in_features=256, out_features=128, bias=True) 284 | (2): Linear(in_features=128, out_features=512, bias=True) 285 | ) 286 | ) 287 | (conv_layers): ModuleList( 288 | (0): TransformerConv(512, 512, heads=1) 289 | (1): TransformerConv(512, 512, heads=1) 290 | (2): TransformerConv(512, 512, heads=1) 291 | (3): TransformerConv(512, 512, heads=1) 292 | (4): TransformerConv(512, 512, heads=1) 293 | (5): TransformerConv(512, 512, heads=1) 294 | (6): TransformerConv(512, 512, heads=1) 295 | ) 296 | (jkn): JumpingKnowledge(max) 297 | (gate_nn): Sequential( 298 | (0): Linear(in_features=512, out_features=512, bias=True) 299 | (1): ReLU() 300 | (2): Linear(in_features=512, out_features=1, bias=True) 301 | ) 302 | (glob): MyGlobalAttention(gate_nn=Sequential( 303 | (0): Linear(in_features=512, out_features=512, bias=True) 304 | (1): ReLU() 305 | (2): Linear(in_features=512, out_features=1, bias=True) 306 | ), nn=None) 307 | (loss_function): CrossEntropyLoss() 308 | (MLPs): ModuleDict( 309 | (perf): MLP( 310 | (activation): ELU(alpha=1.0) 311 | (layers): ModuleList( 312 | (0): Linear(in_features=512, out_features=256, bias=True) 313 | (1): Linear(in_features=256, out_features=128, bias=True) 314 | (2): Linear(in_features=128, out_features=64, bias=True) 315 | (3): Linear(in_features=64, out_features=32, bias=True) 316 | (4): Linear(in_features=32, out_features=16, bias=True) 317 | (5): Linear(in_features=16, out_features=2, bias=True) 318 | ) 319 | ) 320 | ) 321 | ) 322 | -------------------------------------------------------------------------------- /baselines/configs_for_reproducibility/gnn_dse_2L_v1_c.txt: -------------------------------------------------------------------------------- 1 | model : our 2 | v2_db : False 3 | use_redis : False 4 | task : class 5 | subtask : train 6 | sequence_modeling : False 7 | multi_modality : False 8 | load_db : None 9 | load_model : None 10 | dataset : programl 11 | benchmarks : ['machsuite', 12 | : 'poly'] 13 | tag : wmp-d 14 | round_num : 13 15 | graph_type : 16 | graph_transformer_option : None 17 | gae_T : False 18 | gae_P : False 19 | SSL : False 20 | load_pretrained_GNN : False 21 | class_model_path : None 22 | all_kernels : True 23 | sample_finetune : False 24 | FT_extra : False 25 | new_speedup : True 26 | feature_extract : False 27 | ignore_kernels : [] 28 | test_kernels : ['stencil-3d', 29 | : 'doitgen-red', 30 | : 'gemm-ncubed', 31 | : 'fdtd-2d', 32 | : 'jacobi-2d', 33 | : 'trmm-opt'] 34 | val_ratio_in_test_kernels : 0 35 | val_ratio_in_train_kernels : 0.15 36 | test_ratio_in_train_kernels : 0.15 37 | tvt_split_by : kernels_inductive 38 | itype_mask_perc : 0 39 | gtype : programl 40 | only_pragma_nodes : False 41 | encode_full_text : None 42 | fulltext_dim : None 43 | MAML : False 44 | force_regen : True 45 | load_encoders_label : None 46 | encoder_path : None 47 | model_tag : test 48 | activation : elu 49 | outlier_removal : None 50 | no_pragma : False 51 | num_layers : 2 52 | D : 512 53 | target : ['perf', 54 | : 'util-LUT', 55 | : 'util-FF', 56 | : 'util-DSP', 57 | : 'util-BRAM'] 58 | gnn_type : transformer 59 | min_allowed_latency : 100.0 60 | encode_edge : True 61 | encode_edge_position : False 62 | ptrans : False 63 | jkn_mode : max 64 | jkn_enable : True 65 | node_attention : True 66 | node_attention_MLP : False 67 | separate_P_T : False 68 | pragma_encoder : True 69 | pragma_uniform_encoder : True 70 | epsilon : 0.001 71 | normalizer : 10000000.0 72 | util_normalizer : 1 73 | max_number : 10000000000.0 74 | norm_method : speedup-log2 75 | target_preproc : None 76 | target_convert_back : True 77 | invalid : False 78 | multi_target : True 79 | activation_type : elu 80 | margin_loss : False 81 | save_model : True 82 | save_every_epoch : 10000 83 | encode_log : False 84 | baseline_no_graph : False 85 | target_factor : 1 86 | target_transform : None 87 | loss_scale : {'perf': 1.0, 88 | : 'util-DSP': 1.0, 89 | : 'util-BRAM': 1.0, 90 | : 'util-LUT': 1.0, 91 | : 'util-FF': 1.0} 92 | pairwise_class : False 93 | batch_size : 64 94 | data_loader_num_workers : 0 95 | device : cuda:7 96 | epoch_num : 200 97 | debug_iter : -1 98 | ignore_testing : True 99 | ignore_validation : False 100 | shuffle : True 101 | opt_type : AdamW 102 | lr : 0.0001 103 | max_grad_norm : None 104 | weight_decay : 0.01 105 | plot_pred_points : True 106 | fix_randomness : True 107 | random_seed : 123 108 | user : xxx 109 | hostname : xxxx 110 | exp_name : 111 | ts : 2023-06-02T11-25-09.905485 112 | 113 | Net( 114 | (conv_first): TransformerConv(158, 512, heads=1) 115 | (conv_layers): ModuleList( 116 | (0): TransformerConv(512, 512, heads=1) 117 | ) 118 | (jkn): JumpingKnowledge(max) 119 | (gate_nn): Sequential( 120 | (0): Linear(in_features=512, out_features=512, bias=True) 121 | (1): ReLU() 122 | (2): Linear(in_features=512, out_features=1, bias=True) 123 | ) 124 | (glob): MyGlobalAttention(gate_nn=Sequential( 125 | (0): Linear(in_features=512, out_features=512, bias=True) 126 | (1): ReLU() 127 | (2): Linear(in_features=512, out_features=1, bias=True) 128 | ), nn=None) 129 | (loss_function): CrossEntropyLoss() 130 | (MLPs): ModuleDict( 131 | (perf): MLP( 132 | (activation): ELU(alpha=1.0) 133 | (layers): ModuleList( 134 | (0): Linear(in_features=512, out_features=256, bias=True) 135 | (1): Linear(in_features=256, out_features=128, bias=True) 136 | (2): Linear(in_features=128, out_features=64, bias=True) 137 | (3): Linear(in_features=64, out_features=32, bias=True) 138 | (4): Linear(in_features=32, out_features=16, bias=True) 139 | (5): Linear(in_features=16, out_features=2, bias=True) 140 | ) 141 | ) 142 | ) 143 | ) 144 | -------------------------------------------------------------------------------- /baselines/configs_for_reproducibility/gnn_dse_2L_v1_r.txt: -------------------------------------------------------------------------------- 1 | model : our 2 | v2_db : False 3 | use_redis : False 4 | task : regression 5 | subtask : train 6 | sequence_modeling : False 7 | multi_modality : False 8 | load_db : None 9 | load_model : None 10 | dataset : programl 11 | benchmarks : ['machsuite', 12 | : 'poly'] 13 | tag : wmp-d 14 | round_num : 13 15 | graph_type : 16 | graph_transformer_option : None 17 | gae_T : False 18 | gae_P : False 19 | SSL : False 20 | load_pretrained_GNN : False 21 | class_model_path : None 22 | all_kernels : True 23 | sample_finetune : False 24 | FT_extra : False 25 | new_speedup : True 26 | feature_extract : False 27 | ignore_kernels : [] 28 | test_kernels : ['stencil-3d', 29 | : 'doitgen-red', 30 | : 'gemm-ncubed', 31 | : 'fdtd-2d', 32 | : 'jacobi-2d', 33 | : 'trmm-opt'] 34 | val_ratio_in_test_kernels : 0 35 | val_ratio_in_train_kernels : 0.15 36 | test_ratio_in_train_kernels : 0.15 37 | tvt_split_by : kernels_inductive 38 | itype_mask_perc : 0 39 | gtype : programl 40 | only_pragma_nodes : False 41 | encode_full_text : None 42 | fulltext_dim : None 43 | MAML : False 44 | force_regen : False 45 | load_encoders_label : None 46 | encoder_path : None 47 | model_tag : test 48 | activation : elu 49 | outlier_removal : None 50 | no_pragma : False 51 | num_layers : 2 52 | D : 512 53 | target : ['perf', 54 | : 'util-LUT', 55 | : 'util-FF', 56 | : 'util-DSP', 57 | : 'util-BRAM'] 58 | gnn_type : transformer 59 | min_allowed_latency : 100.0 60 | encode_edge : True 61 | encode_edge_position : False 62 | ptrans : False 63 | jkn_mode : max 64 | jkn_enable : True 65 | node_attention : True 66 | node_attention_MLP : False 67 | separate_P_T : False 68 | pragma_encoder : True 69 | pragma_uniform_encoder : True 70 | epsilon : 0.001 71 | normalizer : 10000000.0 72 | util_normalizer : 1 73 | max_number : 10000000000.0 74 | norm_method : speedup-log2 75 | target_preproc : None 76 | target_convert_back : True 77 | invalid : False 78 | multi_target : True 79 | activation_type : elu 80 | margin_loss : False 81 | save_model : True 82 | save_every_epoch : 10000 83 | encode_log : False 84 | baseline_no_graph : False 85 | target_factor : 1 86 | target_transform : None 87 | loss_scale : {'perf': 1.0, 88 | : 'util-DSP': 1.0, 89 | : 'util-BRAM': 1.0, 90 | : 'util-LUT': 1.0, 91 | : 'util-FF': 1.0} 92 | pairwise_class : False 93 | batch_size : 64 94 | data_loader_num_workers : 0 95 | device : cuda:6 96 | epoch_num : 1000 97 | debug_iter : -1 98 | ignore_testing : True 99 | ignore_validation : False 100 | shuffle : True 101 | opt_type : AdamW 102 | lr : 0.0001 103 | max_grad_norm : None 104 | weight_decay : 0.01 105 | plot_pred_points : True 106 | fix_randomness : True 107 | random_seed : 123 108 | user : xxx 109 | hostname : xxxx 110 | exp_name : 111 | ts : 2023-05-30T22-43-20.088387 112 | 113 | Net( 114 | (conv_first): TransformerConv(158, 512, heads=1) 115 | (conv_layers): ModuleList( 116 | (0): TransformerConv(512, 512, heads=1) 117 | ) 118 | (jkn): JumpingKnowledge(max) 119 | (gate_nn): Sequential( 120 | (0): Linear(in_features=512, out_features=512, bias=True) 121 | (1): ReLU() 122 | (2): Linear(in_features=512, out_features=1, bias=True) 123 | ) 124 | (glob): MyGlobalAttention(gate_nn=Sequential( 125 | (0): Linear(in_features=512, out_features=512, bias=True) 126 | (1): ReLU() 127 | (2): Linear(in_features=512, out_features=1, bias=True) 128 | ), nn=None) 129 | (loss_function): MSELoss() 130 | (MLPs): ModuleDict( 131 | (perf): MLP( 132 | (activation): ELU(alpha=1.0) 133 | (layers): ModuleList( 134 | (0): Linear(in_features=512, out_features=256, bias=True) 135 | (1): Linear(in_features=256, out_features=128, bias=True) 136 | (2): Linear(in_features=128, out_features=64, bias=True) 137 | (3): Linear(in_features=64, out_features=32, bias=True) 138 | (4): Linear(in_features=32, out_features=16, bias=True) 139 | (5): Linear(in_features=16, out_features=1, bias=True) 140 | ) 141 | ) 142 | (util-LUT): MLP( 143 | (activation): ELU(alpha=1.0) 144 | (layers): ModuleList( 145 | (0): Linear(in_features=512, out_features=256, bias=True) 146 | (1): Linear(in_features=256, out_features=128, bias=True) 147 | (2): Linear(in_features=128, out_features=64, bias=True) 148 | (3): Linear(in_features=64, out_features=32, bias=True) 149 | (4): Linear(in_features=32, out_features=16, bias=True) 150 | (5): Linear(in_features=16, out_features=1, bias=True) 151 | ) 152 | ) 153 | (util-FF): MLP( 154 | (activation): ELU(alpha=1.0) 155 | (layers): ModuleList( 156 | (0): Linear(in_features=512, out_features=256, bias=True) 157 | (1): Linear(in_features=256, out_features=128, bias=True) 158 | (2): Linear(in_features=128, out_features=64, bias=True) 159 | (3): Linear(in_features=64, out_features=32, bias=True) 160 | (4): Linear(in_features=32, out_features=16, bias=True) 161 | (5): Linear(in_features=16, out_features=1, bias=True) 162 | ) 163 | ) 164 | (util-DSP): MLP( 165 | (activation): ELU(alpha=1.0) 166 | (layers): ModuleList( 167 | (0): Linear(in_features=512, out_features=256, bias=True) 168 | (1): Linear(in_features=256, out_features=128, bias=True) 169 | (2): Linear(in_features=128, out_features=64, bias=True) 170 | (3): Linear(in_features=64, out_features=32, bias=True) 171 | (4): Linear(in_features=32, out_features=16, bias=True) 172 | (5): Linear(in_features=16, out_features=1, bias=True) 173 | ) 174 | ) 175 | (util-BRAM): MLP( 176 | (activation): ELU(alpha=1.0) 177 | (layers): ModuleList( 178 | (0): Linear(in_features=512, out_features=256, bias=True) 179 | (1): Linear(in_features=256, out_features=128, bias=True) 180 | (2): Linear(in_features=128, out_features=64, bias=True) 181 | (3): Linear(in_features=64, out_features=32, bias=True) 182 | (4): Linear(in_features=32, out_features=16, bias=True) 183 | (5): Linear(in_features=16, out_features=1, bias=True) 184 | ) 185 | ) 186 | ) 187 | ) 188 | -------------------------------------------------------------------------------- /baselines/configs_for_reproducibility/gnn_dse_2L_v2_c.txt: -------------------------------------------------------------------------------- 1 | model : our 2 | v2_db : True 3 | use_redis : False 4 | task : class 5 | subtask : train 6 | sequence_modeling : False 7 | multi_modality : False 8 | load_db : None 9 | load_model : None 10 | dataset : programl 11 | benchmarks : ['machsuite', 12 | : 'poly'] 13 | tag : wmp-d 14 | round_num : 3 15 | graph_type : 16 | graph_transformer_option : None 17 | gae_T : False 18 | gae_P : False 19 | SSL : False 20 | load_pretrained_GNN : False 21 | class_model_path : None 22 | all_kernels : True 23 | sample_finetune : False 24 | FT_extra : False 25 | only_common_db : False 26 | test_extra : False 27 | only_new_points : False 28 | new_speedup : True 29 | feature_extract : False 30 | ignore_kernels : [] 31 | test_kernels : ['gemm-p-large', 32 | : 'covariance', 33 | : 'gemm-ncubed', 34 | : 'symm', 35 | : 'fdtd-2d-large', 36 | : 'trmm-opt'] 37 | val_ratio_in_test_kernels : 0 38 | val_ratio_in_train_kernels : 0.15 39 | test_ratio_in_train_kernels : 0.15 40 | tvt_split_by : kernels_inductive 41 | itype_mask_perc : 0 42 | gtype : programl 43 | only_pragma_nodes : False 44 | encode_full_text : None 45 | fulltext_dim : None 46 | MAML : False 47 | force_regen : False 48 | load_encoders_label : None 49 | encoder_path : None 50 | model_tag : test 51 | activation : elu 52 | outlier_removal : None 53 | no_pragma : False 54 | num_layers : 2 55 | D : 512 56 | target : ['perf', 57 | : 'util-LUT', 58 | : 'util-FF', 59 | : 'util-DSP', 60 | : 'util-BRAM'] 61 | gnn_type : transformer 62 | min_allowed_latency : 100.0 63 | encode_edge : True 64 | encode_edge_position : False 65 | ptrans : False 66 | jkn_mode : max 67 | jkn_enable : True 68 | node_attention : True 69 | node_attention_MLP : False 70 | separate_P_T : False 71 | pragma_encoder : True 72 | pragma_uniform_encoder : True 73 | epsilon : 0.001 74 | normalizer : 10000000.0 75 | util_normalizer : 1 76 | max_number : 10000000000.0 77 | norm_method : speedup-log2 78 | target_preproc : None 79 | target_convert_back : True 80 | invalid : False 81 | multi_target : True 82 | activation_type : elu 83 | margin_loss : False 84 | save_model : True 85 | save_every_epoch : 10000 86 | encode_log : False 87 | baseline_no_graph : False 88 | target_factor : 1 89 | target_transform : None 90 | loss_scale : {'perf': 1.0, 91 | : 'util-DSP': 1.0, 92 | : 'util-BRAM': 1.0, 93 | : 'util-LUT': 1.0, 94 | : 'util-FF': 1.0} 95 | pairwise_class : False 96 | batch_size : 64 97 | data_loader_num_workers : 0 98 | device : cuda:3 99 | epoch_num : 200 100 | debug_iter : -1 101 | ignore_testing : True 102 | ignore_validation : False 103 | shuffle : True 104 | opt_type : AdamW 105 | lr : 0.0001 106 | max_grad_norm : None 107 | weight_decay : 0.01 108 | plot_pred_points : True 109 | fix_randomness : True 110 | random_seed : 123 111 | user : xxx 112 | hostname : xxxx 113 | exp_name : 114 | ts : 2023-05-30T22-45-11.537613 115 | 116 | Net( 117 | (conv_first): TransformerConv(150, 512, heads=1) 118 | (conv_layers): ModuleList( 119 | (0): TransformerConv(512, 512, heads=1) 120 | ) 121 | (jkn): JumpingKnowledge(max) 122 | (gate_nn): Sequential( 123 | (0): Linear(in_features=512, out_features=512, bias=True) 124 | (1): ReLU() 125 | (2): Linear(in_features=512, out_features=1, bias=True) 126 | ) 127 | (glob): MyGlobalAttention(gate_nn=Sequential( 128 | (0): Linear(in_features=512, out_features=512, bias=True) 129 | (1): ReLU() 130 | (2): Linear(in_features=512, out_features=1, bias=True) 131 | ), nn=None) 132 | (loss_function): CrossEntropyLoss() 133 | (MLPs): ModuleDict( 134 | (perf): MLP( 135 | (activation): ELU(alpha=1.0) 136 | (layers): ModuleList( 137 | (0): Linear(in_features=512, out_features=256, bias=True) 138 | (1): Linear(in_features=256, out_features=128, bias=True) 139 | (2): Linear(in_features=128, out_features=64, bias=True) 140 | (3): Linear(in_features=64, out_features=32, bias=True) 141 | (4): Linear(in_features=32, out_features=16, bias=True) 142 | (5): Linear(in_features=16, out_features=2, bias=True) 143 | ) 144 | ) 145 | ) 146 | ) 147 | -------------------------------------------------------------------------------- /baselines/configs_for_reproducibility/gnn_dse_2L_v2_r.txt: -------------------------------------------------------------------------------- 1 | model : our 2 | v2_db : True 3 | use_redis : False 4 | task : regression 5 | subtask : train 6 | sequence_modeling : False 7 | multi_modality : False 8 | load_db : None 9 | load_model : None 10 | dataset : programl 11 | benchmarks : ['machsuite', 12 | : 'poly'] 13 | tag : wmp-d 14 | round_num : 3 15 | graph_type : 16 | graph_transformer_option : None 17 | gae_T : False 18 | gae_P : False 19 | SSL : False 20 | load_pretrained_GNN : False 21 | class_model_path : None 22 | all_kernels : True 23 | sample_finetune : False 24 | FT_extra : False 25 | only_common_db : False 26 | test_extra : False 27 | only_new_points : False 28 | new_speedup : True 29 | feature_extract : False 30 | ignore_kernels : [] 31 | test_kernels : ['gemm-p-large', 32 | : 'covariance', 33 | : 'gemm-ncubed', 34 | : 'symm', 35 | : 'fdtd-2d-large', 36 | : 'trmm-opt'] 37 | val_ratio_in_test_kernels : 0 38 | val_ratio_in_train_kernels : 0.15 39 | test_ratio_in_train_kernels : 0.15 40 | tvt_split_by : kernels_inductive 41 | itype_mask_perc : 0 42 | gtype : programl 43 | only_pragma_nodes : False 44 | encode_full_text : None 45 | fulltext_dim : None 46 | MAML : False 47 | force_regen : False 48 | load_encoders_label : None 49 | encoder_path : None 50 | model_tag : test 51 | activation : elu 52 | outlier_removal : None 53 | no_pragma : False 54 | num_layers : 2 55 | D : 512 56 | target : ['perf', 57 | : 'util-LUT', 58 | : 'util-FF', 59 | : 'util-DSP', 60 | : 'util-BRAM'] 61 | gnn_type : transformer 62 | min_allowed_latency : 100.0 63 | encode_edge : True 64 | encode_edge_position : False 65 | ptrans : False 66 | jkn_mode : max 67 | jkn_enable : True 68 | node_attention : True 69 | node_attention_MLP : False 70 | separate_P_T : False 71 | pragma_encoder : True 72 | pragma_uniform_encoder : True 73 | epsilon : 0.001 74 | normalizer : 10000000.0 75 | util_normalizer : 1 76 | max_number : 10000000000.0 77 | norm_method : speedup-log2 78 | target_preproc : None 79 | target_convert_back : True 80 | invalid : False 81 | multi_target : True 82 | activation_type : elu 83 | margin_loss : False 84 | save_model : True 85 | save_every_epoch : 10000 86 | encode_log : False 87 | baseline_no_graph : False 88 | target_factor : 1 89 | target_transform : None 90 | loss_scale : {'perf': 1.0, 91 | : 'util-DSP': 1.0, 92 | : 'util-BRAM': 1.0, 93 | : 'util-LUT': 1.0, 94 | : 'util-FF': 1.0} 95 | pairwise_class : False 96 | batch_size : 64 97 | data_loader_num_workers : 0 98 | device : cuda:1 99 | epoch_num : 1000 100 | debug_iter : -1 101 | ignore_testing : True 102 | ignore_validation : False 103 | shuffle : True 104 | opt_type : AdamW 105 | lr : 0.0001 106 | max_grad_norm : None 107 | weight_decay : 0.01 108 | plot_pred_points : True 109 | fix_randomness : True 110 | random_seed : 123 111 | user : xxx 112 | hostname : xxxx 113 | exp_name : 114 | ts : 2023-05-30T22-45-36.671731 115 | 116 | Net( 117 | (conv_first): TransformerConv(150, 512, heads=1) 118 | (conv_layers): ModuleList( 119 | (0): TransformerConv(512, 512, heads=1) 120 | ) 121 | (jkn): JumpingKnowledge(max) 122 | (gate_nn): Sequential( 123 | (0): Linear(in_features=512, out_features=512, bias=True) 124 | (1): ReLU() 125 | (2): Linear(in_features=512, out_features=1, bias=True) 126 | ) 127 | (glob): MyGlobalAttention(gate_nn=Sequential( 128 | (0): Linear(in_features=512, out_features=512, bias=True) 129 | (1): ReLU() 130 | (2): Linear(in_features=512, out_features=1, bias=True) 131 | ), nn=None) 132 | (loss_function): MSELoss() 133 | (MLPs): ModuleDict( 134 | (perf): MLP( 135 | (activation): ELU(alpha=1.0) 136 | (layers): ModuleList( 137 | (0): Linear(in_features=512, out_features=256, bias=True) 138 | (1): Linear(in_features=256, out_features=128, bias=True) 139 | (2): Linear(in_features=128, out_features=64, bias=True) 140 | (3): Linear(in_features=64, out_features=32, bias=True) 141 | (4): Linear(in_features=32, out_features=16, bias=True) 142 | (5): Linear(in_features=16, out_features=1, bias=True) 143 | ) 144 | ) 145 | (util-LUT): MLP( 146 | (activation): ELU(alpha=1.0) 147 | (layers): ModuleList( 148 | (0): Linear(in_features=512, out_features=256, bias=True) 149 | (1): Linear(in_features=256, out_features=128, bias=True) 150 | (2): Linear(in_features=128, out_features=64, bias=True) 151 | (3): Linear(in_features=64, out_features=32, bias=True) 152 | (4): Linear(in_features=32, out_features=16, bias=True) 153 | (5): Linear(in_features=16, out_features=1, bias=True) 154 | ) 155 | ) 156 | (util-FF): MLP( 157 | (activation): ELU(alpha=1.0) 158 | (layers): ModuleList( 159 | (0): Linear(in_features=512, out_features=256, bias=True) 160 | (1): Linear(in_features=256, out_features=128, bias=True) 161 | (2): Linear(in_features=128, out_features=64, bias=True) 162 | (3): Linear(in_features=64, out_features=32, bias=True) 163 | (4): Linear(in_features=32, out_features=16, bias=True) 164 | (5): Linear(in_features=16, out_features=1, bias=True) 165 | ) 166 | ) 167 | (util-DSP): MLP( 168 | (activation): ELU(alpha=1.0) 169 | (layers): ModuleList( 170 | (0): Linear(in_features=512, out_features=256, bias=True) 171 | (1): Linear(in_features=256, out_features=128, bias=True) 172 | (2): Linear(in_features=128, out_features=64, bias=True) 173 | (3): Linear(in_features=64, out_features=32, bias=True) 174 | (4): Linear(in_features=32, out_features=16, bias=True) 175 | (5): Linear(in_features=16, out_features=1, bias=True) 176 | ) 177 | ) 178 | (util-BRAM): MLP( 179 | (activation): ELU(alpha=1.0) 180 | (layers): ModuleList( 181 | (0): Linear(in_features=512, out_features=256, bias=True) 182 | (1): Linear(in_features=256, out_features=128, bias=True) 183 | (2): Linear(in_features=128, out_features=64, bias=True) 184 | (3): Linear(in_features=64, out_features=32, bias=True) 185 | (4): Linear(in_features=32, out_features=16, bias=True) 186 | (5): Linear(in_features=16, out_features=1, bias=True) 187 | ) 188 | ) 189 | ) 190 | ) 191 | -------------------------------------------------------------------------------- /baselines/configs_for_reproducibility/gnn_dse_v1_c.txt: -------------------------------------------------------------------------------- 1 | model : our 2 | v2_db : False 3 | use_redis : False 4 | task : class 5 | subtask : train 6 | sequence_modeling : False 7 | multi_modality : False 8 | load_db : None 9 | load_model : None 10 | dataset : programl 11 | benchmarks : ['machsuite', 12 | : 'poly'] 13 | tag : wmp-d 14 | round_num : 13 15 | graph_type : 16 | graph_transformer_option : None 17 | gae_T : False 18 | gae_P : False 19 | SSL : False 20 | load_pretrained_GNN : False 21 | class_model_path : None 22 | all_kernels : True 23 | sample_finetune : False 24 | FT_extra : False 25 | new_speedup : True 26 | feature_extract : False 27 | ignore_kernels : [] 28 | test_kernels : ['stencil-3d', 29 | : 'doitgen-red', 30 | : 'gemm-ncubed', 31 | : 'fdtd-2d', 32 | : 'jacobi-2d', 33 | : 'trmm-opt'] 34 | val_ratio_in_test_kernels : 0 35 | val_ratio_in_train_kernels : 0.15 36 | test_ratio_in_train_kernels : 0.15 37 | tvt_split_by : kernels_inductive 38 | itype_mask_perc : 0 39 | gtype : programl 40 | only_pragma_nodes : False 41 | encode_full_text : None 42 | fulltext_dim : None 43 | MAML : False 44 | force_regen : True 45 | load_encoders_label : None 46 | encoder_path : None 47 | model_tag : test 48 | activation : elu 49 | outlier_removal : None 50 | no_pragma : False 51 | num_layers : 8 52 | D : 512 53 | target : ['perf', 54 | : 'util-LUT', 55 | : 'util-FF', 56 | : 'util-DSP', 57 | : 'util-BRAM'] 58 | gnn_type : transformer 59 | min_allowed_latency : 100.0 60 | encode_edge : True 61 | encode_edge_position : False 62 | ptrans : False 63 | jkn_mode : max 64 | jkn_enable : True 65 | node_attention : True 66 | node_attention_MLP : False 67 | separate_P_T : False 68 | pragma_encoder : True 69 | pragma_uniform_encoder : True 70 | epsilon : 0.001 71 | normalizer : 10000000.0 72 | util_normalizer : 1 73 | max_number : 10000000000.0 74 | norm_method : speedup-log2 75 | target_preproc : None 76 | target_convert_back : True 77 | invalid : False 78 | multi_target : True 79 | activation_type : elu 80 | margin_loss : False 81 | save_model : True 82 | save_every_epoch : 10000 83 | encode_log : False 84 | baseline_no_graph : False 85 | target_factor : 1 86 | target_transform : None 87 | loss_scale : {'perf': 1.0, 88 | : 'util-DSP': 1.0, 89 | : 'util-BRAM': 1.0, 90 | : 'util-LUT': 1.0, 91 | : 'util-FF': 1.0} 92 | pairwise_class : False 93 | batch_size : 64 94 | data_loader_num_workers : 0 95 | device : cuda:5 96 | epoch_num : 200 97 | debug_iter : -1 98 | ignore_testing : True 99 | ignore_validation : False 100 | shuffle : True 101 | opt_type : AdamW 102 | lr : 0.0001 103 | max_grad_norm : None 104 | weight_decay : 0.01 105 | plot_pred_points : True 106 | fix_randomness : True 107 | random_seed : 123 108 | user : xxx 109 | hostname : xxxx 110 | exp_name : 111 | ts : 2023-05-30T21-42-52.763973 112 | 113 | Net( 114 | (conv_first): TransformerConv(158, 512, heads=1) 115 | (conv_layers): ModuleList( 116 | (0): TransformerConv(512, 512, heads=1) 117 | (1): TransformerConv(512, 512, heads=1) 118 | (2): TransformerConv(512, 512, heads=1) 119 | (3): TransformerConv(512, 512, heads=1) 120 | (4): TransformerConv(512, 512, heads=1) 121 | (5): TransformerConv(512, 512, heads=1) 122 | (6): TransformerConv(512, 512, heads=1) 123 | ) 124 | (jkn): JumpingKnowledge(max) 125 | (gate_nn): Sequential( 126 | (0): Linear(in_features=512, out_features=512, bias=True) 127 | (1): ReLU() 128 | (2): Linear(in_features=512, out_features=1, bias=True) 129 | ) 130 | (glob): MyGlobalAttention(gate_nn=Sequential( 131 | (0): Linear(in_features=512, out_features=512, bias=True) 132 | (1): ReLU() 133 | (2): Linear(in_features=512, out_features=1, bias=True) 134 | ), nn=None) 135 | (loss_function): CrossEntropyLoss() 136 | (MLPs): ModuleDict( 137 | (perf): MLP( 138 | (activation): ELU(alpha=1.0) 139 | (layers): ModuleList( 140 | (0): Linear(in_features=512, out_features=256, bias=True) 141 | (1): Linear(in_features=256, out_features=128, bias=True) 142 | (2): Linear(in_features=128, out_features=64, bias=True) 143 | (3): Linear(in_features=64, out_features=32, bias=True) 144 | (4): Linear(in_features=32, out_features=16, bias=True) 145 | (5): Linear(in_features=16, out_features=2, bias=True) 146 | ) 147 | ) 148 | ) 149 | ) 150 | -------------------------------------------------------------------------------- /baselines/configs_for_reproducibility/gnn_dse_v1_r.txt: -------------------------------------------------------------------------------- 1 | model : our 2 | v2_db : False 3 | use_redis : False 4 | task : regression 5 | subtask : train 6 | sequence_modeling : False 7 | multi_modality : False 8 | load_db : None 9 | load_model : None 10 | dataset : programl 11 | benchmarks : ['machsuite', 12 | : 'poly'] 13 | tag : wmp-d 14 | round_num : 13 15 | graph_type : 16 | graph_transformer_option : None 17 | gae_T : False 18 | gae_P : False 19 | SSL : False 20 | load_pretrained_GNN : False 21 | class_model_path : None 22 | all_kernels : True 23 | sample_finetune : False 24 | FT_extra : False 25 | new_speedup : True 26 | feature_extract : False 27 | ignore_kernels : [] 28 | test_kernels : ['stencil-3d', 29 | : 'doitgen-red', 30 | : 'gemm-ncubed', 31 | : 'fdtd-2d', 32 | : 'jacobi-2d', 33 | : 'trmm-opt'] 34 | val_ratio_in_test_kernels : 0 35 | val_ratio_in_train_kernels : 0.15 36 | test_ratio_in_train_kernels : 0.15 37 | tvt_split_by : kernels_inductive 38 | itype_mask_perc : 0 39 | gtype : programl 40 | only_pragma_nodes : False 41 | encode_full_text : None 42 | fulltext_dim : None 43 | MAML : False 44 | force_regen : True 45 | load_encoders_label : None 46 | encoder_path : None 47 | model_tag : test 48 | activation : elu 49 | outlier_removal : None 50 | no_pragma : False 51 | num_layers : 8 52 | D : 512 53 | target : ['perf', 54 | : 'util-LUT', 55 | : 'util-FF', 56 | : 'util-DSP', 57 | : 'util-BRAM'] 58 | gnn_type : transformer 59 | min_allowed_latency : 100.0 60 | encode_edge : True 61 | encode_edge_position : False 62 | ptrans : False 63 | jkn_mode : max 64 | jkn_enable : True 65 | node_attention : True 66 | node_attention_MLP : False 67 | separate_P_T : False 68 | pragma_encoder : True 69 | pragma_uniform_encoder : True 70 | epsilon : 0.001 71 | normalizer : 10000000.0 72 | util_normalizer : 1 73 | max_number : 10000000000.0 74 | norm_method : speedup-log2 75 | target_preproc : None 76 | target_convert_back : True 77 | invalid : False 78 | multi_target : True 79 | activation_type : elu 80 | margin_loss : False 81 | save_model : True 82 | save_every_epoch : 10000 83 | encode_log : False 84 | baseline_no_graph : False 85 | target_factor : 1 86 | target_transform : None 87 | loss_scale : {'perf': 1.0, 88 | : 'util-DSP': 1.0, 89 | : 'util-BRAM': 1.0, 90 | : 'util-LUT': 1.0, 91 | : 'util-FF': 1.0} 92 | pairwise_class : False 93 | batch_size : 64 94 | data_loader_num_workers : 0 95 | device : cuda:4 96 | epoch_num : 1000 97 | debug_iter : -1 98 | ignore_testing : True 99 | ignore_validation : False 100 | shuffle : True 101 | opt_type : AdamW 102 | lr : 0.0001 103 | max_grad_norm : None 104 | weight_decay : 0.01 105 | plot_pred_points : True 106 | fix_randomness : True 107 | random_seed : 123 108 | user : xxx 109 | hostname : xxxx 110 | exp_name : 111 | ts : 2023-05-30T21-43-11.606154 112 | 113 | Net( 114 | (conv_first): TransformerConv(158, 512, heads=1) 115 | (conv_layers): ModuleList( 116 | (0): TransformerConv(512, 512, heads=1) 117 | (1): TransformerConv(512, 512, heads=1) 118 | (2): TransformerConv(512, 512, heads=1) 119 | (3): TransformerConv(512, 512, heads=1) 120 | (4): TransformerConv(512, 512, heads=1) 121 | (5): TransformerConv(512, 512, heads=1) 122 | (6): TransformerConv(512, 512, heads=1) 123 | ) 124 | (jkn): JumpingKnowledge(max) 125 | (gate_nn): Sequential( 126 | (0): Linear(in_features=512, out_features=512, bias=True) 127 | (1): ReLU() 128 | (2): Linear(in_features=512, out_features=1, bias=True) 129 | ) 130 | (glob): MyGlobalAttention(gate_nn=Sequential( 131 | (0): Linear(in_features=512, out_features=512, bias=True) 132 | (1): ReLU() 133 | (2): Linear(in_features=512, out_features=1, bias=True) 134 | ), nn=None) 135 | (loss_function): MSELoss() 136 | (MLPs): ModuleDict( 137 | (perf): MLP( 138 | (activation): ELU(alpha=1.0) 139 | (layers): ModuleList( 140 | (0): Linear(in_features=512, out_features=256, bias=True) 141 | (1): Linear(in_features=256, out_features=128, bias=True) 142 | (2): Linear(in_features=128, out_features=64, bias=True) 143 | (3): Linear(in_features=64, out_features=32, bias=True) 144 | (4): Linear(in_features=32, out_features=16, bias=True) 145 | (5): Linear(in_features=16, out_features=1, bias=True) 146 | ) 147 | ) 148 | (util-LUT): MLP( 149 | (activation): ELU(alpha=1.0) 150 | (layers): ModuleList( 151 | (0): Linear(in_features=512, out_features=256, bias=True) 152 | (1): Linear(in_features=256, out_features=128, bias=True) 153 | (2): Linear(in_features=128, out_features=64, bias=True) 154 | (3): Linear(in_features=64, out_features=32, bias=True) 155 | (4): Linear(in_features=32, out_features=16, bias=True) 156 | (5): Linear(in_features=16, out_features=1, bias=True) 157 | ) 158 | ) 159 | (util-FF): MLP( 160 | (activation): ELU(alpha=1.0) 161 | (layers): ModuleList( 162 | (0): Linear(in_features=512, out_features=256, bias=True) 163 | (1): Linear(in_features=256, out_features=128, bias=True) 164 | (2): Linear(in_features=128, out_features=64, bias=True) 165 | (3): Linear(in_features=64, out_features=32, bias=True) 166 | (4): Linear(in_features=32, out_features=16, bias=True) 167 | (5): Linear(in_features=16, out_features=1, bias=True) 168 | ) 169 | ) 170 | (util-DSP): MLP( 171 | (activation): ELU(alpha=1.0) 172 | (layers): ModuleList( 173 | (0): Linear(in_features=512, out_features=256, bias=True) 174 | (1): Linear(in_features=256, out_features=128, bias=True) 175 | (2): Linear(in_features=128, out_features=64, bias=True) 176 | (3): Linear(in_features=64, out_features=32, bias=True) 177 | (4): Linear(in_features=32, out_features=16, bias=True) 178 | (5): Linear(in_features=16, out_features=1, bias=True) 179 | ) 180 | ) 181 | (util-BRAM): MLP( 182 | (activation): ELU(alpha=1.0) 183 | (layers): ModuleList( 184 | (0): Linear(in_features=512, out_features=256, bias=True) 185 | (1): Linear(in_features=256, out_features=128, bias=True) 186 | (2): Linear(in_features=128, out_features=64, bias=True) 187 | (3): Linear(in_features=64, out_features=32, bias=True) 188 | (4): Linear(in_features=32, out_features=16, bias=True) 189 | (5): Linear(in_features=16, out_features=1, bias=True) 190 | ) 191 | ) 192 | ) 193 | ) 194 | -------------------------------------------------------------------------------- /baselines/configs_for_reproducibility/gnn_dse_v2_r.txt: -------------------------------------------------------------------------------- 1 | model : our 2 | v2_db : True 3 | use_redis : False 4 | task : regression 5 | subtask : train 6 | sequence_modeling : False 7 | multi_modality : False 8 | load_db : None 9 | load_model : None 10 | dataset : programl 11 | benchmarks : ['machsuite', 12 | : 'poly'] 13 | tag : wmp-d 14 | round_num : 3 15 | graph_type : 16 | graph_transformer_option : None 17 | gae_T : False 18 | gae_P : False 19 | SSL : False 20 | load_pretrained_GNN : False 21 | class_model_path : None 22 | all_kernels : True 23 | sample_finetune : False 24 | FT_extra : False 25 | only_common_db : False 26 | test_extra : False 27 | only_new_points : False 28 | new_speedup : True 29 | feature_extract : False 30 | ignore_kernels : [] 31 | test_kernels : ['gemm-p-large', 32 | : 'covariance', 33 | : 'gemm-ncubed', 34 | : 'symm', 35 | : 'fdtd-2d-large', 36 | : 'trmm-opt'] 37 | val_ratio_in_test_kernels : 0 38 | val_ratio_in_train_kernels : 0.15 39 | test_ratio_in_train_kernels : 0.15 40 | tvt_split_by : kernels_inductive 41 | itype_mask_perc : 0 42 | gtype : programl 43 | only_pragma_nodes : False 44 | encode_full_text : None 45 | fulltext_dim : None 46 | MAML : False 47 | force_regen : True 48 | load_encoders_label : None 49 | encoder_path : None 50 | model_tag : test 51 | activation : elu 52 | outlier_removal : None 53 | no_pragma : False 54 | num_layers : 8 55 | D : 512 56 | target : ['perf', 57 | : 'util-LUT', 58 | : 'util-FF', 59 | : 'util-DSP', 60 | : 'util-BRAM'] 61 | gnn_type : transformer 62 | min_allowed_latency : 100.0 63 | encode_edge : True 64 | encode_edge_position : False 65 | ptrans : False 66 | jkn_mode : max 67 | jkn_enable : True 68 | node_attention : True 69 | node_attention_MLP : False 70 | separate_P_T : False 71 | pragma_encoder : True 72 | pragma_uniform_encoder : True 73 | epsilon : 0.001 74 | normalizer : 10000000.0 75 | util_normalizer : 1 76 | max_number : 10000000000.0 77 | norm_method : speedup-log2 78 | target_preproc : None 79 | target_convert_back : True 80 | invalid : False 81 | multi_target : True 82 | activation_type : elu 83 | margin_loss : False 84 | save_model : True 85 | save_every_epoch : 10000 86 | encode_log : False 87 | baseline_no_graph : False 88 | target_factor : 1 89 | target_transform : None 90 | loss_scale : {'perf': 1.0, 91 | : 'util-DSP': 1.0, 92 | : 'util-BRAM': 1.0, 93 | : 'util-LUT': 1.0, 94 | : 'util-FF': 1.0} 95 | pairwise_class : False 96 | batch_size : 64 97 | data_loader_num_workers : 0 98 | device : cuda:7 99 | epoch_num : 1000 100 | debug_iter : -1 101 | ignore_testing : True 102 | ignore_validation : False 103 | shuffle : True 104 | opt_type : AdamW 105 | lr : 0.0001 106 | max_grad_norm : None 107 | weight_decay : 0.01 108 | plot_pred_points : True 109 | fix_randomness : True 110 | random_seed : 123 111 | user : xxx 112 | hostname : xxxx 113 | exp_name : 114 | ts : 2023-05-30T21-41-16.437018 115 | 116 | Net( 117 | (conv_first): TransformerConv(150, 512, heads=1) 118 | (conv_layers): ModuleList( 119 | (0): TransformerConv(512, 512, heads=1) 120 | (1): TransformerConv(512, 512, heads=1) 121 | (2): TransformerConv(512, 512, heads=1) 122 | (3): TransformerConv(512, 512, heads=1) 123 | (4): TransformerConv(512, 512, heads=1) 124 | (5): TransformerConv(512, 512, heads=1) 125 | (6): TransformerConv(512, 512, heads=1) 126 | ) 127 | (jkn): JumpingKnowledge(max) 128 | (gate_nn): Sequential( 129 | (0): Linear(in_features=512, out_features=512, bias=True) 130 | (1): ReLU() 131 | (2): Linear(in_features=512, out_features=1, bias=True) 132 | ) 133 | (glob): MyGlobalAttention(gate_nn=Sequential( 134 | (0): Linear(in_features=512, out_features=512, bias=True) 135 | (1): ReLU() 136 | (2): Linear(in_features=512, out_features=1, bias=True) 137 | ), nn=None) 138 | (loss_function): MSELoss() 139 | (MLPs): ModuleDict( 140 | (perf): MLP( 141 | (activation): ELU(alpha=1.0) 142 | (layers): ModuleList( 143 | (0): Linear(in_features=512, out_features=256, bias=True) 144 | (1): Linear(in_features=256, out_features=128, bias=True) 145 | (2): Linear(in_features=128, out_features=64, bias=True) 146 | (3): Linear(in_features=64, out_features=32, bias=True) 147 | (4): Linear(in_features=32, out_features=16, bias=True) 148 | (5): Linear(in_features=16, out_features=1, bias=True) 149 | ) 150 | ) 151 | (util-LUT): MLP( 152 | (activation): ELU(alpha=1.0) 153 | (layers): ModuleList( 154 | (0): Linear(in_features=512, out_features=256, bias=True) 155 | (1): Linear(in_features=256, out_features=128, bias=True) 156 | (2): Linear(in_features=128, out_features=64, bias=True) 157 | (3): Linear(in_features=64, out_features=32, bias=True) 158 | (4): Linear(in_features=32, out_features=16, bias=True) 159 | (5): Linear(in_features=16, out_features=1, bias=True) 160 | ) 161 | ) 162 | (util-FF): MLP( 163 | (activation): ELU(alpha=1.0) 164 | (layers): ModuleList( 165 | (0): Linear(in_features=512, out_features=256, bias=True) 166 | (1): Linear(in_features=256, out_features=128, bias=True) 167 | (2): Linear(in_features=128, out_features=64, bias=True) 168 | (3): Linear(in_features=64, out_features=32, bias=True) 169 | (4): Linear(in_features=32, out_features=16, bias=True) 170 | (5): Linear(in_features=16, out_features=1, bias=True) 171 | ) 172 | ) 173 | (util-DSP): MLP( 174 | (activation): ELU(alpha=1.0) 175 | (layers): ModuleList( 176 | (0): Linear(in_features=512, out_features=256, bias=True) 177 | (1): Linear(in_features=256, out_features=128, bias=True) 178 | (2): Linear(in_features=128, out_features=64, bias=True) 179 | (3): Linear(in_features=64, out_features=32, bias=True) 180 | (4): Linear(in_features=32, out_features=16, bias=True) 181 | (5): Linear(in_features=16, out_features=1, bias=True) 182 | ) 183 | ) 184 | (util-BRAM): MLP( 185 | (activation): ELU(alpha=1.0) 186 | (layers): ModuleList( 187 | (0): Linear(in_features=512, out_features=256, bias=True) 188 | (1): Linear(in_features=256, out_features=128, bias=True) 189 | (2): Linear(in_features=128, out_features=64, bias=True) 190 | (3): Linear(in_features=64, out_features=32, bias=True) 191 | (4): Linear(in_features=32, out_features=16, bias=True) 192 | (5): Linear(in_features=16, out_features=1, bias=True) 193 | ) 194 | ) 195 | ) 196 | ) 197 | -------------------------------------------------------------------------------- /baselines/src/adapt.py: -------------------------------------------------------------------------------- 1 | from train import train, create_optimizer 2 | from test import test 3 | from config import FLAGS 4 | from saver import saver 5 | from data import split_li, create_dataloader 6 | 7 | from model import feature_extract 8 | from model_factory import create_model 9 | 10 | import torch 11 | from tqdm import tqdm 12 | from collections import OrderedDict, defaultdict 13 | import random 14 | import numpy as np 15 | import copy 16 | 17 | PLUSMINUS_SYMBOL = u"\u00B1" 18 | 19 | 20 | def adapt_main(dataset, pragma_dim): 21 | # The key characteristic of adaptation is that it is done per kernel, 22 | # i.e. for each new/unseen test kernel (inductive), adaptation is performed on it 23 | # by copying the model's weights and doing some adaptation. 24 | if FLAGS.tvt_split_by != 'kernels_inductive': 25 | saver.log_info(f'adapt_main: FLAGS.tvt_split_by != "kernels_inductive ({FLAGS.tvt_split_by})"') 26 | return 27 | 28 | dataloaders = _split_into_indiviual_dataloaders(dataset) 29 | 30 | eval_dict = OrderedDict() 31 | eval_dict['before_adaptation'] = OrderedDict() 32 | eval_dict['after_adaptation'] = OrderedDict() 33 | for gname, dls in dataloaders.items(): 34 | for k in eval_dict.keys(): 35 | eval_dict[k][gname] = [] 36 | for j, dl in enumerate(dls): 37 | eval_dict = _adapt_and_test(dl, pragma_dim, f'{gname}_{j}', eval_dict, gname) 38 | 39 | for key, eval_d in eval_dict.items(): 40 | saver.log_info(f'-' * 20) 41 | final_print_dict = OrderedDict() 42 | saver.log_info(f'{key}:') 43 | all_rmses = [] 44 | for gname, rmse_list in eval_d.items(): 45 | rmse_list_s = [f'{x:.4f}' for x in rmse_list] 46 | fps = _avg_plusminus_std_for_li(rmse_list) 47 | saver.log_info(f'\t{gname}: {rmse_list_s} --> {fps}') 48 | final_print_dict[gname] = fps 49 | all_rmses += rmse_list 50 | all_agg_key = f'all/aggregated({len(all_rmses)})' 51 | keys_to_use = [all_agg_key] + list(final_print_dict.keys()) 52 | final_print_dict[all_agg_key] = _avg_plusminus_std_for_li(all_rmses) 53 | ks = "\t".join(keys_to_use) 54 | vs = "\t".join([final_print_dict[k] for k in keys_to_use]) 55 | saver.log_info(f'Summary of adaptation for {key}:') 56 | saver.log_info(f'{ks}') 57 | saver.log_info(f'{vs}') 58 | saver.log_info(f'-' * 20) 59 | saver.log_info(f'Adaptation done') 60 | # exit() 61 | 62 | 63 | def _avg_plusminus_std_for_li(li): 64 | avg = np.mean(li) 65 | std = np.std(li) 66 | return f'{avg:.4f}{PLUSMINUS_SYMBOL}{std:.2f}' 67 | 68 | 69 | def _adapt_and_test(dl, pragma_dim, test_name, eval_dict, gname): 70 | model = create_model(dl.adaptation_dl, pragma_dim) 71 | 72 | if FLAGS.sequence_modeling and not FLAGS.finetune_bert: 73 | for name, param in model.bert_model.named_parameters(): 74 | param.requires_grad = False 75 | saver.log_info(f'No fine tune bert: Freezing param {name}') 76 | 77 | if FLAGS.load_model != None and FLAGS.load_model != 'None': 78 | model, _ = saver.load_trained_model(FLAGS.load_model, model) 79 | 80 | if FLAGS.feature_extract: 81 | feature_extract(model, 'MLPs', FLAGS.fix_gnn_layer) 82 | 83 | if len(eval_dict['before_adaptation'][gname]) == 0: # otherwise, no need to test the model again and again 84 | r_before_adapt = _test(model, dl, f'{test_name}_before') 85 | eval_dict['before_adaptation'][gname].append(r_before_adapt) 86 | 87 | if FLAGS.num_mini_epochs >= 1: 88 | model = _adapt(model, dl) 89 | r_after_adapt = _test(model, dl, f'{test_name}_after') 90 | eval_dict['after_adaptation'][gname].append(r_after_adapt) 91 | 92 | return eval_dict 93 | 94 | 95 | def _test(model, dl, test_name): 96 | testr, loss_dict, encode_loss, eval_df = test(dl.final_hold_out_dl, 'test', model, epoch=0, plot_test=True, 97 | csv_dict=None, 98 | data_dict=None, forward_pairwise=False, 99 | eval_pairwise=False, test_name=f'{test_name}_holdout') 100 | 101 | # saver.log_info(f'testr={testr}') 102 | # saver.log_info(f'loss_dict={loss_dict}') 103 | # saver.log_info(f'encode_loss={encode_loss}') 104 | # print('@@@', eval_df) 105 | # exit(-1) 106 | if FLAGS.task == 'regression': 107 | assert eval_df.iloc[-1]['target'] == 'tot/avg' 108 | assert FLAGS.task == 'regression', f'Other task has not been implemented {FLAGS.task}' 109 | rmse = eval_df.iloc[-1]['rmse'] 110 | rtn = rmse 111 | elif FLAGS.task == 'class': 112 | accuracy = eval_df['accuracy'] 113 | assert 0 <= accuracy <= 1 114 | rtn = accuracy 115 | else: 116 | raise NotImplementedError() 117 | 118 | 119 | return rtn 120 | 121 | 122 | def _adapt(model, dl): 123 | optimizer = create_optimizer(model) 124 | assert FLAGS.num_mini_epochs >= 1 125 | train_losses = [] 126 | model_best = None 127 | for epoch in range(FLAGS.num_mini_epochs): 128 | loss, loss_dict_train, gae_loss_train, num_data = train(epoch, model, dl.adaptation_dl, optimizer, 129 | forward_pairwise=False) 130 | if FLAGS.test_which_adapted_model == 'best_train' and (model_best is None or (train_losses and loss < min(train_losses))): 131 | print('FLAGS.test_which_adapted_model', FLAGS.test_which_adapted_model) 132 | model_best = copy.deepcopy(model) 133 | train_losses.append(loss) 134 | saver.log_info(f'Mini epoch {epoch}: loss={loss:.4f} num_data={num_data}') 135 | saver.log_info(f'min train loss at epoch: {train_losses.index(min(train_losses))}') 136 | if FLAGS.test_which_adapted_model == 'last_epoch': 137 | pass 138 | elif FLAGS.test_which_adapted_model == 'best_train': 139 | assert model_best is not None 140 | model = model_best # return the best model in training 141 | saver.log_info(f'Returning the best model according to train loss') 142 | else: 143 | raise NotImplementedError() 144 | return model 145 | 146 | 147 | class AdaptationDataLoaders(object): 148 | def __init__(self, final_hold_out_dl, adaptation_dl): 149 | self.final_hold_out_dl = final_hold_out_dl 150 | self.adaptation_dl = adaptation_dl 151 | 152 | 153 | def _split_into_indiviual_dataloaders(dataset): 154 | saver.log_info(f'Split the dataset by invidual kernels according to {FLAGS.test_kernels}') 155 | # train_file_li = [] 156 | test_kernel_li_map = defaultdict(list) 157 | file_li = dataset.processed_file_names 158 | for file in tqdm(file_li): 159 | data = torch.load(file) 160 | if data.gname in FLAGS.test_kernels: 161 | test_kernel_li_map[data.gname].append(file) 162 | # train_file_li.append(file) 163 | saver.log_info(f'-' * 20) 164 | 165 | # saver.log_info(f'Found {len(train_file_li)} files for training') 166 | 167 | def _shuffle_split_li_into_2_chunks(li_, ratio=None, num=None, force_shuffle=False): 168 | 169 | if FLAGS.shuffle or force_shuffle: 170 | random.Random(123).shuffle(li_) 171 | if ratio is not None: 172 | r1 = int(len(li_) * ratio) 173 | elif num is not None: 174 | r1 = num 175 | else: 176 | assert False 177 | lengths = [r1, len(li_) - r1] 178 | rtn = split_li(li_, lengths) 179 | return rtn 180 | 181 | dataloaders = OrderedDict() 182 | for gname, test_file_li in sorted(test_kernel_li_map.items()): 183 | li = _shuffle_split_li_into_2_chunks(test_file_li, ratio=FLAGS.test_holdout_ratio) 184 | saver.log_info( 185 | f'\tval/test: \t{gname} has {len(test_file_li)} designs in total; split into others {len(li[1])} and final test {len(li[0])}') 186 | final_hold_out_data_points = li[0] 187 | final_hold_out_dl = create_dataloader(final_hold_out_data_points, shuffle=False) 188 | 189 | assert FLAGS.repeat_times >= 1 190 | rest_for_choosing = li[1] 191 | saver.log_info(f'Samoling {FLAGS.adaptation_num_dp} out of {len(rest_for_choosing)} for {FLAGS.repeat_times} ' 192 | f'times to get adaptation data points') 193 | assert len(rest_for_choosing) >= FLAGS.adaptation_num_dp, \ 194 | f'Not enough rest designs to choose {len(rest_for_choosing)} < {FLAGS.adaptation_num_dp}' 195 | dataloaders[gname] = [] 196 | for i in range(FLAGS.repeat_times): 197 | li = _shuffle_split_li_into_2_chunks(rest_for_choosing, num=FLAGS.adaptation_num_dp, force_shuffle=True) 198 | adaptation_data_points = li[0] 199 | assert len(adaptation_data_points) == FLAGS.adaptation_num_dp 200 | adaptation_dl = create_dataloader(adaptation_data_points, shuffle=FLAGS.shuffle) 201 | 202 | dataloaders[gname].append(AdaptationDataLoaders(final_hold_out_dl, adaptation_dl)) 203 | 204 | return dataloaders 205 | -------------------------------------------------------------------------------- /baselines/src/aug21autodse/__pycache__/result.cpython-39.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UCLA-DM/HLSyn/88ed5321811e5c9d838215f0a4f95f89b3ffcd02/baselines/src/aug21autodse/__pycache__/result.cpython-39.pyc -------------------------------------------------------------------------------- /baselines/src/aug21autodse/result.py: -------------------------------------------------------------------------------- 1 | """ 2 | The definition of evaluation results 3 | """ 4 | 5 | from enum import Enum 6 | from typing import Dict, List, NamedTuple, Optional, Union 7 | 8 | 9 | DesignPoint = Dict[str, Union[int, str]] 10 | 11 | class Job(object): 12 | """The info and properties of a job""" 13 | 14 | class Status(Enum): 15 | INIT = 0 16 | APPLIED = 1 17 | 18 | def __init__(self, path: str): 19 | self.path: str = path 20 | self.key: str = 'NotAPPLIED' 21 | self.point: Optional[DesignPoint] = None 22 | self.status: Job.Status = Job.Status.INIT 23 | 24 | 25 | class Result(object): 26 | """The base module of evaluation result""" 27 | 28 | class RetCode(Enum): 29 | PASS = 0 30 | UNAVAILABLE = -1 31 | ANALYZE_ERROR = -2 32 | EARLY_REJECT = -3 33 | TIMEOUT = -4 34 | DUPLICATED = -5 35 | 36 | def __init__(self, ret_code_str: str = 'PASS'): 37 | 38 | # The design point of this result. 39 | self.point: Optional[DesignPoint] = None 40 | 41 | # The return code of the evaluation 42 | self.ret_code: Result.RetCode = self.RetCode[ret_code_str] 43 | 44 | # Indicate if this result is valid to be a final output. For example, a result that 45 | # out-of-resource is invalid. 46 | self.valid: bool = False 47 | 48 | # The job path for this result (if available) 49 | self.path: Optional[str] = None 50 | 51 | # The quantified QoR value. Larger the better. 52 | self.quality: float = -float('inf') 53 | 54 | # Performance in terms of estimated cycle or onboard runtime. 55 | self.perf: float = 0.0 56 | 57 | # Resource utilizations 58 | self.res_util: Dict[str, float] = { 59 | 'util-BRAM': 0, 60 | 'util-DSP': 0, 61 | 'util-LUT': 0, 62 | 'util-FF': 0, 63 | 'total-BRAM': 0, 64 | 'total-DSP': 0, 65 | 'total-LUT': 0, 66 | 'total-FF': 0 67 | } 68 | 69 | # Elapsed time for evaluation 70 | self.eval_time: float = 0.0 71 | 72 | 73 | class MerlinResult(Result): 74 | """The result after running Merlin transformations""" 75 | 76 | def __init__(self, ret_code_str: str = 'PASS'): 77 | super(MerlinResult, self).__init__(ret_code_str) 78 | 79 | # Critical messages from the Merlin transformations 80 | self.criticals: List[str] = [] 81 | 82 | # The kernel code hash for recognizing duplications 83 | self.code_hash: Optional[str] = None 84 | 85 | 86 | class HierPathNode(NamedTuple): 87 | """The datastructure of hierarchy path node""" 88 | nid: str 89 | latency: float 90 | is_compute_bound: bool 91 | 92 | 93 | class HLSResult(Result): 94 | """The result after running the HLS""" 95 | 96 | def __init__(self, ret_code_str: str = 'PASS'): 97 | super(HLSResult, self).__init__(ret_code_str) 98 | 99 | # A list of hierarchy paths in the order of importance 100 | self.ordered_paths: Optional[List[List[HierPathNode]]] = None 101 | 102 | 103 | class BitgenResult(Result): 104 | """The result after bit-stream generation""" 105 | 106 | def __init__(self, ret_code_str: str = 'PASS'): 107 | super(BitgenResult, self).__init__(ret_code_str) 108 | 109 | # Frequency 110 | self.freq: float = 0.0 111 | -------------------------------------------------------------------------------- /baselines/src/autodse/__pycache__/result.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UCLA-DM/HLSyn/88ed5321811e5c9d838215f0a4f95f89b3ffcd02/baselines/src/autodse/__pycache__/result.cpython-37.pyc -------------------------------------------------------------------------------- /baselines/src/autodse/__pycache__/result.cpython-39.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UCLA-DM/HLSyn/88ed5321811e5c9d838215f0a4f95f89b3ffcd02/baselines/src/autodse/__pycache__/result.cpython-39.pyc -------------------------------------------------------------------------------- /baselines/src/autodse/result.py: -------------------------------------------------------------------------------- 1 | """ 2 | The definition of evaluation results 3 | """ 4 | 5 | from enum import Enum 6 | from typing import Dict, List, NamedTuple, Optional, Union 7 | 8 | 9 | DesignPoint = Dict[str, Union[int, str]] 10 | 11 | class Job(object): 12 | """The info and properties of a job""" 13 | 14 | class Status(Enum): 15 | INIT = 0 16 | APPLIED = 1 17 | 18 | def __init__(self, path: str): 19 | self.path: str = path 20 | self.key: str = 'NotAPPLIED' 21 | self.point: Optional[DesignPoint] = None 22 | self.status: Job.Status = Job.Status.INIT 23 | 24 | 25 | class Result(object): 26 | """The base module of evaluation result""" 27 | 28 | class RetCode(Enum): 29 | PASS = 0 30 | UNAVAILABLE = -1 31 | ANALYZE_ERROR = -2 32 | EARLY_REJECT = -3 33 | TIMEOUT = -4 34 | DUPLICATED = -5 35 | 36 | def __init__(self, ret_code_str: str = 'PASS'): 37 | 38 | # The design point of this result. 39 | self.point: Optional[DesignPoint] = None 40 | 41 | # The return code of the evaluation 42 | self.ret_code: Result.RetCode = self.RetCode[ret_code_str] 43 | 44 | # Indicate if this result is valid to be a final output. For example, a result that 45 | # out-of-resource is invalid. 46 | self.valid: bool = False 47 | 48 | # The job path for this result (if available) 49 | self.path: Optional[str] = None 50 | 51 | # The quantified QoR value. Larger the better. 52 | self.quality: float = -float('inf') 53 | 54 | # Performance in terms of estimated cycle or onboard runtime. 55 | self.perf: float = 0.0 56 | 57 | # Resource utilizations 58 | self.res_util: Dict[str, float] = { 59 | 'util-BRAM': 0, 60 | 'util-DSP': 0, 61 | 'util-LUT': 0, 62 | 'util-FF': 0, 63 | 'total-BRAM': 0, 64 | 'total-DSP': 0, 65 | 'total-LUT': 0, 66 | 'total-FF': 0 67 | } 68 | 69 | # Elapsed time for evaluation 70 | self.eval_time: float = 0.0 71 | 72 | 73 | class MerlinResult(Result): 74 | """The result after running Merlin transformations""" 75 | 76 | def __init__(self, ret_code_str: str = 'PASS'): 77 | super(MerlinResult, self).__init__(ret_code_str) 78 | 79 | # Critical messages from the Merlin transformations 80 | self.criticals: List[str] = [] 81 | 82 | # The kernel code hash for recognizing duplications 83 | self.code_hash: Optional[str] = None 84 | 85 | 86 | class HierPathNode(NamedTuple): 87 | """The datastructure of hierarchy path node""" 88 | nid: str 89 | latency: float 90 | is_compute_bound: bool 91 | 92 | 93 | class HLSResult(Result): 94 | """The result after running the HLS""" 95 | 96 | def __init__(self, ret_code_str: str = 'PASS'): 97 | super(HLSResult, self).__init__(ret_code_str) 98 | 99 | # A list of hierarchy paths in the order of importance 100 | self.ordered_paths: Optional[List[List[HierPathNode]]] = None 101 | 102 | 103 | class BitgenResult(Result): 104 | """The result after bit-stream generation""" 105 | 106 | def __init__(self, ret_code_str: str = 'PASS'): 107 | super(BitgenResult, self).__init__(ret_code_str) 108 | 109 | # Frequency 110 | self.freq: float = 0.0 111 | -------------------------------------------------------------------------------- /baselines/src/check_model.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from utils import get_root_path 3 | from glob import iglob 4 | from os.path import join 5 | 6 | models_path = [f for f in iglob(join(get_root_path(), 'save/programl/**'), recursive = True) if f.endswith('.pth') and 'node-att' in f] 7 | 8 | for model in models_path: 9 | loaded_model = torch.load(model, map_location=torch.device('cpu')) 10 | #loaded_model.eval() 11 | print(model) 12 | for param in loaded_model: 13 | print(param) 14 | 15 | print() 16 | print('##########################################################') 17 | print() 18 | # break -------------------------------------------------------------------------------- /baselines/src/config_ds.py: -------------------------------------------------------------------------------- 1 | """ 2 | DSE config settings 3 | """ 4 | from typing import Any, Dict, Optional 5 | 6 | 7 | # All configurable attributes. Please follow the following rules if you want to add new config. 8 | # 1) Follow the naming: .. 9 | # 2) 'require' is necessary for every config. 10 | # 3) If the config is optional ('require' is False), then 'default' is necessary. 11 | # 4) If the config is limited to certain options, add 'options' to the config attribute. 12 | CONFIG_SETTING: Dict[str, Dict[str, Any]] = { 13 | 'project.name': { 14 | 'require': False, 15 | 'default': 'project' 16 | }, 17 | 'project.backup': { 18 | 'require': False, 19 | 'default': 'NO_BACKUP', 20 | 'options': ['NO_BACKUP', 'BACKUP_ERROR', 'BACKUP_ALL'] 21 | }, 22 | 'project.fast-output-num': { 23 | 'require': False, 24 | 'default': 4 25 | }, 26 | 'design-space.definition': { 27 | 'require': True 28 | }, 29 | 'design-space.max-part-num': { 30 | 'require': False, 31 | 'default': 4 32 | }, 33 | 'evaluate.worker-per-part': { 34 | 'require': False, 35 | 'default': 2 36 | }, 37 | 'evaluate.command.transform': { 38 | 'require': True, 39 | }, 40 | 'evaluate.command.hls': { 41 | 'require': True, 42 | }, 43 | 'evaluate.max-util.BRAM': { 44 | 'require': False, 45 | 'default': 0.8 46 | }, 47 | 'evaluate.max-util.DSP': { 48 | 'require': False, 49 | 'default': 0.8 50 | }, 51 | 'evaluate.max-util.LUT': { 52 | 'require': False, 53 | 'default': 0.8 54 | }, 55 | 'evaluate.max-util.FF': { 56 | 'require': False, 57 | 'default': 0.8 58 | }, 59 | 'evaluate.command.bitgen': { 60 | 'require': True, 61 | }, 62 | 'search.algorithm.name': { 63 | 'require': False, 64 | 'default': 'gradient', 65 | 'options': ['exhaustive', 'gradient', 'hybrid'] 66 | }, 67 | 'search.algorithm.exhaustive.batch-size': { 68 | 'require': False, 69 | 'default': 2 70 | }, 71 | 'search.algorithm.gradient.latency-threshold': { 72 | 'require': False, 73 | 'default': 64 74 | }, 75 | 'search.algorithm.gradient.fine-grained-first': { 76 | 'require': False, 77 | 'default': True 78 | }, 79 | 'search.algorithm.gradient.quality-type': { 80 | 'require': False, 81 | 'default': 'performance', 82 | 'options': ['finite-difference', 'performance', 'resource-efficiency'] 83 | }, 84 | 'search.algorithm.gradient.compute-bound-order': { 85 | 'require': False, 86 | 'default': ['PARALLEL', 'PIPELINE'] 87 | }, 88 | 'search.algorithm.gradient.memory-bound-order': { 89 | 'require': False, 90 | 'default': ['INTERFACE', 'CACHE', 'PIPELINE', 'TILE', 'TILING'] 91 | }, 92 | 'timeout.exploration': { 93 | 'require': True, 94 | }, 95 | 'timeout.transform': { 96 | 'require': True, 97 | }, 98 | 'timeout.hls': { 99 | 'require': True, 100 | }, 101 | 'timeout.bitgen': { 102 | 'require': True, 103 | } 104 | } 105 | 106 | 107 | def build_config(user_config: Dict[str, Any], log) -> Optional[Dict[str, Any]]: 108 | """Check user config and apply default value to optional configs. 109 | 110 | Args: 111 | user_config: The user config to be referred. 112 | 113 | Returns: 114 | A nested dict of configs, or None if there has any errors. 115 | """ 116 | 117 | 118 | # Check user config and make up optional values 119 | error = 0 120 | for key, attr in CONFIG_SETTING.items(): 121 | if key in user_config: 122 | if 'options' in attr: 123 | # Specified config, check if it is legal 124 | if user_config[key] not in attr['options']: 125 | log.error((f'"{user_config[key]}" is not a valid option for {key}')) 126 | error += 1 127 | else: 128 | # Missing config, check if it is optional (set to default if so) 129 | if attr['require']: 130 | log.error((f'Missing "{key}" in the config which is required')) 131 | error += 1 132 | else: 133 | d = attr['default'] 134 | log.info((f'Use default value for {key}: {str(d)}')) 135 | user_config[key] = d 136 | 137 | for key in user_config.keys(): 138 | if key not in CONFIG_SETTING: 139 | log.error((f'Unrecognized config key: {key}')) 140 | error += 1 141 | 142 | if error > 0: 143 | return None 144 | 145 | # Build config 146 | config: Dict[str, Any] = {} 147 | for key, attr in user_config.items(): 148 | curr = config 149 | levels = key.split('.') 150 | for level in levels[:-1]: 151 | if level not in curr: 152 | curr[level] = {} 153 | curr = curr[level] 154 | curr[levels[-1]] = attr 155 | 156 | return config 157 | -------------------------------------------------------------------------------- /baselines/src/convert_db_to_pkl.py: -------------------------------------------------------------------------------- 1 | ''' 2 | 3 | 05/29/2023: Convert db files to pickle files so that no need to set up redis on server. 4 | 5 | ''' 6 | from utils import get_root_path, save_pickle 7 | from glob import glob 8 | from tqdm import tqdm 9 | from os.path import basename, dirname 10 | import redis, pickle 11 | from collections import OrderedDict 12 | 13 | db_files = glob(f'/{get_root_path()}/dse_database/**/*.db', recursive=True) 14 | database = redis.StrictRedis(host='localhost', port=6379) 15 | database.flushdb() 16 | 17 | for db_file in tqdm(db_files): 18 | print(db_file) 19 | bn = basename(db_file) 20 | assert '.db' in bn 21 | fnb = bn.split('.db')[0] 22 | new_file = f'{dirname(db_file)}/{fnb}.pkl' 23 | print(new_file) 24 | 25 | database.flushdb() 26 | 27 | d = OrderedDict() 28 | try: 29 | with open(db_file, 'rb') as f_db: 30 | database.hmset(0, pickle.load(f_db)) 31 | keys = [k.decode('utf-8') for k in database.hkeys(0)] 32 | print(len(keys)) 33 | for key in sorted(keys): 34 | # obj = pickle.loads(database.hget(0, key)) 35 | d[key] = database.hget(0, key) 36 | # print() 37 | save_pickle(d, new_file, print_msg=True) 38 | except Exception as e: 39 | print(e) 40 | 41 | -------------------------------------------------------------------------------- /baselines/src/data_analysis.py: -------------------------------------------------------------------------------- 1 | from model import Net, get_y_with_target 2 | from config import FLAGS 3 | import torch 4 | from torch_geometric.data import DataLoader 5 | from saver import saver 6 | 7 | from os.path import join 8 | import pandas as pd 9 | from sklearn.manifold import TSNE 10 | import matplotlib 11 | 12 | matplotlib.use('pdf') 13 | import matplotlib.pyplot as plt 14 | import seaborn as sns 15 | 16 | import learn2learn as l2l 17 | import numpy as np 18 | from collections import defaultdict 19 | 20 | from tqdm import tqdm 21 | 22 | from torch_scatter import scatter_add 23 | 24 | 25 | def analyze_data(dataset): 26 | setattr(FLAGS, 'target', ['perf', 'util-DSP', 'util-BRAM', 'util-LUT', 'util-FF']) 27 | # model = Net(dataset.num_features).to(FLAGS.device) 28 | # print(model) 29 | # print(model.state_dict().keys()) 30 | # assert FLAGS.load_model != 'None' 31 | # if not FLAGS.only_pragma_nodes: 32 | # saver.log_info(f'Loading model: {FLAGS.load_model}') 33 | # ld = torch.load(FLAGS.load_model, map_location=FLAGS.device) 34 | # 35 | # if FLAGS.MAML and FLAGS.learning_algorithm == 'MAML': 36 | # model = l2l.algorithms.MAML(model, lr=FLAGS.fast_lr, first_order=False) 37 | # 38 | # model.load_state_dict(ld) 39 | # saver.log_info('Model loaded') 40 | # else: 41 | # saver.log_info('Skipping loading model since only pragma nodes') 42 | 43 | setattr(FLAGS, 'target', 44 | ['perf', 'actual_perf', 'util_DSP', 'util_BRAM', 'util_LUT', 'util_FF']) 45 | designs_by_kernel = defaultdict(list) 46 | for i, data in enumerate(tqdm(dataset)): 47 | for target in FLAGS.target: 48 | setattr(data, target, getattr(data, target).item()) 49 | designs_by_kernel[data.gname].append(data) 50 | 51 | for kernel in ['spmv-crs', 'spmv-ellpack']: 52 | designs = designs_by_kernel[kernel] 53 | saver.log_info(f'Found {len(designs)} designs for kernel {kernel}') 54 | designs.sort(key=lambda x: x.perf, reverse=True) 55 | for j, d in enumerate(designs): 56 | saver.log_info(f'\t{j}: {d}') 57 | if j == 3: 58 | break 59 | -------------------------------------------------------------------------------- /baselines/src/data_code2vec.py: -------------------------------------------------------------------------------- 1 | from config import FLAGS 2 | from saver import saver 3 | from utils import get_root_path, load_pickle 4 | from torch_geometric.data import Data 5 | import torch 6 | from collections import defaultdict 7 | from os.path import join 8 | import numpy as np 9 | 10 | 11 | class Code2VecData(object): 12 | pass 13 | 14 | 15 | did_to_emb_dict = {} 16 | did_to_point_dict = {} 17 | kernel_design_lookup_dict = defaultdict(dict) 18 | 19 | miss_did_cnt = 0 20 | 21 | 22 | def _load_code2vec_data(): 23 | global did_to_emb_dict 24 | global did_to_point_dict 25 | global kernel_design_lookup_dict 26 | global miss_did_cnt 27 | if FLAGS.v2_db: 28 | ver = 'v20' 29 | else: 30 | ver = 'v18' 31 | 32 | if not did_to_emb_dict or not did_to_point_dict: 33 | p = join(get_root_path(), 'file', 'code2vec', FLAGS.code2vec_data_version, f'{ver}-emb', 'emb.pkl') 34 | did_to_emb_dict = load_pickle(p) 35 | if not did_to_emb_dict: 36 | raise FileNotFoundError(f'Cannot load did_to_emb_dict from {p}; double check the path') 37 | p = join(get_root_path(), 'file', 'code2vec', FLAGS.code2vec_data_version, f'{ver}-emb', 'fname_to_config.pkl') 38 | did_to_point_dict = load_pickle(p) 39 | if not did_to_point_dict: 40 | raise FileNotFoundError(f'Cannot load did_to_point_dict from {p}; double check the path') 41 | if len(did_to_emb_dict) != len(did_to_point_dict): 42 | saver.log_info( 43 | f'len(did_to_emb_dict)={len(did_to_emb_dict)} != len(did_to_point_dict)={len(did_to_point_dict)}') 44 | for did, emb in did_to_emb_dict.items(): 45 | spt = did.split('_') 46 | gname, _ = spt[0], spt[1] 47 | did = did.split('.c')[0] 48 | point = did_to_point_dict.get(did) 49 | if point is None: 50 | miss_did_cnt += 1 51 | saver.log_info(f'miss_did_cnt={miss_did_cnt} {did}!') 52 | else: 53 | kernel_design_lookup_dict[gname][str(point)] = emb 54 | 55 | saver.log_info( 56 | f'Loaded {len(kernel_design_lookup_dict)} kernels\'s {len(did_to_emb_dict)} designs from disk: {kernel_design_lookup_dict.keys()}') 57 | 58 | 59 | def init_preprocessors_code2vec(): 60 | return {'encoders': {}} 61 | 62 | 63 | def read_code2vec_data(gexf_file): 64 | return Code2VecData() 65 | 66 | 67 | def encode_feat_dict_code2vec(mmo, preprocessors, point=None): 68 | return {}, {} 69 | 70 | 71 | def fit_preprocessors_code2vec(preprocessors): 72 | pass 73 | 74 | 75 | not_found_cnt = 0 76 | 77 | 78 | def encode_X_torch_code2vec(g, d_node, d_edge, preprocessors, gname, vname, return_Data=True): 79 | global not_found_cnt 80 | _load_code2vec_data() 81 | 82 | point_this_design = str(d_node['point']) 83 | emb = kernel_design_lookup_dict[gname].get(point_this_design) 84 | if emb is None: 85 | not_found_cnt += 1 86 | saver.log_info(f'not_found_cnt={not_found_cnt} {gname}\'s design {point_this_design} not found in code2vec!') 87 | emb = np.ones((384,)) 88 | assert emb.shape == (384,) 89 | emb = torch.tensor(emb).view(1, -1) 90 | 91 | if return_Data: 92 | d_node.pop('point') 93 | 94 | if FLAGS.task == 'regression': 95 | return Data( 96 | gname=gname, 97 | key=vname, 98 | 99 | x=emb, 100 | 101 | perf=d_node['perf'], 102 | actual_perf=d_node['actual_perf'], 103 | kernel_speedup=d_node['kernel_speedup'], 104 | quality=d_node['quality'], 105 | util_BRAM=d_node['util-BRAM'], 106 | util_DSP=d_node['util-DSP'], 107 | util_LUT=d_node['util-LUT'], 108 | util_FF=d_node['util-FF'], 109 | total_BRAM=d_node['total-BRAM'], 110 | total_DSP=d_node['total-DSP'], 111 | total_LUT=d_node['total-LUT'], 112 | total_FF=d_node['total-FF'], 113 | ) 114 | elif FLAGS.task == 'class': 115 | return Data( 116 | gname=gname, 117 | key=vname, 118 | 119 | x=emb, 120 | 121 | perf=d_node['perf'], 122 | ) 123 | else: 124 | raise NotImplementedError() 125 | 126 | 127 | 128 | 129 | else: 130 | assert False 131 | -------------------------------------------------------------------------------- /baselines/src/database.py: -------------------------------------------------------------------------------- 1 | ''' 2 | 3 | 05/29/2023: A database object to use either the redis database or a custom object. 4 | 5 | ''' 6 | 7 | from config import FLAGS 8 | from saver import saver 9 | from collections import OrderedDict 10 | 11 | class OurDatabase(object): 12 | def __int__(self): 13 | self.flushdb() 14 | 15 | def flushdb(self): 16 | self.d = {} 17 | self.d_decoded = {} 18 | 19 | def hmset(self, n, data): 20 | self.d[n] = data 21 | new_data = OrderedDict() 22 | assert type(data) is dict 23 | for k, v in data.items(): 24 | new_data[k.decode('utf-8')] = v 25 | self.d_decoded[n] = new_data 26 | 27 | def hkeys(self, n): 28 | if n not in self.d: 29 | return [] 30 | return list(self.d[n].keys()) 31 | 32 | def hget(self, n, key): 33 | try: 34 | return self.d_decoded[n][key] 35 | except KeyError as e: 36 | exit() 37 | 38 | 39 | def create_database(): 40 | if FLAGS.use_redis: 41 | import redis 42 | rtn = redis.StrictRedis(host='localhost', port=6379) 43 | saver.log_info(f'Connection established to port 6379') 44 | return rtn 45 | else: 46 | return OurDatabase() 47 | -------------------------------------------------------------------------------- /baselines/src/early_stopping.py: -------------------------------------------------------------------------------- 1 | ''' 2 | From https://gist.github.com/stefanonardo/693d96ceb2f531fa05db530f3e21517d 3 | ''' 4 | 5 | 6 | # MIT License 7 | # 8 | # Copyright (c) 2018 Stefano Nardo https://gist.github.com/stefanonardo 9 | # 10 | # Permission is hereby granted, free of charge, to any person obtaining a copy 11 | # of this software and associated documentation files (the "Software"), to deal 12 | # in the Software without restriction, including without limitation the rights 13 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 14 | # copies of the Software, and to permit persons to whom the Software is 15 | # furnished to do so, subject to the following conditions: 16 | # 17 | # The above copyright notice and this permission notice shall be included in all 18 | # copies or substantial portions of the Software. 19 | # 20 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 21 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 22 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 23 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 24 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 25 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 26 | # SOFTWARE. 27 | 28 | from saver import saver 29 | 30 | class EarlyStopping(object): 31 | def __init__(self, mode='min', min_delta=0, patience=10, percentage=False): 32 | self.mode = mode 33 | self.min_delta = min_delta 34 | self.patience = patience 35 | self.best = None 36 | self.num_bad_epochs = 0 37 | self.is_better = None 38 | self._init_is_better(mode, min_delta, percentage) 39 | 40 | if patience == 0: 41 | self.is_better = lambda a, b: True 42 | self.step = lambda a: False 43 | 44 | def step(self, metrics): 45 | if self.best is None: 46 | self.best = metrics 47 | saver.log_info(f'early stopper: self.best is None') 48 | return False 49 | 50 | # if torch.isnan(metrics): 51 | # return True 52 | 53 | if self.is_better(metrics, self.best): 54 | saver.log_info(f'early stopper: is_better {metrics:.4f} {self.best:.4f}') 55 | 56 | self.num_bad_epochs = 0 57 | self.best = metrics 58 | else: 59 | saver.log_info(f'early stopper: NOT is_better {metrics:.4f} {self.best:.4f} ' 60 | f'self.num_bad_epochs={self.num_bad_epochs}') 61 | 62 | self.num_bad_epochs += 1 63 | 64 | if self.num_bad_epochs >= self.patience: 65 | saver.log_info(f'early stopper: triggered!' 66 | f' {self.num_bad_epochs} >= {self.patience}') 67 | 68 | return True 69 | 70 | return False 71 | 72 | def _init_is_better(self, mode, min_delta, percentage): 73 | if mode not in {'min', 'max'}: 74 | raise ValueError('mode ' + mode + ' is unknown!') 75 | if not percentage: 76 | if mode == 'min': 77 | self.is_better = lambda a, best: a < best - min_delta 78 | if mode == 'max': 79 | self.is_better = lambda a, best: a > best + min_delta 80 | else: 81 | if mode == 'min': 82 | self.is_better = lambda a, best: a < best - ( 83 | best * min_delta / 100) 84 | if mode == 'max': 85 | self.is_better = lambda a, best: a > best + ( 86 | best * min_delta / 100) -------------------------------------------------------------------------------- /baselines/src/link_graph.py: -------------------------------------------------------------------------------- 1 | from utils import get_root_path 2 | from saver import saver 3 | 4 | from os.path import basename 5 | from pathlib import Path 6 | 7 | def find_ast(programl_gexf_path): 8 | bn = basename(programl_gexf_path) 9 | source_path = Path(f'{get_root_path()}/dse_database/machsuite/dot-files/original-size') 10 | # source_path = Path(f'{get_root_path()}/dse_database/machsuite/sources/original-size') 11 | source_path_polly = Path(f'{get_root_path()}/dse_database/polly/dot-files') 12 | 13 | if 'gemm-blocked' in bn: 14 | return Path(f'{source_path}/gemm-blocked_kernel.c.gexf') 15 | elif 'gemm-ncubed' in bn: 16 | return Path(f'{source_path}/gemm-ncubed_kernel.c.gexf') 17 | elif 'stencil_stencil2d' in bn: 18 | return Path(f'{source_path}/stencil_kernel.c.gexf') 19 | elif 'aes' in bn: 20 | return Path(f'{source_path}/aes_kernel.c.gexf') 21 | elif 'nw' in bn: 22 | return Path(f'{source_path}/nw_kernel.c.gexf') 23 | elif 'spmv-crs' in bn: 24 | return Path(f'{source_path}/spmv-crs_kernel.c.gexf') 25 | elif 'spmv-ellpack' in bn: 26 | return Path(f'{source_path}/spmv-ellpack_kernel.c.gexf') 27 | elif 'atax' in bn: 28 | return Path(f'{source_path_polly}/atax_kernel.c.gexf') 29 | elif 'mvt' in bn: 30 | return Path(f'{source_path_polly}/mvt_kernel.c.gexf') 31 | else: 32 | saver.log_info(f'Cannot find ast gexf for {programl_gexf_path}') 33 | return None -------------------------------------------------------------------------------- /baselines/src/main.py: -------------------------------------------------------------------------------- 1 | #!/opt/conda/bin/python 2 | from config import FLAGS 3 | from data import get_data_list, MyOwnDataset 4 | from data import update_save_dir 5 | import data 6 | from data_src_code import update_tokenizer 7 | from train import train_main 8 | from test import inference 9 | from adapt import adapt_main 10 | from saver import saver 11 | from utils import OurTimer, load_replace_flags, load 12 | from os.path import join, basename 13 | 14 | import torch, numpy as np, random 15 | import traceback 16 | 17 | if FLAGS.fix_randomness: 18 | saver.log_info('Critical! Fix random seed for torch and numpy') 19 | torch.manual_seed(FLAGS.random_seed) 20 | np.random.seed(FLAGS.random_seed) 21 | random.seed(FLAGS.random_seed) 22 | 23 | torch.backends.cudnn.deterministic = True 24 | torch.backends.cudnn.benchmark = False 25 | 26 | torch.cuda.manual_seed(FLAGS.random_seed) 27 | 28 | from torch import softmax 29 | 30 | import config 31 | 32 | TARGETS = config.TARGETS 33 | MACHSUITE_KERNEL = config.MACHSUITE_KERNEL 34 | poly_KERNEL = config.poly_KERNEL 35 | 36 | 37 | class HandleNodeAttention(object): 38 | def __call__(self, data): 39 | data.attn = softmax(data.x[:, 0], dim=0) 40 | data.x = data.x[:, 1:] 41 | return data 42 | 43 | 44 | timer = OurTimer() 45 | 46 | 47 | def main(): 48 | 49 | if FLAGS.load_model != 'None': 50 | saver.log_info(f'Loading model\'s config/FLAGS: {FLAGS.load_model}') 51 | load_replace_flags(FLAGS) 52 | saver.log_new_FLAGS_to_model_info() 53 | update_save_dir(FLAGS) 54 | update_tokenizer() 55 | print(basename(saver.logdir)) 56 | 57 | if not FLAGS.force_regen: 58 | dataset = MyOwnDataset() 59 | pragma_dim = load(join(data.SAVE_DIR, 'pragma_dim')) 60 | print('read dataset') 61 | else: 62 | dataset, pragma_dim = get_data_list() 63 | saver.log_info(f'pragma_dim: {pragma_dim}') 64 | 65 | if len(dataset) == 0: 66 | raise ValueError('Empty dataset! Check config.py; Maybe use force_regen') 67 | saver.log_info(f'Dataset contains {len(dataset)} designs ') 68 | 69 | saver.log_info(f'dataset[0].num_features={dataset[0].num_features}') 70 | 71 | 72 | # if FLAGS.task == 'regression': 73 | if FLAGS.subtask == 'inference': 74 | if FLAGS.adaptation_needed: 75 | adapt_main(dataset, pragma_dim) 76 | inference(dataset, pragma_dim) 77 | else: 78 | train_main(dataset, pragma_dim) 79 | 80 | 81 | if __name__ == '__main__': 82 | 83 | timer = OurTimer() 84 | 85 | try: 86 | main() 87 | status = 'Complete' 88 | except: 89 | traceback.print_exc() 90 | s = '\n'.join(traceback.format_exc(limit=-1).split('\n')[1:]) 91 | saver.log_info(traceback.format_exc(), silent=True) 92 | saver.save_exception_msg(traceback.format_exc()) 93 | status = 'Error' 94 | 95 | tot_time = timer.time_and_clear() 96 | saver.log_info(f'Total time: {tot_time}') 97 | saver.close() 98 | -------------------------------------------------------------------------------- /baselines/src/model_code2vec.py: -------------------------------------------------------------------------------- 1 | from model import Model 2 | from config import FLAGS 3 | import torch 4 | 5 | from collections import OrderedDict 6 | 7 | 8 | class Code2VecNet(Model): 9 | def __init__(self, init_pragma_dict=None, dataset=None): 10 | super(Code2VecNet, self).__init__() 11 | self.task = FLAGS.task 12 | 13 | self.out_dim, self.loss_function = self._create_loss() 14 | 15 | self.target_list = self._get_target_list() 16 | 17 | self.decoder, _ = self._create_decoder_MLPs(384, 384, self.target_list, 18 | self.out_dim, hidden_channels=None) 19 | 20 | def forward(self, data, forward_pairwise, tvt=None, epoch=None, iter=None, test_name=None): 21 | total_loss = torch.tensor(0.0, device=FLAGS.device) 22 | out_dict = OrderedDict() 23 | loss_dict = OrderedDict() 24 | 25 | self._apply_target_MLPs_with_loss(self.decoder, data.x.float(), data, total_loss, out_dict, 26 | loss_dict, 27 | 'normal') 28 | 29 | return out_dict, total_loss, loss_dict, torch.tensor(0.0) 30 | -------------------------------------------------------------------------------- /baselines/src/model_factory.py: -------------------------------------------------------------------------------- 1 | from config import FLAGS 2 | from model import Net 3 | from model_multi_modality import MultiModalityNet 4 | from model_code2vec import Code2VecNet 5 | from pretrained_GNN_handler import create_and_load_pretrained_GNN 6 | 7 | 8 | def create_model(data_loader, pragma_dim): 9 | # saver.log_info(f'edge_dim={edge_dim}') 10 | # if FLAGS.dataset == 'simple-programl' or FLAGS.target_kernel is not None: 11 | # init_pragma_dict = {'all': [1, 21]} 12 | if FLAGS.model == 'code2vec': 13 | c = Code2VecNet 14 | else: 15 | assert FLAGS.model == 'our' 16 | if FLAGS.multi_modality: 17 | c = MultiModalityNet 18 | else: 19 | c = Net 20 | model = c(init_pragma_dict=pragma_dim, dataset=data_loader.dataset).to( 21 | FLAGS.device) 22 | if FLAGS.load_pretrained_GNN: 23 | model.pretrained_GNN_encoder = create_and_load_pretrained_GNN().to(FLAGS.device) 24 | return model 25 | -------------------------------------------------------------------------------- /baselines/src/model_node_token_interact.py: -------------------------------------------------------------------------------- 1 | from utils_nn import create_graph_att_module, LpModule 2 | from config import FLAGS 3 | from our_transformer_conv import OurTransformerConv 4 | from utils import MLP 5 | from saver import saver 6 | 7 | from torch_geometric.nn.norm import GraphNorm 8 | 9 | import torch 10 | import torch.nn as nn 11 | 12 | 13 | def create_node_token_interaction_module(D, code_encoder_num_layers): 14 | def _create_bridge_GNN(): 15 | return OurTransformerConv(D, D, edge_dim=edge_dim, root_weight=False) 16 | 17 | if FLAGS.pc_links_aug in ['all_line_sw', 's_grease']: 18 | edge_dim = 2 19 | elif FLAGS.pc_links_aug == 'all_line_swp': 20 | edge_dim = 3 21 | elif FLAGS.pc_links_aug == 'all_line_swp_grease': 22 | edge_dim = 4 23 | else: 24 | edge_dim = None 25 | 26 | if FLAGS.interleave_GNN_transformer and not FLAGS.weight_tying_for_interaction_GNN: 27 | bridge_GNN_layers = nn.ModuleList() 28 | num_bridge_GNNs = code_encoder_num_layers 29 | if FLAGS.interact_after_all_layers: 30 | num_bridge_GNNs += 1 31 | for _ in range(num_bridge_GNNs): 32 | bridge_GNN_layers.append(_create_bridge_GNN()) 33 | return bridge_GNN_layers 34 | else: 35 | return _create_bridge_GNN() 36 | 37 | 38 | def _get_bridge_GNN_layer(nti_objs, layer_id): 39 | if FLAGS.interleave_GNN_transformer and not FLAGS.weight_tying_for_interaction_GNN: 40 | assert 0 <= layer_id < len(nti_objs['node_token_interaction_GNN']) 41 | return nti_objs['node_token_interaction_GNN'][layer_id] # index into a list of modules 42 | else: 43 | return nti_objs['node_token_interaction_GNN'] # just one module 44 | 45 | 46 | def create_collaboration(D, code_encoder_num_layers): 47 | if FLAGS.interleave_GNN_transformer and not FLAGS.weight_tying_for_interaction_GNN: 48 | collaboration_layers = nn.ModuleList() 49 | num_collab_layers = code_encoder_num_layers 50 | if FLAGS.interact_after_all_layers: 51 | num_collab_layers += 1 52 | for _ in range(num_collab_layers): 53 | collaboration_layers.append(create_graph_att_module(D)) 54 | return collaboration_layers 55 | else: 56 | return create_graph_att_module(D) 57 | 58 | 59 | def _get_collaboration_layer(nti_objs, layer_id): 60 | if FLAGS.interleave_GNN_transformer and not FLAGS.weight_tying_for_interaction_GNN: 61 | assert 0 <= layer_id < len(nti_objs['node_token_collaboration']) == len(nti_objs['node_token_interaction_GNN']) 62 | return nti_objs['node_token_collaboration'][layer_id] # index into a list of modules 63 | else: 64 | return nti_objs['node_token_collaboration'] # just one module 65 | 66 | 67 | def create_graphnorm_layers(D, code_encoder_num_layers): 68 | def _get_norm(norm_name, dim): 69 | if norm_name == 'layer': 70 | return torch.nn.LayerNorm(dim) 71 | elif norm_name == 'graph': 72 | return GraphNorm(dim) 73 | elif norm_name == 'L2': 74 | return LpModule(2) 75 | elif norm_name == 'L1': 76 | return LpModule(1) 77 | else: 78 | raise NotImplementedError(f'{norm_name}') 79 | 80 | rtn = nn.ModuleList() 81 | use = code_encoder_num_layers 82 | if FLAGS.interact_after_all_layers: 83 | use = code_encoder_num_layers + 1 84 | for _ in range(use): 85 | rtn.append(_get_norm(FLAGS.emb_norm_method, D)) 86 | return rtn 87 | 88 | 89 | def _get_graphnorm_layer(nti_objs, layer_id): 90 | assert 0 <= layer_id < len(nti_objs['graphnorm_layers_collab']) 91 | return nti_objs['graphnorm_layers_collab'][layer_id] # index into a list of modules 92 | 93 | 94 | def create_mlp_afterwards_layers(D, code_encoder_num_layers): 95 | rtn = nn.ModuleList() 96 | use = code_encoder_num_layers 97 | if FLAGS.interact_after_all_layers: 98 | use = code_encoder_num_layers + 1 99 | for _ in range(use): 100 | rtn.append(MLP(D, D, activation_type=FLAGS.activation_type, num_hidden_lyr=2)) 101 | return rtn 102 | 103 | 104 | def _get_mlp_afterwards_layer(nti_objs, layer_id): 105 | assert 0 <= layer_id < len(nti_objs['mlp_layers_afterwards']) 106 | return nti_objs['mlp_layers_afterwards'][layer_id] # index into a list of modules 107 | 108 | 109 | def interact_the_two_modalities(inputs_embeds, nti_objs, layer_id): 110 | ''' 111 | Handles the interaction between programl graph and source code sequence. 112 | Perform intra-graph GNN for programl and interaction across the two. 113 | The transformer applied to source code sequence is done by the caller of this function in our_codet5. 114 | ''' 115 | if FLAGS.code_encoder == 'codet5': 116 | assert 0 <= layer_id <= 6 117 | else: 118 | raise NotImplementedError() 119 | 120 | if hasattr(FLAGS, 'node_token_interaction') and FLAGS.node_token_interaction: 121 | 122 | # Run (intra-modality/programl) GNN (possibly). 123 | if FLAGS.interleave_GNN_transformer and layer_id > 0: # no need to run GNN for the very initial interaction 124 | 125 | if 'outs' not in nti_objs: 126 | nti_objs['outs'] = [] 127 | nti_objs['outs'].append(nti_objs['out_programl']) 128 | 129 | assert FLAGS.graph_transformer_option is None, 'not supported yet' 130 | GNN_layer_id = layer_id - 1 131 | GNN_layer = nti_objs['interleaved_GNN_layers'][GNN_layer_id] 132 | nti_objs['out_programl'] = GNN_layer( 133 | nti_objs['out_programl'], edge_index=nti_objs['edge_index_programl'], 134 | edge_attr=nti_objs['edge_attr_programl']) 135 | 136 | if GNN_layer_id != len(nti_objs['interleaved_GNN_layers']) - 1: 137 | nti_objs['out_programl'] = nti_objs['act'](nti_objs['out_programl']) 138 | nti_objs['outs'].append(nti_objs['out_programl']) 139 | 140 | # Interact (possibly). 141 | if layer_id == 0: 142 | do_interaction = True 143 | else: 144 | if FLAGS.interleave_GNN_transformer: 145 | do_interaction = True 146 | if layer_id == 6: 147 | # Special for after the last layer. 148 | if FLAGS.jkn_enable: 149 | # This line updates out_programl which will be used by the outside. 150 | nti_objs['out_programl'] = nti_objs['jkn_programl_interleave'](nti_objs['outs']) 151 | do_interaction = FLAGS.interact_after_all_layers 152 | else: 153 | do_interaction = False 154 | if do_interaction: 155 | inputs_embeds = _forward_node_token_interaction(inputs_embeds, nti_objs, 156 | layer_id) # TODO: both directions msg passing --> ie update nti_objs['out_programl']; use the right bridge-GNN; TODO: apply proj MLP to node & token embds for alignment????? check final alignment loss for which node embeddings are taken 157 | 158 | return inputs_embeds 159 | 160 | 161 | def _forward_node_token_interaction(inputs_embeds_src_code, nti_objs, layer_id): 162 | out_programl = nti_objs['out_programl'] 163 | num_chunks, num_tokens, embed_dim = inputs_embeds_src_code.shape 164 | inputs_embeds_src_code_flat = inputs_embeds_src_code.view(num_chunks * num_tokens, embed_dim) 165 | 166 | num_nodes_total = out_programl.shape[0] 167 | # num_tokens_total = inputs_embeds_src_code_flat.shape[0] 168 | x = torch.cat((out_programl, inputs_embeds_src_code_flat), dim=0) 169 | 170 | edge_msg_batch = None 171 | if (FLAGS.pc_links_holdout_ratio > 0 and nti_objs['training']) or \ 172 | (hasattr(FLAGS, 'pc_links_force_use_train_when_inference') 173 | and FLAGS.pc_links_force_use_train_when_inference): 174 | pos_node_ids = nti_objs['data'].pos_node_ids_train 175 | pos_token_global_ids_in_chunks = nti_objs['data'].pos_token_global_ids_in_chunks_train 176 | if hasattr(FLAGS, 'collaboration_btw_modalities') and FLAGS.collaboration_btw_modalities == 'edge_msgs': 177 | edge_msg_batch = nti_objs['data'].pos_node_ids_train_batch 178 | saver.log_info_once(f'model_node_token_interact.py: Use pos_token_global_ids_in_chunks_train') 179 | else: 180 | pos_node_ids = nti_objs['data'].pos_node_ids 181 | pos_token_global_ids_in_chunks = nti_objs['data'].pos_token_global_ids_in_chunks 182 | if hasattr(FLAGS, 'collaboration_btw_modalities') and FLAGS.collaboration_btw_modalities == 'edge_msgs': 183 | edge_msg_batch = nti_objs['data'].pos_node_ids_batch 184 | node_ids = pos_node_ids.view(1, -1) 185 | # Because cat (, ) (see above), need to increment the token ids by number of nodes 186 | # in order to get the right id. 187 | shifted_token_ids = pos_token_global_ids_in_chunks.view(1, -1) + num_nodes_total 188 | edge_index = torch.cat((node_ids, shifted_token_ids), dim=0) # from node --> token; no other direction needed 189 | 190 | # edge_index = torch.LongTensor([(0, 1)]).t().contiguous() 191 | if FLAGS.pc_links_aug in ['all_line_sw', 'all_line_swp', 'all_line_swp_grease', 's_grease']: 192 | if nti_objs['training'] and FLAGS.pc_links_holdout_ratio > 0: 193 | edge_attr = nti_objs['data'].pos_token_sw_type_in_chunks_train 194 | else: 195 | edge_attr = nti_objs['data'].pos_token_sw_type_in_chunks 196 | else: 197 | edge_attr = None 198 | 199 | if FLAGS.interleave_GNN_transformer: 200 | # Bi-directional information ezchange across the two modalities. 201 | edge_index_token_to_node = torch.cat((shifted_token_ids, node_ids), dim=0) 202 | edge_index = torch.cat((edge_index, edge_index_token_to_node), dim=1) 203 | if FLAGS.pc_links_aug not in ['grease', None, 'all_line']: 204 | edge_attr = torch.cat((edge_attr, edge_attr), dim=0) # edge for the other direction has the same edge type 205 | 206 | # Bridge-GNN. 207 | bridge_GNN = _get_bridge_GNN_layer(nti_objs, layer_id) 208 | x_output = bridge_GNN(x, edge_index, edge_attr=edge_attr) 209 | 210 | if hasattr(FLAGS, 'collaboration_btw_modalities') and FLAGS.collaboration_btw_modalities == 'edge_msgs': 211 | if 'edge_msgs_aggregated_list' not in nti_objs: 212 | nti_objs['edge_msgs_aggregated_list'] = [] 213 | edge_msgs = bridge_GNN.edge_msgs_saved_in_propagation 214 | assert edge_msgs.shape[1] == 1 215 | edge_msgs = edge_msgs.view(edge_msgs.shape[0], edge_msgs.shape[2]) 216 | collab_layer = _get_collaboration_layer(nti_objs, layer_id) 217 | if FLAGS.interleave_GNN_transformer: 218 | edge_msg_batch = torch.cat((edge_msg_batch, edge_msg_batch)) 219 | edge_msgs_aggregated, _ = collab_layer(edge_msgs, edge_msg_batch) 220 | nti_objs['edge_msgs_aggregated_list'].append(edge_msgs_aggregated) 221 | # The following two lines should return identical results, indicating how many tokens are actually updated. 222 | # set(nti_objs['data'].pos_token_global_ids_in_chunks.tolist()) 223 | # sum([1 if x > 0 else 0 for x in (x_output.numpy() != 0).sum(1)]) 224 | 225 | if hasattr(FLAGS, 'actually_aggregate') and not FLAGS.actually_aggregate: 226 | x_new = x 227 | else: 228 | x_new = x + x_output 229 | 230 | if FLAGS.interleave_GNN_transformer: 231 | x_programl = x_new[0:num_nodes_total] 232 | nti_objs['out_programl'] = x_programl # critical: need to let later layers use the updated node embeddings 233 | if FLAGS.apply_norm_after_interaction: 234 | graphnorm = _get_graphnorm_layer(nti_objs, layer_id) 235 | assert len(nti_objs['data'].x_programl_batch) == x_programl.shape[0] 236 | nti_objs['out_programl'] = graphnorm(nti_objs['out_programl'], batch=nti_objs['data'].x_programl_batch) 237 | if FLAGS.apply_mlp_after_interaction: 238 | mlp = _get_mlp_afterwards_layer(nti_objs, layer_id) 239 | out_programl_temp = mlp(nti_objs['out_programl']) 240 | nti_objs['out_programl'] = nti_objs[ 241 | 'out_programl'] + out_programl_temp # similar to feed forward in transformer 242 | 243 | x_src_code = x_new[num_nodes_total:] 244 | x_src_code = x_src_code.view(num_chunks, num_tokens, embed_dim) 245 | assert x_src_code.shape == inputs_embeds_src_code.shape 246 | return x_src_code 247 | -------------------------------------------------------------------------------- /baselines/src/nn.py: -------------------------------------------------------------------------------- 1 | 2 | import torch 3 | from torch_scatter import scatter_add 4 | from torch_geometric.utils import softmax 5 | 6 | from torch_geometric.nn.inits import reset 7 | 8 | 9 | class MyGlobalAttention(torch.nn.Module): 10 | r"""Global soft attention layer from the `"Gated Graph Sequence Neural 11 | Networks" `_ paper 12 | 13 | .. math:: 14 | \mathbf{r}_i = \sum_{n=1}^{N_i} \mathrm{softmax} \left( 15 | h_{\mathrm{gate}} ( \mathbf{x}_n ) \right) \odot 16 | h_{\mathbf{\Theta}} ( \mathbf{x}_n ), 17 | 18 | where :math:`h_{\mathrm{gate}} \colon \mathbb{R}^F \to 19 | \mathbb{R}` and :math:`h_{\mathbf{\Theta}}` denote neural networks, *i.e.* 20 | MLPS. 21 | 22 | Args: 23 | gate_nn (torch.nn.Module): A neural network :math:`h_{\mathrm{gate}}` 24 | that computes attention scores by mapping node features :obj:`x` of 25 | shape :obj:`[-1, in_channels]` to shape :obj:`[-1, 1]`, *e.g.*, 26 | defined by :class:`torch.nn.Sequential`. 27 | nn (torch.nn.Module, optional): A neural network 28 | :math:`h_{\mathbf{\Theta}}` that maps node features :obj:`x` of 29 | shape :obj:`[-1, in_channels]` to shape :obj:`[-1, out_channels]` 30 | before combining them with the attention scores, *e.g.*, defined by 31 | :class:`torch.nn.Sequential`. (default: :obj:`None`) 32 | """ 33 | def __init__(self, gate_nn, nn=None): 34 | super(MyGlobalAttention, self).__init__() 35 | self.gate_nn = gate_nn 36 | self.nn = nn 37 | 38 | self.reset_parameters() 39 | 40 | def reset_parameters(self): 41 | reset(self.gate_nn) 42 | reset(self.nn) 43 | 44 | def forward(self, x, batch, set_zeros_ids=None, size=None): 45 | """""" 46 | x = x.unsqueeze(-1) if x.dim() == 1 else x 47 | size = batch[-1].item() + 1 if size is None else size 48 | 49 | gate = self.gate_nn(x).view(-1, 1) 50 | x = self.nn(x) if self.nn is not None else x 51 | assert gate.dim() == x.dim() and gate.size(0) == x.size(0) 52 | 53 | gate = softmax(gate, batch, num_nodes=size) 54 | 55 | if set_zeros_ids is not None: 56 | # x[set_zeros_ids] = 0 57 | # x.masked_fill(set_zeros_ids, 0) 58 | x = x.permute((1, 0)) 59 | x = x * set_zeros_ids 60 | x = x.permute((1, 0)) 61 | 62 | out = scatter_add(gate * x, batch, dim=0, dim_size=size) 63 | 64 | return out, gate 65 | 66 | def __repr__(self): 67 | return '{}(gate_nn={}, nn={})'.format(self.__class__.__name__, 68 | self.gate_nn, self.nn) 69 | 70 | -------------------------------------------------------------------------------- /baselines/src/nn_att.py: -------------------------------------------------------------------------------- 1 | 2 | import torch 3 | from torch_scatter import scatter_add 4 | from torch_geometric.utils import softmax 5 | 6 | from torch_geometric.nn.inits import reset 7 | 8 | 9 | class MyGlobalAttention(torch.nn.Module): 10 | r"""Global soft attention layer from the `"Gated Graph Sequence Neural 11 | Networks" `_ paper 12 | 13 | .. math:: 14 | \mathbf{r}_i = \sum_{n=1}^{N_i} \mathrm{softmax} \left( 15 | h_{\mathrm{gate}} ( \mathbf{x}_n ) \right) \odot 16 | h_{\mathbf{\Theta}} ( \mathbf{x}_n ), 17 | 18 | where :math:`h_{\mathrm{gate}} \colon \mathbb{R}^F \to 19 | \mathbb{R}` and :math:`h_{\mathbf{\Theta}}` denote neural networks, *i.e.* 20 | MLPS. 21 | 22 | Args: 23 | gate_nn (torch.nn.Module): A neural network :math:`h_{\mathrm{gate}}` 24 | that computes attention scores by mapping node features :obj:`x` of 25 | shape :obj:`[-1, in_channels]` to shape :obj:`[-1, 1]`, *e.g.*, 26 | defined by :class:`torch.nn.Sequential`. 27 | nn (torch.nn.Module, optional): A neural network 28 | :math:`h_{\mathbf{\Theta}}` that maps node features :obj:`x` of 29 | shape :obj:`[-1, in_channels]` to shape :obj:`[-1, out_channels]` 30 | before combining them with the attention scores, *e.g.*, defined by 31 | :class:`torch.nn.Sequential`. (default: :obj:`None`) 32 | """ 33 | def __init__(self, gate_nn, nn=None): 34 | super(MyGlobalAttention, self).__init__() 35 | self.gate_nn = gate_nn 36 | self.nn = nn 37 | 38 | self.reset_parameters() 39 | 40 | def reset_parameters(self): 41 | reset(self.gate_nn) 42 | reset(self.nn) 43 | 44 | def forward(self, x, batch, set_zeros_ids=None, size=None, att_mask=None): 45 | """""" 46 | x = x.unsqueeze(-1) if x.dim() == 1 else x 47 | size = batch[-1].item() + 1 if size is None else size 48 | 49 | gate = self.gate_nn(x).view(-1, 1) 50 | x = self.nn(x) if self.nn is not None else x 51 | assert gate.dim() == x.dim() and gate.size(0) == x.size(0) 52 | 53 | if att_mask is not None: 54 | gate = gate + att_mask # additive! 55 | 56 | gate = softmax(gate, batch, num_nodes=size) 57 | 58 | if set_zeros_ids is not None: 59 | # x[set_zeros_ids] = 0 60 | # x.masked_fill(set_zeros_ids, 0) 61 | x = x.permute((1, 0)) 62 | x = x * set_zeros_ids 63 | x = x.permute((1, 0)) 64 | 65 | out = scatter_add(gate * x, batch, dim=0, dim_size=size) 66 | 67 | return out, gate 68 | 69 | def __repr__(self): 70 | return '{}(gate_nn={}, nn={})'.format(self.__class__.__name__, 71 | self.gate_nn, self.nn) 72 | -------------------------------------------------------------------------------- /baselines/src/our_transformer_conv.py: -------------------------------------------------------------------------------- 1 | ''' 2 | 04/26/2023: 3 | Adapted from /Users/xxx/.conda/envs/software/lib/python3.9/site-packages/torch_geometric/nn/conv/transformer_conv.py 4 | torch_geometric==2.0.4 5 | ''' 6 | 7 | import math 8 | from typing import Optional, Tuple, Union 9 | 10 | import torch 11 | import torch.nn.functional as F 12 | from torch import Tensor 13 | from torch_sparse import SparseTensor 14 | 15 | from torch_geometric.nn.conv import MessagePassing 16 | from torch_geometric.nn.dense.linear import Linear 17 | from torch_geometric.typing import Adj, OptTensor, PairTensor 18 | from torch_geometric.utils import softmax 19 | 20 | 21 | class OurTransformerConv(MessagePassing): 22 | r"""The graph transformer operator from the `"Masked Label Prediction: 23 | Unified Message Passing Model for Semi-Supervised Classification" 24 | `_ paper 25 | 26 | .. math:: 27 | \mathbf{x}^{\prime}_i = \mathbf{W}_1 \mathbf{x}_i + 28 | \sum_{j \in \mathcal{N}(i)} \alpha_{i,j} \mathbf{W}_2 \mathbf{x}_{j}, 29 | 30 | where the attention coefficients :math:`\alpha_{i,j}` are computed via 31 | multi-head dot product attention: 32 | 33 | .. math:: 34 | \alpha_{i,j} = \textrm{softmax} \left( 35 | \frac{(\mathbf{W}_3\mathbf{x}_i)^{\top} (\mathbf{W}_4\mathbf{x}_j)} 36 | {\sqrt{d}} \right) 37 | 38 | Args: 39 | in_channels (int or tuple): Size of each input sample, or :obj:`-1` to 40 | derive the size from the first input(s) to the forward method. 41 | A tuple corresponds to the sizes of source and target 42 | dimensionalities. 43 | out_channels (int): Size of each output sample. 44 | heads (int, optional): Number of multi-head-attentions. 45 | (default: :obj:`1`) 46 | concat (bool, optional): If set to :obj:`False`, the multi-head 47 | attentions are averaged instead of concatenated. 48 | (default: :obj:`True`) 49 | beta (bool, optional): If set, will combine aggregation and 50 | skip information via 51 | 52 | .. math:: 53 | \mathbf{x}^{\prime}_i = \beta_i \mathbf{W}_1 \mathbf{x}_i + 54 | (1 - \beta_i) \underbrace{\left(\sum_{j \in \mathcal{N}(i)} 55 | \alpha_{i,j} \mathbf{W}_2 \vec{x}_j \right)}_{=\mathbf{m}_i} 56 | 57 | with :math:`\beta_i = \textrm{sigmoid}(\mathbf{w}_5^{\top} 58 | [ \mathbf{W}_1 \mathbf{x}_i, \mathbf{m}_i, \mathbf{W}_1 59 | \mathbf{x}_i - \mathbf{m}_i ])` (default: :obj:`False`) 60 | dropout (float, optional): Dropout probability of the normalized 61 | attention coefficients which exposes each node to a stochastically 62 | sampled neighborhood during training. (default: :obj:`0`) 63 | edge_dim (int, optional): Edge feature dimensionality (in case 64 | there are any). Edge features are added to the keys after 65 | linear transformation, that is, prior to computing the 66 | attention dot product. They are also added to final values 67 | after the same linear transformation. The model is: 68 | 69 | .. math:: 70 | \mathbf{x}^{\prime}_i = \mathbf{W}_1 \mathbf{x}_i + 71 | \sum_{j \in \mathcal{N}(i)} \alpha_{i,j} \left( 72 | \mathbf{W}_2 \mathbf{x}_{j} + \mathbf{W}_6 \mathbf{e}_{ij} 73 | \right), 74 | 75 | where the attention coefficients :math:`\alpha_{i,j}` are now 76 | computed via: 77 | 78 | .. math:: 79 | \alpha_{i,j} = \textrm{softmax} \left( 80 | \frac{(\mathbf{W}_3\mathbf{x}_i)^{\top} 81 | (\mathbf{W}_4\mathbf{x}_j + \mathbf{W}_6 \mathbf{e}_{ij})} 82 | {\sqrt{d}} \right) 83 | 84 | (default :obj:`None`) 85 | bias (bool, optional): If set to :obj:`False`, the layer will not learn 86 | an additive bias. (default: :obj:`True`) 87 | root_weight (bool, optional): If set to :obj:`False`, the layer will 88 | not add the transformed root node features to the output and the 89 | option :attr:`beta` is set to :obj:`False`. (default: :obj:`True`) 90 | **kwargs (optional): Additional arguments of 91 | :class:`torch_geometric.nn.conv.MessagePassing`. 92 | """ 93 | _alpha: OptTensor 94 | 95 | def __init__( 96 | self, 97 | in_channels: Union[int, Tuple[int, int]], 98 | out_channels: int, 99 | heads: int = 1, 100 | concat: bool = True, 101 | beta: bool = False, 102 | dropout: float = 0., 103 | edge_dim: Optional[int] = None, 104 | bias: bool = True, 105 | root_weight: bool = True, 106 | **kwargs, 107 | ): 108 | kwargs.setdefault('aggr', 'add') 109 | super(OurTransformerConv, self).__init__(node_dim=0, **kwargs) 110 | 111 | self.in_channels = in_channels 112 | self.out_channels = out_channels 113 | self.heads = heads 114 | self.beta = beta and root_weight 115 | self.root_weight = root_weight 116 | self.concat = concat 117 | self.dropout = dropout 118 | self.edge_dim = edge_dim 119 | self._alpha = None 120 | 121 | if isinstance(in_channels, int): 122 | in_channels = (in_channels, in_channels) 123 | 124 | self.lin_key = Linear(in_channels[0], heads * out_channels) 125 | self.lin_query = Linear(in_channels[1], heads * out_channels) 126 | self.lin_value = Linear(in_channels[0], heads * out_channels) 127 | if edge_dim is not None: 128 | self.lin_edge = Linear(edge_dim, heads * out_channels, bias=False) 129 | else: 130 | self.lin_edge = self.register_parameter('lin_edge', None) 131 | 132 | if concat: 133 | self.lin_skip = Linear(in_channels[1], heads * out_channels, 134 | bias=bias) 135 | if self.beta: 136 | self.lin_beta = Linear(3 * heads * out_channels, 1, bias=False) 137 | else: 138 | self.lin_beta = self.register_parameter('lin_beta', None) 139 | else: 140 | self.lin_skip = Linear(in_channels[1], out_channels, bias=bias) 141 | if self.beta: 142 | self.lin_beta = Linear(3 * out_channels, 1, bias=False) 143 | else: 144 | self.lin_beta = self.register_parameter('lin_beta', None) 145 | 146 | self.reset_parameters() 147 | 148 | def reset_parameters(self): 149 | self.lin_key.reset_parameters() 150 | self.lin_query.reset_parameters() 151 | self.lin_value.reset_parameters() 152 | if self.edge_dim: 153 | self.lin_edge.reset_parameters() 154 | self.lin_skip.reset_parameters() 155 | if self.beta: 156 | self.lin_beta.reset_parameters() 157 | 158 | def forward(self, x: Union[Tensor, PairTensor], edge_index: Adj, 159 | edge_attr: OptTensor = None, return_attention_weights=None): 160 | # type: (Union[Tensor, PairTensor], Tensor, OptTensor, NoneType) -> Tensor # noqa 161 | # type: (Union[Tensor, PairTensor], SparseTensor, OptTensor, NoneType) -> Tensor # noqa 162 | # type: (Union[Tensor, PairTensor], Tensor, OptTensor, bool) -> Tuple[Tensor, Tuple[Tensor, Tensor]] # noqa 163 | # type: (Union[Tensor, PairTensor], SparseTensor, OptTensor, bool) -> Tuple[Tensor, SparseTensor] # noqa 164 | r""" 165 | Args: 166 | return_attention_weights (bool, optional): If set to :obj:`True`, 167 | will additionally return the tuple 168 | :obj:`(edge_index, attention_weights)`, holding the computed 169 | attention weights for each edge. (default: :obj:`None`) 170 | """ 171 | 172 | H, C = self.heads, self.out_channels 173 | 174 | if isinstance(x, Tensor): 175 | x: PairTensor = (x, x) 176 | 177 | query = self.lin_query(x[1]).view(-1, H, C) 178 | key = self.lin_key(x[0]).view(-1, H, C) 179 | value = self.lin_value(x[0]).view(-1, H, C) 180 | 181 | # propagate_type: (query: Tensor, key:Tensor, value: Tensor, edge_attr: OptTensor) # noqa 182 | out = self.propagate(edge_index, query=query, key=key, value=value, 183 | edge_attr=edge_attr, size=None) 184 | 185 | alpha = self._alpha 186 | self._alpha = None 187 | 188 | if self.concat: 189 | out = out.view(-1, self.heads * self.out_channels) 190 | else: 191 | out = out.mean(dim=1) 192 | 193 | if self.root_weight: 194 | x_r = self.lin_skip(x[1]) 195 | if self.lin_beta is not None: 196 | beta = self.lin_beta(torch.cat([out, x_r, out - x_r], dim=-1)) 197 | beta = beta.sigmoid() 198 | out = beta * x_r + (1 - beta) * out 199 | else: 200 | out += x_r 201 | 202 | if isinstance(return_attention_weights, bool): 203 | assert alpha is not None 204 | if isinstance(edge_index, Tensor): 205 | return out, (edge_index, alpha) 206 | elif isinstance(edge_index, SparseTensor): 207 | return out, edge_index.set_value(alpha, layout='coo') 208 | else: 209 | return out 210 | 211 | def message(self, query_i: Tensor, key_j: Tensor, value_j: Tensor, 212 | edge_attr: OptTensor, index: Tensor, ptr: OptTensor, 213 | size_i: Optional[int]) -> Tensor: 214 | 215 | if self.lin_edge is not None: 216 | assert edge_attr is not None 217 | edge_attr = self.lin_edge(edge_attr).view(-1, self.heads, 218 | self.out_channels) 219 | key_j += edge_attr 220 | 221 | alpha = (query_i * key_j).sum(dim=-1) / math.sqrt(self.out_channels) 222 | alpha = softmax(alpha, index, ptr, size_i) 223 | self._alpha = alpha 224 | alpha = F.dropout(alpha, p=self.dropout, training=self.training) 225 | 226 | out = value_j 227 | if edge_attr is not None: 228 | out += edge_attr 229 | 230 | out *= alpha.view(-1, self.heads, 1) 231 | 232 | self.edge_msgs_saved_in_propagation = out # critical: assign the edge messages so we can retrieve it later 233 | 234 | return out 235 | 236 | def __repr__(self) -> str: 237 | return (f'{self.__class__.__name__}({self.in_channels}, ' 238 | f'{self.out_channels}, heads={self.heads})') 239 | -------------------------------------------------------------------------------- /baselines/src/parameter.py: -------------------------------------------------------------------------------- 1 | """ 2 | The definition of supported design parameters 3 | """ 4 | import ast 5 | from typing import Dict, List, Optional, Tuple, Type, Union, Set 6 | # from saver import saver 7 | 8 | 9 | class DesignParameter(object): 10 | """A tunable design parameter""" 11 | 12 | def __init__(self, name: str = ''): 13 | self.name: str = name 14 | self.default: Union[str, int] = 1 15 | self.option_expr: str = '' 16 | self.scope: List[str] = [] 17 | self.order: Dict[str, str] = {} 18 | self.deps: List[str] = [] 19 | self.child: List[str] = [] 20 | self.value: Union[str, int] = 1 21 | 22 | 23 | DesignSpace = Dict[str, DesignParameter] 24 | DesignPoint = Dict[str, Union[int, str]] 25 | 26 | 27 | def gen_key_from_design_point(point: DesignPoint) -> str: 28 | """Generate a unique key from the given design point. 29 | 30 | Args: 31 | point: The given design point. 32 | 33 | Returns: 34 | The generated key in the format of "param1-value1.param2-value2". 35 | """ 36 | 37 | return '.'.join([ 38 | '{0}-{1}'.format(pid, 39 | str(point[pid]) if point[pid] else 'NA') for pid in sorted(point.keys()) 40 | ]) 41 | 42 | 43 | def check_option_syntax(option_expr: str, log) -> Tuple[bool, List[str]]: 44 | """Check the syntax of design options and extract dependent design parameter IDs. 45 | 46 | Args: 47 | option_expr: The design space option expression. 48 | 49 | Returns: 50 | Indicate if the expression is valid or not; A list of dependent design parameter IDs. 51 | """ 52 | 53 | try: 54 | stree = ast.parse(option_expr) 55 | except SyntaxError: 56 | log.error((f'"options" error: Illegal option list {option_expr}')) 57 | return (False, []) 58 | 59 | # Traverse AST of the option_expression for all variables 60 | names = set() 61 | iter_val = None 62 | for node in ast.walk(stree): 63 | if isinstance(node, ast.ListComp): 64 | funcs = [ 65 | n.func.id for n in ast.walk(node.elt) 66 | if isinstance(n, ast.Call) and isinstance(n.func, ast.Name) 67 | ] 68 | elt_vals = [ 69 | n.id for n in ast.walk(node.elt) 70 | if isinstance(n, ast.Name) and n.id not in funcs and n.id != '_' 71 | ] 72 | assert len(elt_vals) <= 1, 'Found more than one iterators in {0}'.format(option_expr) 73 | if len(elt_vals) == 1: 74 | iter_val = elt_vals[0] 75 | elif isinstance(node, ast.Name): 76 | names.add(node.id) 77 | 78 | # Ignore the list comprehension iterator 79 | if iter_val: 80 | names.remove(iter_val) 81 | 82 | # # Ignore legal builtin functions 83 | # for func in SAFE_LIST: 84 | # if func in names: 85 | # names.remove(func) 86 | 87 | # Ignore builtin primitive type casting 88 | for ptype in ['int', 'str', 'float']: 89 | if ptype in names: 90 | names.remove(ptype) 91 | 92 | return (True, list(names)) 93 | 94 | 95 | def check_order_syntax(order_expr: str, log) -> Tuple[bool, str]: 96 | """Check the syntax of the partition rule and extract the variable name. 97 | 98 | Args: 99 | order_expr: The design space option expression. 100 | 101 | Returns: 102 | Indicate if the expression is valid or not; The single variable name in the expression. 103 | """ 104 | try: 105 | stree = ast.parse(order_expr) 106 | except SyntaxError: 107 | log.error((f'"order" error: Illegal order expression {order_expr}')) 108 | return (False, '') 109 | 110 | # Traverse AST of the expression for the variable 111 | names = set() 112 | for node in ast.walk(stree): 113 | if isinstance(node, ast.Name): 114 | names.add(node.id) 115 | 116 | if len(names) != 1: 117 | log.error((f'"order" should have one and only one variable in {order_expr} but found {len(names)}')) 118 | return (False, '') 119 | return (True, names.pop()) 120 | 121 | 122 | def create_design_parameter(param_id: str, ds_config: Dict[str, Union[str, int]], 123 | param_cls: Type[DesignParameter], 124 | log) -> Optional[DesignParameter]: 125 | """Create DesignParameter from the string. 126 | 127 | Args: 128 | param_id: The unique parameter ID. 129 | attr_str: The design space string in the auto pragma. 130 | param_cls: The class of parameter we will create. 131 | 132 | Returns: 133 | The created DesignParameter object. 134 | """ 135 | 136 | if param_cls == DesignParameter: 137 | param = DesignParameter(param_id) 138 | 139 | # Type checking 140 | if 'ds_type' not in ds_config: 141 | log.warning(( 142 | f'Missing attribute "ds_type" in {param_id}. Some optimization may not be triggered')) 143 | else: 144 | param.ds_type = str(ds_config['ds_type']).upper() 145 | else: 146 | log.error('Unrecognized parameter type') 147 | return None 148 | 149 | # General settings for parameters 150 | # Option checking 151 | if 'options' not in ds_config: 152 | log.error('Missing attribute "options" in %s', param_id) 153 | return None 154 | if 'TIL-' in param.ds_type: 155 | param.option_expr = str([1]) ###### pruning DS: removing the options on tile 156 | else: 157 | param.option_expr = str(ds_config['options']) 158 | check, param.deps = check_option_syntax(param.option_expr, log) 159 | if not check: 160 | return None 161 | 162 | # Partition checking 163 | if 'order' in ds_config: 164 | check, var = check_order_syntax(str(ds_config['order']), log) 165 | if not check: 166 | log.warning((f'Failed to parse "order" of {param_id}, ignore.')) 167 | else: 168 | param.order = {'expr': str(ds_config['order']), 'var': var} 169 | 170 | # Default checking 171 | if 'default' not in ds_config: 172 | log.error((f'Missing attribute "default" in {param_id}')) 173 | return None 174 | param.default = ds_config['default'] 175 | 176 | return param 177 | 178 | 179 | def get_default_point(ds: DesignSpace) -> DesignPoint: 180 | """Generate a design point with all default values. 181 | 182 | Returns: 183 | The design point with all default value applied. 184 | """ 185 | 186 | point: DesignPoint = {} 187 | for pid, param in ds.items(): 188 | point[pid] = param.default 189 | return point 190 | 191 | 192 | def check_design_space(params: DesignSpace, log) -> int: 193 | """Check design space for missing dependency and duplication. 194 | 195 | Args: 196 | params: The overall design space. 197 | 198 | Returns: 199 | The number of errors found in the design space. 200 | """ 201 | 202 | error = 0 203 | 204 | for pid, param in params.items(): 205 | has_error = False 206 | 207 | # Check dependencies 208 | for dep in param.deps: 209 | if dep == pid: 210 | log.error((f'Parameter {pid} cannot depend on itself')) 211 | error += 1 212 | has_error = True 213 | if dep not in params.keys(): 214 | log.error((f'Parameter {pid} depends on {dep} which is undefined or not allowed')) 215 | error += 1 216 | has_error = True 217 | 218 | if has_error: 219 | continue 220 | 221 | # Check expression types 222 | # Assign default value to dependent parameters 223 | local = {} 224 | for dep in param.deps: 225 | local[dep] = params[dep].default 226 | 227 | # Try to get an option list 228 | options: Optional[List[Union[int, str]]] = None 229 | try: 230 | options = eval(param.option_expr, local) 231 | except (NameError, ValueError, TypeError, ZeroDivisionError) as err: 232 | log.error('Failed to get the options of parameter %s: %s', pid, str(err)) 233 | error += 1 234 | 235 | # Try to get the order of options 236 | if options is not None and param.order and isinstance(param, DesignParameter): 237 | for option in options: 238 | if eval(param.order['expr'], {param.order['var']: option}) is None: 239 | log.error('Failed to evaluate the order of option %s in parameter %s', option, 240 | pid) 241 | error += 1 242 | return error 243 | 244 | 245 | def analyze_child_in_design_space(params: DesignSpace) -> None: 246 | """Traverse design parameter dependency and build child list for each parameter in place. 247 | 248 | Args: 249 | params: The overall design space 250 | """ 251 | 252 | # Setup child for each parameter 253 | for pid, param in params.items(): 254 | for dep in param.deps: 255 | params[dep].child.append(pid) 256 | 257 | # Remove duplications 258 | for param in params.values(): 259 | param.child = list(dict.fromkeys(param.child)) 260 | 261 | 262 | def topo_sort_param_ids(space: DesignSpace) -> List[str]: 263 | """Sort the parameter IDs topologically. 264 | 265 | Args: 266 | space: The design space to be sorted. 267 | 268 | Returns: 269 | The sorted IDs. 270 | """ 271 | 272 | def helper(curr_id: str, visited: Set[str], stack: List[str]) -> None: 273 | """The helper function for topological sort.""" 274 | visited.add(curr_id) 275 | for dep in space[curr_id].deps: 276 | if dep not in visited: 277 | helper(dep, visited, stack) 278 | stack.append(curr_id) 279 | # saver.info((f'added, {curr_id}')) 280 | 281 | visited: Set[str] = set() 282 | stack: List[str] = [] 283 | for pid in space.keys(): 284 | # saver.info(pid) 285 | if pid not in visited: 286 | helper(pid, visited, stack) 287 | return stack 288 | 289 | def count_design_points(ds: DesignSpace, log) -> int: 290 | """Count the valid points in a given design space. 291 | 292 | Args: 293 | ds: Design space to be counted. 294 | 295 | Returns: 296 | Number of valid design points. 297 | """ 298 | 299 | def helper(ds: DesignSpace, sorted_ids: List[str], idx: int, point: DesignPoint) -> int: 300 | """Count the deisgn points of parameters by traversing topological sorted parameters.""" 301 | 302 | # Reach to the end 303 | if idx == len(sorted_ids): 304 | return 1 305 | 306 | pid = sorted_ids[idx] 307 | param = ds[pid] 308 | options = eval(param.option_expr, point) 309 | 310 | counter = 0 311 | if param.child: 312 | # Sum all points under each option 313 | for option in options: 314 | point[pid] = option 315 | counter += helper(ds, sorted_ids, idx + 1, point) 316 | else: 317 | # Product the number of options with the rest points 318 | counter = len(options) * helper(ds, sorted_ids, idx + 1, point) 319 | log.debug((f'Node {pid}: {counter}')) 320 | return counter 321 | 322 | point = get_default_point(ds) 323 | sorted_ids = topo_sort_param_ids(ds) 324 | return helper(ds, sorted_ids, 0, point) 325 | 326 | 327 | def compile_design_space(user_ds_config: Dict[str, Dict[str, Union[str, int]]], 328 | scope_map: Optional[Dict[str, List[str]]], 329 | log) -> Optional[DesignSpace]: 330 | """Compile the design space from the config JSON file. 331 | 332 | Args: 333 | user_ds_config: The input design space configure loaded from a JSON file. 334 | Note that the duplicated ID checking should be done when 335 | loading the JSON file and here we assume no duplications. 336 | scope_map: The scope map that maps design parameter ID to its scope. 337 | 338 | Returns: 339 | The design space compiled from the kernel code; or None if failed. 340 | """ 341 | params: Dict[str, DesignParameter] = {} 342 | for param_id, param_config in user_ds_config.items(): 343 | param = create_design_parameter(param_id, param_config, DesignParameter, log) 344 | if param: 345 | if param.ds_type not in [ 346 | 'PARALLEL', 'PIPELINE', 'TILING', 'TILE' 347 | ]: 348 | param.scope.append('GLOBAL') 349 | else: 350 | if scope_map and param_id in scope_map: 351 | param.scope = scope_map[param_id] 352 | params[param_id] = param 353 | 354 | error = check_design_space(params, log) 355 | if error > 0: 356 | log.error((f'Design space has {error} errors')) 357 | return None 358 | 359 | analyze_child_in_design_space(params) 360 | 361 | num_ds = count_design_points(params, log) 362 | log.info((f'Design space contains {num_ds} valid design points')) 363 | log.info('Finished design space compilation') 364 | return params, num_ds 365 | 366 | 367 | 368 | -------------------------------------------------------------------------------- /baselines/src/pretrained_GNN_handler.py: -------------------------------------------------------------------------------- 1 | 2 | from config import FLAGS 3 | from saver import saver 4 | from utils import get_root_path 5 | import torch 6 | import torch.nn.functional as F 7 | from torch_geometric.nn import GATConv, JumpingKnowledge, TransformerConv, GCNConv, GatedGraphConv 8 | import torch.nn as nn 9 | 10 | from collections import OrderedDict 11 | 12 | 13 | class PretrainedGNNEncoder(nn.Module): 14 | def __init__(self, in_channels, edge_dim=0, init_pragma_dict=None, task=FLAGS.task, num_layers=FLAGS.num_layers, 15 | D=64, target=FLAGS.target): # TODO: debug and check if edge dim/attr is used properly -- yes due to pretrained_model.load_state_dict loading the correct lin_edge 16 | super(PretrainedGNNEncoder, self).__init__() 17 | 18 | conv_class = TransformerConv # default choice so far (05/18/2023) 19 | 20 | self.conv_first = conv_class(in_channels, D, edge_dim=edge_dim) 21 | 22 | self.conv_layers = nn.ModuleList() 23 | 24 | for _ in range(num_layers - 1): 25 | conv = conv_class(D, D, edge_dim=edge_dim) 26 | self.conv_layers.append(conv) 27 | 28 | self.jkn = JumpingKnowledge(FLAGS.jkn_mode, channels=D, num_layers=2) 29 | 30 | def get_node_emb(self, data): 31 | x, edge_index, edge_attr, batch = \ 32 | data.x, data.edge_index, data.edge_attr, data.batch 33 | 34 | if x is None: 35 | x, edge_index, edge_attr, batch = \ 36 | data.x_programl, data.edge_index_programl, getattr(data, 'edge_attr_programl'), data.x_programl_batch 37 | 38 | # if hasattr(data, 'kernel'): 39 | # gname = data.kernel[0] 40 | # print(gname) 41 | # print(edge_attr.shape) 42 | outs = [] 43 | out_dict = OrderedDict() 44 | if FLAGS.activation == 'relu': 45 | activation = F.relu 46 | elif FLAGS.activation == 'elu': 47 | activation = F.elu 48 | else: 49 | raise NotImplementedError() 50 | 51 | out = activation(self.conv_first(x, edge_index, edge_attr=edge_attr)) 52 | 53 | outs.append(out) 54 | 55 | for i, conv in enumerate(self.conv_layers): 56 | out = conv(out, edge_index, edge_attr=edge_attr) 57 | if i != len(self.conv_layers) - 1: 58 | out = activation(out) 59 | 60 | outs.append(out) 61 | 62 | if FLAGS.jkn_enable: 63 | out = self.jkn(outs) 64 | out = torch.tanh(out) 65 | node_emb = out 66 | 67 | return node_emb, outs 68 | 69 | 70 | def create_and_load_pretrained_GNN(): 71 | if not hasattr(FLAGS, 'pretrained_GNN_name') or FLAGS.pretrained_GNN_name == 'our': 72 | load_model = f'{get_root_path()}/file/xxx_pretrain/test_model_state_dict.pth' 73 | elif FLAGS.pretrained_GNN_name == 'GraphMAE': 74 | load_model = f'{get_root_path()}/file/xxx_pretrain/GraphMAE/test_model_state_dict.pth' 75 | else: 76 | raise NotImplementedError() 77 | 78 | pretrained_model = PretrainedGNNEncoder(in_channels=158, num_layers=6) 79 | ld = torch.load(load_model, map_location=torch.device('cpu')) 80 | # weights_cur = dict(pretrained_model.state_dict()) 81 | pretrained_model.load_state_dict(ld, strict=False) 82 | saver.log_info(f'Loaded GNN encoder from {load_model}') 83 | return pretrained_model 84 | 85 | 86 | if __name__ == '__main__': 87 | create_and_load_pretrained_GNN() 88 | -------------------------------------------------------------------------------- /baselines/src/protonet_miniimagenet.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import numpy as np 5 | 6 | import torch 7 | import torch.nn as nn 8 | import torch.nn.functional as F 9 | from torch.utils.data import DataLoader 10 | 11 | import learn2learn as l2l 12 | from learn2learn.data.transforms import NWays, KShots, LoadData, RemapLabels 13 | from tqdm import tqdm 14 | 15 | 16 | def pairwise_distances_logits(a, b): 17 | n = a.shape[0] 18 | m = b.shape[0] 19 | logits = -((a.unsqueeze(1).expand(n, m, -1) - 20 | b.unsqueeze(0).expand(n, m, -1)) ** 2).sum(dim=2) 21 | return logits 22 | 23 | 24 | def accuracy(predictions, targets): 25 | predictions = predictions.argmax(dim=1).view(targets.shape) 26 | return (predictions == targets).sum().float() / targets.size(0) 27 | 28 | 29 | class Convnet(nn.Module): 30 | 31 | def __init__(self, x_dim=3, hid_dim=64, z_dim=64): 32 | super().__init__() 33 | self.encoder = l2l.vision.models.CNN4Backbone( 34 | hidden_size=hid_dim, 35 | channels=x_dim, 36 | max_pool=True, 37 | ) 38 | self.out_channels = 1600 39 | 40 | def forward(self, x): 41 | x = self.encoder(x) 42 | return x.view(x.size(0), -1) 43 | 44 | 45 | def fast_adapt(model, batch, ways, shot, query_num, metric=None, device=None): 46 | if metric is None: 47 | metric = pairwise_distances_logits 48 | if device is None: 49 | device = model.device() 50 | data, labels = batch 51 | data = data.to(device) 52 | labels = labels.to(device) 53 | n_items = shot * ways 54 | 55 | # Sort data samples by labels 56 | # TODO: Can this be replaced by ConsecutiveLabels ? 57 | sort = torch.sort(labels) 58 | data = data.squeeze(0)[sort.indices].squeeze(0) 59 | labels = labels.squeeze(0)[sort.indices].squeeze(0) 60 | 61 | # Compute support and query embeddings 62 | embeddings = model(data) 63 | support_indices = np.zeros(data.size(0), dtype=bool) 64 | selection = np.arange(ways) * (shot + query_num) 65 | for offset in range(shot): 66 | support_indices[selection + offset] = True 67 | query_indices = torch.from_numpy(~support_indices) 68 | support_indices = torch.from_numpy(support_indices) 69 | support = embeddings[support_indices] 70 | support = support.reshape(ways, shot, -1).mean(dim=1) 71 | query = embeddings[query_indices] 72 | labels = labels[query_indices].long() 73 | 74 | logits = pairwise_distances_logits(query, support) 75 | loss = F.cross_entropy(logits, labels) 76 | acc = accuracy(logits, labels) 77 | return loss, acc 78 | 79 | 80 | if __name__ == '__main__': 81 | parser = argparse.ArgumentParser() 82 | parser.add_argument('--max-epoch', type=int, default=250) 83 | parser.add_argument('--shot', type=int, default=1) 84 | parser.add_argument('--test-way', type=int, default=5) 85 | parser.add_argument('--test-shot', type=int, default=1) 86 | parser.add_argument('--test-query', type=int, default=30) 87 | parser.add_argument('--train-query', type=int, default=15) 88 | parser.add_argument('--train-way', type=int, default=30) 89 | parser.add_argument('--gpu', default=0) 90 | args = parser.parse_args() 91 | print(args) 92 | 93 | device = torch.device('cpu') 94 | if args.gpu and torch.cuda.device_count(): 95 | print("Using gpu") 96 | torch.cuda.manual_seed(43) 97 | device = torch.device('cuda:0') 98 | 99 | model = Convnet() 100 | model.to(device) 101 | 102 | path_data = '~/data' 103 | train_dataset = l2l.vision.datasets.MiniImagenet( 104 | root=path_data, mode='train') 105 | valid_dataset = l2l.vision.datasets.MiniImagenet( 106 | root=path_data, mode='validation') 107 | test_dataset = l2l.vision.datasets.MiniImagenet( 108 | root=path_data, mode='test') 109 | 110 | train_dataset = l2l.data.MetaDataset(train_dataset) 111 | train_transforms = [ 112 | NWays(train_dataset, args.train_way), 113 | KShots(train_dataset, args.train_query + args.shot), 114 | LoadData(train_dataset), 115 | RemapLabels(train_dataset), 116 | ] 117 | train_tasks = l2l.data.TaskDataset(train_dataset, task_transforms=train_transforms) 118 | train_loader = DataLoader(train_tasks, pin_memory=True, shuffle=True) 119 | 120 | valid_dataset = l2l.data.MetaDataset(valid_dataset) 121 | valid_transforms = [ 122 | NWays(valid_dataset, args.test_way), 123 | KShots(valid_dataset, args.test_query + args.test_shot), 124 | LoadData(valid_dataset), 125 | RemapLabels(valid_dataset), 126 | ] 127 | valid_tasks = l2l.data.TaskDataset(valid_dataset, 128 | task_transforms=valid_transforms, 129 | num_tasks=200) 130 | valid_loader = DataLoader(valid_tasks, pin_memory=True, shuffle=True) 131 | 132 | test_dataset = l2l.data.MetaDataset(test_dataset) 133 | test_transforms = [ 134 | NWays(test_dataset, args.test_way), 135 | KShots(test_dataset, args.test_query + args.test_shot), 136 | LoadData(test_dataset), 137 | RemapLabels(test_dataset), 138 | ] 139 | test_tasks = l2l.data.TaskDataset(test_dataset, 140 | task_transforms=test_transforms, 141 | num_tasks=2000) 142 | test_loader = DataLoader(test_tasks, pin_memory=True, shuffle=True) 143 | 144 | optimizer = torch.optim.Adam(model.parameters(), lr=0.001) 145 | lr_scheduler = torch.optim.lr_scheduler.StepLR( 146 | optimizer, step_size=20, gamma=0.5) 147 | 148 | for epoch in range(1, args.max_epoch + 1): 149 | model.train() 150 | 151 | loss_ctr = 0 152 | n_loss = 0 153 | n_acc = 0 154 | 155 | for i in tqdm(range(100)): 156 | batch = next(iter(train_loader)) 157 | 158 | loss, acc = fast_adapt(model, 159 | batch, 160 | args.train_way, 161 | args.shot, 162 | args.train_query, 163 | metric=pairwise_distances_logits, 164 | device=device) 165 | 166 | loss_ctr += 1 167 | n_loss += loss.item() 168 | n_acc += acc 169 | 170 | optimizer.zero_grad() 171 | loss.backward() 172 | optimizer.step() 173 | print(f'Epoch {epoch} Iter {i} loss={loss.item():.4f}') 174 | lr_scheduler.step() 175 | 176 | print('epoch {}, train, loss={:.4f} acc={:.4f}'.format( 177 | epoch, n_loss / loss_ctr, n_acc / loss_ctr)) 178 | 179 | model.eval() 180 | 181 | loss_ctr = 0 182 | n_loss = 0 183 | n_acc = 0 184 | for i, batch in enumerate(valid_loader): 185 | loss, acc = fast_adapt(model, 186 | batch, 187 | args.test_way, 188 | args.test_shot, 189 | args.test_query, 190 | metric=pairwise_distances_logits, 191 | device=device) 192 | 193 | loss_ctr += 1 194 | n_loss += loss.item() 195 | n_acc += acc 196 | 197 | print('epoch {}, val, loss={:.4f} acc={:.4f}'.format( 198 | epoch, n_loss / loss_ctr, n_acc / loss_ctr)) 199 | 200 | loss_ctr = 0 201 | n_acc = 0 202 | 203 | for i, batch in enumerate(test_loader, 1): 204 | loss, acc = fast_adapt(model, 205 | batch, 206 | args.test_way, 207 | args.test_shot, 208 | args.test_query, 209 | metric=pairwise_distances_logits, 210 | device=device) 211 | loss_ctr += 1 212 | n_acc += acc 213 | print('batch {}: {:.2f}({:.2f})'.format( 214 | i, n_acc / loss_ctr * 100, acc * 100)) 215 | -------------------------------------------------------------------------------- /baselines/src/result.py: -------------------------------------------------------------------------------- 1 | """ 2 | The definition of evaluation results 3 | """ 4 | 5 | from enum import Enum 6 | from typing import Dict, List, NamedTuple, Optional 7 | import pickle 8 | 9 | from parameter import DesignPoint 10 | 11 | def persist(database, db_file_path) -> bool: 12 | #pylint:disable=missing-docstring 13 | 14 | dump_db = { 15 | key: database.hget(0, key) 16 | for key in database.hgetall(0) 17 | } 18 | with open(db_file_path, 'wb') as filep: 19 | pickle.dump(dump_db, filep, pickle.HIGHEST_PROTOCOL) 20 | 21 | return True 22 | 23 | class Job(object): 24 | """The info and properties of a job""" 25 | 26 | class Status(Enum): 27 | INIT = 0 28 | APPLIED = 1 29 | 30 | def __init__(self, path: str): 31 | self.path: str = path 32 | self.key: str = 'NotAPPLIED' 33 | self.point: Optional[DesignPoint] = None 34 | self.status: Job.Status = Job.Status.INIT 35 | 36 | 37 | class Result(object): 38 | """The base module of evaluation result""" 39 | 40 | class RetCode(Enum): 41 | PASS = 0 42 | UNAVAILABLE = -1 43 | ANALYZE_ERROR = -2 44 | EARLY_REJECT = -3 45 | TIMEOUT = -4 46 | DUPLICATED = -5 47 | 48 | def __init__(self, ret_code_str: str = 'PASS'): 49 | 50 | # The design point of this result. 51 | self.point: Optional[DesignPoint] = None 52 | 53 | # The return code of the evaluation 54 | self.ret_code: Result.RetCode = self.RetCode[ret_code_str] 55 | 56 | # Indicate if this result is valid to be a final output. For example, a result that 57 | # out-of-resource is invalid. 58 | self.valid: bool = False 59 | 60 | # The job path for this result (if available) 61 | self.path: Optional[str] = None 62 | 63 | # The quantified QoR value. Larger the better. 64 | self.quality: float = -float('inf') 65 | 66 | # Performance in terms of estimated cycle or onboard runtime. 67 | self.perf: float = 0.0 68 | 69 | # Resource utilizations 70 | self.res_util: Dict[str, float] = { 71 | 'util-BRAM': 0, 72 | 'util-DSP': 0, 73 | 'util-LUT': 0, 74 | 'util-FF': 0, 75 | 'total-BRAM': 0, 76 | 'total-DSP': 0, 77 | 'total-LUT': 0, 78 | 'total-FF': 0 79 | } 80 | 81 | # Elapsed time for evaluation 82 | self.eval_time: float = 0.0 83 | 84 | 85 | class MerlinResult(Result): 86 | """The result after running Merlin transformations""" 87 | 88 | def __init__(self, ret_code_str: str = 'PASS'): 89 | super(MerlinResult, self).__init__(ret_code_str) 90 | 91 | # Critical messages from the Merlin transformations 92 | self.criticals: List[str] = [] 93 | 94 | # The kernel code hash for recognizing duplications 95 | self.code_hash: Optional[str] = None 96 | 97 | 98 | class HierPathNode(NamedTuple): 99 | """The datastructure of hierarchy path node""" 100 | nid: str 101 | latency: float 102 | is_compute_bound: bool 103 | 104 | 105 | class HLSResult(Result): 106 | """The result after running the HLS""" 107 | 108 | def __init__(self, ret_code_str: str = 'PASS'): 109 | super(HLSResult, self).__init__(ret_code_str) 110 | 111 | # A list of hierarchy paths in the order of importance 112 | self.ordered_paths: Optional[List[List[HierPathNode]]] = None 113 | 114 | 115 | class BitgenResult(Result): 116 | """The result after bit-stream generation""" 117 | 118 | def __init__(self, ret_code_str: str = 'PASS'): 119 | super(BitgenResult, self).__init__(ret_code_str) 120 | 121 | # Frequency 122 | self.freq: float = 0.0 -------------------------------------------------------------------------------- /baselines/src/rose_gen_dot.py: -------------------------------------------------------------------------------- 1 | from config import FLAGS 2 | from saver import saver 3 | 4 | from utils import print_stats, print_g 5 | from subprocess import check_output, STDOUT, CalledProcessError 6 | 7 | from tqdm import tqdm 8 | from os.path import dirname, join, isfile, basename, splitext 9 | from os import remove 10 | import networkx as nx 11 | 12 | 13 | def get_ast_graph(code_text, source_C_file, gname): 14 | dot_file_path = _get_dot_file(code_text, source_C_file, gname) 15 | 16 | if dot_file_path is None: 17 | return None 18 | 19 | return read_dot_file_as_graph(code_text, dot_file_path, gname) 20 | 21 | 22 | def read_dot_file_as_graph(code_text, dot_file_path, gname): 23 | assert dot_file_path is not None 24 | g = _read_preproc_ast_graph(dot_file_path, code_text) 25 | g = nx.convert_node_labels_to_integers(g, 26 | ordering='sorted') # super tricky! be careful; otherwise egde_index will be messed up 27 | # g.gname = gname 28 | p = f'{saver.get_obj_dir()}/{gname}_ast.gexf' 29 | nx.write_gexf(g, p) 30 | saver.log_info(f'Saved AST gexf to {p}') 31 | 32 | assert type(g) is nx.MultiDiGraph 33 | g = nx.DiGraph( 34 | g) # critical: no need to use MultiDiGraph; DiGraph works better with later edge index generation code 35 | 36 | if FLAGS.bi_directional_AST: 37 | # critical: turns into an undirected graph with edge label 38 | g_new = g.copy() 39 | for nid1, nid2, edata in g.edges(data=True): 40 | g_new.add_edge(nid2, nid1, **edata) 41 | nx.set_edge_attributes(g_new, {(nid1, nid2): {"direction": 0}}) 42 | nx.set_edge_attributes(g_new, {(nid2, nid1): {"direction": 1}}) 43 | assert g_new.number_of_edges() == 2 * g.number_of_edges() # to debug, list(g_new.edges(data=True)) 44 | assert len(g) == len(g_new) 45 | g = g_new 46 | 47 | g.gname = gname 48 | return g 49 | 50 | 51 | def _read_preproc_ast_graph(dot_file, code_text): 52 | g = nx.drawing.nx_pydot.read_dot(dot_file) 53 | print_g(f'ast loaded from {dot_file}', g, print_func=saver.log_info) 54 | 55 | g_new = nx.MultiDiGraph() 56 | lines = code_text.split('\n') 57 | None_nodes = set() 58 | # print(g.number_of_nodes()) 59 | for node, ndata in g.nodes(data=True): 60 | # print(ndata) 61 | g_new.add_node(node) 62 | assert 'label' in ndata, f'label not in ndata for {node}: {ndata}' 63 | nds = ndata['label'].split('\\n') 64 | attrs = {'shape': ndata.get('shape', '')} 65 | # try: 66 | attrs['type'] = nds[1] if len(nds) >= 2 else '' 67 | # except: 68 | # print() 69 | 70 | # if 'Pragma' in ndata['label']: 71 | # print() 72 | 73 | content = _find_content(nds, lines) 74 | if content == 'None': 75 | None_nodes.add(node) 76 | attrs['content'] = content 77 | # print(content) 78 | nx.set_node_attributes(g_new, {node: attrs}) 79 | for nid1, nid2, edata in g.edges(data=True): 80 | elabel = edata['label'] 81 | g_new.add_edge(nid1, nid2, elabel=elabel) 82 | 83 | while True: 84 | remove_nodes = set() 85 | for node in g_new.nodes(): 86 | if len(list(g_new.neighbors(node))) == 0 and node in None_nodes: 87 | remove_nodes.add(node) 88 | for node in remove_nodes: 89 | g_new.remove_node(node) 90 | if not remove_nodes: 91 | break 92 | 93 | print_g(f'ast g_new after loading code content etc.', g_new, print_func=saver.log_info) 94 | return g_new 95 | 96 | 97 | def _find_content(nds, lines): 98 | # f = strip_(f) 99 | for s_id, s in enumerate(nds): 100 | if 'compilerGenerated' in s or 'NULL_FILE' in s or 'SgSourceFile' in s: 101 | return 'None' 102 | if 'physical line=' in s: 103 | if '.cpp' in s or '.c' in s: 104 | line_num = int(s.split('physical line=')[1].split(')')[0]) 105 | begin = int(s.split('raw line:col=')[1].split(')')[0].split(':')[1]) 106 | # try: 107 | end = int(nds[s_id + 1].split('raw line:col=')[1].split(')')[0].split(':')[1]) 108 | # except: 109 | # print(nds) 110 | # exit(-1) 111 | line = lines[line_num - 1] 112 | content = line[begin - 1:end] 113 | # print() 114 | return content 115 | return 'None' 116 | 117 | 118 | # def strip_(s): 119 | # return s.replace('_', '') 120 | 121 | 122 | def _get_dot_file(code_text, source_C_file, gname): 123 | ext = splitext(source_C_file)[1] 124 | dot_file_path = join(dirname(source_C_file), f'{gname}_rose{ext}.dot') 125 | 126 | # if gname == 'aes': 127 | # return None # TODO: fix aes 128 | 129 | if isfile(dot_file_path): 130 | saver.log_info(f'dot_file_path found; load') 131 | else: 132 | saver.log_info(f'dot_file_path not found; generate using rose (docker -- ensure docker is started)') 133 | # Step 1: Save the temporary c file. 134 | code_text_processed = code_text.replace('#include "merlin_type_define.h"', '//#include "merlin_type_define.h"') 135 | temp_code_file_path = join(dirname(source_C_file), f'{gname}_rose{ext}') 136 | with open(temp_code_file_path, 'w') as f: 137 | f.write(code_text_processed) 138 | # Step 2: Run docker rose. 139 | assert isfile(temp_code_file_path) 140 | cmd = f'docker run --rm -v {dirname(source_C_file)}:/root gleisonsdm/rose dotGenerator -c {basename(temp_code_file_path)}' 141 | saver.log_info(cmd) 142 | output = None 143 | try: 144 | output = check_output(cmd, shell=True, stderr=STDOUT).decode('utf-8') 145 | except CalledProcessError as e: 146 | saver.log_info(f'Error of rose docker: {e.output}') 147 | raise e # just continue 148 | saver.log_info(output) 149 | try: 150 | remove(temp_code_file_path) 151 | except OSError: 152 | pass 153 | # exit() 154 | 155 | return dot_file_path 156 | 157 | 158 | def main(): 159 | cmd = 'docker run --rm -it -v $(pwd):/root gleisonsdm/rose dotGenerator -c main.cpp' 160 | 161 | print(cmd) 162 | output = check_output(cmd, shell=True).decode('utf-8') 163 | print(output) 164 | 165 | 166 | if __name__ == '__main__': 167 | main() 168 | -------------------------------------------------------------------------------- /baselines/src/test.sh: -------------------------------------------------------------------------------- 1 | # run on 2nd instance of u5 2 | for kernel in gemm-ncubed # stencil 3 | do 4 | python3 run_tool_dse.py --kernel $kernel --benchmark machsuite --root-dir ../ --redis-port "6379" 5 | done 6 | -------------------------------------------------------------------------------- /baselines/src/text_encoder.py: -------------------------------------------------------------------------------- 1 | from config import FLAGS 2 | from transformers import RobertaTokenizer, RobertaModel 3 | import torch 4 | from tqdm import tqdm 5 | from saver import saver 6 | 7 | from gensim.models.doc2vec import Doc2Vec, TaggedDocument 8 | 9 | class TextEncoder(object): 10 | def encode(self, fulltext): 11 | raise NotImplementedError() 12 | 13 | def dim(self): 14 | raise NotImplementedError() 15 | 16 | 17 | class RobertaTextEncoder(TextEncoder): 18 | def __init__(self): 19 | self.roberta_model = RobertaModel.from_pretrained('roberta-base') 20 | self.roberta_tokenizer = RobertaTokenizer.from_pretrained('roberta-base') 21 | 22 | # def encode(self, list_of_text): 23 | # X = [] 24 | # for fulltext in list_of_text: 25 | # inputs = self.roberta_tokenizer(fulltext, return_tensors="pt") 26 | # output_emb = self.roberta_model(**inputs)['pooler_output'][:,:4] 27 | # X.append(output_emb) 28 | # X = torch.cat(X, dim=0) 29 | # return X 30 | 31 | def encode(self, fulltext): 32 | inputs = self.roberta_tokenizer(fulltext, return_tensors="pt") 33 | output_emb = self.roberta_model(**inputs)['pooler_output'] #[:, :4] 34 | return output_emb 35 | 36 | def dim(self): 37 | return 768 38 | 39 | 40 | class Word2VecTextEncoder(TextEncoder): 41 | def __init__(self): 42 | self.sentences = [] 43 | self.model = None 44 | self.dim_ = FLAGS.fulltext_dim 45 | assert self.dim_ > 0 46 | 47 | 48 | def add_sentence(self, sentence): 49 | assert type(sentence) is str 50 | self.sentences.append(TaggedDocument(sentence.split(), [len(self.sentences)])) 51 | 52 | 53 | def encode(self, fulltext): 54 | if self.model is None: # has not been trained 55 | saver.log_info('Training Doc2Vec...') 56 | self.model = Doc2Vec(self.sentences, vector_size=self.dim_, window=2, min_count=1, workers=4) 57 | saver.log_info(f'Doc2Vec trained with dv {len(self.model.dv)} and wv {len(self.model.wv)}') 58 | saver.log_info(f'Example sentence: {self.sentences[0]}') 59 | 60 | vector = self.model.infer_vector(fulltext.split()) 61 | return torch.tensor(vector.reshape(1, len(vector))) 62 | 63 | def dim(self): 64 | return self.dim_ 65 | 66 | 67 | def create_text_encoder(t): 68 | if t == 'roberta': 69 | return RobertaTextEncoder() 70 | elif t == 'word2vec': 71 | return Word2VecTextEncoder() 72 | else: 73 | return None 74 | 75 | 76 | -------------------------------------------------------------------------------- /baselines/src/utils_nn.py: -------------------------------------------------------------------------------- 1 | from nn import MyGlobalAttention 2 | from config import FLAGS 3 | import torch 4 | import torch.nn as nn 5 | from nn_att import MyGlobalAttention 6 | from torch.nn import Sequential, Linear, ReLU 7 | from utils import MLP 8 | 9 | def create_graph_att_module(D, return_gate_nn=False): 10 | def _node_att_gate_nn(D): 11 | if FLAGS.node_attention_MLP: 12 | return MLP(D, 1, 13 | activation_type=FLAGS.activation_type, 14 | hidden_channels=[D // 2, D // 4, D // 8], 15 | num_hidden_lyr=3) 16 | else: 17 | return Sequential(Linear(D, D), ReLU(), Linear(D, 1)) 18 | 19 | gate_nn = _node_att_gate_nn(D) 20 | glob = MyGlobalAttention(gate_nn, None) 21 | if return_gate_nn: 22 | return gate_nn, glob 23 | else: 24 | return glob 25 | 26 | 27 | class LpModule(torch.nn.Module): 28 | def __init__(self, p): 29 | super(LpModule, self).__init__() 30 | self.p = p 31 | 32 | def forward(self, x, **kwargs): 33 | x = x / (torch.norm(x, p=self.p, dim=-1).unsqueeze(-1) + 1e-12) 34 | return x 35 | -------------------------------------------------------------------------------- /data/HLSyn_data.tar.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/UCLA-DM/HLSyn/88ed5321811e5c9d838215f0a4f95f89b3ffcd02/data/HLSyn_data.tar.gz --------------------------------------------------------------------------------