├── CDC ├── 2016_DATA.json ├── 2017_DATA.json ├── 2018_DATA.json ├── 2019_DATA.json ├── 2020_DATA.json └── 2021_DATA.json ├── ICRA ├── 2016_DATA.json ├── 2017_DATA.json ├── 2018_DATA.json ├── 2019_DATA.json ├── 2020_DATA.json └── 2021_DATA.json ├── NeurIPS ├── 2016_DATA.json ├── 2017_DATA.json ├── 2018_DATA.json ├── 2019_DATA.json ├── 2020_DATA.json └── 2021_DATA.json ├── README.md ├── analyze.py ├── citation_count_box_plot.py ├── column_ids.py ├── config.yaml ├── json_to_csv.py ├── main.py └── readme-figures ├── Figure_1.png ├── Figure_2.png ├── Figure_3.png ├── Figure_4.png ├── Figure_5.png ├── Figure_6.png └── Figure_7.png /README.md: -------------------------------------------------------------------------------- 1 | # code-release 2 | 3 | This repository contains all the data and plotting scripts required to reproduce the plots in our paper ["What is the Impact of Releasing Code with Publications? Statistics from the Machine Learning, Robotics, and Control Communities."](https://ieeexplore.ieee.org/document/10621946) The preprint is available [here](https://arxiv.org/abs/2308.10008). 4 | 5 | ## Installation 6 | 7 | Install/upgrade Python3 dependencies: 8 | 9 | ```sh 10 | pip3 install --upgrade pip 11 | pip3 install pyyaml 12 | pip3 install tikzplotlib 13 | pip3 install matplotlib --upgrade 14 | ``` 15 | 16 | This was tested on macOS 13.3 with the following: 17 | 18 | ```sh 19 | anaconda 2022.10 20 | matplotlib 3.7.1 21 | pip 23.0.1 22 | python 3.9.13 23 | pyyaml 6.0 24 | tikzplotlib 0.10.1 25 | ``` 26 | 27 | ## Use 28 | 29 | Clone this repository and run its `main.py` script: 30 | 31 | ```sh 32 | git clone https://github.com/utiasDSL/code-release.git 33 | cd code-release/ 34 | python3 main.py 35 | ``` 36 | 37 | ## Output 38 | 39 | The script will sequentially generate the following figures: 40 | 41 | ![fig1](./readme-figures/Figure_1.png) 42 | ![fig2](./readme-figures/Figure_2.png) 43 | 44 | ![fig3](./readme-figures/Figure_3.png) 45 | ![fig4](./readme-figures/Figure_4.png) 46 | 47 | ![fig5](./readme-figures/Figure_5.png) 48 | ![fig6](./readme-figures/Figure_6.png) 49 | 50 | ![fig7](./readme-figures/Figure_7.png) 51 | 52 | ## Contribution 53 | 54 | Our determination of available open-source code for publications is not perfect. If we incorrectly associated your publication with or without code, please open a pull request with the correction. We appreciate your contributions! 55 | 56 | ## Citation 57 | 58 | Please cite our work [(paper)](https://ieeexplore.ieee.org/document/10621946) or [(preprint)](https://arxiv.org/abs/2308.10008) as: 59 | 60 | ```bibtex 61 | @ARTICLE{oscrelease2024, 62 | author={Zhou, Siqi and Brunke, Lukas and Tao, Allen and Hall, Adam W. and Bejarano, Federico Pizarro and Panerati, Jacopo and Schoellig, Angela P.}, 63 | journal={IEEE Control Systems Magazine}, 64 | title={What Is the Impact of Releasing Code With Publications? Statistics from the Machine Learning, Robotics, and Control Communities}, 65 | year={2024}, 66 | volume={44}, 67 | number={4}, 68 | pages={38-46}, 69 | doi={10.1109/MCS.2024.3402888} 70 | } 71 | ``` 72 | 73 | ----- 74 | > [Learning Systems and Robotics Lab](https://www.learnsyslab.org/) at the Technical University of Munich (TUM) and the University of Toronto 75 | -------------------------------------------------------------------------------- /analyze.py: -------------------------------------------------------------------------------- 1 | '''Plotting functions for histograms and stars-vs-citations ellipses 2 | 3 | ''' 4 | import matplotlib.pyplot as plt 5 | import numpy as np 6 | from sympy import * 7 | from sklearn.covariance import MinCovDet 8 | import tikzplotlib 9 | import json 10 | import pandas as pd 11 | 12 | 13 | # Fixes AttributeError when using a legend in matplotlib for tikzplotlib 14 | from matplotlib.lines import Line2D 15 | from matplotlib.legend import Legend 16 | Line2D._us_dashSeq = property(lambda self: self._dash_pattern[1]) 17 | Line2D._us_dashOffset = property(lambda self: self._dash_pattern[0]) 18 | Legend._ncol = property(lambda self: self._ncols) 19 | 20 | 21 | from column_ids import ColumnIDs 22 | 23 | 24 | def get_conf_year(file_name): 25 | ''' Returns the conference year from a file name, e.g., 26 | extracts 2016 from '2016_DATA.json' ''' 27 | return int(''.join(filter(str.isdigit, file_name))) 28 | 29 | 30 | def historical_papers(conf, spreadsheets, years, cfg): 31 | ''' Returns number of papers with and without code over the years 32 | 33 | Args: 34 | None 35 | ''' 36 | output_folder = cfg['SAVE_DIR'] 37 | plt_filename = output_folder + 'Percentage of Papers with Code at ' + conf 38 | 39 | for i in range(len(spreadsheets)): 40 | spreadsheets[i] = './' + conf + '/' + spreadsheets[i] 41 | 42 | ## Define data 43 | with_code = [] 44 | without_code = [] 45 | tot_papers = [] 46 | 47 | for sheet in spreadsheets: 48 | ## Open spreadsheet 49 | try: 50 | with open(sheet) as json_file: 51 | parsed_json = json.load(json_file) 52 | parsed_json = json.loads(parsed_json) 53 | df = pd.DataFrame(parsed_json) 54 | 55 | code_key = df.columns[ColumnIDs.CODE - 1] 56 | except: 57 | print('ERROR: file {} does not exist'.format(sheet)) 58 | return 59 | 60 | print('Opened:', sheet) 61 | 62 | with_code_num = 0 63 | without_code_num = 0 64 | 65 | last_row = len(df.index) 66 | year_tot = 0 67 | 68 | for r in range(last_row): 69 | year_tot += 1 70 | 71 | val = df[code_key].iloc[r] 72 | try: 73 | list_value = val.splitlines() 74 | except: 75 | without_code_num += 1 76 | continue 77 | 78 | found_code = False 79 | 80 | if conf in ['CDC', 'ICRA']: 81 | for item in list_value: 82 | if 'github.com' in item.lower(): 83 | found_code = True 84 | break 85 | else: 86 | for item in list_value: 87 | if item != 'None' and item != '[]': 88 | found_code = True 89 | 90 | if found_code: 91 | with_code_num += 1 92 | else: 93 | without_code_num += 1 94 | 95 | with_code.append(with_code_num) 96 | without_code.append(without_code_num) 97 | tot_papers.append(year_tot) 98 | year_tot = 0 99 | 100 | x_axis = np.arange(len(years)) 101 | 102 | # Graph double bar chart across time 103 | plt.bar(x_axis +0.2, with_code, width=0.4, label='Papers With Code', color='green') 104 | plt.bar(x_axis -0.2, tot_papers, width=0.4, label='Total Papers', color='grey') 105 | 106 | for i in range(len(years)): 107 | with_code_txt = str(round(with_code[i] / tot_papers[i] * 100, 1)) + '%' 108 | plt.text((x_axis +0.2)[i], with_code[i] + 1, with_code_txt, ha='center', fontsize=12) 109 | plt.text((x_axis -0.2)[i], tot_papers[i] + 1, tot_papers[i], ha='center', fontsize=12) 110 | 111 | plt.xticks(x_axis, years, fontsize=12) 112 | plt.yticks(fontsize=12) 113 | plt.legend(fontsize=12) 114 | 115 | plt_title = 'Percentage of Papers with Code at ' + conf 116 | plt.title(plt_title, fontsize=18) 117 | plt.xlabel('Conference Year', fontsize=14) 118 | plt.ylabel('Number of Papers', fontsize=14) 119 | 120 | tikzplotlib.save(plt_filename + '.tex') 121 | plt.savefig(plt_filename + '.png') 122 | plt.show() 123 | 124 | 125 | def plot_ellipsoid(mean, covariance, s, circle, line, year, color): 126 | eig_values, eig_vectors = np.linalg.eig(s * covariance) 127 | 128 | area = np.pi * np.sqrt(eig_values[0] * eig_values[1]) 129 | 130 | print('Area {}: {:.2f}'.format(year, area)) 131 | 132 | if max(eig_values) > eig_values[0]: 133 | eig_values = np.array([eig_values[1], eig_values[0]]).T 134 | eig_vectors = np.array([eig_vectors[:, 1], eig_vectors[:, 0]]) 135 | 136 | rotated_ellipse = eig_vectors @ np.diag(np.sqrt(eig_values)) @ np.vstack((circle[0], circle[1])) 137 | plt.plot(rotated_ellipse[0, :] + mean[0], rotated_ellipse[1, :] + mean[1], color + '-', 138 | label=year) 139 | 140 | rotated_line = eig_vectors @ np.diag(np.sqrt(eig_values)) @ np.vstack((line[0], line[1])) 141 | slope = np.abs(rotated_line[1, -1]/ rotated_line[0, -1]) 142 | 143 | print('Slope {}: {}'.format(year, slope)) 144 | plt.plot(rotated_line[0, :] + mean[0], rotated_line[1, :] + mean[1], color + '-') 145 | 146 | plt.plot(mean[0], mean[1], color + 'x') 147 | 148 | return area, slope 149 | 150 | 151 | def stars_vs_citations(conf, spreadsheets, cfg): 152 | ''' Plots a scatter plot of github stars versus citations for papers 153 | with github code. 154 | ''' 155 | output_folder = cfg['SAVE_DIR'] 156 | plt_filename = output_folder + 'Github Stars vs Paper Citations in ' + conf 157 | 158 | colors = ['b', 'g', 'r', 'c', 'm', 'y'] 159 | 160 | confidence = 0.99 161 | s = -2 * np.log(1 - confidence) 162 | 163 | angle = np.linspace(0, 2 * np.pi) 164 | 165 | areas = [] 166 | slopes = [] 167 | 168 | circle = [np.cos(angle), np.sin(angle)] 169 | line = [np.array([-1, 1]), np.array([0, 0])] 170 | 171 | for i in range(len(spreadsheets)): 172 | spreadsheets[i] = './' + conf + '/' + spreadsheets[i] 173 | 174 | for i, sheet in enumerate(spreadsheets): 175 | ## Define data 176 | paper_stars = [] 177 | paper_citations = [] 178 | paper_titles = [] 179 | 180 | ## Open spreadsheet 181 | try: 182 | with open(sheet) as json_file: 183 | parsed_json = json.load(json_file) 184 | parsed_json = json.loads(parsed_json) 185 | df = pd.DataFrame(parsed_json) 186 | 187 | citation_key = df.columns[ColumnIDs.CITATION - 1] 188 | star_key = df.columns[ColumnIDs.STAR - 1] 189 | title_key = df.columns[ColumnIDs.TITLE - 1] 190 | except: 191 | print('ERROR: file {} does not exist'.format(sheet)) 192 | return 193 | 194 | print('Opened:', sheet) 195 | year = get_conf_year(sheet) 196 | last_row = len(df.index) 197 | 198 | for r in range(last_row): 199 | val = df[star_key].iloc[r] 200 | val2 = df[citation_key].iloc[r] 201 | 202 | if val2 == -1 or val == -1: 203 | continue 204 | 205 | # Store the number of citations 206 | paper_citations.append(val2) 207 | 208 | # Store the number of stars 209 | paper_stars.append(val) 210 | 211 | # Store paper title 212 | val3 = df[title_key].iloc[r] 213 | paper_titles.append(val3) 214 | 215 | data = np.vstack((np.array(paper_stars), np.array(paper_citations))).T 216 | result = MinCovDet(assume_centered=False).fit(data) 217 | 218 | cov = result.covariance_ 219 | mean = result.location_ 220 | 221 | area, slope = plot_ellipsoid(mean, cov, s, circle, line, year, colors[i]) 222 | areas.append(area) 223 | slopes.append(slope) 224 | 225 | areas = np.array(areas[::-1]) 226 | diff_percentages_area = np.diff(areas) / areas[:-1] * 100.0 227 | print('diff changes area: ', ['{:.2f} %'.format(percent) for percent in diff_percentages_area]) 228 | 229 | slopes = np.array(slopes[::-1]) 230 | diff_percentages_slope = np.diff(slopes) / slopes[:-1] * 100.0 231 | print('diff changes slope: ', ['{:.2f} %'.format(percent) for percent in diff_percentages_slope]) 232 | 233 | plt.legend(fontsize=12) 234 | 235 | plt.xlabel('Github Stars', fontsize=14) 236 | plt.ylabel('Semantic Scholar Citations', fontsize=14) 237 | 238 | plt.xlim([0, 500]) 239 | plt.ylim([0, 500]) 240 | 241 | plt.title(conf) 242 | 243 | tikzplotlib.save(plt_filename + '.tex') 244 | plt.savefig(plt_filename + '.png') 245 | plt.show() 246 | -------------------------------------------------------------------------------- /citation_count_box_plot.py: -------------------------------------------------------------------------------- 1 | '''Plotting functions for box plots 2 | 3 | ''' 4 | import matplotlib.pyplot as plt 5 | import matplotlib.patheffects as path_effects 6 | import seaborn as sns 7 | import pandas as pd 8 | import tikzplotlib 9 | 10 | 11 | # Fixes AttributeError when using a legend in matplotlib for tikzplotlib 12 | from matplotlib.lines import Line2D 13 | from matplotlib.legend import Legend 14 | Line2D._us_dashSeq = property(lambda self: self._dash_pattern[1]) 15 | Line2D._us_dashOffset = property(lambda self: self._dash_pattern[0]) 16 | Legend._ncol = property(lambda self: self._ncols) 17 | 18 | 19 | def calc_percentage(minuend, subtrahend, denom): 20 | if abs(denom) <= 1e-8: 21 | return float('inf') 22 | return (minuend - subtrahend) / denom * 100.0 23 | 24 | 25 | def percentual_change(percentages): 26 | percentage_diff = [] 27 | for i in range(len(percentages)-1): 28 | percentage_diff.append(calc_percentage(percentages[i + 1], percentages[i], percentages[i])) 29 | 30 | return percentage_diff 31 | 32 | 33 | def print_percentages(stats_code, stats_no_code, name=''): 34 | percent_no_code = percentual_change(stats_no_code) 35 | percent_code = percentual_change(stats_code) 36 | print('{} NO code: {}'.format(name, stats_no_code)) 37 | print('Changes: ', ['{:.2f} %'.format(percent) for percent in percent_no_code]) 38 | print('{} W/ code: {}'.format(name, stats_code)) 39 | print('Changes: ', ['{:.2f} %'.format(percent) for percent in percent_code]) 40 | diff_percentage_quartile = [calc_percentage(percent_code[i], percent_no_code[i], percent_no_code[i]) for i in range(len(percent_no_code))] 41 | print('diff Changes: ', ['{:.2f} %'.format(percent) for percent in diff_percentage_quartile]) 42 | 43 | 44 | def add_percentile_labels(ax, percentile_name='Median', fmt='.1f'): 45 | # adapted from https://stackoverflow.com/a/63295846 by Christian Karcher 46 | no_code = [] 47 | code = [] 48 | 49 | # Get the lines and boxes in the box plot 50 | lines = ax.get_lines() 51 | boxes = [c for c in ax.get_children() if type(c).__name__ == 'PathPatch'] 52 | lines_per_box = int(len(lines) / len(boxes)) 53 | 54 | # Determine the elements to loop over (lines for medians, boxes for quartiles) 55 | if percentile_name == 'Median': 56 | plot_elements = lines[4:len(lines):lines_per_box] 57 | elif percentile_name == 'Third Quartile': 58 | plot_elements = boxes 59 | else: 60 | raise NotImplementedError 61 | 62 | for plot_element in plot_elements: 63 | # Determine x and y data for median or quartile 64 | if percentile_name == 'Median': 65 | x, y = (data.mean() for data in plot_element.get_data()) 66 | foreground = plot_element.get_color() 67 | elif percentile_name == 'Third Quartile': 68 | top_left = plot_element.get_path().vertices[2, :] 69 | top_right = plot_element.get_path().vertices[3, :] 70 | y = top_left[1] 71 | x = (top_right[0] - top_left[0]) / 2.0 + top_left[0] 72 | foreground = plot_element.get_edgecolor() 73 | else: 74 | raise NotImplementedError 75 | 76 | # Alternate between without OSC and with OSC data 77 | if len(code) >= len(no_code): 78 | no_code.append(y) 79 | else: 80 | code.append(y) 81 | 82 | # Add text for percentile 83 | text = ax.text(x, y, f'{y:{fmt}}', ha='center', va='center', 84 | fontweight='bold', color='white') 85 | 86 | # Create colored border around white text for contrast 87 | text.set_path_effects([ 88 | path_effects.Stroke(linewidth=3, foreground=foreground), 89 | path_effects.Normal(), 90 | ]) 91 | 92 | return code, no_code 93 | 94 | 95 | def citation_count_w_wo_code(conf, cfg): 96 | add_statistics = cfg['ADD_STATISTICS'] 97 | 98 | # Print conference name 99 | print('Conference {}'.format(conf)) 100 | 101 | # Set paths for data and plotting 102 | csv_file_name = conf + '/ALL_DATA.csv' 103 | figure_output_file = 'plots/Code Availability vs Citations in {} Box Plot.'.format(conf) 104 | 105 | # Read CSV file 106 | data_plt = pd.read_csv(csv_file_name) 107 | 108 | # The number of years for plotting the data 109 | years = list(set(data_plt.Year)) # get the years 110 | reversed_years = years[::-1] # reverse order of years 111 | 112 | # Create box plots 113 | ax = sns.boxplot(x='Year', y='Citations', hue='With Code', data=data_plt, showfliers=False, order=reversed_years) 114 | 115 | # Set labels 116 | ax.set_xticklabels(['{} ({})'.format(reversed_years[0] + 1 - year, year) for year in reversed_years]) 117 | ax.set_xlabel('Years since Publication (from {})'.format(reversed_years[0] + 1)) 118 | ax.set_ylabel('Semantic Scholar Citations') 119 | plt.title(conf) 120 | 121 | if add_statistics: 122 | percentile_names = ['Median', 'Third Quartile'] 123 | for percentile_name in percentile_names: 124 | print(percentile_name) 125 | # label the median and third quartiles on the plot 126 | code, no_code = add_percentile_labels(ax, percentile_name=percentile_name) 127 | # print changes in percentile for publications with and without OSC over the years 128 | print_percentages(code, no_code, name=percentile_name) 129 | 130 | # Create tikz figure and save PNG 131 | tikzplotlib.clean_figure() 132 | tikzplotlib.save(figure_output_file + 'tex') 133 | plt.savefig(figure_output_file + 'png') 134 | plt.show() 135 | -------------------------------------------------------------------------------- /column_ids.py: -------------------------------------------------------------------------------- 1 | from enum import IntEnum, unique 2 | 3 | 4 | @unique 5 | class ColumnIDs(IntEnum): 6 | '''A class that creates ids for relevant columns in the spreadsheet.''' 7 | 8 | CONFERENCE = 1 9 | YEAR = 2 10 | TITLE = 3 11 | AUTHOR = 4 12 | KEYWORD = 5 13 | BENCHMARK = 6 14 | RESULT = 7 15 | CODE = 8 16 | STAR = 9 17 | FORK = 10 18 | CITATION = 11 19 | CIT_OVER_TIME = 12 20 | -------------------------------------------------------------------------------- /config.yaml: -------------------------------------------------------------------------------- 1 | CONFERENCES: 2 | - CDC 3 | - NeurIPS 4 | - ICRA 5 | 6 | YEARS: 7 | - 2016 8 | - 2017 9 | - 2018 10 | - 2019 11 | - 2020 12 | - 2021 13 | 14 | SAVE_DIR: plots/ 15 | 16 | # Print statistics and plot median and third quartiles 17 | ADD_STATISTICS: True 18 | -------------------------------------------------------------------------------- /json_to_csv.py: -------------------------------------------------------------------------------- 1 | '''Data format utility functions. 2 | 3 | ''' 4 | import os 5 | import json 6 | import pandas as pd 7 | import ast 8 | 9 | from column_ids import ColumnIDs 10 | 11 | 12 | def drop_columns(sheet : str, columns_to_drop : list, conf : str) -> None: 13 | # Open sheet 14 | print('Opening', sheet) 15 | with open(sheet) as json_file: 16 | parsed_json = json.load(json_file) 17 | parsed_json = json.loads(parsed_json) 18 | df = pd.DataFrame(parsed_json) 19 | 20 | # Drop columns 21 | columns_to_drop = [i - 1 for i in columns_to_drop] 22 | columns_to_drop.sort(reverse=True) 23 | 24 | for i in columns_to_drop: 25 | df.drop(df.columns[i], axis=1, inplace=True) 26 | 27 | # Rename headers 28 | if conf in ['CDC', 'ICRA']: 29 | df = df.rename(columns={'Code Link': 'With Code'}) 30 | df = df.rename(columns={'Crossref Citations' : 'Citations'}) 31 | else: 32 | df = df.rename(columns={'Github' : 'With Code'}) 33 | 34 | # Drop rows with -1 citations 35 | df.drop(df[df.Citations == -1].index, inplace=True) 36 | 37 | # Swap order of With Code and Citations 38 | new_order = ['Year', 'Citations', 'With Code'] 39 | df = df.reindex(columns=new_order) 40 | 41 | # Set With Code column to True False basis 42 | if conf in ['CDC', 'ICRA']: 43 | df['With Code'] = [True if 'github' in str(row).lower() else False for row in df['With Code']] 44 | else: 45 | df['With Code'] = [True if len(ast.literal_eval(row)) > 0 else False for row in df['With Code']] 46 | 47 | # Reformat years if conf is IEEE 48 | if conf in ['CDC', 'ICRA']: 49 | df['Year'] = [str(row)[-4:] for row in df['Year']] 50 | 51 | print('-- Saving...') 52 | df.to_csv('.' + sheet.rsplit('.')[1] + '_code.csv', index=False, encoding='utf-8') 53 | 54 | 55 | def merge(spreadsheets : list, conf : str) -> None: 56 | df = [] 57 | 58 | for sheet in spreadsheets: 59 | sheet = '.' + sheet.rsplit('.')[1] + '_code.csv' 60 | print('Opening', sheet) 61 | df_tmp = pd.read_csv(sheet) 62 | 63 | print(df_tmp.loc[df_tmp['Citations'].idxmax()]) 64 | print(df_tmp.loc[df_tmp['Citations'].idxmin()]) 65 | 66 | df.append(df_tmp) 67 | 68 | # Merge sheets 69 | for i in range(1, len(df)): 70 | df[0] = pd.concat([df[0], df[i]], ignore_index=True) 71 | 72 | # Save sheet 73 | print('-- Saving...') 74 | df[0].to_csv('./' + conf + '/' + 'ALL_DATA.csv', index=False) 75 | 76 | 77 | def merge_sheets(spreadsheets, conf): 78 | columns_to_drop = [ColumnIDs.CONFERENCE, ColumnIDs.TITLE, ColumnIDs.AUTHOR, 79 | ColumnIDs.KEYWORD, ColumnIDs.BENCHMARK, ColumnIDs.RESULT, 80 | ColumnIDs.STAR, ColumnIDs.FORK, ColumnIDs.CIT_OVER_TIME] 81 | for i in range(len(spreadsheets)): 82 | spreadsheets[i] = './' + conf + '/' + spreadsheets[i] 83 | 84 | for sheet in spreadsheets: 85 | if conf in ['CDC', 'ICRA']: 86 | columns_to_drop = columns_to_drop[:-1] 87 | drop_columns(sheet, columns_to_drop, conf) 88 | 89 | # Merge all years into one csv 90 | merge(spreadsheets, conf) 91 | 92 | for sheet in spreadsheets: 93 | os.remove('.' + sheet.rsplit('.')[1] + '_code.csv') 94 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | '''Main plotting script 2 | 3 | Run as: 4 | 5 | $ python3 main.py 6 | 7 | ''' 8 | import yaml 9 | import os 10 | 11 | 12 | from analyze import historical_papers, stars_vs_citations 13 | from citation_count_box_plot import citation_count_w_wo_code 14 | from json_to_csv import merge_sheets 15 | 16 | 17 | def main(config): 18 | conferences = config['CONFERENCES'] 19 | years = config['YEARS'] 20 | spreadsheets = ['{}_DATA.json'.format(year) for year in years] 21 | 22 | for conf in conferences: 23 | # Figure 2 24 | historical_papers(conf, spreadsheets.copy(), years, config) 25 | # Figure 3 26 | merge_sheets(spreadsheets.copy(), conf) 27 | citation_count_w_wo_code(conf, cfg) 28 | 29 | # Figure 4 30 | stars_vs_citations('NeurIPS', spreadsheets.copy(), config) 31 | 32 | 33 | if __name__ == '__main__': 34 | # Configure correct function args with .yaml 35 | with open('config.yaml') as f: 36 | cfg = yaml.load(f, Loader=yaml.FullLoader) 37 | 38 | # Create folder for plots 39 | if not os.path.isdir(cfg['SAVE_DIR']): 40 | os.makedirs(cfg['SAVE_DIR']) 41 | 42 | main(cfg) 43 | -------------------------------------------------------------------------------- /readme-figures/Figure_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/utiasDSL/code-release/0b0f5275dd24c7851bcb7d08a1e30d8325b67ccf/readme-figures/Figure_1.png -------------------------------------------------------------------------------- /readme-figures/Figure_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/utiasDSL/code-release/0b0f5275dd24c7851bcb7d08a1e30d8325b67ccf/readme-figures/Figure_2.png -------------------------------------------------------------------------------- /readme-figures/Figure_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/utiasDSL/code-release/0b0f5275dd24c7851bcb7d08a1e30d8325b67ccf/readme-figures/Figure_3.png -------------------------------------------------------------------------------- /readme-figures/Figure_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/utiasDSL/code-release/0b0f5275dd24c7851bcb7d08a1e30d8325b67ccf/readme-figures/Figure_4.png -------------------------------------------------------------------------------- /readme-figures/Figure_5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/utiasDSL/code-release/0b0f5275dd24c7851bcb7d08a1e30d8325b67ccf/readme-figures/Figure_5.png -------------------------------------------------------------------------------- /readme-figures/Figure_6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/utiasDSL/code-release/0b0f5275dd24c7851bcb7d08a1e30d8325b67ccf/readme-figures/Figure_6.png -------------------------------------------------------------------------------- /readme-figures/Figure_7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/utiasDSL/code-release/0b0f5275dd24c7851bcb7d08a1e30d8325b67ccf/readme-figures/Figure_7.png --------------------------------------------------------------------------------