├── LICENSE.txt ├── README.md ├── data ├── EVAL1 │ ├── MethodA.list │ ├── MethodA │ │ ├── SF3-TF1 │ │ │ ├── 1.wav │ │ │ └── 2.wav │ │ └── SM3-TF1 │ │ │ ├── 1.wav │ │ │ └── 2.wav │ ├── MethodB.list │ ├── MethodB │ │ ├── SF3-TF1 │ │ │ ├── 1.wav │ │ │ └── 2.wav │ │ └── SM3-TF1 │ │ │ ├── 1.wav │ │ │ └── 2.wav │ ├── Target.list │ ├── Target │ │ └── TF1 │ │ │ ├── 1.wav │ │ │ └── 2.wav │ └── Target_ref.list └── example │ ├── MOS │ ├── exp1.wav │ └── exp2.wav │ └── XAB │ ├── neg.wav │ ├── pos.wav │ └── ref.wav ├── requirements.txt ├── run_post.py ├── run_pre.py ├── run_test.py ├── src ├── __init__.py ├── bin │ ├── __init__.py │ ├── eval_io.py │ └── eval_statistic.py ├── user │ ├── __init__.py │ └── info_class.py └── utils │ ├── __init__.py │ ├── dict_filter.py │ └── eval_io.py └── yml ├── EVAL1 ├── evaluation.yml ├── record.yml ├── test_system.yml └── text.yml └── template ├── MOS.yml ├── PK.yml ├── SIM.yml └── XAB.yml /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [2019] [Yi-Chiao Wu] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Python Version](https://img.shields.io/badge/Python-3.5%2C%203.6-green.svg)](https://img.shields.io/badge/Python-3.5%2C%203.6-green.svg) 2 | 3 | 4 | Speech Quality Subjective Evaluation 5 | ====== 6 | 7 |

This repository provides four general subjective tests (MOS: mean opinion score; PK: preference test; SIM: speaker similarity test; XAB: XAB test) for voice conversion(VC) and speech synthesis (SS) evaluations. The evaluation results will output in an Excel file with statistic results.

8 | 9 | ## Subjective evaluations 10 | 11 |

For VC/SS, subjective evaluations are usually conducted. The general performance measurements of VC/SS are speech quality (the naturalness of converted/synthesized speech) and speaker similarity (the similarity between converted speech and target speech). The repository includes four types of subjective tests for VC/SS performance evaluation. Each test plays several speech files to listeners and ask them to give a specific score to evaluate each converted/synthesized file. The system will output an average score with a confidence interval of each test into a Excel file.

12 | 13 | ### Mean opinion score (MOS): 14 | 15 | - Speech quality (naturalness) evaluation 16 | - System plays a speech file to a listener and asks him/her to give a score (1-5) to evaluate the speech quality of this speech file. 17 | 18 | ### Speaker similarity test (SIM): 19 | 20 | - Speaker similarity evaluation 21 | - Proposed by the [Voice Conversion Challenge (VCC)](http://www.vc-challenge.org/) organize 22 | - System plays a golden and a predicted speech files to a listener and asks him/her to measure that these two speeches are from the same speaker or not. 23 | - Four measurements are given: 1. *Definitely the same*; 2. *Maybe the same*; 3. *Maybe different*; 4. *Definitely different*. 24 | 25 | ### Preference test (PK): 26 | 27 | - Speech quality or speaker similarity evaluations 28 | - System plays two speech files of different methods to a listener and asks him/her to pick up the file with better performance. 29 | 30 | ### XAB test: 31 | 32 | - Speaker similarity or speech quality evaluation 33 | - System plays a golden file and two speech files of different methods to a listener and ask him/her to pick up the file that is more similar to the golden speech. 34 | 35 | --- 36 | ## Setup 37 | 38 | ### Install requirements 39 | 40 | ``` 41 | pip install -r requirements.txt 42 | 43 | # For Mac user 44 | pip install -U PyObjC 45 | ``` 46 | 47 | ### Folder structure 48 | 49 | - data/{project}/: testing data of {project}. 50 | - data/example/: example speech files of XAB and MOS tests. 51 | - results: output Excel file. 52 | - src: source code. 53 | - yml: testing configs and profiles. 54 | 55 | --- 56 | ## Usage 57 | - Take project `EVAL1` as an example. 58 | 59 | ### 1. Data preparation 60 | - Create `data/EVAL1/` folder. 61 | - Put testing data of each method in `data/EVAL1/{method}`. 62 | - Create data lists of all methods and put them in the same folder. 63 | 64 | ### 2. Config initialization 65 | - Create `yml/EVAL1/` folder 66 | - Create `evaluation.yml`, `record.yml`, `test_system.yml`, and `text.yml` in `yml/EVAL1/` folder. 67 | 68 | ### 3. Parse testing profile 69 | - Create .yml format testing profile files for each testing type and subset corresponding to the `evaluation.yml` config. 70 | ``` 71 | python run_pre.py EVAL1 72 | ``` 73 | 74 | ### 4. Run evaluation 75 | - Run evaluation and the results will be in the `/result/EVAL1_{type}.xlsx` files. Each listener's results will be in the `yml/EVAL1/{type}_{subset}_{userID}.yml` files. 76 | - **-o {mode}**: set mode to control the output information; choice:[vc, as, others]. 77 | - Set mode **'vc'** will output complete details and mode **'others'** will only output overall results. 78 | ``` 79 | python run_test.py EVAL1 -o {mode} 80 | ``` 81 | 82 | ### 5. Get statistics 83 | - Output evaluation results with statistics to the `/result/EVAL1_final_{type}.xlsx` files. 84 | ``` 85 | python run_post.py EVAL1 -o {mode} 86 | ``` 87 | 88 | --- 89 | ## COPYRIGHT 90 | 91 | Copyright (c) 2020 Yi-Chiao Wu @ Nagoya University ([@bigpon](https://github.com/bigpon)) 92 | E-mail: `yichiao.wu@g.sp.m.is.nagoya-u.ac.jp` 93 | Released under [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0) 94 | 95 | 96 | -------------------------------------------------------------------------------- /data/EVAL1/MethodA.list: -------------------------------------------------------------------------------- 1 | data/EVAL1/MethodA/SF3-TF1/1.wav 2 | data/EVAL1/MethodA/SF3-TF1/2.wav 3 | data/EVAL1/MethodA/SM3-TF1/1.wav 4 | data/EVAL1/MethodA/SM3-TF1/2.wav -------------------------------------------------------------------------------- /data/EVAL1/MethodA/SF3-TF1/1.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/EVAL1/MethodA/SF3-TF1/1.wav -------------------------------------------------------------------------------- /data/EVAL1/MethodA/SF3-TF1/2.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/EVAL1/MethodA/SF3-TF1/2.wav -------------------------------------------------------------------------------- /data/EVAL1/MethodA/SM3-TF1/1.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/EVAL1/MethodA/SM3-TF1/1.wav -------------------------------------------------------------------------------- /data/EVAL1/MethodA/SM3-TF1/2.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/EVAL1/MethodA/SM3-TF1/2.wav -------------------------------------------------------------------------------- /data/EVAL1/MethodB.list: -------------------------------------------------------------------------------- 1 | data/EVAL1/MethodB/SF3-TF1/1.wav 2 | data/EVAL1/MethodB/SF3-TF1/2.wav 3 | data/EVAL1/MethodB/SM3-TF1/1.wav 4 | data/EVAL1/MethodB/SM3-TF1/2.wav -------------------------------------------------------------------------------- /data/EVAL1/MethodB/SF3-TF1/1.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/EVAL1/MethodB/SF3-TF1/1.wav -------------------------------------------------------------------------------- /data/EVAL1/MethodB/SF3-TF1/2.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/EVAL1/MethodB/SF3-TF1/2.wav -------------------------------------------------------------------------------- /data/EVAL1/MethodB/SM3-TF1/1.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/EVAL1/MethodB/SM3-TF1/1.wav -------------------------------------------------------------------------------- /data/EVAL1/MethodB/SM3-TF1/2.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/EVAL1/MethodB/SM3-TF1/2.wav -------------------------------------------------------------------------------- /data/EVAL1/Target.list: -------------------------------------------------------------------------------- 1 | data/EVAL1/Target/TF1/1.wav 2 | data/EVAL1/Target/TF1/2.wav -------------------------------------------------------------------------------- /data/EVAL1/Target/TF1/1.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/EVAL1/Target/TF1/1.wav -------------------------------------------------------------------------------- /data/EVAL1/Target/TF1/2.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/EVAL1/Target/TF1/2.wav -------------------------------------------------------------------------------- /data/EVAL1/Target_ref.list: -------------------------------------------------------------------------------- 1 | data/EVAL1/Target/TF1/1.wav 2 | data/EVAL1/Target/TF1/2.wav 3 | data/EVAL1/Target/TF1/1.wav 4 | data/EVAL1/Target/TF1/2.wav -------------------------------------------------------------------------------- /data/example/MOS/exp1.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/example/MOS/exp1.wav -------------------------------------------------------------------------------- /data/example/MOS/exp2.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/example/MOS/exp2.wav -------------------------------------------------------------------------------- /data/example/XAB/neg.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/example/XAB/neg.wav -------------------------------------------------------------------------------- /data/example/XAB/pos.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/example/XAB/pos.wav -------------------------------------------------------------------------------- /data/example/XAB/ref.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/data/example/XAB/ref.wav -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | numpy 2 | scipy 3 | h5py 4 | docopt 5 | pyyaml 6 | playsound 7 | openpyxl 8 | pathlib 9 | argparse -------------------------------------------------------------------------------- /run_post.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | # Copyright 2020 Wu Yi-Chiao (Nagoya University) 4 | # Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0) 5 | 6 | """VCSS subjective evaluation 7 | 8 | Usage: run_post.py [-h] [-o MODE] EVALNAME 9 | 10 | Options: 11 | -h, --help Show the help 12 | -o, MODE The output mode choice:["vc", "as", and others] 13 | EVALNAME The name of the evaluation 14 | 15 | """ 16 | import os 17 | import sys 18 | import numpy as np 19 | import yaml 20 | from docopt import docopt 21 | from src.user.info_class import UserResult 22 | from src.bin.eval_statistic import xlsx_fSCORE, StatisticResult 23 | from src.utils.dict_filter import tpair_filter 24 | 25 | CONFIDENCE = 0.95 26 | 27 | # FINAL OUTPUT TEMPLATE CLASS 28 | class final_output(object): 29 | 30 | def __init__(self, test_systems, 31 | test_type, test_pairs, 32 | fxlsx, conf_user, 33 | yml_path, c_text, 34 | confidence=0.95): 35 | """CLASS TO OUTPUT THE FINAL RESULT INTO XLSX 36 | Args: 37 | test_systems (list): the list of test systems 38 | test_type (str): the type of the subjective test 39 | ['MOS', 'XAB', 'SIM', 'PK'] 40 | test_pairs (list): the list of test pair 41 | ['summary', 'female', 'male', 'xgender', 'sgender', 42 | 'F-F', 'F-M', 'M-F', 'M-M'] 43 | fxlsx (str): the file name of the xslx file 44 | conf_user (dict): the config file of the information of all users 45 | yml_path (str): the path of the .yml files 46 | c_text (dixt): the dict of config text 47 | confidence (float): the confidence value of the confidence interval 48 | """ 49 | self.c_xlsx = xlsx_fSCORE(fxlsx, test_pairs, test_systems, c_text) 50 | self.t_systems = test_systems 51 | self.t_pairs = test_pairs 52 | self.t_type = test_type 53 | self.yml_path = yml_path 54 | self.c_text = c_text 55 | self.confidence = confidence 56 | self.user_results = [] 57 | self._load_user_result(conf_user[c_text['user_name']], 58 | conf_user[c_text['t_subset']]) 59 | self.stat = {} 60 | for t_pair in self.t_pairs: 61 | self.stat = {**self.stat, t_pair:StatisticResult(self.c_xlsx, t_pair)} 62 | 63 | def output_result(self): 64 | """OUTPUT FINAL SUBJECTIVE RESULTS: 65 | Output each user's result and the total result 66 | """ 67 | self.total_results = [] 68 | # output result of each user 69 | for user_result in self.user_results: 70 | if user_result.finished: 71 | t_name = user_result.name 72 | for t_pair in self.t_pairs: 73 | t_dict = tpair_filter(t_pair, user_result.recordf_user) 74 | self.c_xlsx.output_result(t_name, t_pair, t_dict) 75 | else: 76 | print("%s didn't finish all tests." % user_result.name) 77 | # output total result 78 | if self.t_type == 'MOS': # utterance-based score 79 | scores = self.c_xlsx.utt_score 80 | elif self.t_systems == 'SIM': 81 | raise NotImplementedError() 82 | else: # user-based score 83 | scores = self.c_xlsx.user_score 84 | for t_system in self.t_systems: 85 | for t_pair in self.t_pairs: 86 | self.stat[t_pair].push_result(scores[t_pair][t_system], self.confidence) 87 | for t_pair in self.t_pairs: 88 | self.stat[t_pair].output_result() 89 | 90 | def _load_user_result(self, user_names, user_sets): 91 | for user_name, user_set in zip(user_names, user_sets): 92 | self.user_results.append(UserResult(user_name, user_set, 93 | self.t_type, self.yml_path, 94 | self.c_text)) 95 | 96 | def main(eval_name, yml_path, results_path, test_pair, confidence): 97 | #LOAD CONSTANT TEXT 98 | c_textf = '%stext.yml'%(yml_path) 99 | if not os.path.exists(c_textf): 100 | print("%s doesn't exist!" % c_textf) 101 | sys.exit(0) 102 | with open(c_textf, 'r', encoding='utf-8') as yf: 103 | constant_text = yaml.safe_load(yf) 104 | c_text = constant_text['c_text'] 105 | #LOAD EXPERIMENTAL INFO 106 | # fconf_user: the config file of the information of all users 107 | # conf_user (dict): 108 | # MAX_count: the total number of the testing subsets 109 | # Subject_name: the list of users 110 | # Subject_set: the corresponding subset index of each user 111 | # count: the index of the current subset 112 | # time: last updated time 113 | fconf_user = "%s%s.yml" % (yml_path, c_text['recordf']) 114 | if not os.path.exists(fconf_user): 115 | print("%s doesn't exist!"%fconf_user) 116 | sys.exit(0) 117 | with open(fconf_user, "r", encoding='utf-8') as yf: 118 | conf_user = yaml.safe_load(yf) 119 | # fconf: the config file of all evaluations information 120 | # conf (list of dict): 121 | # method: evaluated methods 122 | # set: the name of each subset 123 | # type: the type of the test (MOS, SIM, XAB, PK) 124 | fconf = "%s%s.yml"%(yml_path, c_text['configf']) 125 | if not os.path.exists(fconf): 126 | print("%s doesn't exist!"%fconf) 127 | sys.exit(0) 128 | with open(fconf, "r", encoding='utf-8') as yf: 129 | conf = yaml.safe_load(yf) 130 | 131 | for tconf in conf: 132 | if tconf[c_text['t_type']] == 'MOS': 133 | mos_systems = tconf[c_text['system']] 134 | mos_fxlsx = "%s%s_final_MOS.xlsx" % (results_path, eval_name) 135 | f_output = final_output(mos_systems, 136 | 'MOS', test_pair, 137 | mos_fxlsx, conf_user, 138 | yml_path, c_text, 139 | confidence) 140 | elif tconf[c_text['t_type']] == 'XAB': 141 | xab_systems = tconf[c_text['system']] 142 | xab_fxlsx = "%s%s_final_XAB.xlsx" % (results_path, eval_name) 143 | f_output = final_output(xab_systems, 144 | 'XAB', test_pair, 145 | xab_fxlsx, conf_user, 146 | yml_path, c_text, 147 | confidence) 148 | elif tconf[c_text['t_type']] == 'SIM': 149 | sim_systems = tconf[c_text['system']] 150 | sim_fxlsx = "%s%s_final_SIM.xlsx" % (results_path, eval_name) 151 | f_output = final_output(sim_systems, 152 | 'SIM', test_pair, 153 | sim_fxlsx, conf_user, 154 | yml_path, c_text, 155 | confidence) 156 | elif tconf[c_text['t_type']] == 'PK': 157 | pk_systems = tconf[c_text['system']] 158 | pk_fxlsx = "%s%s_final_PK.xlsx" % (results_path, eval_name) 159 | f_output = final_output(pk_systems, 160 | 'PK', test_pair, 161 | pk_fxlsx, conf_user, 162 | yml_path, c_text, 163 | confidence) 164 | else: 165 | print("type %s is not defined!" % tconf[c_text['t_type']]) 166 | return 0 167 | 168 | f_output.output_result() 169 | 170 | # MAIN 171 | if __name__ == "__main__": 172 | args = docopt(__doc__) # pylint: disable=invalid-name 173 | eval_name = args['EVALNAME'] 174 | yml_path = "yml/%s/" % eval_name 175 | result_path = "results/" 176 | if args['-o'] == 'vc': 177 | test_pair = ['summary', 'xgender', 'sgender', 'F-F', 'F-M', 'M-F', 'M-M'] 178 | elif args['-o'] == 'as': 179 | test_pair = ['summary', 'female', 'male'] 180 | else: 181 | test_pair = ['summary'] 182 | main(eval_name, yml_path, result_path, test_pair, CONFIDENCE) 183 | 184 | 185 | 186 | 187 | 188 | 189 | -------------------------------------------------------------------------------- /run_pre.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | # Copyright 2020 Wu Yi-Chiao (Nagoya University) 4 | # Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0) 5 | 6 | """VCSS subjective evaluation pre processing 7 | 8 | Usage: run_pre.py [-h] EVALNAME 9 | 10 | Options: 11 | -h, --help Show the help 12 | EVALNAME The name of the evaluation 13 | 14 | """ 15 | import os 16 | import sys 17 | import yaml 18 | import numpy as np 19 | from docopt import docopt 20 | from src.user.info_class import ParserConf 21 | 22 | # MAIN 23 | if __name__ == "__main__": 24 | args = docopt(__doc__) 25 | # print(args) 26 | # PATH INITIALIZATION 27 | eval_name = args['EVALNAME'] 28 | yml_path = "yml/%s/" % eval_name 29 | data_path = "data/%s/" % eval_name 30 | template_path = "yml/template/" 31 | #LOAD CONSTANT TEXT 32 | c_textf = '%stext.yml'%(yml_path) 33 | if not os.path.exists(c_textf): 34 | print("%s doesn't exist!" % c_textf) 35 | sys.exit(0) 36 | with open(c_textf, 'r', encoding='utf-8') as yf: 37 | constant_text = yaml.safe_load(yf) 38 | text = constant_text['text'] 39 | c_text = constant_text['c_text'] 40 | # PARSE SUBSET .yml 41 | t_parser = ParserConf(data_path, template_path, yml_path, c_text) 42 | t_parser.subset_gen() 43 | 44 | 45 | 46 | -------------------------------------------------------------------------------- /run_test.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | # Copyright 2020 Wu Yi-Chiao (Nagoya University) 4 | # Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0) 5 | 6 | """VC subjective evaluation 7 | 8 | Usage: run_test.py [-h] [-o MODE] EVALNAME 9 | 10 | Options: 11 | -h, --help Show the help 12 | -o, MODE The output mode choice:["vc", "as", and others] 13 | EVALNAME The name of the evaluation 14 | 15 | """ 16 | import os 17 | import sys 18 | import numpy as np 19 | import yaml 20 | from docopt import docopt 21 | from src.user.info_class import UserInfo 22 | from src.bin.eval_io import main_XAB, main_PK, main_MOS, main_SIM 23 | from src.bin.eval_io import playspeech, introduction, xlsx_SCORE, xlsx_SIM 24 | from src.utils.dict_filter import tpair_filter 25 | 26 | def evaluation(eval_dict, test_type, text): 27 | """RUN EVALUATIONS 28 | Args: 29 | eval_dict (list of dict): the result list of dicts of the test 30 | test_type (str): the type of the test 31 | text (dict): the dictionary of constant text 32 | Return: 33 | ans : 'q' for quit, float or int values for scores 34 | """ 35 | ans = 'initial' 36 | r_idx = -len(eval_dict) 37 | for d_idx in np.random.permutation(len(eval_dict)): 38 | if test_type == 'PK': 39 | ans = main_PK(eval_dict[d_idx], r_idx, text) 40 | elif test_type == 'SIM': 41 | ans = main_SIM(eval_dict[d_idx], r_idx, text) 42 | elif test_type == 'MOS': 43 | ans = main_MOS(eval_dict[d_idx], r_idx, text) 44 | elif test_type == 'XAB': 45 | ans = main_XAB(eval_dict[d_idx], r_idx, text) 46 | else: 47 | print("test type %s is not in (MOS, SIM, PK, XAB)!"%(test_type)) 48 | sys.exit(0) 49 | r_idx += 1 50 | if ans == 'q': 51 | print("Quit evaluation...") 52 | return ans 53 | return ans 54 | 55 | # OUTPUT RESULTS 56 | def output_results(total_dict, test_type, test_system, 57 | user_name, eval_name, result_path, 58 | text, test_pairs): 59 | """OUTPUT SUBJECTIVE RESULTS to a EXCEL FILE 60 | Args: 61 | total_dict (list of dict): the result dicts 62 | test_type (str): the type of the test [SIM, XAB, PK, MOS] 63 | test_system (list): the list of systems corresponding to the test type 64 | user_name (str): the name of the listener 65 | eval_name (str): the name of the evaluation 66 | result_path (str): the path to output result .xlsx file 67 | text (dict): the dictionary of constant text 68 | test_pairs (list): the list of the output conditions 69 | """ 70 | 71 | fxlsx = "%s%s_%s.xlsx" % (result_path, eval_name, test_type) 72 | if test_type == 'PK' or test_type == 'MOS' or test_type == 'XAB': 73 | c_xlsx = xlsx_SCORE(user_name, fxlsx, test_pairs, test_system) 74 | elif test_type == 'SIM': 75 | c_xlsx = xlsx_SIM(user_name, fxlsx, test_pairs, test_system, text) 76 | else: 77 | print("test type %s is not in (MOS, SIM, PK, XAB)!" % (test_type)) 78 | sys.exit(0) 79 | # output results 80 | for t_pair in test_pairs: 81 | t_dict = tpair_filter(t_pair, total_dict) 82 | c_xlsx.output_xlsx(t_pair, t_dict) 83 | 84 | def main(eval_name, yml_path, results_path, test_pair): 85 | print("\n##############################################################\n" + 86 | "#"+("%s Subjective Test" % eval_name).center(60) + "#\n" + 87 | "##############################################################\n") 88 | #LOAD CONSTANT TEXT 89 | c_textf = '%stext.yml'%(yml_path) 90 | if not os.path.exists(c_textf): 91 | print("%s doesn't exist!" % c_textf) 92 | sys.exit(0) 93 | with open(c_textf, 'r', encoding='utf-8') as yf: 94 | constant_text = yaml.safe_load(yf) 95 | text = constant_text['text'] 96 | c_text = constant_text['c_text'] 97 | examples = constant_text['examples'] 98 | #GET USER INFORMATION 99 | user = UserInfo(yml_path, c_text) 100 | while user.flag: 101 | user_name = input("Please enter your name or 'q' to quit: ") 102 | while user_name == "": # empty input, ask user inputs again 103 | user_name = input("Please enter your name or 'q' to quit: ") 104 | if user_name == 'q': # quit 105 | print('Quit evaluation.') 106 | return 107 | # progess check 108 | if user.check_user(user_name): # load and check the progress of the user 109 | action = input("The user name already exists !!" + 110 | "please enter any key to change the name, " + 111 | "or enter 'q' to quit") 112 | if action == 'q': 113 | print('Quit evaluation.') 114 | return 115 | #EVALUATION 116 | for t_idx in user.t_idxs: 117 | if user.finished[t_idx]: 118 | print("PART %d is already finished!\n" % (t_idx)) 119 | else: 120 | print("\n--------------------------------------------------------------") 121 | print("-" + ("PART %d: %s Test" % 122 | (t_idx, user.test_type[t_idx])).center(60) + "-") 123 | print("--------------------------------------------------------------") 124 | print("PART %d Progress: %d/%d\n" % 125 | (t_idx, len(user.total_dict[t_idx]) - len(user.eval_dict[t_idx]), 126 | len(user.total_dict[t_idx]))) 127 | if user.initial[t_idx]: 128 | introduction(user.test_type[t_idx], text, examples) 129 | input("Press any key to start the evaluations!\n") 130 | action = evaluation(user.eval_dict[t_idx], user.test_type[t_idx], text) 131 | user.save_result(t_idx) 132 | if user.finished[t_idx]: 133 | output_results(user.total_dict[t_idx], user.test_type[t_idx], 134 | user.test_system[t_idx], user_name, 135 | eval_name, results_path, 136 | text, test_pair) 137 | if action == 'q': 138 | return 139 | if t_idx != user.t_idxs[-1]: 140 | action = input("Press any key to start the next PART or 'q' to quit!") 141 | if action == 'q': 142 | return 143 | print("All PARTs are completed. Thank you!!") 144 | 145 | # MAIN 146 | if __name__ == "__main__": 147 | args = docopt(__doc__) # pylint: disable=invalid-name 148 | eval_name = args['EVALNAME'] 149 | yml_path = "yml/%s/" % eval_name 150 | result_path = "results/" 151 | if args['-o'] == 'vc': # voice conversion mode 152 | test_pair = ['summary', 'xgender', 'sgender', 'F-F', 'F-M', 'M-F', 'M-M'] 153 | vc_flag = True 154 | elif args['-o'] == 'as': # anasis-synthesis mode 155 | test_pair = ['summary', 'female', 'male'] 156 | else: # basic mode 157 | test_pair = ['summary'] 158 | main(eval_name, yml_path, result_path, test_pair) 159 | 160 | 161 | 162 | 163 | 164 | 165 | -------------------------------------------------------------------------------- /src/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/src/__init__.py -------------------------------------------------------------------------------- /src/bin/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/src/bin/__init__.py -------------------------------------------------------------------------------- /src/bin/eval_io.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | # Copyright 2020 Wu Yi-Chiao (Nagoya University) 4 | # Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0) 5 | 6 | import sys 7 | import os 8 | import random 9 | import time 10 | import platform 11 | from openpyxl import Workbook 12 | from openpyxl import load_workbook 13 | try: 14 | import winsound 15 | except ImportError: 16 | from playsound import playsound 17 | 18 | def playspeech(filename): 19 | if os.path.exists(filename): 20 | if platform.system() == 'Windows': 21 | winsound.PlaySound(filename, winsound.SND_ALIAS) 22 | else: 23 | playsound(filename) 24 | else: 25 | print("%s doesn't exists!!"%filename) 26 | sys.exit(0) 27 | 28 | def introduction(test_type, text, examples): 29 | """TEST INTRODUCTION 30 | Args: 31 | test_type (str): the type of the test 32 | """ 33 | if test_type == 'PK': 34 | info = "'Each PK' test will play two files," + \ 35 | " and please select the answer" + \ 36 | " with %s %s.\n" % (text['PK']['Ans'], text['PK']['C']) 37 | print(info) 38 | elif test_type == 'SIM': 39 | info = "Each 'SIM' test will play two files" + \ 40 | " and please select the answer" + \ 41 | " correspoinding to the question.\n" 42 | print(info) 43 | elif test_type == 'MOS': 44 | info = "Each 'MOS' test will play one file," + \ 45 | " and please select the score" + \ 46 | " correspoinding to the question." 47 | print(info) 48 | print("There are several examples with score 5\n") 49 | ans = input("Press any key to play the examples or 'q' to skip it!\n") 50 | if ans == "q": 51 | print("skip the examples.") 52 | else: 53 | for i, file in enumerate(examples['MOS']): 54 | print("example%d" % i) 55 | playspeech(file) 56 | print('\n') 57 | elif test_type == 'XAB': 58 | info = "Each 'XAB' test will play 'reference', 'A'" + \ 59 | " and 'B' files, please selct the %s" % text['XAB']['C']+ \ 60 | " of 'A' or 'B' is %s to that of the reference." % text['XAB']['Ans'] 61 | print(info) 62 | print("There is an example\n") 63 | ans = input("Press any key to play the examples or 'q' to skip it!\n") 64 | if ans == "q": 65 | print("skip the examples.") 66 | else: 67 | print("example reference") 68 | playspeech(examples['XAB']['ref']) 69 | print("postive example (correct answer)") 70 | playspeech(examples['XAB']['pos']) 71 | print("negative example (wrong answer)") 72 | playspeech(examples['XAB']['neg']) 73 | print('\n') 74 | else: 75 | raise ValueError( 76 | "test type %s is not in (MOS, SIM, PK, XAB)!"%(test_type)) 77 | 78 | # XAB test 79 | def main_XAB(eval_dict, r_idx, text): 80 | # MethodA(expected answer) v.s. MethodB with the reference MethodX 81 | fileX = eval_dict['FileX'] 82 | answer = random.randint(0, 1) 83 | #print(answer) 84 | if answer: 85 | # expected answer is fileA 86 | fileA = eval_dict['FileA'] 87 | fileB = eval_dict['FileB'] 88 | else: 89 | # expected answer is fileB 90 | fileA = eval_dict['FileB'] 91 | fileB = eval_dict['FileA'] 92 | #print(fileX) 93 | #print(fileA) 94 | #print(fileB) 95 | print("*"+("[No. %4d]-0 reference wav file" % (r_idx)).center(20)+"*") 96 | playspeech(fileX) 97 | time.sleep(0.3) 98 | print("*"+("[No. %4d]-1 'A' wav file" % (r_idx)).center(20)+"*") 99 | playspeech(fileA) 100 | time.sleep(0.3) 101 | print("*"+("[No. %4d]-2 'B' wav file" % (r_idx)).center(20)+"*") 102 | playspeech(fileB) 103 | ans_flag = True 104 | Criteria = text['XAB']['C'] 105 | Ans = text['XAB']['Ans'] 106 | Q = text['XAB']['Q'].replace('Ans', Ans).replace('Criteria', Criteria) 107 | while ans_flag: 108 | print("\n%s" % Q) 109 | ans = input(("%-6s[1].%-22s[2].%-22s\n" % 110 | ("Enter ", " 'A' is %s" % Ans, " 'B' is %s" % Ans)) + \ 111 | ("%-6s[6].%-22s[7].%-22s\n" % 112 | ("", " replay 'A' wav", " replay 'B' wav")) + \ 113 | ("%-6s[8].%-22s[9].%-22s\n" % 114 | ("", " replay reference wav", " replay all files")) + \ 115 | ("%-6s[q].%-22s\n: " % 116 | ("", " quit evaluation"))) 117 | 118 | if ans == "1": 119 | eval_dict['Score'] = answer 120 | eval_dict['Finished'] = True 121 | ans_flag = False 122 | elif ans == "2": 123 | eval_dict['Score'] = (1 - answer) 124 | eval_dict['Finished'] = True 125 | ans_flag = False 126 | elif ans == "6": 127 | print("'A' wav") 128 | playspeech(fileA) 129 | elif ans == "7": 130 | print("'B' wav") 131 | playspeech(fileB) 132 | elif ans == "8": 133 | print("reference wav") 134 | playspeech(fileX) 135 | elif ans == "9": 136 | print("reference wav") 137 | playspeech(fileX) 138 | time.sleep(0.3) 139 | print("'A' wav") 140 | playspeech(fileA) 141 | time.sleep(0.3) 142 | print("'B' wav") 143 | playspeech(fileB) 144 | elif ans == "q": 145 | return ans 146 | else: 147 | print("ans should be 1,2,6,7,8,9 or q. %s is not included!!" % ans) 148 | return ans 149 | 150 | # preference test 151 | def main_PK(eval_dict, r_idx, text): 152 | # MethodA(expected answer) v.s. MethodB 153 | answer = random.randint(0, 1) 154 | #print(answer) 155 | if answer: 156 | # expected answer is fileA 157 | fileA = eval_dict['FileA'] 158 | fileB = eval_dict['FileB'] 159 | else: 160 | # expected answer is fileB 161 | fileA = eval_dict['FileB'] 162 | fileB = eval_dict['FileA'] 163 | #print(fileA) 164 | #print(fileB) 165 | print("*"+("[No. %4d]-1 'A' wav file" % (r_idx)).center(20)+"*") 166 | playspeech(fileA) 167 | time.sleep(0.3) 168 | print("*"+("[No. %4d]-2 'B' wav file" % (r_idx)).center(20)+"*") 169 | playspeech(fileB) 170 | ans_flag = True 171 | Criteria = text['PK']['C'] 172 | Ans = text['PK']['Ans'] 173 | Q = text['PK']['Q'].replace('Ans', Ans).replace('Criteria', Criteria) 174 | while ans_flag: 175 | print("\n%s" % Q) 176 | ans = input(("%-6s[1].%-22s[2].%-22s\n" % 177 | ("Enter ", " 'A' is %s" % Ans, " 'B' is %s" % Ans)) + \ 178 | ("%-6s[7].%-22s[8].%-22s\n" % 179 | ("", " replay 'A' wav", " replay 'B' wav")) + \ 180 | ("%-6s[9].%-22s[q].%-22s\n: " % 181 | ("", " replay both files", " quit evaluation"))) 182 | if ans == "1": 183 | eval_dict['Score'] = answer 184 | eval_dict['Finished'] = True 185 | ans_flag = False 186 | elif ans == "2": 187 | eval_dict['Score'] = (1 - answer) 188 | eval_dict['Finished'] = True 189 | ans_flag = False 190 | elif ans == "7": 191 | print("'A' wav") 192 | playspeech(fileA) 193 | elif ans == "8": 194 | print("'B' wav") 195 | playspeech(fileB) 196 | elif ans == "9": 197 | print("'A' wav") 198 | playspeech(fileA) 199 | time.sleep(0.3) 200 | print("'B' wav") 201 | playspeech(fileB) 202 | elif ans == "q": 203 | return ans 204 | else: 205 | print("ans should be 1,2,7,8,9 or q. %s is not included!!" % ans) 206 | return ans 207 | 208 | # MOS test 209 | def main_MOS(eval_dict, r_idx, text): 210 | # PLAY FILES 211 | file = eval_dict['File'] 212 | #print(file) 213 | print("*"+("[No. %4d] wav file" % (r_idx)).center(20)+"*") 214 | playspeech(file) 215 | # INPUT ANSWER 216 | ans_flag = True 217 | scores = ["1", "2", "3", "4", "5"] 218 | Criteria = text['MOS']['C'] 219 | Q = text['MOS']['Q'].replace('Criteria', Criteria) 220 | while ans_flag: 221 | print("\n%s" % Q) 222 | ans = input(("%-6s[1].%-22s[2].%-22s\n" % 223 | ("Enter ", " Bad", " Poor")) + \ 224 | ("%-6s[3].%-22s[4].%-22s\n" % 225 | ("", " Fair", " Good")) + \ 226 | ("%-6s[5].%-22s[7].%-22s\n" % 227 | ("", " Excellent", " replay wav")) + \ 228 | ("%-6s[q].%-22s\n: " % 229 | ("", " quit evaluation"))) 230 | if ans in scores: 231 | eval_dict['Score'] = float(ans) 232 | eval_dict['Finished'] = True 233 | ans_flag = False 234 | elif ans == "7": 235 | print("replay wav") 236 | playspeech(file) 237 | elif ans == "q": 238 | return ans 239 | else: 240 | print("ans should be 1,2,3,4,5,7 or q. %s is not included!!" % ans) 241 | return ans 242 | 243 | # similarity test 244 | def main_SIM(eval_dict, r_idx, text): 245 | # PLAY FILES 246 | if random.randint(0, 1): 247 | file1 = eval_dict['File'] 248 | file2 = eval_dict['File_ans'] 249 | else: 250 | file1 = eval_dict['File_ans'] 251 | file2 = eval_dict['File'] 252 | #print(file1) 253 | #print(file2) 254 | print("*"+("[No. %4d]-1 wav file" % (r_idx)).center(20)+"*") 255 | playspeech(file1) 256 | time.sleep(0.1) 257 | print("*"+("[No. %4d]-2 wav file" % (r_idx)).center(20)+"*") 258 | playspeech(file2) 259 | # INPUT ANSWER 260 | ans_flag = True 261 | scores = ["1", "2", "3", "4"] 262 | Q = text['SIM']['Q'] 263 | posAns = text['SIM']['posAns'] 264 | negAns = text['SIM']['negAns'] 265 | while ans_flag: 266 | print("\n%s" % Q) 267 | ans = input(("%-6s[1].%-22s[2].%-22s\n" % 268 | ("Enter ", " definitely %s" % posAns, " maybe %s" % posAns)) + \ 269 | ("%-6s[3].%-22s[4].%-22s\n" % 270 | ("", " maybe %s" % negAns, " definitely %s" % negAns)) + \ 271 | ("%-6s[7].%-22s[8].%-22s\n" % 272 | ("", " replay 1st wav", " replay 2nd wav")) + \ 273 | ("%-6s[9].%-22s[q].%-22s\n: " % 274 | ("", " replay both files", " quit evaluation"))) 275 | if ans in scores: 276 | eval_dict['Score'] = float(ans) 277 | eval_dict['Finished'] = True 278 | ans_flag = False 279 | elif ans == "7": 280 | print("1st wav") 281 | playspeech(file1) 282 | elif ans == "8": 283 | print("2nd wav") 284 | playspeech(file2) 285 | elif ans == "9": 286 | print("1st wav") 287 | playspeech(file1) 288 | time.sleep(0.1) 289 | print("2nd wav") 290 | playspeech(file2) 291 | elif ans == "q": 292 | return ans 293 | else: 294 | print("ans should be 1,2,3,4,7,8,9 or q. %s is not included!!" % ans) 295 | return ans 296 | 297 | # XLSX OUTPUT TEMPLATE 298 | class xlsx_output(object): 299 | def __init__(self, username, filename, sheetnames, testsystems): 300 | self.u_name = username 301 | self.fxlsx = filename 302 | self.t_sheets = sheetnames 303 | self.t_systems = testsystems 304 | # column index of xslx (alphabet list = ['A', 'B', ..., 'Z']) 305 | self.alphabet = [] 306 | for i in range(26): 307 | self.alphabet.append(chr(ord('A')+i)) 308 | 309 | def _add_username(self, sheet, c_row, t_name): 310 | # add user in the new row 311 | sheet.cell(row=c_row, column=1).value = t_name 312 | sheet.cell(row=(c_row+1), column=1).value = 'AVG.' 313 | 314 | def _add_data(self, sheet, c_row, col_chr, t_score): 315 | # add new scores 316 | sheet['%s%d' % (col_chr, c_row)].value = t_score 317 | start_idx = "%s%d" % (col_chr, 2) 318 | end_idx = "%s%d" % (col_chr, c_row) 319 | # update the average scores 320 | sheet['%s%d' % (col_chr, c_row+1)].value = \ 321 | "=AVERAGE(%s:%s)" % (start_idx, end_idx) 322 | 323 | # XAB, PREFERENCE, MOS XLSX OUTPUT CLASS 324 | class xlsx_SCORE(xlsx_output): 325 | def __init__(self, username, filename, sheetnames, testsystems): 326 | super().__init__(username, filename, sheetnames, testsystems) 327 | if not os.path.exists(self.fxlsx): 328 | self._initial_xlsx() 329 | 330 | def output_xlsx(self, t_sheet, t_dict): 331 | """OUTPUT SUBJECTIVE RESULTS INTO A SHEET OF EXCEL FILE 332 | Args: 333 | t_sheet (str): the subset name of the subjective results 334 | [summary, xgender, sgender, F-F, F-M, M-F, M-M] 335 | t_dict (list of dict): the result dicts 336 | """ 337 | if len(t_dict) == 0: 338 | print("%s is empty!!" % t_sheet) 339 | return 340 | wb = load_workbook(self.fxlsx) # load workspace of the excel file 341 | sheet = wb['%s' % t_sheet] # load sheet 342 | c_row = sheet.max_row # get latest row index 343 | # add new user 344 | self._add_username(sheet, c_row, self.u_name) 345 | # parse results 346 | t_score = self._score(t_dict) 347 | # update sheet 348 | for i in range(len(t_score)): 349 | self._add_data(sheet, c_row, self.alphabet[i+1], t_score[i]) 350 | wb.save(self.fxlsx) 351 | 352 | def _initial_xlsx(self): 353 | wb = Workbook() 354 | first = True 355 | for s_name in self.t_sheets: 356 | if first: 357 | sheet = wb.active 358 | sheet.title = s_name 359 | first = False 360 | else: 361 | wb.create_sheet(title=s_name) 362 | sheet = wb['%s' % s_name] 363 | sheet['A1'].value = 'USER' 364 | sheet['A2'].value = 'AVG.' 365 | for i in range(len(self.t_systems)): 366 | sheet.cell(row=1, column=(2+i)).value = self.t_systems[i] 367 | wb.save(self.fxlsx) 368 | 369 | def _score(self, t_dict): 370 | t_score = [] 371 | for t_method in self.t_systems: 372 | score = 0.0 373 | t_num = 0 374 | f_t_dict = filter(lambda item: item['method'] == t_method, t_dict) 375 | for t_file in f_t_dict: 376 | score += t_file['Score'] 377 | t_num+=1 378 | if t_num==0: 379 | t_score.append(-1.0) 380 | else: 381 | t_score.append(score/t_num) 382 | return t_score 383 | 384 | # SIMILARITY XLSX OUTPUT CLASS 385 | class xlsx_SIM(xlsx_output): 386 | def __init__(self, username, filename, sheetnames, testsystems, text): 387 | super().__init__(username, filename, sheetnames, testsystems) 388 | self.posAns_sure = '%s' % text['SIM']['posAns'] 389 | self.posAns_weak = 'maybe_%s' % text['SIM']['posAns'] 390 | self.negAns_weak = 'maybe_%s' % text['SIM']['negAns'] 391 | self.negAns_sure = '%s' % text['SIM']['negAns'] 392 | if not os.path.exists(self.fxlsx): 393 | self._initial_xlsx() 394 | 395 | def output_xlsx(self, t_sheet, t_dict): 396 | """OUTPUT SUBJECTIVE RESULTS INTO A SHEET OF EXCEL FILE 397 | Args: 398 | t_sheet (str): the subset name of the subjective results 399 | [summary, xgender, sgender, F-F, F-M, M-F, M-M] 400 | t_dict (list of dict): the result dicts 401 | """ 402 | if len(t_dict) == 0: 403 | print("%s is empty!!" % t_sheet) 404 | return 405 | wb = load_workbook(self.fxlsx) # load workspace of the excel file 406 | sheet = wb['%s' % t_sheet] # load sheet 407 | c_row = sheet.max_row # get latest row index 408 | # add new user 409 | self._add_username(sheet, c_row, self.u_name) 410 | # parse results 411 | t_score = self._score(t_dict) 412 | # update sheet 413 | for i in range(len(t_score)): 414 | self._add_data(sheet, c_row, self.alphabet[i*4+1], t_score[i][self.posAns_sure]) 415 | self._add_data(sheet, c_row, self.alphabet[i*4+2], t_score[i][self.posAns_weak]) 416 | self._add_data(sheet, c_row, self.alphabet[i*4+3], t_score[i][self.negAns_weak]) 417 | self._add_data(sheet, c_row, self.alphabet[i*4+4], t_score[i][self.negAns_sure]) 418 | #self._final_result(sheet) 419 | wb.save(self.fxlsx) 420 | 421 | def _final_result(self, sheet): 422 | c_row = sheet.max_row + 1 423 | sheet.cell(row=c_row, column=1).value = 'Result' 424 | m_idx = range(1, len(self.t_systems)*4+1, 4) 425 | for i in range(len(self.t_systems)): 426 | # SAME SPEAKER 427 | merge_range = "%s%d:%s%d" % ( 428 | chr(ord('A')+m_idx[i]), c_row, chr(ord('B')+m_idx[i]), c_row) 429 | sheet.merge_cells(merge_range) 430 | sum_range = "%s%d:%s%d" % ( 431 | chr(ord('A')+m_idx[i]), c_row-1, chr(ord('B')+m_idx[i]), c_row-1) 432 | sheet.cell(row=c_row, column=(2+i*4)).value = "=SUM(%s)" % (sum_range) 433 | # DIFFERENT SPEAKER 434 | merge_range = "%s%d:%s%d" % ( 435 | chr(ord('C')+m_idx[i]), c_row, chr(ord('D')+m_idx[i]), c_row) 436 | sheet.merge_cells(merge_range) 437 | sum_range = "%s%d:%s%d" % ( 438 | chr(ord('C')+m_idx[i]), c_row-1, chr(ord('D')+m_idx[i]), c_row-1) 439 | sheet.cell(row=c_row, column=(4+i*4)).value = "=SUM(%s)" % (sum_range) 440 | 441 | def _initial_xlsx(self): 442 | wb = Workbook() 443 | first = True 444 | for s_name in self.t_sheets: 445 | if first: 446 | sheet = wb.active 447 | sheet.title = s_name 448 | first = False 449 | else: 450 | wb.create_sheet(title=s_name) 451 | sheet = wb['%s' % s_name] 452 | sheet['A1'].value = 'USER' 453 | sheet['A2'].value = '' 454 | sheet['A3'].value = 'AVG.' 455 | m_idx = range(1, len(self.t_systems)*4+1, 4) 456 | for i in range(len(self.t_systems)): 457 | merge_range = "%s%d:%s%d" % (chr(ord('A')+m_idx[i]), 1, chr(ord('D')+m_idx[i]), 1) 458 | sheet.merge_cells(merge_range) 459 | sheet.cell(row=1, column=(2+i*4)).value = self.t_systems[i] 460 | sheet.cell(row=2, column=(2+i*4)).value = self.posAns_sure 461 | sheet.cell(row=2, column=(3+i*4)).value = self.posAns_weak 462 | sheet.cell(row=2, column=(4+i*4)).value = self.negAns_weak 463 | sheet.cell(row=2, column=(5+i*4)).value = self.negAns_sure 464 | wb.save(self.fxlsx) 465 | 466 | def _score(self, t_dict): 467 | t_score = [] 468 | for t_method in self.t_systems: 469 | score = {self.posAns_sure: 0.0, self.posAns_weak: 0.0, 470 | self.negAns_sure: 0.0, self.negAns_weak: 0.0} 471 | f_t_dict = filter(lambda item: item['method'] == t_method, t_dict) 472 | t_num = 0 473 | for t_file in f_t_dict: 474 | if t_file['Score'] == 1.0: 475 | ans = self.posAns_sure 476 | elif t_file['Score'] == 2.0: 477 | ans = self.posAns_weak 478 | elif t_file['Score'] == 3.0: 479 | ans = self.negAns_weak 480 | elif t_file['Score'] == 4.0: 481 | ans = self.negAns_sure 482 | else: 483 | raise ValueError("%s method score %.2f is out of range" % ( 484 | t_method, t_file['Score'])) 485 | score[ans] += 1.0 486 | t_num += 1 487 | if t_num == 0: 488 | t_score.append(score) 489 | else: 490 | score = {key: value / t_num for key, value in score.items()} 491 | t_score.append(score) 492 | return t_score -------------------------------------------------------------------------------- /src/bin/eval_statistic.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | # Copyright 2020 Wu Yi-Chiao (Nagoya University) 4 | # Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0) 5 | 6 | import os 7 | import sys 8 | import numpy as np 9 | import yaml 10 | import scipy.stats 11 | import copy 12 | from openpyxl import Workbook 13 | from openpyxl import load_workbook 14 | 15 | # STATISTICAL RESULTS CLASS 16 | class StatisticResult(object): 17 | """ 18 | Class to output statistics of evaluation results 19 | """ 20 | def __init__(self, c_xlsx, sheetname): 21 | self.c_xlsx = c_xlsx 22 | self.t_sheet = sheetname 23 | self.mean = [] 24 | self.std = [] 25 | self.ci = [] 26 | self.clow = [] 27 | self.cup = [] 28 | self.data_length = 0 29 | 30 | def push_result(self, scores, confidence=0.95): 31 | """PUSH SUBJECTIVE RESULTS OF A USER, PAIR, OR SYSTEM TO THE QUEUE OF OVERALL RESULTS 32 | Args: 33 | scores (list): score list 34 | confidence (float): the level of confidence (0~1.0) (default: 0.95 (95%)) 35 | """ 36 | self.data_length = len(scores) 37 | if self.data_length == 0: 38 | #print('(push) data_length of %s is 0!' % self.t_sheet) 39 | return 0 40 | mean, std, ci, clow, cup = self._mean_confidence_interval(scores, confidence) 41 | self.mean.append(mean) 42 | self.std.append(std) 43 | self.ci.append(ci) 44 | self.clow.append(clow) 45 | self.cup.append(cup) 46 | 47 | def output_result(self): 48 | """OUTPUT STATISTICS OF ALL EVALUATION RESULTS INTO A EXCEL FILE 49 | mean: average of results 50 | std: standard deviation of results 51 | ci: confidence interval 52 | clow: lowerbound of the confidence interval 53 | cup: upperbound of the confidence interval 54 | """ 55 | if self.data_length == 0: 56 | return 0 57 | self.c_xlsx.output_xlsx('mean', self.t_sheet, self.mean) 58 | self.c_xlsx.output_xlsx('std', self.t_sheet, self.std) 59 | self.c_xlsx.output_xlsx('ci', self.t_sheet, self.ci) 60 | self.c_xlsx.output_xlsx('clow', self.t_sheet, self.clow) 61 | self.c_xlsx.output_xlsx('cup', self.t_sheet, self.cup) 62 | 63 | def _mean_confidence_interval(self, data, confidence=0.95): 64 | a = 1.0 * np.array(data) 65 | #n = len(a) 66 | mean = np.mean(a) 67 | std = np.std(a) 68 | std_err = scipy.stats.sem(a) 69 | ci = std_err * scipy.stats.norm.ppf((1 + confidence) / 2.) # confidence interval 70 | return mean, std, ci, mean-ci, mean+ci, 71 | 72 | # XLSX FINAL OUTPUT TEMPLATE 73 | class xlsx_foutput(object): 74 | def __init__(self, filename, sheetnames, testsystems): 75 | self.fxlsx = filename 76 | self.t_sheets = sheetnames 77 | self.t_systems = testsystems 78 | # column index of xslx (alphabet list = ['A', 'B', ..., 'Z']) 79 | self.alphabet = [] 80 | for i in range(26): 81 | self.alphabet.append(chr(ord('A')+i)) 82 | 83 | def _add_username(self, sheet, c_row, t_name): 84 | # add user in the new row 85 | sheet.cell(row=c_row, column=1).value = t_name 86 | 87 | def _add_data(self, sheet, c_row, col_chr, t_score): 88 | # add new scores 89 | sheet['%s%d' % (col_chr, c_row)].value = t_score 90 | 91 | # XAB, PREFERENCE, MOS XLSX FINAL OUTPUT CLASS 92 | class xlsx_fSCORE(xlsx_foutput): 93 | """ 94 | Class to output evaluation results 95 | """ 96 | def __init__(self, filename, sheetnames, testsystems, c_text): 97 | super().__init__(filename, sheetnames, testsystems) 98 | if os.path.exists(self.fxlsx): 99 | print("overwrite %s" % self.fxlsx) 100 | self._initial_xlsx() 101 | # initialize score list dictionary of each test system 102 | system_dict = {} 103 | for t_system in self.t_systems: 104 | system_dict = {**system_dict, t_system:[]} 105 | # initialize score list dictionary of each test pair and test system 106 | self.user_score={} # user-based score 107 | self.utt_score={} # utterance-based score 108 | for t_pair in self.t_sheets: 109 | self.user_score = {**self.user_score, t_pair:copy.deepcopy(system_dict)} 110 | self.utt_score = {**self.utt_score, t_pair:copy.deepcopy(system_dict)} 111 | # load config text 112 | self.c_text = c_text 113 | 114 | def output_xlsx(self, t_name, t_sheet, t_score): 115 | """OUTPUT SUBJECTIVE RESULTS INTO A SHEET OF EXCEL FILE 116 | Args: 117 | t_name (str): the user name 118 | t_sheet (str): the subset name of the subjective results 119 | [summary, xgender, sgender, F-F, F-M, M-F, M-M] 120 | t_score (list of score): the list of scores of the subset 121 | """ 122 | if len(t_score) == 0: 123 | print("%-10s - %-6s is empty!!" % (t_name, t_sheet)) 124 | return 125 | wb = load_workbook(self.fxlsx) # load workspace of the excel file 126 | sheet = wb['%s' % t_sheet] # load sheet 127 | c_row = sheet.max_row # get latest row index 128 | c_row += 1 129 | # add new user 130 | self._add_username(sheet, c_row, t_name) 131 | # update sheet 132 | for i, score in enumerate(t_score): 133 | self._add_data(sheet, c_row, self.alphabet[i+1], score) 134 | wb.save(self.fxlsx) 135 | 136 | def output_result(self, t_name, t_sheet, t_dict): 137 | """OUTPUT SUBJECTIVE RESULTS 138 | Args: 139 | t_name (str): the user name 140 | t_sheet (str): the subset name of the subjective results 141 | [summary, xgender, sgender, F-F, F-M, M-F, M-M] 142 | t_dict (list of dict): the result list of dicts of the subset 143 | """ 144 | if len(t_dict) == 0: 145 | print("%-10s - %-6s is empty!!" % (t_name, t_sheet)) 146 | return 147 | else: 148 | # parse results 149 | t_score = self._score(t_sheet, t_dict) 150 | self.output_xlsx(t_name, t_sheet, t_score) 151 | 152 | def _initial_xlsx(self): 153 | wb = Workbook() 154 | first = True 155 | for t_pair in self.t_sheets: 156 | if first: 157 | sheet = wb.active 158 | sheet.title = t_pair 159 | first = False 160 | else: 161 | wb.create_sheet(title=t_pair) 162 | sheet = wb['%s' % t_pair] 163 | sheet['A1'].value = 'USER' 164 | for i in range(len(self.t_systems)): 165 | sheet.cell(row=1, column=(2+i)).value = self.t_systems[i] 166 | wb.save(self.fxlsx) 167 | 168 | def _score(self, t_pair, t_dict): 169 | t_score = [] 170 | for t_system in self.t_systems: 171 | score = 0.0 172 | t_num = 0 173 | f_t_dict = filter(lambda item: item[self.c_text['system']] == t_system, t_dict) 174 | for t_file in f_t_dict: 175 | u_score = t_file[self.c_text['r_score']] # score of the utterance 176 | self.utt_score[t_pair][t_system] += [u_score] 177 | score += u_score 178 | t_num+=1 179 | if t_num==0: 180 | score = -1.0 181 | self.utt_score[t_pair][t_system] += [-1.0] 182 | else: 183 | score /= t_num 184 | t_score.append(score) 185 | # update score of each system under this test pair 186 | self.user_score[t_pair][t_system] += [score] 187 | return t_score 188 | 189 | -------------------------------------------------------------------------------- /src/user/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/src/user/__init__.py -------------------------------------------------------------------------------- /src/user/info_class.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | # Copyright 2020 Wu Yi-Chiao (Nagoya University) 4 | # Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0) 5 | 6 | import os 7 | import sys 8 | import yaml 9 | import copy 10 | import fnmatch 11 | import datetime 12 | import numpy as np 13 | 14 | class ConfigInfo(object): 15 | def __init__(self, yml_path, c_text): 16 | self.flag = True 17 | self.yml_path = yml_path # file path of yml format config file 18 | self.c_text = c_text # dictionary of text 19 | conf_user, self.fconf_user = self._load_record() 20 | self.conf, self.fconf = self._load_config() 21 | # check the number of sub set 22 | n_subset = conf_user[c_text['n_subset']] 23 | for t_idx in range(len(self.conf)): 24 | n_setlist = len(self.conf[t_idx][c_text['t_set']]) 25 | if n_setlist != n_subset: 26 | print('The number of sub set lists of "%s(%d)" is not same as %s in %s.yml(%d) !!' \ 27 | % (self.conf[t_idx][c_text['t_type']], n_setlist, 28 | c_text['n_subset'], c_text['recordf'], n_subset)) 29 | sys.exit(0) 30 | self.n_subset = n_subset 31 | 32 | def _check_file_exist(self, filename): 33 | if not os.path.exists(filename): 34 | print("%s doesn't exist!"%filename) 35 | return False 36 | return True 37 | 38 | def _yaml_dump(self, filename, data, encoding='utf-8'): 39 | with open(filename, "w", encoding=encoding) as yf: 40 | yaml.safe_dump(data, yf) 41 | 42 | def _yaml_load(self, filename, encoding='utf-8'): 43 | if not os.path.exists(filename): 44 | print("%s doesn't exist!" % filename) 45 | sys.exit(0) 46 | with open(filename, "r", encoding=encoding) as yf: 47 | return yaml.safe_load(yf) 48 | 49 | def _load_record(self): 50 | # fconf_user: the config file of the information of all users 51 | # conf_user (dict): 52 | # MAX_count: the total number of the testing subsets 53 | # Subject_name: the list of users 54 | # Subject_set: the corresponding subset index of each user 55 | # count: the index of the current subset 56 | # time: last updated time 57 | fconf_user = "%s%s.yml" % (self.yml_path, self.c_text['recordf']) 58 | if not self._check_file_exist(fconf_user): 59 | sys.exit(0) 60 | conf_user = self._yaml_load(fconf_user) 61 | return conf_user, fconf_user 62 | 63 | def _load_config(self): 64 | # fconf: the config file of all evaluations information 65 | # conf (list of dict): 66 | # method: evaluated methods 67 | # set: the name of each subset 68 | # type: the type of the test (MOS, SIM, XAB, PK) 69 | fconf = "%s%s.yml"%(self.yml_path, self.c_text['configf']) 70 | if not self._check_file_exist(fconf): 71 | sys.exit(0) 72 | conf = self._yaml_load(fconf) 73 | return conf, fconf 74 | 75 | class ParserConf(ConfigInfo): 76 | """ 77 | Class to parse config files 78 | """ 79 | def __init__(self, data_path, template_path, yml_path, c_text): 80 | super().__init__(yml_path, c_text) 81 | self.data_path = data_path 82 | self.template_path = template_path 83 | self.t_info = self._load_sysinfo() 84 | 85 | def _get_spkidx(self, t_path, spk): 86 | for idx, item in enumerate(t_path): 87 | if spk == item: 88 | break 89 | if idx == 0: 90 | print('The path format of %s or %s is wrong! It should be "/spk/"!') 91 | sys.exit(0) 92 | return idx 93 | 94 | def _get_pairidx(self, t_path, srcspk, tarspk): 95 | for idx, item in enumerate(t_path): 96 | if fnmatch.fnmatch(item, '%s*%s' % (srcspk, tarspk)): 97 | break 98 | if idx == 0: 99 | print('The path format of %s and %s is wrong! It should be "/spk/spk/" or "/spk-spk/"!') 100 | sys.exit(0) 101 | return idx 102 | 103 | def _check_idx(self, item, name, idx): 104 | if idx >= len(item): 105 | msg = "idx '%s' (%d) is out of the length of %s\n" % (name, idx, str(item)) 106 | msg += "The template path cannot be correctly splitted by slash.\n" 107 | msg += "Please check the template path in 'test_system.yml'." 108 | raise ValueError(msg) 109 | 110 | def _load_sysinfo(self): 111 | sysinfof = "%s%s.yml"%(self.yml_path, self.c_text['systemf']) 112 | if not self._check_file_exist(sysinfof): 113 | sys.exit(0) 114 | sysinfo = self._yaml_load(sysinfof) 115 | t_paths = sysinfo[self.c_text['templatedir']] 116 | srcspk = self.c_text['srcspk'] 117 | tarspk = self.c_text['tarspk'] 118 | for system in list(t_paths.keys()): 119 | t_paths[system] = t_paths[system].replace('\\', '/') 120 | t_path = t_paths[system].split('/') 121 | if srcspk in t_paths[system]: 122 | if tarspk in t_paths[system]: # voice conversion 123 | if tarspk in t_path and srcspk in t_path: 124 | # path format: /srcspk/tarspk/ or /tarspk/srcspk/ 125 | srcidx = t_path.index(srcspk) 126 | taridx = t_path.index(tarspk) 127 | t_paths[system] = {'src':srcidx, 'tar':taridx, 'split':None, 128 | 'src_sub':None, 'tar_sub':None } 129 | else: 130 | # path format /srcspk-tarspk/ or /tarspk-srcspk/ 131 | pairidx = self._get_pairidx(t_path, srcspk, tarspk) 132 | symbol = t_path[pairidx].replace(srcspk, "").replace(tarspk, "") 133 | subsrc = t_path[pairidx].split(symbol).index(srcspk) 134 | subtar = t_path[pairidx].split(symbol).index(tarspk) 135 | t_paths[system] = {'src':pairidx, 'tar':pairidx, 'split':symbol, 136 | 'src_sub':subsrc, 'tar_sub':subtar } 137 | else: # source speaker only 138 | spkidx = self._get_spkidx(t_path, srcspk) 139 | t_paths[system] = {'src':spkidx, 'tar':None, 'split':None, 140 | 'src_sub':None, 'tar_sub':None } 141 | elif tarspk in t_paths[system]: # target speaker only 142 | spkidx = self._get_spkidx(t_path, tarspk) 143 | t_paths[system] = {'src':None, 'tar':spkidx, 'split':None, 144 | 'src_sub':None, 'tar_sub':None } 145 | else: 146 | print('%s or %s is not in the template path of %s of %s!'\ 147 | % (srcspk, tarspk, system, sysinfof)) 148 | sys.exit(0) 149 | return sysinfo 150 | 151 | def _load_divide_list(self, flistf, ref=None, reflen=0): 152 | # load file list and divide it into sub set lists 153 | if not self._check_file_exist(flistf): 154 | sys.exit(0) 155 | with open(flistf, "r") as f: 156 | file_list = f.readlines() 157 | flen = len(file_list) 158 | if ref != None: 159 | if reflen != flen: 160 | print('The list lengths of %s(%d) and %s(%d) should be the same!' \ 161 | % (ref, reflen, flistf, flen)) 162 | sys.exit(0) 163 | file_list= [file.strip() for file in file_list] 164 | file_lists = np.reshape(file_list, (-1, self.n_subset)) 165 | file_lists = [file_lists[:,i].tolist() for i in range(self.n_subset)] 166 | return file_lists, flen 167 | 168 | def _parse_spkinfo(self, pathinfo, genderinfo, t_dict, filename): 169 | filename = filename.replace('\\', '/') 170 | item = filename.split('/') 171 | if pathinfo['tar'] == None: # only source speaker 172 | self._check_idx(item, 'src', pathinfo['src']) 173 | t_dict['srcspk'] = item[pathinfo['src']] 174 | t_dict['gender'] = genderinfo[t_dict['srcspk']] 175 | elif pathinfo['src'] == None: # only target speaker 176 | self._check_idx(item, 'tar', pathinfo['tar']) 177 | t_dict['tarspk'] = item[pathinfo['tar']] 178 | t_dict['gender'] = genderinfo[t_dict['tarspk']] 179 | else: # voice conversion 180 | if pathinfo['src'] == pathinfo['tar']: # format /srcspk/tarspk/ or /tarspk/srcspk/ 181 | self._check_idx(item, 'src', pathinfo['src']) 182 | spkpair = item[pathinfo['src']].split(pathinfo['split']) 183 | t_dict['srcspk'] = spkpair[pathinfo['src_sub']] 184 | t_dict['tarspk'] = spkpair[pathinfo['tar_sub']] 185 | else: # format /srcspk-tarspk/ or /tarspk-srcspk/ 186 | self._check_idx(item, 'src', pathinfo['src']) 187 | self._check_idx(item, 'tar', pathinfo['tar']) 188 | t_dict['srcspk'] = item[pathinfo['src']] 189 | t_dict['tarspk'] = item[pathinfo['tar']] 190 | t_dict['conversion'] = True 191 | srcgender = genderinfo[t_dict['srcspk']] 192 | targender = genderinfo[t_dict['tarspk']] 193 | if srcgender != targender: 194 | t_dict['xgender'] = True 195 | t_dict['gender'] = genderinfo[t_dict['tarspk']] 196 | t_dict['pair'] = '%s-%s' % (srcgender, targender) 197 | 198 | def _parse_mos(self, sysinfo, template, t_system, file_lists, conf): 199 | for set_idx, file_list in enumerate(file_lists): # for each sub set 200 | for filename in file_list: # for each file 201 | t_dict = copy.deepcopy(template) 202 | t_dict['File'] = filename 203 | t_dict[self.c_text['system']] = t_system 204 | self._parse_spkinfo(sysinfo[self.c_text['templatedir']][t_system], 205 | sysinfo[self.c_text['spk']], 206 | t_dict, filename) 207 | conf[set_idx].append(t_dict) 208 | 209 | def _parse_sim(self, sysinfo, template, t_system, file_lists, ref_lists, conf): 210 | for set_idx, file_list in enumerate(file_lists): # for each sub set 211 | for filename, refname in zip(file_list, ref_lists[set_idx]): # for each file 212 | t_dict = copy.deepcopy(template) 213 | t_dict['File'] = filename 214 | t_dict['File_ans'] = refname 215 | t_dict[self.c_text['system']] = t_system 216 | self._parse_spkinfo(sysinfo[self.c_text['templatedir']][t_system], 217 | sysinfo[self.c_text['spk']], 218 | t_dict, filename) 219 | conf[set_idx].append(t_dict) 220 | 221 | def _parse_xab(self, sysinfo, template, t_system, 222 | systemA, systemB, systemX, 223 | fileA_lists, fileB_lists, fileX_lists, conf): 224 | for set_idx in range(len(fileA_lists)): # for each sub set 225 | listA = fileA_lists[set_idx] 226 | listB = fileB_lists[set_idx] 227 | listX = fileX_lists[set_idx] 228 | assert len(listA)==len(listB)==len(listX) 229 | for file_idx in range(len(listA)): # for each file 230 | t_dict = copy.deepcopy(template) 231 | t_dict['FileA'] = listA[file_idx] 232 | t_dict['FileB'] = listB[file_idx] 233 | t_dict['FileX'] = listX[file_idx] 234 | t_dict[self.c_text['system']] = t_system 235 | t_dict[self.c_text['system']+'A'] = systemA 236 | t_dict[self.c_text['system']+'B'] = systemB 237 | t_dict[self.c_text['system']+'X'] = systemX 238 | self._parse_spkinfo(sysinfo[self.c_text['templatedir']][systemA], 239 | sysinfo[self.c_text['spk']], 240 | t_dict, listA[file_idx]) 241 | conf[set_idx].append(t_dict) 242 | 243 | def _parse_pk(self, sysinfo, template, t_system, 244 | systemA, systemB, 245 | fileA_lists, fileB_lists, conf): 246 | for set_idx in range(len(fileA_lists)): # for each sub set 247 | listA = fileA_lists[set_idx] 248 | listB = fileB_lists[set_idx] 249 | assert len(listA)==len(listB) 250 | for file_idx in range(len(listA)): # for each file 251 | t_dict = copy.deepcopy(template) 252 | t_dict['FileA'] = listA[file_idx] 253 | t_dict['FileB'] = listB[file_idx] 254 | t_dict[self.c_text['system']] = t_system 255 | t_dict[self.c_text['system']+'A'] = systemA 256 | t_dict[self.c_text['system']+'B'] = systemB 257 | self._parse_spkinfo(sysinfo[self.c_text['templatedir']][systemA], 258 | sysinfo[self.c_text['spk']], 259 | t_dict, listA[file_idx]) 260 | conf[set_idx].append(t_dict) 261 | 262 | def subset_gen(self): 263 | print('Test files will be divided into %d sub-sets.' % self.n_subset) 264 | for t_idx in range(len(self.conf)): # for each evaluation type 265 | t_type = self.conf[t_idx][self.c_text['t_type']] 266 | t_systems = self.conf[t_idx][self.c_text['system']] 267 | t_sets = self.conf[t_idx][self.c_text['t_set']] 268 | assert len(t_sets) == self.n_subset 269 | tempf = '%s/%s.yml' % (self.template_path, t_type) 270 | template = self._yaml_load(tempf) 271 | conf = [[] for i in range(self.n_subset)] 272 | for t_system in t_systems: # for each test method 273 | if t_type == 'MOS' or t_type == 'SIM': 274 | # load divied file list 275 | flistf = '%s%s.list' % (self.data_path, t_system) 276 | file_lists, flen = self._load_divide_list(flistf) 277 | if t_type == 'MOS': 278 | self._parse_mos(self.t_info, template, t_system, file_lists, conf) 279 | elif t_type == 'SIM': 280 | t_reference = self.conf[t_idx][self.c_text['reference']] 281 | reflistf = '%s%s.list' % (self.data_path, t_reference) 282 | ref_lists, _ = self._load_divide_list(reflistf, flistf, flen) 283 | self._parse_sim(self.t_info, template, t_system, file_lists, ref_lists, conf) 284 | elif t_type == 'XAB' or t_type == 'PK': 285 | # prase sytemA, systemB 286 | items = t_system.split('-') 287 | systemA = items[0].strip('*') 288 | systemB = items[1] 289 | # load divied file list 290 | listAf = '%s%s.list' % (self.data_path, systemA) 291 | fileA_lists, flen = self._load_divide_list(listAf) 292 | listBf = '%s%s.list' % (self.data_path, systemB) 293 | fileB_lists, _ = self._load_divide_list(listBf, listAf, flen) 294 | if t_type == 'PK': 295 | self._parse_pk(self.t_info, template, t_system, 296 | systemA, systemB, 297 | fileA_lists, fileB_lists, conf) 298 | elif t_type == 'XAB': 299 | systemX = items[2] 300 | listXf = '%s%s.list' % (self.data_path, systemX) 301 | fileX_lists, _ = self._load_divide_list(listXf, listAf, flen) 302 | self._parse_xab(self.t_info, template, t_system, 303 | systemA, systemB, systemX, 304 | fileA_lists, fileB_lists, fileX_lists, conf) 305 | else: 306 | print('Type %s is not supported! Please check %s!!' % (t_type, self.fconf)) 307 | for set_idx, t_set in enumerate(t_sets): # for each sub set 308 | subsetf = '%s%s' % (self.yml_path, t_set) 309 | self._yaml_dump(subsetf, conf[set_idx]) 310 | 311 | class UserInfo(ConfigInfo): 312 | """ 313 | Class of all information of each user 314 | """ 315 | def __init__(self, yml_path, c_text): 316 | super().__init__(yml_path, c_text) 317 | self.t_idxs = range(len(self.conf)) 318 | 319 | def _conf_user_save(self, conf_user, fconf_user, name, t_set): 320 | conf_user[self.c_text['user_name']].append(name) 321 | conf_user[self.c_text['t_subset']].append(t_set) 322 | conf_user[self.c_text['date']] = datetime.datetime.now() 323 | self._yaml_dump(fconf_user, conf_user) 324 | 325 | def _check_progress(self, recordf_user): 326 | return list(filter(lambda item: item[self.c_text['t_finish']] == False, recordf_user)) 327 | 328 | def _load_test_data(self, conf, fconf, name, t_set): 329 | test_type = [] # the list of test type 330 | test_set = [] # the list of user yml corresponding to each test type 331 | test_system = [] # the list of methods corresponding to each test type 332 | total_dict = [] # the result list of dicts of all tests 333 | eval_dict = [] # the result list of dicts of the uncompleted tests 334 | 335 | for t_idx in range(len(conf)): 336 | test_type.append(conf[t_idx][self.c_text['t_type']]) 337 | test_system.append(conf[t_idx][self.c_text['system']]) 338 | # load template record yml file 339 | if not self._check_file_exist(self.yml_path + conf[t_idx][self.c_text['t_set']][t_set]): 340 | print('Please check the "set" setting in %s' % fconf) 341 | sys.exit(0) 342 | recordf_user = self._yaml_load(self.yml_path + conf[t_idx][self.c_text['t_set']][t_set]) 343 | # create user record yml file 344 | user_set = conf[t_idx][self.c_text['t_set']][t_set].replace( 345 | ".yml", "_%s.yml" % name) 346 | test_set.append(self.yml_path + user_set) 347 | if not self._check_file_exist(os.path.dirname(test_set[t_idx])): 348 | print('Please check the "set" setting in %s' % fconf) 349 | sys.exit(0) 350 | self._yaml_dump(test_set[t_idx], recordf_user) 351 | 352 | total_dict.append(recordf_user) 353 | eval_dict.append(recordf_user) 354 | 355 | self.test_type = test_type 356 | self.test_system = test_system 357 | self.test_set = test_set 358 | self.eval_dict = eval_dict 359 | self.total_dict = total_dict 360 | 361 | def _reload_test_data(self, conf, fconf_user, name, t_set): 362 | test_type = [] # the list of test type 363 | test_set = [] # the list of user yml corresponding to each test type 364 | test_system = [] # the list of methods corresponding to each test type 365 | total_dict = [] # the result list of dicts of all tests 366 | eval_dict = [] # the result list of dicts of the uncompleted tests 367 | 368 | for t_idx in range(len(conf)): 369 | test_type.append(conf[t_idx][self.c_text['t_type']]) 370 | test_system.append(conf[t_idx][self.c_text['system']]) 371 | # load user record yml file (record_user) 372 | user_set = conf[t_idx][self.c_text['t_set']][t_set].replace( 373 | ".yml", "_%s.yml"%name) 374 | test_set.append(self.yml_path + user_set) 375 | if not self._check_file_exist(test_set[t_idx]): 376 | print('User %s data lost!! Please check %s' % (name, fconf_user)) 377 | sys.exit(0) 378 | recordf_user = self._yaml_load(test_set[t_idx]) 379 | total_dict.append(recordf_user) 380 | #CHECK PROGRESS 381 | # remaining unfinished parts 382 | r_recordf_user = self._check_progress(recordf_user) 383 | eval_dict.append(r_recordf_user) 384 | if len(r_recordf_user) == 0: 385 | self.finished[t_idx] = True 386 | if len(r_recordf_user) < len(recordf_user): 387 | self.initial[t_idx] = False 388 | 389 | self.test_type = test_type 390 | self.test_system = test_system 391 | self.test_set = test_set 392 | self.eval_dict = eval_dict 393 | self.total_dict = total_dict 394 | 395 | def check_user(self, name): 396 | """LOAD AND CHECK USER PROGRESS 397 | Args: 398 | name (str): the name of the user 399 | Return: 400 | flag (bool): True: all tests of the user have been completed 401 | False: new user or the tests are not completed 402 | """ 403 | #LOAD RECORD 404 | conf_user, _ = self._load_record() 405 | #LOAD USER INFO 406 | self.finished = [False]*len(self.conf) 407 | self.initial = [True]*len(self.conf) 408 | if name in conf_user[self.c_text['user_name']]: 409 | # load user (conf_user) 410 | t_set = conf_user[self.c_text['t_subset']][conf_user[self.c_text['user_name']].index(name)] 411 | # reload testing data 412 | self._reload_test_data(self.conf, self.fconf_user, name, t_set) 413 | else: 414 | # create new user (conf_user) 415 | conf_user[self.c_text['subset_idx']] += 1 416 | if conf_user[self.c_text['subset_idx']] >= conf_user[self.c_text['n_subset']]: 417 | conf_user[self.c_text['subset_idx']] = 0 418 | t_set = conf_user[self.c_text['subset_idx']] 419 | # load testing data 420 | self._load_test_data(self.conf, self.fconf, name, t_set) 421 | self._conf_user_save(conf_user, self.fconf_user, name, t_set) 422 | 423 | self.flag = (sum(self.finished)==len(self.conf)) 424 | return self.flag 425 | 426 | def save_result(self, t_idx): 427 | """SAVE RESULTS 428 | Args: 429 | t_idx (int): the index of test 430 | """ 431 | self._yaml_dump(self.test_set[t_idx], self.total_dict[t_idx]) 432 | #CHECK PROGRESS 433 | if len(self._check_progress(self.total_dict[t_idx])) == 0: 434 | self.finished[t_idx] = True 435 | 436 | # USER RESULT CLASS 437 | class UserResult(UserInfo): 438 | """ 439 | Class of results of each user 440 | """ 441 | def __init__(self, user_name, user_set, test_type, yml_path, c_text): 442 | super().__init__(yml_path, c_text) 443 | self.name = user_name 444 | self.t_set = user_set 445 | self.t_type = test_type 446 | self._load_result() 447 | # remaining unfinished parts 448 | r_recordf_user = self._check_progress(self.recordf_user) 449 | if len(r_recordf_user) == 0: 450 | self.finished = True 451 | else: 452 | self.finished = False 453 | 454 | def _load_result(self): 455 | # load user yml 456 | frecordf_user = "%s%s_%d_%s.yml" % (self.yml_path, 457 | self.t_type, 458 | self.t_set, 459 | self.name) 460 | if not self._check_file_exist(frecordf_user): 461 | sys.exit(0) 462 | self.recordf_user = self._yaml_load(frecordf_user) -------------------------------------------------------------------------------- /src/utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bigpon/SpeechSubjectiveTest/3e5056121096969d5f61c9daeb70543e3d4fd95b/src/utils/__init__.py -------------------------------------------------------------------------------- /src/utils/dict_filter.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | # Copyright 2020 Wu Yi-Chiao (Nagoya University) 4 | # Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0) 5 | 6 | import sys 7 | 8 | def tpair_filter(t_pair, t_dict): 9 | """FILTER DICTIONARY 10 | Args: 11 | t_pair (str): the filter condition 12 | t_dict (dict): the test dictionary 13 | """ 14 | if t_pair == 'summary': 15 | return t_dict 16 | elif t_pair == 'female': 17 | return list(filter(lambda item: item['gender'] == 'F', t_dict)) 18 | elif t_pair == 'male': 19 | return list(filter(lambda item: item['gender'] == 'M', t_dict)) 20 | elif t_pair == 'xgender': # cross gender 21 | vc_dict = list(filter(lambda item: item['conversion'], t_dict)) 22 | return list(filter(lambda item: item['xgender'], vc_dict)) 23 | elif t_pair == 'sgender': # same gender 24 | vc_dict = list(filter(lambda item: item['conversion'], t_dict)) 25 | return list(filter(lambda item: item['xgender'] == False, vc_dict)) 26 | elif t_pair in ['F-F', 'F-M', 'M-F', 'M-M']: # gender pair 27 | vc_dict = list(filter(lambda item: item['conversion'], t_dict)) 28 | return list(filter(lambda item: item['pair'] == t_pair, vc_dict)) 29 | else: 30 | print("(tpair_filter) pair %s is not defined!" % t_pair) 31 | sys.exit(0) -------------------------------------------------------------------------------- /src/utils/eval_io.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | import random 4 | import time 5 | import platform 6 | import winsound 7 | from playsound import playsound 8 | from openpyxl import Workbook 9 | from openpyxl import load_workbook 10 | 11 | def playspeech(filename): 12 | if os.path.exists(filename): 13 | if platform.system() == 'Windows': 14 | winsound.PlaySound(filename, winsound.SND_ALIAS) 15 | else: 16 | playsound(filename) 17 | else: 18 | print("%s doesn't exists!!"%filename) 19 | sys.exit(0) 20 | 21 | def introduction(test_type, text, examples): 22 | """TEST INTRODUCTION 23 | Args: 24 | test_type (str): the type of the test 25 | """ 26 | if test_type == 'PK': 27 | info = "'Each PK' test will play two files," + \ 28 | " and please select the answer" + \ 29 | " with %s %s.\n" % (text['PK']['Ans'], text['PK']['C']) 30 | print(info) 31 | elif test_type == 'SIM': 32 | info = "Each 'SIM' test will play two files" + \ 33 | " and please select the answer" + \ 34 | " correspoinding to the question.\n" 35 | print(info) 36 | elif test_type == 'MOS': 37 | info = "Each 'MOS' test will play one file," + \ 38 | " and please select the score" + \ 39 | " correspoinding to the question." 40 | print(info) 41 | print("There are several examples with score 5\n") 42 | input("Press any key to play the examples!\n") 43 | for i, file in enumerate(examples['MOS']): 44 | print("example%d" % i) 45 | playspeech(file) 46 | print('\n') 47 | elif test_type == 'XAB': 48 | info = "Each 'XAB' test will play 'reference', 'A'" + \ 49 | " and 'B' files, please selct the %s" % text['XAB']['C']+ \ 50 | " of 'A' or 'B' is %s to that of the reference." % text['XAB']['Ans'] 51 | print(info) 52 | print("There is an example\n") 53 | input("Press any key to play the example!\n") 54 | print("example reference") 55 | playspeech(examples['XAB']['ref']) 56 | print("postive example (correct answer)") 57 | playspeech(examples['XAB']['pos']) 58 | print("negative example (wrong answer)") 59 | playspeech(examples['XAB']['neg']) 60 | print('\n') 61 | else: 62 | raise ValueError( 63 | "test type %s is not in (MOS, SIM, PK, XAB)!"%(test_type)) 64 | 65 | # XAB test 66 | def main_XAB(eval_dict, r_idx, text): 67 | """MAIN OF XAB TEST 68 | Args: 69 | eval_dict (dict): evaluation dict 70 | r_idx (int): index of unfinished files 71 | text (dict): dict of text information 72 | """ 73 | # MethodA(expected answer) v.s. MethodB with the reference MethodX 74 | fileX = eval_dict['FileX'] 75 | answer = random.randint(0, 1) 76 | #print(answer) 77 | if answer: 78 | # expected answer is fileA 79 | fileA = eval_dict['FileA'] 80 | fileB = eval_dict['FileB'] 81 | else: 82 | # expected answer is fileB 83 | fileA = eval_dict['FileB'] 84 | fileB = eval_dict['FileA'] 85 | #print(fileX) 86 | #print(fileA) 87 | #print(fileB) 88 | print("*"+("[No. %4d]-0 reference wav file" % (r_idx)).center(20)+"*") 89 | playspeech(fileX) 90 | time.sleep(0.3) 91 | print("*"+("[No. %4d]-1 'A' wav file" % (r_idx)).center(20)+"*") 92 | playspeech(fileA) 93 | time.sleep(0.3) 94 | print("*"+("[No. %4d]-2 'B' wav file" % (r_idx)).center(20)+"*") 95 | playspeech(fileB) 96 | ans_flag = True 97 | Criteria = text['XAB']['C'] 98 | Ans = text['XAB']['Ans'] 99 | Q = text['XAB']['Q'].replace('Ans', Ans).replace('Criteria', Criteria) 100 | while ans_flag: 101 | print("\n%s" % Q) 102 | ans = input(("%-6s[1].%-22s[2].%-22s\n" % 103 | ("Enter ", " 'A' is %s" % Ans, " 'B' is %s" % Ans)) + \ 104 | ("%-6s[6].%-22s[7].%-22s\n" % 105 | ("", " replay 'A' wav", " replay 'B' wav")) + \ 106 | ("%-6s[8].%-22s[9].%-22s\n" % 107 | ("", " replay reference wav", " replay all files")) + \ 108 | ("%-6s[q].%-22s\n: " % 109 | ("", " quit evaluation"))) 110 | 111 | if ans == "1": 112 | eval_dict['Score'] = answer 113 | eval_dict['Finished'] = True 114 | ans_flag = False 115 | elif ans == "2": 116 | eval_dict['Score'] = (1 - answer) 117 | eval_dict['Finished'] = True 118 | ans_flag = False 119 | elif ans == "6": 120 | print("'A' wav") 121 | playspeech(fileA) 122 | elif ans == "7": 123 | print("'B' wav") 124 | playspeech(fileB) 125 | elif ans == "8": 126 | print("reference wav") 127 | playspeech(fileX) 128 | elif ans == "9": 129 | print("reference wav") 130 | playspeech(fileX) 131 | time.sleep(0.3) 132 | print("'A' wav") 133 | playspeech(fileA) 134 | time.sleep(0.3) 135 | print("'B' wav") 136 | playspeech(fileB) 137 | elif ans == "q": 138 | return ans 139 | else: 140 | print("ans should be 1,2,6,7,8,9 or q. %s is not included!!" % ans) 141 | return ans 142 | 143 | # preference test 144 | def main_PK(eval_dict, r_idx, text): 145 | """MAIN OF PK TEST 146 | Args: 147 | eval_dict (dict): evaluation dict 148 | r_idx (int): index of unfinished files 149 | text (dict): dict of text information 150 | """ 151 | # MethodA(expected answer) v.s. MethodB 152 | answer = random.randint(0, 1) 153 | #print(answer) 154 | if answer: 155 | # expected answer is fileA 156 | fileA = eval_dict['FileA'] 157 | fileB = eval_dict['FileB'] 158 | else: 159 | # expected answer is fileB 160 | fileA = eval_dict['FileB'] 161 | fileB = eval_dict['FileA'] 162 | #print(fileA) 163 | #print(fileB) 164 | print("*"+("[No. %4d]-1 'A' wav file" % (r_idx)).center(20)+"*") 165 | playspeech(fileA) 166 | time.sleep(0.3) 167 | print("*"+("[No. %4d]-2 'B' wav file" % (r_idx)).center(20)+"*") 168 | playspeech(fileB) 169 | ans_flag = True 170 | Criteria = text['PK']['C'] 171 | Ans = text['PK']['Ans'] 172 | Q = text['PK']['Q'].replace('Ans', Ans).replace('Criteria', Criteria) 173 | while ans_flag: 174 | print("\n%s" % Q) 175 | ans = input(("%-6s[1].%-22s[2].%-22s\n" % 176 | ("Enter ", " 'A' is %s" % Ans, " 'B' is %s" % Ans)) + \ 177 | ("%-6s[7].%-22s[8].%-22s\n" % 178 | ("", " replay 'A' wav", " replay 'B' wav")) + \ 179 | ("%-6s[9].%-22s[q].%-22s\n: " % 180 | ("", " replay both files", " quit evaluation"))) 181 | if ans == "1": 182 | eval_dict['Score'] = answer 183 | eval_dict['Finished'] = True 184 | ans_flag = False 185 | elif ans == "2": 186 | eval_dict['Score'] = (1 - answer) 187 | eval_dict['Finished'] = True 188 | ans_flag = False 189 | elif ans == "7": 190 | print("'A' wav") 191 | playspeech(fileA) 192 | elif ans == "8": 193 | print("'B' wav") 194 | playspeech(fileB) 195 | elif ans == "9": 196 | print("'A' wav") 197 | playspeech(fileA) 198 | time.sleep(0.3) 199 | print("'B' wav") 200 | playspeech(fileB) 201 | elif ans == "q": 202 | return ans 203 | else: 204 | print("ans should be 1,2,7,8,9 or q. %s is not included!!" % ans) 205 | return ans 206 | 207 | # MOS test 208 | def main_MOS(eval_dict, r_idx, text): 209 | """MAIN OF MOS TEST 210 | Args: 211 | eval_dict (dict): evaluation dict 212 | r_idx (int): index of unfinished files 213 | text (dict): dict of text information 214 | """ 215 | # PLAY FILES 216 | file = eval_dict['File'] 217 | #print(file) 218 | print("*"+("[No. %4d] wav file" % (r_idx)).center(20)+"*") 219 | playspeech(file) 220 | # INPUT ANSWER 221 | ans_flag = True 222 | scores = ["1", "2", "3", "4", "5"] 223 | Criteria = text['MOS']['C'] 224 | Q = text['MOS']['Q'].replace('Criteria', Criteria) 225 | while ans_flag: 226 | print("\n%s" % Q) 227 | ans = input(("%-6s[1].%-22s[2].%-22s\n" % 228 | ("Enter ", " Bad", " Poor")) + \ 229 | ("%-6s[3].%-22s[4].%-22s\n" % 230 | ("", " Fair", " Good")) + \ 231 | ("%-6s[5].%-22s[7].%-22s\n" % 232 | ("", " Excellent", " replay wav")) + \ 233 | ("%-6s[q].%-22s\n: " % 234 | ("", " quit evaluation"))) 235 | if ans in scores: 236 | eval_dict['Score'] = float(ans) 237 | eval_dict['Finished'] = True 238 | ans_flag = False 239 | elif ans == "7": 240 | print("replay wav") 241 | playspeech(file) 242 | elif ans == "q": 243 | return ans 244 | else: 245 | print("ans should be 1,2,3,4,5,7 or q. %s is not included!!" % ans) 246 | return ans 247 | 248 | # similarity test 249 | def main_SIM(eval_dict, r_idx, text): 250 | """MAIN OF SIM TEST 251 | Args: 252 | eval_dict (dict): evaluation dict 253 | r_idx (int): index of unfinished files 254 | text (dict): dict of text information 255 | """ 256 | # PLAY FILES 257 | if random.randint(0, 1): 258 | file1 = eval_dict['File'] 259 | file2 = eval_dict['File_ans'] 260 | else: 261 | file1 = eval_dict['File_ans'] 262 | file2 = eval_dict['File'] 263 | #print(file1) 264 | #print(file2) 265 | print("*"+("[No. %4d]-1 wav file" % (r_idx)).center(20)+"*") 266 | playspeech(file1) 267 | time.sleep(0.1) 268 | print("*"+("[No. %4d]-2 wav file" % (r_idx)).center(20)+"*") 269 | playspeech(file2) 270 | # INPUT ANSWER 271 | ans_flag = True 272 | scores = ["1", "2", "3", "4"] 273 | Q = text['SIM']['Q'] 274 | posAns = text['SIM']['posAns'] 275 | negAns = text['SIM']['negAns'] 276 | while ans_flag: 277 | print("\n%s" % Q) 278 | ans = input(("%-6s[1].%-22s[2].%-22s\n" % 279 | ("Enter ", " definitely %s" % posAns, " maybe %s" % posAns)) + \ 280 | ("%-6s[3].%-22s[4].%-22s\n" % 281 | ("", " maybe %s" % negAns, " definitely %s" % negAns)) + \ 282 | ("%-6s[7].%-22s[8].%-22s\n" % 283 | ("", " replay 1st wav", " replay 2nd wav")) + \ 284 | ("%-6s[9].%-22s[q].%-22s\n: " % 285 | ("", " replay both files", " quit evaluation"))) 286 | if ans in scores: 287 | eval_dict['Score'] = float(ans) 288 | eval_dict['Finished'] = True 289 | ans_flag = False 290 | elif ans == "7": 291 | print("1st wav") 292 | playspeech(file1) 293 | elif ans == "8": 294 | print("2nd wav") 295 | playspeech(file2) 296 | elif ans == "9": 297 | print("1st wav") 298 | playspeech(file1) 299 | time.sleep(0.1) 300 | print("2nd wav") 301 | playspeech(file2) 302 | elif ans == "q": 303 | return ans 304 | else: 305 | print("ans should be 1,2,3,4,7,8,9 or q. %s is not included!!" % ans) 306 | return ans 307 | 308 | # XLSX OUTPUT TEMPLATE 309 | class xlsx_output(object): 310 | def __init__(self, username, filename, sheetnames, testsystems): 311 | self.u_name = username 312 | self.fxlsx = filename 313 | self.t_sheets = sheetnames 314 | self.t_systems = testsystems 315 | # column index of xslx (alphabet list = ['A', 'B', ..., 'Z']) 316 | self.alphabet = [] 317 | for i in range(26): 318 | self.alphabet.append(chr(ord('A')+i)) 319 | 320 | def _add_username(self, sheet, c_row, t_name): 321 | # add user in the new row 322 | sheet.cell(row=c_row, column=1).value = t_name 323 | sheet.cell(row=(c_row+1), column=1).value = 'AVG.' 324 | 325 | def _add_data(self, sheet, c_row, col_chr, t_score): 326 | # add new scores 327 | sheet['%s%d' % (col_chr, c_row)].value = t_score 328 | start_idx = "%s%d" % (col_chr, 2) 329 | end_idx = "%s%d" % (col_chr, c_row) 330 | # update the average scores 331 | sheet['%s%d' % (col_chr, c_row+1)].value = \ 332 | "=AVERAGE(%s:%s)" % (start_idx, end_idx) 333 | 334 | # XAB, PREFERENCE, MOS XLSX OUTPUT CLASS 335 | class xlsx_SCORE(xlsx_output): 336 | """ 337 | Class of preference, MOS, and XAB tests 338 | """ 339 | def __init__(self, username, filename, sheetnames, testsystems): 340 | super().__init__(username, filename, sheetnames, testsystems) 341 | if not os.path.exists(self.fxlsx): 342 | self._initial_xlsx() 343 | 344 | def output_xlsx(self, t_sheet, t_dict): 345 | """OUTPUT SUBJECTIVE RESULTS INTO A SHEET OF EXCEL FILE 346 | Args: 347 | t_sheet (str): the subset name of the subjective results 348 | [summary, xgender, sgender, F-F, F-M, M-F, M-M] 349 | t_dict (list of dict): the result dicts 350 | """ 351 | if len(t_dict) == 0: 352 | print("%s is empty!!" % t_sheet) 353 | return 354 | wb = load_workbook(self.fxlsx) # load workspace of the excel file 355 | sheet = wb['%s' % t_sheet] # load sheet 356 | c_row = sheet.max_row # get latest row index 357 | # add new user 358 | self._add_username(sheet, c_row, self.u_name) 359 | # parse results 360 | t_score = self._score(t_dict) 361 | # update sheet 362 | for i in range(len(t_score)): 363 | self._add_data(sheet, c_row, self.alphabet[i+1], t_score[i]) 364 | wb.save(self.fxlsx) 365 | 366 | def _initial_xlsx(self): 367 | wb = Workbook() 368 | first = True 369 | for s_name in self.t_sheets: 370 | if first: 371 | sheet = wb.active 372 | sheet.title = s_name 373 | first = False 374 | else: 375 | wb.create_sheet(title=s_name) 376 | sheet = wb['%s' % s_name] 377 | sheet['A1'].value = 'USER' 378 | sheet['A2'].value = 'AVG.' 379 | for i in range(len(self.t_systems)): 380 | sheet.cell(row=1, column=(2+i)).value = self.t_systems[i] 381 | wb.save(self.fxlsx) 382 | 383 | def _score(self, t_dict): 384 | t_score = [] 385 | for t_method in self.t_systems: 386 | score = 0.0 387 | t_num = 0 388 | f_t_dict = filter(lambda item: item['method'] == t_method, t_dict) 389 | for t_file in f_t_dict: 390 | score += t_file['Score'] 391 | t_num+=1 392 | if t_num==0: 393 | t_score.append(-1.0) 394 | else: 395 | t_score.append(score/t_num) 396 | return t_score 397 | 398 | # SIMILARITY XLSX OUTPUT CLASS 399 | class xlsx_SIM(xlsx_output): 400 | """ 401 | Class of similarity test 402 | """ 403 | def __init__(self, username, filename, sheetnames, testsystems, text): 404 | super().__init__(username, filename, sheetnames, testsystems) 405 | self.posAns_sure = '%s' % text['SIM']['posAns'] 406 | self.posAns_weak = 'maybe_%s' % text['SIM']['posAns'] 407 | self.negAns_weak = 'maybe_%s' % text['SIM']['negAns'] 408 | self.negAns_sure = '%s' % text['SIM']['negAns'] 409 | if not os.path.exists(self.fxlsx): 410 | self._initial_xlsx() 411 | 412 | def output_xlsx(self, t_sheet, t_dict): 413 | """OUTPUT SUBJECTIVE RESULTS INTO A SHEET OF EXCEL FILE 414 | Args: 415 | t_sheet (str): the subset name of the subjective results 416 | [summary, xgender, sgender, F-F, F-M, M-F, M-M] 417 | t_dict (list of dict): the result dicts 418 | """ 419 | if len(t_dict) == 0: 420 | print("%s is empty!!" % t_sheet) 421 | return 422 | wb = load_workbook(self.fxlsx) # load workspace of the excel file 423 | sheet = wb['%s' % t_sheet] # load sheet 424 | c_row = sheet.max_row # get latest row index 425 | # add new user 426 | self._add_username(sheet, c_row, self.u_name) 427 | # parse results 428 | t_score = self._score(t_dict) 429 | # update sheet 430 | for i in range(len(t_score)): 431 | self._add_data(sheet, c_row, self.alphabet[i*4+1], t_score[i][self.posAns_sure]) 432 | self._add_data(sheet, c_row, self.alphabet[i*4+2], t_score[i][self.posAns_weak]) 433 | self._add_data(sheet, c_row, self.alphabet[i*4+3], t_score[i][self.negAns_weak]) 434 | self._add_data(sheet, c_row, self.alphabet[i*4+4], t_score[i][self.negAns_sure]) 435 | #self._final_result(sheet) 436 | wb.save(self.fxlsx) 437 | 438 | def _final_result(self, sheet): 439 | c_row = sheet.max_row + 1 440 | sheet.cell(row=c_row, column=1).value = 'Result' 441 | m_idx = range(1, len(self.t_systems)*4+1, 4) 442 | for i in range(len(self.t_systems)): 443 | # SAME SPEAKER 444 | merge_range = "%s%d:%s%d" % ( 445 | chr(ord('A')+m_idx[i]), c_row, chr(ord('B')+m_idx[i]), c_row) 446 | sheet.merge_cells(merge_range) 447 | sum_range = "%s%d:%s%d" % ( 448 | chr(ord('A')+m_idx[i]), c_row-1, chr(ord('B')+m_idx[i]), c_row-1) 449 | sheet.cell(row=c_row, column=(2+i*4)).value = "=SUM(%s)" % (sum_range) 450 | # DIFFERENT SPEAKER 451 | merge_range = "%s%d:%s%d" % ( 452 | chr(ord('C')+m_idx[i]), c_row, chr(ord('D')+m_idx[i]), c_row) 453 | sheet.merge_cells(merge_range) 454 | sum_range = "%s%d:%s%d" % ( 455 | chr(ord('C')+m_idx[i]), c_row-1, chr(ord('D')+m_idx[i]), c_row-1) 456 | sheet.cell(row=c_row, column=(4+i*4)).value = "=SUM(%s)" % (sum_range) 457 | 458 | def _initial_xlsx(self): 459 | wb = Workbook() 460 | first = True 461 | for s_name in self.t_sheets: 462 | if first: 463 | sheet = wb.active 464 | sheet.title = s_name 465 | first = False 466 | else: 467 | wb.create_sheet(title=s_name) 468 | sheet = wb['%s' % s_name] 469 | sheet['A1'].value = 'USER' 470 | sheet['A2'].value = '' 471 | sheet['A3'].value = 'AVG.' 472 | m_idx = range(1, len(self.t_systems)*4+1, 4) 473 | for i in range(len(self.t_systems)): 474 | merge_range = "%s%d:%s%d" % (chr(ord('A')+m_idx[i]), 1, chr(ord('D')+m_idx[i]), 1) 475 | sheet.merge_cells(merge_range) 476 | sheet.cell(row=1, column=(2+i*4)).value = self.t_systems[i] 477 | sheet.cell(row=2, column=(2+i*4)).value = self.posAns_sure 478 | sheet.cell(row=2, column=(3+i*4)).value = self.posAns_weak 479 | sheet.cell(row=2, column=(4+i*4)).value = self.negAns_weak 480 | sheet.cell(row=2, column=(5+i*4)).value = self.negAns_sure 481 | wb.save(self.fxlsx) 482 | 483 | def _score(self, t_dict): 484 | t_score = [] 485 | for t_method in self.t_systems: 486 | score = {self.posAns_sure: 0.0, self.posAns_weak: 0.0, 487 | self.negAns_sure: 0.0, self.negAns_weak: 0.0} 488 | f_t_dict = filter(lambda item: item['method'] == t_method, t_dict) 489 | t_num = 0 490 | for t_file in f_t_dict: 491 | if t_file['Score'] == 1.0: 492 | ans = self.posAns_sure 493 | elif t_file['Score'] == 2.0: 494 | ans = self.posAns_weak 495 | elif t_file['Score'] == 3.0: 496 | ans = self.negAns_weak 497 | elif t_file['Score'] == 4.0: 498 | ans = self.negAns_sure 499 | else: 500 | raise ValueError("%s method score %.2f is out of range" % ( 501 | t_method, t_file['Score'])) 502 | score[ans] += 1.0 503 | t_num += 1 504 | if t_num == 0: 505 | t_score.append(score) 506 | else: 507 | score = {key: value / t_num for key, value in score.items()} 508 | t_score.append(score) 509 | return t_score -------------------------------------------------------------------------------- /yml/EVAL1/evaluation.yml: -------------------------------------------------------------------------------- 1 | - method: 2 | - MethodA 3 | - MethodB 4 | - Target 5 | set: 6 | - MOS_0.yml 7 | - MOS_1.yml 8 | type: MOS 9 | - method: 10 | - MethodB*-MethodA-Target_ref # the method with * is the main system (expected answer) 11 | - MethodA*-MethodB-Target_ref # the method with * is the main system (expected answer) 12 | set: 13 | - XAB_0.yml 14 | - XAB_1.yml 15 | type: XAB 16 | - method: 17 | - MethodA 18 | - MethodB 19 | reference: Target_ref 20 | set: 21 | - SIM_0.yml 22 | - SIM_1.yml 23 | type: SIM 24 | - method: 25 | - MethodB*-MethodA # the method with * is the main system (expected answer) 26 | - MethodA*-MethodB # the method with * is the main system (expected answer) 27 | set: 28 | - PK_0.yml 29 | - PK_1.yml 30 | type: PK -------------------------------------------------------------------------------- /yml/EVAL1/record.yml: -------------------------------------------------------------------------------- 1 | MAX_count: 2 # the number of subsets 2 | Subject_name: [] 3 | Subject_set: [] 4 | count: 0 # the number of start subsets 5 | time: 2019-12-24 14:41:59.596524 6 | -------------------------------------------------------------------------------- /yml/EVAL1/test_system.yml: -------------------------------------------------------------------------------- 1 | path: # the template format of file path of each system 2 | MethodA: data/eval_name/system/srcspk-tarspk/filename 3 | MethodB: data/eval_name/system/srcspk-tarspk/filename 4 | Target: data/eval_name/system/tarspk/filename 5 | speaker: # the information of each speaker in evaluation data 6 | SF3: F 7 | SM3: M 8 | TF1: F -------------------------------------------------------------------------------- /yml/EVAL1/text.yml: -------------------------------------------------------------------------------- 1 | text: 2 | PK: 3 | C: speech QUALITY # criteria 4 | Ans: higher # answer 5 | Q: Which file has Ans Criteria? # question 6 | XAB: 7 | C: "PITCH" # criteria 8 | Ans: closer # answer 9 | Q: The Criteria of which file is Ans to that of the reference file? # question 10 | MOS: 11 | C: speech QUALITY # criteria 12 | Q: How is the Criteria of the file? # question 13 | SIM: 14 | Q: Are the speakers of the two audio files the SAME? # question 15 | posAns: the same # positive answer 16 | negAns: different # negative answer 17 | examples: 18 | MOS: 19 | - data/example/MOS/exp1.wav 20 | - data/example/MOS/exp2.wav 21 | XAB: 22 | ref: data/example/XAB/ref.wav # reference example 23 | pos: data/example/XAB/pos.wav # positive example 24 | neg: data/example/XAB/neg.wav # negative example 25 | c_text: 26 | configf: evaluation # the config file of the evaluations (configf) 27 | system: method # the synonym of test system (configf) 28 | reference: reference # the synonym of reference system (configf) 29 | t_set: set # the synonym of test set (configf) 30 | t_type: type # the synonym of test type (configf) 31 | recordf: record # the result record file (recordf) 32 | n_subset: MAX_count # the synonym of the number of sub sets (recordf) 33 | subset_idx: count # the synonym of the indexes of sub sets (recordf) 34 | user_name: Subject_name # the synonym of the list of user name (recordf) 35 | t_subset: Subject_set # the synonym of the list of test subset (recordf) 36 | date: time # the synonym of date (recordf) 37 | systemf: test_system # the config file of the test systems (systemf) 38 | templatedir: path # the synonym of the template path of the files (systemf) 39 | spk: speaker # the synonym of speaker (systemf) 40 | srcspk: srcspk # the synonym of srcspk (systemf) 41 | tarspk: tarspk # the synonym of tarspk (systemf) 42 | r_score: Score # the synonym of result score 43 | t_finish: Finished # the synonym of test finish flag 44 | -------------------------------------------------------------------------------- /yml/template/MOS.yml: -------------------------------------------------------------------------------- 1 | File: filename.wav 2 | Finished: False 3 | Score: -1.0 4 | conversion: False 5 | xgender: False 6 | gender: None 7 | pair: None 8 | srcspk: None 9 | tarspk: None 10 | method: None 11 | type: MOS 12 | -------------------------------------------------------------------------------- /yml/template/PK.yml: -------------------------------------------------------------------------------- 1 | FileA: filenameA.wav 2 | FileB: filenameB.wav 3 | Finished: False 4 | Score: -1.0 5 | conversion: False 6 | xgender: False 7 | gender: None 8 | pair: None 9 | srcspk: None 10 | tarspk: None 11 | method: None 12 | methodA: None 13 | methodB: None 14 | type: PK 15 | -------------------------------------------------------------------------------- /yml/template/SIM.yml: -------------------------------------------------------------------------------- 1 | File: filename.wav 2 | File_ans: ans_filename.wav 3 | Finished: False 4 | Score: -1.0 5 | conversion: False 6 | xgender: False 7 | gender: None 8 | pair: None 9 | srcspk: None 10 | tarspk: None 11 | method: None 12 | type: SIM 13 | -------------------------------------------------------------------------------- /yml/template/XAB.yml: -------------------------------------------------------------------------------- 1 | FileA: filenameA.wav 2 | FileB: filenameB.wav 3 | FileX: filenameX.wav 4 | Finished: False 5 | Score: -1.0 6 | conversion: False 7 | xgender: False 8 | gender: None 9 | pair: None 10 | srcspk: None 11 | tarspk: None 12 | method: None 13 | methodA: None 14 | methodB: None 15 | methodX: None 16 | type: XAB 17 | --------------------------------------------------------------------------------