├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── data └── README.md └── src └── sae.py /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to making participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, sex characteristics, gender identity and expression, 9 | level of experience, education, socio-economic status, nationality, personal 10 | appearance, race, religion, or sexual identity and orientation. 11 | 12 | ## Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ## Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ## Scope 47 | 48 | This Code of Conduct applies both within project spaces and in public spaces 49 | when an individual is representing the project or its community. Examples of 50 | representing a project or community include using an official project e-mail 51 | address, posting via an official social media account, or acting as an appointed 52 | representative at an online or offline event. Representation of a project may be 53 | further defined and clarified by project maintainers. 54 | 55 | ## Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team at hskim.cvpr@yonsei.ac.kr. All 59 | complaints will be reviewed and investigated and will result in a response that 60 | is deemed necessary and appropriate to the circumstances. The project team is 61 | obligated to maintain confidentiality with regard to the reporter of an incident. 62 | Further details of specific enforcement policies may be posted separately. 63 | 64 | Project maintainers who do not follow or enforce the Code of Conduct in good 65 | faith may face temporary or permanent repercussions as determined by other 66 | members of the project's leadership. 67 | 68 | ## Attribution 69 | 70 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 71 | available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html 72 | 73 | [homepage]: https://www.contributor-covenant.org 74 | 75 | For answers to common questions about this code of conduct, see 76 | https://www.contributor-covenant.org/faq 77 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Introduction 2 | 3 | Thank you for considering contributing to the Explainable Zero-Shot Learning project. 4 | 5 | ### Why We Need This Document? 6 | 7 | Following the document helps to communicate among developers and maintainers. Even this is a small project, we want to minimize the communication costs. 8 | 9 | ### We Need You! 10 | 11 | Explainable Zero-Shot Learning project is an open source project and we welcome someone who can contribute to this project via pull requests, bug reports, or any questions. 12 | 13 | # Ground Rules 14 | 15 | ### Anything is OK. 16 | 17 | We hope everyone could participate easily for this project and I would like to discuss each other freely without any special rules. 18 | 19 | ### Community 20 | 21 | If you have additional questions about the project, please use the issue in the GitHub repository. We will reply as soon as possible. You could also send an e-mail to hskim.cvpr@yonsei.ac.kr. 22 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Hoseong Kim 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # sae-pytorch 2 | 3 | Original : [[MATLAB version]](https://github.com/Elyorcv/SAE) 4 | 5 | PyTorch implementation of Semantic AutoEncoder (SAE). 6 | 7 | ## How to Run 8 | 1. git clone https://github.com/hoseong-kim/sae-pytorch.git 9 | 2. Download 'awa_demo_data.mat' 10 | * https://drive.google.com/file/d/1l0UVhhIU-SmtJ9hqk7OVOG9zNga9qt_I/view 11 | 3. python sae.py 12 | 13 | ## An Implementation of SAE in PyTorch 14 | 1. Set CUB, AwA, aP&Y, SUN, and ImageNet datasets. 15 | * Partially done (only for AwA dataset). 16 | * Other datasets will also be available soon. 17 | 2. Extract deep features from various deep models, e.g., AlexNet, VGG16, VGG19, GoogleNet, Inception_v3, ResNet-18, ResNet-34, ResNet-50, ResNet-101, ResNet-152. 18 | * Done, but tuning my source code to achieve results in this paper. 19 | * The source code will be available after reproducing. 20 | 3. Train a Semantic AutoEncoder (SAE). 21 | * Done. 22 | 4. Test unseen class data. 23 | * Done. 24 | 25 | ## Release Note 26 | 27 | #### v1.0 28 | 29 | * Bug fix 30 | 31 | ## Download Paper 32 | Semantic Autoencoder for Zero-shot Learning: [[Paper Link (arXiv)]](https://arxiv.org/abs/1704.08345) 33 | -------------------------------------------------------------------------------- /data/README.md: -------------------------------------------------------------------------------- 1 | ### Download dataset features 2 | * AwA 3 | * https://drive.google.com/file/d/1l0UVhhIU-SmtJ9hqk7OVOG9zNga9qt_I/view 4 | -------------------------------------------------------------------------------- /src/sae.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import scipy 3 | import scipy.io 4 | import argparse 5 | 6 | def parse_args(): 7 | parser = argparse.ArgumentParser() 8 | parser.add_argument('--ld', type=float, default=500000) # lambda 9 | return parser.parse_args() 10 | 11 | 12 | def normalizeFeature(x): 13 | # x = d x N dims (d: feature dimension, N: the number of features) 14 | x = x + 1e-10 # for avoid RuntimeWarning: invalid value encountered in divide 15 | feature_norm = np.sum(x**2, axis=1)**0.5 # l2-norm 16 | feat = x / feature_norm[:, np.newaxis] 17 | return feat 18 | 19 | def SAE(x, s, ld): 20 | # SAE is Semantic Autoencoder 21 | # INPUTS: 22 | # x: d x N data matrix 23 | # s: k x N semantic matrix 24 | # ld: lambda for regularization parameter 25 | # 26 | # OUTPUT: 27 | # w: kxd projection matrix 28 | 29 | A = np.dot(s, s.transpose()) 30 | B = ld * np.dot(x, x.transpose()) 31 | C = (1+ld) * np.dot(s, x.transpose()) 32 | w = scipy.linalg.solve_sylvester(A,B,C) 33 | return w 34 | 35 | def distCosine(x, y): 36 | xx = np.sum(x**2, axis=1)**0.5 37 | x = x / xx[:, np.newaxis] 38 | yy = np.sum(y**2, axis=1)**0.5 39 | y = y / yy[:, np.newaxis] 40 | dist = 1 - np.dot(x, y.transpose()) 41 | return dist 42 | 43 | 44 | 45 | def zsl_acc(semantic_predicted, semantic_gt, opts): 46 | # zsl_acc calculates zero-shot classification accruacy 47 | # 48 | # INPUTS: 49 | # semantic_prediced: predicted semantic labels 50 | # semantic_gt: ground truth semantic labels 51 | # opts: other parameters 52 | # 53 | # OUTPUT: 54 | # zsl_accuracy: zero-shot classification accuracy (per-sample) 55 | 56 | dist = 1 - distCosine(semantic_predicted, normalizeFeature(semantic_gt.transpose()).transpose()) 57 | y_hit_k = np.zeros((dist.shape[0], opts.HITK)) 58 | for idx in range(0, dist.shape[0]): 59 | sorted_id = sorted(range(len(dist[idx,:])), key=lambda k: dist[idx,:][k], reverse=True) 60 | y_hit_k[idx,:] = opts.test_classes_id[sorted_id[0:opts.HITK]] 61 | 62 | n = 0 63 | for idx in range(0, dist.shape[0]): 64 | if opts.test_labels[idx] in y_hit_k[idx,:]: 65 | n = n + 1 66 | zsl_accuracy = float(n) / dist.shape[0] * 100 67 | return zsl_accuracy, y_hit_k 68 | 69 | 70 | def main(): 71 | # for AwA dataset: Perfectly works. 72 | opts = parse_args() 73 | awa = scipy.io.loadmat('awa_demo_data.mat') 74 | train_data = awa['X_tr'] 75 | test_data = awa['X_te'] 76 | train_class_attributes_labels_continuous_allset = awa['S_tr'] 77 | opts.test_labels = awa['test_labels'] 78 | opts.test_classes_id = awa['testclasses_id'] 79 | test_class_attributes_labels_continuous = awa['S_te_gt'] 80 | 81 | ##### Normalize the data 82 | train_data = normalizeFeature(train_data.transpose()).transpose() 83 | 84 | ##### Training 85 | # SAE 86 | W = SAE(train_data.transpose(), train_class_attributes_labels_continuous_allset.transpose(), opts.ld) 87 | 88 | ##### Test 89 | opts.HITK = 1 90 | 91 | # [F --> S], projecting data from feature space to semantic space: 84.68% for AwA dataset 92 | semantic_predicted = np.dot(test_data, normalizeFeature(W).transpose()) 93 | [zsl_accuracy, y_hit_k] = zsl_acc(semantic_predicted, test_class_attributes_labels_continuous, opts) 94 | print('[1] zsl accuracy for AwA dataset [F >>> S]: {:.2f}%'.format(zsl_accuracy)) 95 | 96 | # [S --> F], projecting from semantic to visual space: 84.00% for AwA dataset 97 | test_predicted = np.dot(normalizeFeature(test_class_attributes_labels_continuous.transpose()).transpose(), normalizeFeature(W)) 98 | [zsl_accuracy, y_hit_k] = zsl_acc(test_data, test_predicted, opts) 99 | print('[2] zsl accuracy for AwA dataset [S >>> F]: {:.2f}%'.format(zsl_accuracy)) 100 | 101 | if __name__ == '__main__': 102 | main() 103 | --------------------------------------------------------------------------------