'
13 |
14 | hr_faded: '
'
15 | hr_shaded: '
'
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | Copyright 2018 Finn.no / Joakim Rishaug
2 |
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
4 |
5 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
6 |
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
8 |
--------------------------------------------------------------------------------
/docs/licenses/LICENSE:
--------------------------------------------------------------------------------
1 | /* This license pertains to the docs template, except for the Navgoco jQuery component. */
2 |
3 | The MIT License (MIT)
4 |
5 | Original theme: Copyright (c) 2016 Tom Johnson
6 | Modifications: Copyright (c) 2017 onwards fast.ai, Inc
7 |
8 | Permission is hereby granted, free of charge, to any person obtaining a copy
9 | of this software and associated documentation files (the "Software"), to deal
10 | in the Software without restriction, including without limitation the rights
11 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
12 | copies of the Software, and to permit persons to whom the Software is
13 | furnished to do so, subject to the following conditions:
14 |
15 | The above copyright notice and this permission notice shall be included in all
16 | copies or substantial portions of the Software.
17 |
18 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
19 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
20 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
21 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
22 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
23 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
24 | SOFTWARE.
25 |
--------------------------------------------------------------------------------
/.github/workflows/main.yml:
--------------------------------------------------------------------------------
1 | name: CI
2 | on: [push]
3 | jobs:
4 | build:
5 | runs-on: ubuntu-latest
6 | steps:
7 | - uses: actions/checkout@v1
8 | - uses: actions/setup-python@v1
9 | with:
10 | python-version: '3.6'
11 | architecture: 'x64'
12 | - name: Install the library
13 | run: |
14 | pip install nbdev jupyter
15 | pip install -e .
16 | - name: Read all notebooks
17 | run: |
18 | nbdev_read_nbs
19 | - name: Check if all notebooks are cleaned
20 | run: |
21 | echo "Check we are starting with clean git checkout"
22 | if [ -n "$(git status -uno -s)" ]; then echo "git status is not clean"; false; fi
23 | echo "Trying to strip out notebooks"
24 | nbdev_clean_nbs
25 | echo "Check that strip out was unnecessary"
26 | git status -s # display the status to see which nbs need cleaning up
27 | if [ -n "$(git status -uno -s)" ]; then echo -e "!!! Detected unstripped out notebooks\n!!!Remember to run nbdev_install_git_hooks"; false; fi
28 | - name: Check if there is no diff library/notebooks
29 | run: |
30 | if [ -n "$(nbdev_diff_nbs)" ]; then echo -e "!!! Detected difference between the notebooks and the library"; false; fi
31 | - name: Run tests
32 | run: |
33 | nbdev_test_nbs
34 |
--------------------------------------------------------------------------------
/docs/feed.xml:
--------------------------------------------------------------------------------
1 | ---
2 | search: exclude
3 | layout: none
4 | ---
5 |
6 |
7 |
8 |
9 | {{ site.title | xml_escape }}
10 | {{ site.description | xml_escape }}
11 | {{ site.time | date_to_rfc822 }}
14 | {{ site.time | date_to_rfc822 }}
15 | Jekyll v{{ jekyll.version }}
16 | {% for post in site.posts limit:10 %}
17 | -
18 |
{{ post.title | xml_escape }}
19 | {{ post.content | xml_escape }}
20 | {{ post.date | date_to_rfc822 }}
21 | {{ post.url | prepend: site.url }}
23 | {% for tag in post.tags %}
24 | {{ tag | xml_escape }}
25 | {% endfor %}
26 | {% for tag in page.tags %}
27 | {{ cat | xml_escape }}
28 | {% endfor %}
29 |
30 | {% endfor %}
31 |
32 |
33 |
--------------------------------------------------------------------------------
/docs/_config.yml:
--------------------------------------------------------------------------------
1 | repository: NegatioN/online_triplet_loss
2 | output: web
3 | topnav_title: WARP-Pytorch
4 | site_title: WARP-Pytorch
5 | company_name: "Joakim Rishaug"
6 | description: "WARP loss for Pytorch. WSABIE"
7 | # Set to false to disable KaTeX math
8 | use_math: true
9 | # Add Google analytics id if you have one and want to use it here
10 | google_analytics:
11 | # See http://nbdev.fast.ai/search for help with adding Search
12 | google_search:
13 |
14 | host: 127.0.0.1
15 | # the preview server used. Leave as is.
16 | port: 4000
17 | # the port where the preview is rendered.
18 |
19 | exclude:
20 | - .idea/
21 | - .gitignore
22 | - vendor
23 |
24 | exclude: [vendor]
25 |
26 | highlighter: rouge
27 | markdown: kramdown
28 | kramdown:
29 | input: GFM
30 | auto_ids: true
31 | hard_wrap: false
32 | syntax_highlighter: rouge
33 |
34 | collections:
35 | tooltips:
36 | output: false
37 |
38 | defaults:
39 | -
40 | scope:
41 | path: ""
42 | type: "pages"
43 | values:
44 | layout: "page"
45 | comments: true
46 | search: true
47 | sidebar: home_sidebar
48 | topnav: topnav
49 | -
50 | scope:
51 | path: ""
52 | type: "tooltips"
53 | values:
54 | layout: "page"
55 | comments: true
56 | search: true
57 | tooltip: true
58 |
59 | sidebars:
60 | - home_sidebar
61 | permalink: pretty
62 |
63 | theme: jekyll-theme-cayman
64 | baseurl: /WARP-Pytorch/
--------------------------------------------------------------------------------
/docs/_includes/links.html:
--------------------------------------------------------------------------------
1 | {% comment %}Get links from each sidebar, as listed in the _config.yml file under sidebars{% endcomment %}
2 |
3 | {% for sidebar in site.sidebars %}
4 | {% for entry in site.data.sidebars[sidebar].entries %}
5 | {% for folder in entry.folders %}
6 | {% for folderitem in folder.folderitems %}
7 | {% if folderitem.url contains "html#" %}
8 | [{{folderitem.url | remove: "/" }}]: {{folderitem.url | remove: "/"}}
9 | {% else %}
10 | [{{folderitem.url | remove: "/" | remove: ".html"}}]: {{folderitem.url | remove: "/"}}
11 | {% endif %}
12 | {% for subfolders in folderitem.subfolders %}
13 | {% for subfolderitem in subfolders.subfolderitems %}
14 | [{{subfolderitem.url | remove: "/" | remove: ".html"}}]: {{subfolderitem.url | remove: "/"}}
15 | {% endfor %}
16 | {% endfor %}
17 | {% endfor %}
18 | {% endfor %}
19 | {% endfor %}
20 | {% endfor %}
21 |
22 |
23 | {% comment %} Get links from topnav {% endcomment %}
24 |
25 | {% for entry in site.data.topnav.topnav %}
26 | {% for item in entry.items %}
27 | {% if item.external_url == null %}
28 | [{{item.url | remove: "/" | remove: ".html"}}]: {{item.url | remove: "/"}}
29 | {% endif %}
30 | {% endfor %}
31 | {% endfor %}
32 |
33 | {% comment %}Get links from topnav dropdowns {% endcomment %}
34 |
35 | {% for entry in site.data.topnav.topnav_dropdowns %}
36 | {% for folder in entry.folders %}
37 | {% for folderitem in folder.folderitems %}
38 | {% if folderitem.external_url == null %}
39 | [{{folderitem.url | remove: "/" | remove: ".html"}}]: {{folderitem.url | remove: "/"}}
40 | {% endif %}
41 | {% endfor %}
42 | {% endfor %}
43 | {% endfor %}
44 |
45 |
--------------------------------------------------------------------------------
/docs/css/modern-business.css:
--------------------------------------------------------------------------------
1 | /*!
2 | * Start Bootstrap - Modern Business HTML Template (http://startbootstrap.com)
3 | * Code licensed under the Apache License v2.0.
4 | * For details, see http://www.apache.org/licenses/LICENSE-2.0.
5 | */
6 |
7 | /* Global Styles */
8 |
9 | html,
10 | body {
11 | height: 100%;
12 | }
13 |
14 | .img-portfolio {
15 | margin-bottom: 30px;
16 | }
17 |
18 | .img-hover:hover {
19 | opacity: 0.8;
20 | }
21 |
22 | /* Home Page Carousel */
23 |
24 | header.carousel {
25 | height: 50%;
26 | }
27 |
28 | header.carousel .item,
29 | header.carousel .item.active,
30 | header.carousel .carousel-inner {
31 | height: 100%;
32 | }
33 |
34 | header.carousel .fill {
35 | width: 100%;
36 | height: 100%;
37 | background-position: center;
38 | background-size: cover;
39 | }
40 |
41 | /* 404 Page Styles */
42 |
43 | .error-404 {
44 | font-size: 100px;
45 | }
46 |
47 | /* Pricing Page Styles */
48 |
49 | .price {
50 | display: block;
51 | font-size: 50px;
52 | line-height: 50px;
53 | }
54 |
55 | .price sup {
56 | top: -20px;
57 | left: 2px;
58 | font-size: 20px;
59 | }
60 |
61 | .period {
62 | display: block;
63 | font-style: italic;
64 | }
65 |
66 | /* Footer Styles */
67 |
68 | footer {
69 | margin: 50px 0;
70 | }
71 |
72 | /* Responsive Styles */
73 |
74 | @media(max-width:991px) {
75 | .client-img,
76 | .img-related {
77 | margin-bottom: 30px;
78 | }
79 | }
80 |
81 | @media(max-width:767px) {
82 | .img-portfolio {
83 | margin-bottom: 15px;
84 | }
85 |
86 | header.carousel .carousel {
87 | height: 70%;
88 | }
89 | }
90 |
--------------------------------------------------------------------------------
/docs/_includes/head_print.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
{% if page.homepage == true %} {{site.homepage_title}} {% elsif page.title %}{{ page.title }}{% endif %} | {{ site.site_title }}
7 |
8 |
9 |
10 |
11 |
12 |
13 |
14 |
15 |
16 |
17 |
18 |
23 |
24 |
29 |
--------------------------------------------------------------------------------
/docs/licenses/LICENSE-BSD-NAVGOCO.txt:
--------------------------------------------------------------------------------
1 | /* This license pertains to the Navgoco jQuery component used for the sidebar. */
2 |
3 | Copyright (c) 2013, Christodoulos Tsoulloftas, http://www.komposta.net
4 | All rights reserved.
5 |
6 | Redistribution and use in source and binary forms, with or without modification,
7 | are permitted provided that the following conditions are met:
8 |
9 | * Redistributions of source code must retain the above copyright notice,
10 | this list of conditions and the following disclaimer.
11 | * Redistributions in binary form must reproduce the above copyright notice,
12 | this list of conditions and the following disclaimer in the documentation
13 | and/or other materials provided with the distribution.
14 | * Neither the name of the
nor the names of its
15 | contributors may be used to endorse or promote products derived from this
16 | software without specific prior written permission.
17 |
18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
19 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
20 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
21 | IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
22 | INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
23 | BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
24 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
25 | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
26 | OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
27 | OF THE POSSIBILITY OF SUCH DAMAGE.
--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
1 | # Byte-compiled / optimized / DLL files
2 | __pycache__/
3 | *.py[cod]
4 | *$py.class
5 | .idea/
6 | # C extensions
7 | *.so
8 |
9 | .gitattributes
10 | .gitconfig
11 | # Distribution / packaging
12 | .Python
13 | build/
14 | develop-eggs/
15 | dist/
16 | downloads/
17 | eggs/
18 | .eggs/
19 | lib/
20 | lib64/
21 | parts/
22 | sdist/
23 | var/
24 | wheels/
25 | *.egg-info/
26 | .installed.cfg
27 | *.egg
28 | MANIFEST
29 |
30 | # PyInstaller
31 | # Usually these files are written by a python script from a template
32 | # before PyInstaller builds the exe, so as to inject date/other infos into it.
33 | *.manifest
34 | *.spec
35 |
36 | # Installer logs
37 | pip-log.txt
38 | pip-delete-this-directory.txt
39 |
40 | # Unit test / coverage reports
41 | htmlcov/
42 | .tox/
43 | .coverage
44 | .coverage.*
45 | .cache
46 | nosetests.xml
47 | coverage.xml
48 | *.cover
49 | .hypothesis/
50 | .pytest_cache/
51 |
52 | # Translations
53 | *.mo
54 | *.pot
55 |
56 | # Django stuff:
57 | *.log
58 | local_settings.py
59 | db.sqlite3
60 |
61 | # Flask stuff:
62 | instance/
63 | .webassets-cache
64 |
65 | # Scrapy stuff:
66 | .scrapy
67 |
68 | # Sphinx documentation
69 | docs/_build/
70 |
71 | # PyBuilder
72 | target/
73 |
74 | # Jupyter Notebook
75 | .ipynb_checkpoints
76 |
77 | # pyenv
78 | .python-version
79 |
80 | # celery beat schedule file
81 | celerybeat-schedule
82 |
83 | # SageMath parsed files
84 | *.sage.py
85 |
86 | # Environments
87 | .env
88 | .venv
89 | env/
90 | venv/
91 | ENV/
92 | env.bak/
93 | venv.bak/
94 |
95 | # Spyder project settings
96 | .spyderproject
97 | .spyproject
98 |
99 | # Rope project settings
100 | .ropeproject
101 |
102 | # mkdocs documentation
103 | /site
104 |
105 | # mypy
106 | .mypy_cache/
107 |
--------------------------------------------------------------------------------
/docs/_layouts/page.html:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | ---
4 |
5 |
9 |
10 | {% if page.simple_map == true %}
11 |
12 |
17 |
18 | {% include custom/{{page.map_name}}.html %}
19 |
20 | {% elsif page.complex_map == true %}
21 |
22 |
27 |
28 | {% include custom/{{page.map_name}}.html %}
29 |
30 | {% endif %}
31 |
32 |
33 |
34 | {% if page.summary %}
35 |
{{page.summary}}
36 | {% endif %}
37 |
38 | {% unless page.toc == false %}
39 | {% include toc.html %}
40 | {% endunless %}
41 |
42 |
43 | {% if site.github_editme_path %}
44 |
45 |
46 |
47 | {% endif %}
48 |
49 | {{content}}
50 |
51 |
62 |
63 |
21 |
22 |
23 |
24 |
25 |
26 |
27 |
28 |
29 |
30 |
31 |
32 |
33 |
34 |
35 |
36 |
37 |
num_tries_gt_zero(scoresbatch_sizemax_trialsmax_numdevice
38 |
39 |
scores: [batch_size x N] float scores
40 | returns: [batch_size x 1] the lowest indice per row where scores were first greater than 0. plus 1
41 |
42 |
43 |
44 |
45 |
46 |
47 |
48 |
49 |
50 |
51 |
52 |
53 |
54 |
55 |
56 |
57 |
58 |
59 |
warp_loss(positive_predictionsnegative_predictionsnum_labelsdevice
60 |
61 |
positive_predictions: [batch_size x 1] floats between -1 to 1
62 | negative_predictions: [batch_size x N] floats between -1 to 1
63 | num_labels: int total number of labels in dataset (not just the subset you're using for the batch)
64 | device: pytorch.device
65 |
66 |
67 |
68 |
69 |
70 |
71 |
72 |
73 |
74 |
3 |
4 |
13 |
14 |
15 |
16 | {{item.title}} {{item.title}} {{item.title}}
38 | {{ folder.title }}
39 |
50 |
51 | {% endfor %}
52 | {% endfor %}
53 | {% if site.google_search %}
54 |
55 | {% include search_google_custom.html %}
56 |
57 | {% endif %}
58 |
59 |
60 |
63 |
--------------------------------------------------------------------------------
/docs/js/jquery.navgoco.min.js:
--------------------------------------------------------------------------------
1 | /*
2 | * jQuery Navgoco Menus Plugin v0.2.1 (2014-04-11)
3 | * https://github.com/tefra/navgoco
4 | *
5 | * Copyright (c) 2014 Chris T (@tefra)
6 | * BSD - https://github.com/tefra/navgoco/blob/master/LICENSE-BSD
7 | */
8 | !function(a){"use strict";var b=function(b,c,d){return this.el=b,this.$el=a(b),this.options=c,this.uuid=this.$el.attr("id")?this.$el.attr("id"):d,this.state={},this.init(),this};b.prototype={init:function(){var b=this;b._load(),b.$el.find("ul").each(function(c){var d=a(this);d.attr("data-index",c),b.options.save&&b.state.hasOwnProperty(c)?(d.parent().addClass(b.options.openClass),d.show()):d.parent().hasClass(b.options.openClass)?(d.show(),b.state[c]=1):d.hide()});var c=a("{{ page.title }} | {{ site.site_title }}
7 | " + txt + " ";
66 | else if (this_level <= level){ // higher level than before; end parent ol
67 | for(i = this_level; i < level; i++) {
68 | html += "" + txt + " ";
71 | }
72 | else if (this_level > level) { // lower level than before; expand the previous to contain a ol
73 | for(i = this_level; i > level; i--) {
74 | html += "<"+settings.listType+">"+((i-level == 2) ? "" : " ")
75 | }
76 | html += "" + txt + " ";
77 | }
78 | level = this_level; // update for the next one
79 | });
80 | html += "";
81 | if (!settings.noBackToTopLinks) {
82 | $(document).on('click', '.back-to-top', function() {
83 | $(window).scrollTop(0);
84 | window.location.hash = '';
85 | });
86 | }
87 |
88 | render[settings.showEffect]();
89 | };
90 | })(jQuery);
91 |
--------------------------------------------------------------------------------
/docs/_layouts/default.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 | {% include head.html %}
5 |
41 |
46 |
57 | {% if page.datatable == true %}
58 |
59 |
83 |
84 |
85 |
86 | {% assign content_col_size = "col-md-12" %}
87 | {% unless page.hide_sidebar %}
88 |
89 |
92 | {% assign content_col_size = "col-md-9" %}
93 | {% endunless %}
94 |
95 |
96 |
97 | {{content}}
98 |
99 |
100 |
101 |
102 |
103 |
104 |
36 |
37 |
38 | ```
39 | from torch import nn
40 | import torch
41 |
42 | class OurModel(nn.Module):
43 | def __init__(self, num_labels, emb_dim=10):
44 | super(OurModel, self).__init__()
45 | self.emb = nn.Embedding(num_labels, emb_dim)
46 | self.user_embs = nn.Embedding(1, emb_dim)
47 |
48 | def forward(self, pos, neg):
49 | batch_size = neg.size(0)
50 | one_user_vector = self.user_embs(torch.zeros(1).long())
51 | repeated_user_vector = one_user_vector.repeat((batch_size, 1)).view(batch_size, -1, 1)
52 | pos_res = torch.bmm(self.emb(pos), repeated_user_vector).squeeze(2)
53 | neg_res = torch.bmm(self.emb(neg), repeated_user_vector).squeeze(2)
54 |
55 | return pos_res, neg_res
56 |
57 | num_labels = 100
58 | model = OurModel(num_labels)
59 | ```
60 |
61 |
62 |
63 |
65 |
66 |
67 | ```
68 | pos_labels = torch.randint(high=num_labels, size=(3,1)) # our five labels
69 | neg_labels = torch.randint(high=num_labels, size=(3,2)) # a few random negatives per positive
70 |
71 | pos_res, neg_res = model(pos_labels, neg_labels)
72 | print('Positive Labels:', pos_labels)
73 | print('Negative Labels:', neg_labels)
74 | print('Model positive scores:', pos_res)
75 | print('Model negative scores:', neg_res)
76 | loss = warp_loss(pos_res, neg_res, num_labels=num_labels, device=torch.device('cpu'))
77 | print('Loss:', loss)
78 | loss.backward()
79 | ```
80 |
81 |
82 |
83 |
84 | Positive Labels: tensor([[21],
85 | [71],
86 | [26]])
87 | Negative Labels: tensor([[47, 10],
88 | [56, 78],
89 | [44, 55]])
90 | Model positive scores: tensor([[-4.9562],
91 | [-1.6886],
92 | [ 3.3984]], grad_fn=)
93 | Model negative scores: tensor([[ 1.0491, 4.9357],
94 | [-2.1289, 0.4496],
95 | [ 3.4541, 0.0931]], grad_fn=)
96 | Loss: tensor(39.6134, grad_fn=)
97 |
98 |
99 |
100 |
101 |
103 |
104 |
105 | ```
106 | print('We can also see that the gradient is only active for 2x the number of positive labels:', (model.emb.weight.grad.sum(1) != 0).sum().item())
107 | print('Meaning we correctly discard the gradients for all other than the offending negative label.')
108 | ```
109 |
110 |
111 |
112 |
113 | We can also see that the gradient is only active for 2x the number of positive labels: 6
114 | Meaning we correctly discard the gradients for all other than the offending negative label.
115 |
116 |
117 |
118 |
119 |
=0}var self=this;self.matches=function(string,crit){return"string"!=typeof string?!1:(string=string.trim(),doMatch(string,crit))}}module.exports=new LiteralSearchStrategy},{}],4:[function(require,module){module.exports=function(){function findMatches(store,crit,strategy){for(var data=store.get(),i=0;i{title} ',noResultsText:"No results found",limit:10,fuzzy:!1};self.init=function(_opt){validateOptions(_opt),assignOptions(_opt),isJSON(opt.dataSource)?initWithJSON(opt.dataSource):initWithURL(opt.dataSource)}}var Searcher=require("./Searcher"),Templater=require("./Templater"),Store=require("./Store"),JSONLoader=require("./JSONLoader"),searcher=new Searcher,templater=new Templater,store=new Store,jsonLoader=new JSONLoader;window.SimpleJekyllSearch=new SimpleJekyllSearch}(window,document)},{"./JSONLoader":1,"./Searcher":4,"./Store":5,"./Templater":6}]},{},[7]);
2 |
--------------------------------------------------------------------------------
/warp_loss.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {},
7 | "outputs": [],
8 | "source": [
9 | "# default_exp losses"
10 | ]
11 | },
12 | {
13 | "cell_type": "markdown",
14 | "metadata": {},
15 | "source": [
16 | "# Losses\n",
17 | "\n",
18 | "> Where all the losses are situated"
19 | ]
20 | },
21 | {
22 | "cell_type": "code",
23 | "execution_count": null,
24 | "metadata": {},
25 | "outputs": [],
26 | "source": [
27 | "#hide\n",
28 | "from nbdev.showdoc import *"
29 | ]
30 | },
31 | {
32 | "cell_type": "code",
33 | "execution_count": null,
34 | "metadata": {},
35 | "outputs": [],
36 | "source": [
37 | "#export\n",
38 | "import torch\n",
39 | "def num_tries_gt_zero(scores, batch_size, max_trials, max_num, device):\n",
40 | " '''\n",
41 | " scores: [batch_size x N] float scores\n",
42 | " returns: [batch_size x 1] the lowest indice per row where scores were first greater than 0. plus 1\n",
43 | " '''\n",
44 | " tmp = scores.gt(0).nonzero().t()\n",
45 | " # We offset these values by 1 to look for unset values (zeros) later\n",
46 | " values = tmp[1] + 1\n",
47 | " # TODO just allocate normal zero-tensor and fill it?\n",
48 | " # Sparse tensors can't be moved with .to() or .cuda() if you want to send in cuda variables first\n",
49 | " if device.type == 'cuda':\n",
50 | " t = torch.cuda.sparse.LongTensor(tmp, values, torch.Size((batch_size, max_trials+1))).to_dense()\n",
51 | " else:\n",
52 | " t = torch.sparse.LongTensor(tmp, values, torch.Size((batch_size, max_trials+1))).to_dense()\n",
53 | " t[(t == 0)] += max_num # set all unused indices to be max possible number so its not picked by min() call\n",
54 | "\n",
55 | " tries = torch.min(t, dim=1)[0]\n",
56 | " return tries\n",
57 | "\n",
58 | "\n",
59 | "def warp_loss(positive_predictions, negative_predictions, num_labels, device):\n",
60 | " '''\n",
61 | " positive_predictions: [batch_size x 1] floats between -1 to 1\n",
62 | " negative_predictions: [batch_size x N] floats between -1 to 1\n",
63 | " num_labels: int total number of labels in dataset (not just the subset you're using for the batch)\n",
64 | " device: pytorch.device\n",
65 | " '''\n",
66 | " batch_size, max_trials = negative_predictions.size(0), negative_predictions.size(1)\n",
67 | "\n",
68 | " offsets, ones, max_num = (torch.arange(0, batch_size, 1).long().to(device) * (max_trials + 1),\n",
69 | " torch.ones(batch_size, 1).float().to(device),\n",
70 | " batch_size * (max_trials + 1) )\n",
71 | "\n",
72 | " sample_scores = (1 + negative_predictions - positive_predictions)\n",
73 | " # Add column of ones so we know when we used all our attempts, This is used for indexing and computing should_count_loss if no real value is above 0\n",
74 | " sample_scores, negative_predictions = (torch.cat([sample_scores, ones], dim=1),\n",
75 | " torch.cat([negative_predictions, ones], dim=1))\n",
76 | "\n",
77 | " tries = num_tries_gt_zero(sample_scores, batch_size, max_trials, max_num, device)\n",
78 | " attempts, trial_offset = tries.float(), (tries - 1) + offsets\n",
79 | " loss_weights, should_count_loss = ( torch.log(torch.floor((num_labels - 1) / attempts)),\n",
80 | " (attempts <= max_trials).float()) #Don't count loss if we used max number of attempts\n",
81 | "\n",
82 | " losses = loss_weights * ((1 - positive_predictions.view(-1)) + negative_predictions.view(-1)[trial_offset]) * should_count_loss\n",
83 | "\n",
84 | " return losses.sum()\n"
85 | ]
86 | },
87 | {
88 | "cell_type": "code",
89 | "execution_count": null,
90 | "metadata": {},
91 | "outputs": [],
92 | "source": [
93 | "#hide\n",
94 | "import numpy as np\n",
95 | "\n",
96 | "\n",
97 | "cpu_device = torch.device('cpu')\n",
98 | "max_value = num_labels = 10\n",
99 | "\n",
100 | "## warp-loss tests\n",
101 | "def ground_truth(pos_val, neg_val, num_attempts=1, num_labels=10):\n",
102 | " num_labels -= 1\n",
103 | " loss_weight = np.log(np.floor(num_labels / float(num_attempts)))\n",
104 | "\n",
105 | " return loss_weight * ((1-pos_val) + neg_val)\n",
106 | "\n",
107 | "comp_pos = torch.FloatTensor([[0.1], [0.9], [1]])\n",
108 | "comp_neg = torch.FloatTensor([[-1, 0.3, 0.5], [-1, -1, 0.3], [0.3, 0.5, -1]])\n",
109 | "comp_scores = (1 + comp_neg) - comp_pos\n",
110 | "\n",
111 | "def test_num_tries():\n",
112 | " simple = torch.FloatTensor([[0.5, -0.5], [-0.5, 0.5]])\n",
113 | " res = num_tries_gt_zero(simple, 2, 2, max_value, cpu_device)\n",
114 | " ans = torch.LongTensor([1, 2])\n",
115 | " for i, v in enumerate(res.long()):\n",
116 | " assert v == ans[i]\n",
117 | "\n",
118 | " res = num_tries_gt_zero(comp_scores, 3, 3, max_value, cpu_device)\n",
119 | " ans = torch.LongTensor([2, 3, 1])\n",
120 | " for i, v in enumerate(res.long()):\n",
121 | " assert v == ans[i]\n",
122 | "\n",
123 | "\n",
124 | "def test_ground_truth():\n",
125 | " # these variables should always trigger on first index\n",
126 | " pos = torch.rand(2, 1).to(cpu_device)\n",
127 | " neg = torch.rand(2, 3).to(cpu_device)\n",
128 | " res = warp_loss(pos.view(-1, 1), neg, num_labels, cpu_device)\n",
129 | "\n",
130 | " assert res == ground_truth(pos[0], neg[0][0], num_labels=num_labels) + ground_truth(pos[1], neg[1][0], num_labels=num_labels)\n",
131 | "\n",
132 | " res = warp_loss(comp_pos.view(-1, 1), comp_neg, num_labels, cpu_device)\n",
133 | "\n",
134 | " gt = np.sum(np.array([ground_truth(comp_pos[i], comp_neg[i][idx-1], num_attempts=idx, num_labels=num_labels)\n",
135 | " for i, idx in enumerate(num_tries_gt_zero(comp_scores, 3, 3, max_value, cpu_device))]))\n",
136 | "\n",
137 | " assert np.allclose(res.data.numpy(), gt)\n",
138 | "\n",
139 | "def test_no_offending_scores():\n",
140 | " pos = torch.FloatTensor([1, 1])\n",
141 | " neg = torch.FloatTensor([[-1, -1, -1],[-1, -1, -1]])\n",
142 | " res = warp_loss(pos.view(-1, 1), neg, num_labels, cpu_device)\n",
143 | " \n",
144 | " assert res == 0\n",
145 | "\n",
146 | "test_num_tries()\n",
147 | "test_ground_truth()\n",
148 | "test_no_offending_scores()"
149 | ]
150 | },
151 | {
152 | "cell_type": "code",
153 | "execution_count": null,
154 | "metadata": {},
155 | "outputs": [],
156 | "source": []
157 | }
158 | ],
159 | "metadata": {
160 | "kernelspec": {
161 | "display_name": "Python 3",
162 | "language": "python",
163 | "name": "python3"
164 | }
165 | },
166 | "nbformat": 4,
167 | "nbformat_minor": 2
168 | }
169 |
--------------------------------------------------------------------------------
/index.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {},
7 | "outputs": [],
8 | "source": [
9 | "#hide\n",
10 | "from warp_loss.losses import warp_loss"
11 | ]
12 | },
13 | {
14 | "cell_type": "markdown",
15 | "metadata": {},
16 | "source": [
17 | "# WARP-Pytorch\n",
18 | "\n",
19 | "> An implementation of WARP loss which uses matrixes and stays on the GPU in PyTorch."
20 | ]
21 | },
22 | {
23 | "cell_type": "markdown",
24 | "metadata": {},
25 | "source": [
26 | "This means instead of using a for-loop to find the first offending negative sample that ranks above our positive,\n",
27 | "we compute all of them at once. Only later do we find which sample is the first offender, and compute the loss with\n",
28 | "respect to this sample.\n",
29 | "\n",
30 | "The advantage is that it can use the speedups that comes with GPU-usage. \n",
31 | "\n",
32 | "## When is WARP loss advantageous?\n",
33 | "If you're ranking items or making models for recommendations, it's often advantageous to let your loss function directly\n",
34 | "optimize for this case. WARP loss looks at 1 explicit positive up against the implicit negative items that a user never sampled,\n",
35 | "and allows us to adjust weights of the network accordingly.\n"
36 | ]
37 | },
38 | {
39 | "cell_type": "markdown",
40 | "metadata": {},
41 | "source": [
42 | "## Install"
43 | ]
44 | },
45 | {
46 | "cell_type": "markdown",
47 | "metadata": {},
48 | "source": [
49 | "`pip install warp_loss`"
50 | ]
51 | },
52 | {
53 | "cell_type": "markdown",
54 | "metadata": {},
55 | "source": [
56 | "## How to use"
57 | ]
58 | },
59 | {
60 | "cell_type": "markdown",
61 | "metadata": {},
62 | "source": [
63 | "The loss function requires scores for both positive examples, and negative examples to be supplied, such as in the example below."
64 | ]
65 | },
66 | {
67 | "cell_type": "code",
68 | "execution_count": null,
69 | "metadata": {},
70 | "outputs": [],
71 | "source": [
72 | "from torch import nn\n",
73 | "import torch\n",
74 | "\n",
75 | "class OurModel(nn.Module):\n",
76 | " def __init__(self, num_labels, emb_dim=10):\n",
77 | " super(OurModel, self).__init__()\n",
78 | " self.emb = nn.Embedding(num_labels, emb_dim)\n",
79 | " self.user_embs = nn.Embedding(1, emb_dim)\n",
80 | "\n",
81 | " def forward(self, pos, neg):\n",
82 | " batch_size = neg.size(0)\n",
83 | " one_user_vector = self.user_embs(torch.zeros(1).long())\n",
84 | " repeated_user_vector = one_user_vector.repeat((batch_size, 1)).view(batch_size, -1, 1)\n",
85 | " pos_res = torch.bmm(self.emb(pos), repeated_user_vector).squeeze(2)\n",
86 | " neg_res = torch.bmm(self.emb(neg), repeated_user_vector).squeeze(2)\n",
87 | "\n",
88 | " return pos_res, neg_res\n",
89 | " \n",
90 | "num_labels = 100\n",
91 | "model = OurModel(num_labels)"
92 | ]
93 | },
94 | {
95 | "cell_type": "code",
96 | "execution_count": null,
97 | "metadata": {},
98 | "outputs": [
99 | {
100 | "name": "stdout",
101 | "output_type": "stream",
102 | "text": "Positive Labels: tensor([[21],\n [71],\n [26]])\nNegative Labels: tensor([[47, 10],\n [56, 78],\n [44, 55]])\nModel positive scores: tensor([[-4.9562],\n [-1.6886],\n [ 3.3984]], grad_fn=)\nModel negative scores: tensor([[ 1.0491, 4.9357],\n [-2.1289, 0.4496],\n [ 3.4541, 0.0931]], grad_fn=)\nLoss: tensor(39.6134, grad_fn=)\n"
103 | }
104 | ],
105 | "source": [
106 | "pos_labels = torch.randint(high=num_labels, size=(3,1)) # our five labels\n",
107 | "neg_labels = torch.randint(high=num_labels, size=(3,2)) # a few random negatives per positive\n",
108 | "\n",
109 | "pos_res, neg_res = model(pos_labels, neg_labels)\n",
110 | "print('Positive Labels:', pos_labels)\n",
111 | "print('Negative Labels:', neg_labels)\n",
112 | "print('Model positive scores:', pos_res)\n",
113 | "print('Model negative scores:', neg_res)\n",
114 | "loss = warp_loss(pos_res, neg_res, num_labels=num_labels, device=torch.device('cpu'))\n",
115 | "print('Loss:', loss)\n",
116 | "loss.backward()"
117 | ]
118 | },
119 | {
120 | "cell_type": "code",
121 | "execution_count": null,
122 | "metadata": {},
123 | "outputs": [
124 | {
125 | "name": "stdout",
126 | "output_type": "stream",
127 | "text": "We can also see that the gradient is only active for 2x the number of positive labels: 6\nMeaning we correctly discard the gradients for all other than the offending negative label.\n"
128 | }
129 | ],
130 | "source": [
131 | "print('We can also see that the gradient is only active for 2x the number of positive labels:', (model.emb.weight.grad.sum(1) != 0).sum().item())\n",
132 | "print('Meaning we correctly discard the gradients for all other than the offending negative label.')"
133 | ]
134 | },
135 | {
136 | "cell_type": "markdown",
137 | "execution_count": null,
138 | "metadata": {},
139 | "outputs": [
140 | {
141 | "ename": "SyntaxError",
142 | "evalue": "invalid syntax (, line 2)",
143 | "output_type": "error",
144 | "traceback": [
145 | "\u001b[0;36m File \u001b[0;32m\"\"\u001b[0;36m, line \u001b[0;32m2\u001b[0m\n\u001b[0;31m The loss function assumes you have already sampled your negatives randomly.\u001b[0m\n\u001b[0m ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
146 | ]
147 | }
148 | ],
149 | "source": [
150 | "## Assumptions\n",
151 | "The loss function assumes you have already sampled your negatives randomly.\n",
152 | "\n",
153 | "As an example this could be done in your dataloader:\n",
154 | "\n",
155 | "1. Assume we have a total dataset of 100 items\n",
156 | "2. Select a positive sample with index 8\n",
157 | "2. Your negatives should be a random selection from 0-100 excluding 8.\n",
158 | "\n",
159 | "Ex input to loss function: model scores for pos: [8] neg: [88, 3, 99, 7]\n",
160 | "\n",
161 | "Should work on all pytorch-versions from 0.4 and up\n",
162 | "\n",
163 | "### References\n",
164 | "* [WSABIE: Scaling Up To Large Vocabulary Image Annotation](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37180.pdf)\n",
165 | "* [Intro to WARP loss - Automatic differentiation and PyTorch](https://medium.com/@gabrieltseng/intro-to-warp-loss-automatic-differentiation-and-pytorch-b6aa5083187a)\n",
166 | "* [LightFM](https://github.com/lyst/lightfm) as a reference implementaiton"
167 | ]
168 | },
169 | {
170 | "cell_type": "code",
171 | "execution_count": null,
172 | "metadata": {},
173 | "outputs": [],
174 | "source": []
175 | }
176 | ],
177 | "metadata": {
178 | "kernelspec": {
179 | "display_name": "Python 3",
180 | "language": "python",
181 | "name": "python3"
182 | }
183 | },
184 | "nbformat": 4,
185 | "nbformat_minor": 2
186 | }
187 |
--------------------------------------------------------------------------------
/docs/Gemfile.lock:
--------------------------------------------------------------------------------
1 | GEM
2 | remote: https://rubygems.org/
3 | specs:
4 | activesupport (4.2.11.1)
5 | i18n (~> 0.7)
6 | minitest (~> 5.1)
7 | thread_safe (~> 0.3, >= 0.3.4)
8 | tzinfo (~> 1.1)
9 | addressable (2.7.0)
10 | public_suffix (>= 2.0.2, < 5.0)
11 | coffee-script (2.4.1)
12 | coffee-script-source
13 | execjs
14 | coffee-script-source (1.11.1)
15 | colorator (1.1.0)
16 | commonmarker (0.17.13)
17 | ruby-enum (~> 0.5)
18 | concurrent-ruby (1.1.5)
19 | dnsruby (1.61.3)
20 | addressable (~> 2.5)
21 | em-websocket (0.5.1)
22 | eventmachine (>= 0.12.9)
23 | http_parser.rb (~> 0.6.0)
24 | ethon (0.12.0)
25 | ffi (>= 1.3.0)
26 | eventmachine (1.2.7)
27 | execjs (2.7.0)
28 | faraday (0.17.0)
29 | multipart-post (>= 1.2, < 3)
30 | ffi (1.11.3)
31 | forwardable-extended (2.6.0)
32 | gemoji (3.0.1)
33 | github-pages (202)
34 | activesupport (= 4.2.11.1)
35 | github-pages-health-check (= 1.16.1)
36 | jekyll (= 3.8.5)
37 | jekyll-avatar (= 0.6.0)
38 | jekyll-coffeescript (= 1.1.1)
39 | jekyll-commonmark-ghpages (= 0.1.6)
40 | jekyll-default-layout (= 0.1.4)
41 | jekyll-feed (= 0.11.0)
42 | jekyll-gist (= 1.5.0)
43 | jekyll-github-metadata (= 2.12.1)
44 | jekyll-mentions (= 1.4.1)
45 | jekyll-optional-front-matter (= 0.3.0)
46 | jekyll-paginate (= 1.1.0)
47 | jekyll-readme-index (= 0.2.0)
48 | jekyll-redirect-from (= 0.14.0)
49 | jekyll-relative-links (= 0.6.0)
50 | jekyll-remote-theme (= 0.4.0)
51 | jekyll-sass-converter (= 1.5.2)
52 | jekyll-seo-tag (= 2.5.0)
53 | jekyll-sitemap (= 1.2.0)
54 | jekyll-swiss (= 0.4.0)
55 | jekyll-theme-architect (= 0.1.1)
56 | jekyll-theme-cayman (= 0.1.1)
57 | jekyll-theme-dinky (= 0.1.1)
58 | jekyll-theme-hacker (= 0.1.1)
59 | jekyll-theme-leap-day (= 0.1.1)
60 | jekyll-theme-merlot (= 0.1.1)
61 | jekyll-theme-midnight (= 0.1.1)
62 | jekyll-theme-minimal (= 0.1.1)
63 | jekyll-theme-modernist (= 0.1.1)
64 | jekyll-theme-primer (= 0.5.3)
65 | jekyll-theme-slate (= 0.1.1)
66 | jekyll-theme-tactile (= 0.1.1)
67 | jekyll-theme-time-machine (= 0.1.1)
68 | jekyll-titles-from-headings (= 0.5.1)
69 | jemoji (= 0.10.2)
70 | kramdown (= 1.17.0)
71 | liquid (= 4.0.0)
72 | listen (= 3.1.5)
73 | mercenary (~> 0.3)
74 | minima (= 2.5.0)
75 | nokogiri (>= 1.10.4, < 2.0)
76 | rouge (= 3.11.0)
77 | terminal-table (~> 1.4)
78 | github-pages-health-check (1.16.1)
79 | addressable (~> 2.3)
80 | dnsruby (~> 1.60)
81 | octokit (~> 4.0)
82 | public_suffix (~> 3.0)
83 | typhoeus (~> 1.3)
84 | html-pipeline (2.12.2)
85 | activesupport (>= 2)
86 | nokogiri (>= 1.4)
87 | http_parser.rb (0.6.0)
88 | i18n (0.9.5)
89 | concurrent-ruby (~> 1.0)
90 | jekyll (3.8.5)
91 | addressable (~> 2.4)
92 | colorator (~> 1.0)
93 | em-websocket (~> 0.5)
94 | i18n (~> 0.7)
95 | jekyll-sass-converter (~> 1.0)
96 | jekyll-watch (~> 2.0)
97 | kramdown (~> 1.14)
98 | liquid (~> 4.0)
99 | mercenary (~> 0.3.3)
100 | pathutil (~> 0.9)
101 | rouge (>= 1.7, < 4)
102 | safe_yaml (~> 1.0)
103 | jekyll-avatar (0.6.0)
104 | jekyll (~> 3.0)
105 | jekyll-coffeescript (1.1.1)
106 | coffee-script (~> 2.2)
107 | coffee-script-source (~> 1.11.1)
108 | jekyll-commonmark (1.3.1)
109 | commonmarker (~> 0.14)
110 | jekyll (>= 3.7, < 5.0)
111 | jekyll-commonmark-ghpages (0.1.6)
112 | commonmarker (~> 0.17.6)
113 | jekyll-commonmark (~> 1.2)
114 | rouge (>= 2.0, < 4.0)
115 | jekyll-default-layout (0.1.4)
116 | jekyll (~> 3.0)
117 | jekyll-feed (0.11.0)
118 | jekyll (~> 3.3)
119 | jekyll-gist (1.5.0)
120 | octokit (~> 4.2)
121 | jekyll-github-metadata (2.12.1)
122 | jekyll (~> 3.4)
123 | octokit (~> 4.0, != 4.4.0)
124 | jekyll-mentions (1.4.1)
125 | html-pipeline (~> 2.3)
126 | jekyll (~> 3.0)
127 | jekyll-optional-front-matter (0.3.0)
128 | jekyll (~> 3.0)
129 | jekyll-paginate (1.1.0)
130 | jekyll-readme-index (0.2.0)
131 | jekyll (~> 3.0)
132 | jekyll-redirect-from (0.14.0)
133 | jekyll (~> 3.3)
134 | jekyll-relative-links (0.6.0)
135 | jekyll (~> 3.3)
136 | jekyll-remote-theme (0.4.0)
137 | addressable (~> 2.0)
138 | jekyll (~> 3.5)
139 | rubyzip (>= 1.2.1, < 3.0)
140 | jekyll-sass-converter (1.5.2)
141 | sass (~> 3.4)
142 | jekyll-seo-tag (2.5.0)
143 | jekyll (~> 3.3)
144 | jekyll-sitemap (1.2.0)
145 | jekyll (~> 3.3)
146 | jekyll-swiss (0.4.0)
147 | jekyll-theme-architect (0.1.1)
148 | jekyll (~> 3.5)
149 | jekyll-seo-tag (~> 2.0)
150 | jekyll-theme-cayman (0.1.1)
151 | jekyll (~> 3.5)
152 | jekyll-seo-tag (~> 2.0)
153 | jekyll-theme-dinky (0.1.1)
154 | jekyll (~> 3.5)
155 | jekyll-seo-tag (~> 2.0)
156 | jekyll-theme-hacker (0.1.1)
157 | jekyll (~> 3.5)
158 | jekyll-seo-tag (~> 2.0)
159 | jekyll-theme-leap-day (0.1.1)
160 | jekyll (~> 3.5)
161 | jekyll-seo-tag (~> 2.0)
162 | jekyll-theme-merlot (0.1.1)
163 | jekyll (~> 3.5)
164 | jekyll-seo-tag (~> 2.0)
165 | jekyll-theme-midnight (0.1.1)
166 | jekyll (~> 3.5)
167 | jekyll-seo-tag (~> 2.0)
168 | jekyll-theme-minimal (0.1.1)
169 | jekyll (~> 3.5)
170 | jekyll-seo-tag (~> 2.0)
171 | jekyll-theme-modernist (0.1.1)
172 | jekyll (~> 3.5)
173 | jekyll-seo-tag (~> 2.0)
174 | jekyll-theme-primer (0.5.3)
175 | jekyll (~> 3.5)
176 | jekyll-github-metadata (~> 2.9)
177 | jekyll-seo-tag (~> 2.0)
178 | jekyll-theme-slate (0.1.1)
179 | jekyll (~> 3.5)
180 | jekyll-seo-tag (~> 2.0)
181 | jekyll-theme-tactile (0.1.1)
182 | jekyll (~> 3.5)
183 | jekyll-seo-tag (~> 2.0)
184 | jekyll-theme-time-machine (0.1.1)
185 | jekyll (~> 3.5)
186 | jekyll-seo-tag (~> 2.0)
187 | jekyll-titles-from-headings (0.5.1)
188 | jekyll (~> 3.3)
189 | jekyll-watch (2.2.1)
190 | listen (~> 3.0)
191 | jemoji (0.10.2)
192 | gemoji (~> 3.0)
193 | html-pipeline (~> 2.2)
194 | jekyll (~> 3.0)
195 | kramdown (1.17.0)
196 | liquid (4.0.0)
197 | listen (3.1.5)
198 | rb-fsevent (~> 0.9, >= 0.9.4)
199 | rb-inotify (~> 0.9, >= 0.9.7)
200 | ruby_dep (~> 1.2)
201 | mercenary (0.3.6)
202 | mini_portile2 (2.4.0)
203 | minima (2.5.0)
204 | jekyll (~> 3.5)
205 | jekyll-feed (~> 0.9)
206 | jekyll-seo-tag (~> 2.1)
207 | minitest (5.13.0)
208 | multipart-post (2.1.1)
209 | nokogiri (1.10.8)
210 | mini_portile2 (~> 2.4.0)
211 | octokit (4.14.0)
212 | sawyer (~> 0.8.0, >= 0.5.3)
213 | pathutil (0.16.2)
214 | forwardable-extended (~> 2.6)
215 | public_suffix (3.1.1)
216 | rb-fsevent (0.10.3)
217 | rb-inotify (0.10.0)
218 | ffi (~> 1.0)
219 | rouge (3.11.0)
220 | ruby-enum (0.7.2)
221 | i18n
222 | ruby_dep (1.5.0)
223 | rubyzip (2.0.0)
224 | safe_yaml (1.0.5)
225 | sass (3.7.4)
226 | sass-listen (~> 4.0.0)
227 | sass-listen (4.0.0)
228 | rb-fsevent (~> 0.9, >= 0.9.4)
229 | rb-inotify (~> 0.9, >= 0.9.7)
230 | sawyer (0.8.2)
231 | addressable (>= 2.3.5)
232 | faraday (> 0.8, < 2.0)
233 | terminal-table (1.8.0)
234 | unicode-display_width (~> 1.1, >= 1.1.1)
235 | thread_safe (0.3.6)
236 | typhoeus (1.3.1)
237 | ethon (>= 0.9.0)
238 | tzinfo (1.2.5)
239 | thread_safe (~> 0.1)
240 | unicode-display_width (1.6.0)
241 |
242 | PLATFORMS
243 | ruby
244 |
245 | DEPENDENCIES
246 | github-pages
247 | jekyll (~> 3.7)
248 |
249 | BUNDLED WITH
250 | 2.0.2
251 |
--------------------------------------------------------------------------------
/docs/index.html:
--------------------------------------------------------------------------------
1 | ---
2 |
3 | title: WARP-Pytorch
4 |
5 | keywords: fastai
6 | sidebar: home_sidebar
7 |
8 | summary: "An implementation of WARP loss which uses matrixes and stays on the GPU in PyTorch."
9 | ---
10 |
19 |
20 |
21 |
22 |
23 |
24 |
25 |
26 |
27 |
This means instead of using a for-loop to find the first offending negative sample that ranks above our positive,
28 | we compute all of them at once. Only later do we find which sample is the first offender, and compute the loss with
29 | respect to this sample.
30 |
The advantage is that it can use the speedups that comes with GPU-usage.
31 |
When is WARP loss advantageous?¶ If you're ranking items or making models for recommendations, it's often advantageous to let your loss function directly
32 | optimize for this case. WARP loss looks at 1 explicit positive up against the implicit negative items that a user never sampled,
33 | and allows us to adjust weights of the network accordingly.
34 |
35 |
36 |
37 |
38 |
44 |
45 |
46 |
pip install warp_loss
47 |
48 |
49 |
50 |
51 |
57 |
58 |
59 |
The loss function requires scores for both positive examples, and negative examples to be supplied, such as in the example below.
60 |
61 |
62 |
63 |
64 |
96 |
97 |
117 |
118 |
119 |
120 |
121 |
122 |
123 |
124 |
Positive Labels: tensor([[21],
125 | [71],
126 | [26]])
127 | Negative Labels: tensor([[47, 10],
128 | [56, 78],
129 | [44, 55]])
130 | Model positive scores: tensor([[-4.9562],
131 | [-1.6886],
132 | [ 3.3984]], grad_fn=<SqueezeBackward1>)
133 | Model negative scores: tensor([[ 1.0491, 4.9357],
134 | [-2.1289, 0.4496],
135 | [ 3.4541, 0.0931]], grad_fn=<SqueezeBackward1>)
136 | Loss: tensor(39.6134, grad_fn=<SumBackward0>)
137 |
138 |
139 |
140 |
141 |
142 |
143 |
144 |
145 |
146 |
157 |
158 |
159 |
160 |
161 |
162 |
163 |
164 |
We can also see that the gradient is only active for 2x the number of positive labels: 6
165 | Meaning we correctly discard the gradients for all other than the offending negative label.
166 |
167 |
168 |
169 |
170 |
171 |
172 |
173 |
174 |
175 |
176 |
Assumptions¶ The loss function assumes you have already sampled your negatives randomly.
177 |
As an example this could be done in your dataloader:
178 |
179 | Assume we have a total dataset of 100 items
180 | Select a positive sample with index 8
181 | Your negatives should be a random selection from 0-100 excluding 8.
182 |
183 |
Ex input to loss function: model scores for pos: [8] neg: [88, 3, 99, 7]
184 |
Should work on all pytorch-versions from 0.4 and up
185 |
References¶
190 |
191 |
192 |
193 |
194 |