20 | -------------------------------------------------------------------------------- /general-tips.md: -------------------------------------------------------------------------------- 1 | # General Tips 2 | 3 | + **Be a pleasant, friendly person.** 4 | Try to be a pleasant, friendly person to everyone around you, even people you may not know. 5 | For example, greet the cashier who's ringing up your coffee in the morning. 6 | Say "have a nice day!" when you leave. Here's a fun one: 7 | 8 |Every talk at #emnlp2019:
— Vered Shwartz (@VeredShwartz) November 6, 2019
Chair: The next speaker is X from Y university who will speak about Z.
Speaker: Hi, I'm X from Y university, I will be talking about Z.
9 | 10 | + **Treat admin and support staff with respect.** 11 | This one has always baffled me, as I frequently hear about colleagues (and sometimes even students) mistreating staff. 12 | First, it's just not _right_ (see point above). 13 | Second, even from a purely selfish, Machiavellian perspective, that's not even a wise move. 14 | As a faculty member, admin and support staff manage my accounts, handle my reimbursements, help with personnel matters (e.g., appointment paperwork), and generally keep me out of trouble ("No, Jimmy, you can't spend money on that from this account"). 15 | My life would be miserable if I had to do all of that myself. 16 | Be nice to admin and support staff! 17 | 18 | + **Don't do over email what a simple in-person visit could handle more efficiently.** 19 | There's really a reason why offices, labs, and campuses still exist — there are many interactions that still need to happen face to face. 20 | I've seen plenty of long email threads that could have been avoided if the person making the request had just went down the hall for a quick chat. 21 | Ask yourself, "Would this be faster if I just went by Alice's office?" 22 | -------------------------------------------------------------------------------- /thesis-tips.md: -------------------------------------------------------------------------------- 1 | # Thesis Tips 2 | 3 | + **Be more discursive.** 4 | Interestingly, [Merriam-Webster](https://www.merriam-webster.com/dictionary/discursive) defines discursive as either (1) rambling or (2) proceeding coherently from topic to topic, an interesting case of a word in English [also meaning its exact opposite](https://en.wikipedia.org/wiki/Auto-antonym). 5 | But I'm being discursive (in the first sense). 6 | I often find that students are too _terse_ in writing their theses. 7 | I think this stems from the fact that they've been buried in a problem for months (or even years in the case of a Ph.D.) — many aspects of the problem are "obvious". 8 | But this isn't the case for the reader, who may be encountering the problem for the first time. 9 | In my experience, this happens most frequently in the introduction, when explaining the setup of the problem, what the research is trying to accomplish, etc. 10 | Remember, theses don't have page limits! 11 | Walk the reader through your arguments; explicitly spell things out. 12 | 13 | 14 | ## My Thesis Reading Policy 15 | 16 | As a general rule, I will do a _detailed_ read of your thesis _once_. 17 | By detailed read, I mean with a (virtual) pen in hand, marking up your draft. 18 | _You_ tell me when you want me to perform this pass: I can do chapter by chapter, entire thesis at once, or something in between (allowing for a reasonable turnaround time, of course). 19 | The factors to consider are that if you ask too early, the draft might still be very rough — with issues that you could have caught yourself — thus making it a poor use of my time. 20 | If you ask too late, you draft might be too set to make major changes based on my feedback. 21 | 22 | I am, however, happy to skim parts fo your thesis multiple times if you just want a quick sanity check. 23 | By skim, I mean I'll flip through the pages, look at the figures, read the section headers, etc. 24 | Don't expect me to catch typos. 25 | 26 | The only exception to my rule is for the abstract and certain parts of the introduction (e.g., contribution statement). 27 | I will go over, in detail, those parts multiple times with you to get it right. 28 | -------------------------------------------------------------------------------- /general-research-tips.md: -------------------------------------------------------------------------------- 1 | # General Research Tips 2 | 3 | ## Keep a Research Notebook 4 | 5 | TODO: Write something about this. 6 | 7 | ## Change Only One Thing at a Time 8 | 9 | TODO: Write something about this. 10 | 11 | ## Script Your Experiments 12 | 13 | TODO: Write something about this. 14 | 15 | ## Write Early 16 | 17 | Many [1] of my students like to write code more than they like to write latex (i.e., the paper itself) [2]. 18 | Obviously, we desire _everything_: novel contributions, broad claims that are empirically supported in a convincing manner, insightful analyses ‐ and everything well explained in fluent, typo-free prose. 19 | However, as deadlines come close it is necessary to triage. 20 | Otherwise, we end up with [something like this](https://www.reddit.com/r/funny/comments/85p3x2/when_there_is_5_minutes_left_on_the_test/), as [someone has conveyed on Twitter](https://twitter.com/nijfranck/status/977277874460938242). 21 | 22 | My advice: 23 | 24 | + Well-written good results are better than poorly-written amazing results. Stop working on yet another model or technique! 25 | + Strong empirical support for a narrow (perhaps less interesting or novel) contribution is better than unconvincing support for a broad (or interesting, insight, surprising, etc.) contribution. 26 | 27 | Write early! 28 | 29 | In fact, I suggest this heuristic, even long before any paper deadline: ask your self, "Will this be part of the final paper?" 30 | If so, then it lies on the critical path, so you'll have to write it up sooner or later. 31 | Sooner is usually better. 32 | 33 | The corollary is that you should write up baseline results as soon as they are complete. 34 | By "write up" I mean: describe the baseline approach, including data, experimental setup, and results. 35 | At this point, you probably already have an idea of where a paper might go, so you can use, for example, either the \*ACL template or the ACM template. 36 | If you've designed your experiments correctly, this prose will comprise the core of your paper, and your proposed contributions can be captured by additional rows in your results table. 37 | This is also consistent with [my suggested writing order](writing-a-research-paper-with-me.md#writing-order) of a paper. 38 | 39 | A few more advantages of this approach: 40 | Forcing yourself to accurately describe your baseline often pokes holes in your thinking. 41 | Writing is something that you can do while you're waiting for the neural networks to train! 42 | 43 | [1] Most? All? 44 | 45 | [2] Technically, latex _is_ also code. 46 | -------------------------------------------------------------------------------- /what-my-personal-schedule-is-like.md: -------------------------------------------------------------------------------- 1 | # What my personal schedule is like 2 | 3 | + On a weekday, I typically wake up some time between 6am and 7am. 4 | My kids typically wake up at around 8am, so I have a window where I can focus on some task (e.g., paper writing); I am usually quite productive during this time. 5 | Starting around 8am, I help get my kids ready for school, which starts at 9:10am. 6 | In between, I can usually squeeze in quick emails or Slack conversations, but typically not anything that requires concentration. 7 | 8 | + If I'm in lab that day, I'll arrive around 9:30am. 9 | 10 | + I typically leave lab no later than around 4:00pm to pick up kids from school. 11 | 12 | + From when I pick up kids to when the kids go to bed (around 8:30pm) is family time. 13 | I might squeeze in light email and Slack conversations, e.g., when I'm sitting in the bleachers watching the kids' swim lessons, but I don't do anything that requires concentration. 14 | Unless absolutely necessary, I refuse synchronous calls of any type (this is consequential for collaborators on the US west coast, because it basically means that I can't do anything after around 1pm or so west coast time). 15 | 16 | + I'm fully engaged with work again after the kids go to bed, starting around 9pm. 17 | If you'd like to have a synchronous conference call, this time works for me. 18 | I typically work for a bit longer, depending on the urgency of various deadlines. 19 | For example, if there are paper deadlines, I might put in another two solid hours of writing. 20 | If there's nothing particularly urgent, I might do a bit of email and then call it a night. 21 | I'll unwind a bit, and I usually go to bed around 11pm. 22 | 23 | + **I try to get 7–8 hours of sleep every night.** Closer to the 8 hour mark, actually. 24 | 25 | + Family gets priority on weekends. 26 | I'll usually engage in email and Slack conversations. 27 | I'll work when it doesn't interfere with family—for example, if kids are at an activity or are napping. 28 | Typically, I estimate that I can squeeze in five hours of solid, concentrated work. 29 | 30 | + I estimate that I work around 40–50 hours a week. 31 | This doesn't include the little crevices in time when I am able to multitask effectively. 32 | For example, I don't know how to count the time I'm slacking with a student while watching my kids' skating lessons. 33 | 34 | + I break up my time into [maker's schedule, manager's schedule](http://www.paulgraham.com/makersschedule.html) based on the advice of Paul Graham. 35 | I try to pack meetings into consecutive blocks to create large blocks of time for work that requires concentration (e.g., writing papers and coding). 36 | The practical consequence of this is that if you have a scheduled meeting with me, I'm likely to be running from one meeting and going to another right after. 37 | -------------------------------------------------------------------------------- /two-things.md: -------------------------------------------------------------------------------- 1 | # Two Things That Every Graduate Students Needs 2 | 3 | ## A Research Notebook 4 | 5 | > Most scientists keep a research notebook. You should too. You've probably been told this in every science class since fifth grade, but it's true. Different systems work for different people; experiment. 6 | > [[Chapman, 1988](https://dspace.mit.edu/bitstream/handle/1721.1/41487/AI_WP_316.pdf#page=13)] 7 | 8 | ### What is it? 9 | 10 | ### How do I do it? 11 | 12 | What I do: 13 | I keep my research notebook in a private GitHub. 14 | The README.md contains entries in reverse chronological order (latest entries on top). 15 | 16 | At the very top of the notebook, I create a cheat sheet for myself containing all those mysterious command-line invocations that are necessary for setting up my environment. 17 | For example, those cryptic conda commands I can never remember. 18 | 19 | So, I open up README.md in emacs, copy and paste into the shell, and I'm ready to go. 20 | As I'm working, I'm constantly copying things back and forth from my notebook into the shell and vice versa. 21 | I rarely type a complex command (e.g., with many parameters) directly into the shell: typically, I compose the command in emacs, _then_ copy into the shell. 22 | I do this because I'm usually running the same command over and over again with slight variations, e.g., different input files, different output locations, different parameters, etc. 23 | Modifying a long and complex command in emacs is far easier than trying to do it in the shell... and plus, I want to record my commands anyway, so starting in emacs is just easier. 24 | I copy the output of commands (e.g., execution trace, evaluation results, etc.) back into emacs. 25 | 26 | Usually, I'm concurrently writing new code, so I'm tabbing back and forth between emacs and my IDE (and Slack, and Teams, and email, and my calendar, and all the other things that are constantly calling for my attention). 27 | If I encounter a bug (e.g., some exception is thrown) or some really weird result (e.g., MAP of 0.003? There's got to be _something_ obviously wrong), I record the error trace also. 28 | This is important because if (or rather, _when_) I get interrupted, this helps me remember what I was doing, i.e., "oh, I remember, I was in the middle of debugging this exception". 29 | These intermediate traces get cleaned up later (see below). 30 | 31 | At the end of a "working session", which might be a few hours later (if I'm lucky enough to have been working continuously) or might be a few days later (when I get interrupted... which happens _a lot_), I go back and clean up the entry. 32 | 33 | By "cleaning up", I reorganize the entry into something that's actually readable, instead of a random jumble of commands and shell output. 34 | For example, I remove all the intermediate error traces and debug output since by now the bug should have been fixed. 35 | I add some prose around that describes what I was doing. 36 | 37 | The final result is a sample entry that looks this this: 38 | 39 |Canadian politeness never ceases to amaze (just witnessed): man drops object, other man apologies to the first in calling him back; man who dropped object apologies (for dropping the object?) and then says thank you.
— Jimmy Lin (@lintool) May 25, 2019
40 |
41 | Associated files (detailed results that are too verbose to report in the main entry, config files, etc.) are stored in directories by date, e.g., `YYYY-MM-DD`.
42 |
43 | Other possible systems:
44 |
45 | + Google Docs
46 | + [Notion](https://notion.so/)
47 | + Microsoft OneNote
48 |
49 | ## A Literature "Database"
50 |
51 | You'll be reading a lot of papers.
52 | You'll need some system to keep track of them.
53 |
--------------------------------------------------------------------------------
/industry-one-way-street.md:
--------------------------------------------------------------------------------
1 | # Working in industry is a one-way street out of academia
2 |
3 | Standard advice I give to graduating Ph.D. students:
4 | If you're absolutely sure that you don't want an academic position (i.e., become a faculty), then it's fine to go to industry.
5 | If you are _not_ sure, pursue a postdoc (which could be in industry), because a permanent position in industry is a one-way street out of academia.
6 | A faculty member can always get an industry position, but once you're in industry for a while, it's very difficult to get an academic position.
7 |
8 | I've heard countless times a student say something along the lines of:
9 | Oh, I'll just work a few years in industry, see how I like it, and then come back to pursue an academic position.
10 | This is highly unlikely.
11 |
12 | It comes down to publications:
13 | To qualify for a good academic position, you can't stop publishing.
14 | Let's say you work in industry for two years and want to apply for an academic position:
15 | Your competition consists of postdocs who have spent the last two years with a singular focus on pumping out publications to get the best possible academic position.
16 | Who do you think is going to "win"?
17 |
18 | But then I hear the student say:
19 | Oh, this position I've been offered, it's a research scientist position.
20 | The company encourages publications.
21 |
22 | Oh really, I respond:
23 | Who on your future team has published in the last year?
24 | This is a standard recruiting pitch:
25 | We're really changing the culture around here, to open up and encourage publications.
26 | I'm sure they've used the pitch on plenty of other job candidates before, and certainly for those who already joined the team.
27 | I ask again:
28 | What's the median per researcher publication output over the last few years?
29 |
30 | The truth is, for most organizations that recruit and hire Ph.D. graduates, the best case scenario is that they will not _prohibit_ you from publishing.
31 | That is, sure you can work on papers, but that's on your own free time (i.e., nights and weekends).
32 | The key question to ask is: at your performance review with your boss, are publications going to be a metric that factors into your promotion or salary increase?
33 | In other words, are there any parts of the incentive structure tied to publications?
34 |
35 | As a generality, going to industry is a one way street.
36 | There are, of course, certain exceptions: Microsoft Research and certain parts of Google are the two notable ones.
37 | For example, Microsoft Research specifically hires postdocs: fixed-term appointments with a focus on publications that provide natural stepping stones to faculty positions.
38 | I am highly skeptical of companies that are "opening up" or want to "change their publication culture".
39 | Culture is very difficult to change.
40 |
41 | I've had many conversations with students that are basically variations of the above sequence.
42 | Typically, they end with something like:
43 | I'm different, I'll be an exception.
44 | And I say, good luck, really meaning it.
45 | To be an exception requires, I think, incredible inner drive to continue publishing, because the external motivators (e.g., publishing in order to graduate) are no longer there.
46 | How to maintain this "fire" and "hunger" in the presence of pressure to ship products, organizational politics, as well as competing demands outside of work (e.g., family) is challenging.
47 | This is why I say working in industry is a one-way street out of academia.
48 |
49 | (Update, 2023/09/16): [Russ Salakhutdinov](https://en.wikipedia.org/wiki/Russ_Salakhutdinov) basically says the same thing... "Going into industry is usually a one-way street".
50 |
51 | 52 | -------------------------------------------------------------------------------- /us-anti-plugs.md: -------------------------------------------------------------------------------- 1 | # US Anti-Plugs 2 | 3 | Whenever I come across a media mention of how the US is screwing up immigration, tech, etc., I add it to this list. 4 | This is the counterpart to the [Waterloo plugs](waterloo-plugs.md) page. 5 | 6 | Being an American, it pains me to add entries to this page... 7 | 8 | --- 9 | 10 | July 13, 2021 - [Woe, Canada: U.S. lawmakers lament skilled immigrants moving north](https://www.cbc.ca/news/world/us-immigration-canada-1.6101735): With reform efforts stalled, U.S. politicians look at Canadian policies 11 | 12 | September 10, 2020 - [Covid-19 Interrupts Flow of Foreign Students to U.S.](https://www.bloomberg.com/amp/opinion/articles/2020-09-10/covid-19-interrupts-flow-of-foreign-students-to-u-s-colleges): The question for colleges and universities is: Will they come back when the pandemic is over? 13 | 14 | August 6, 2020 - [The Trump administration’s no-blanks policy is the latest Kafkaesque plan designed to curb immigration](https://www.washingtonpost.com/opinions/the-trump-administration-imposes-yet-another-arbitrary-absurd-modification-to-the-immigration-system/2020/08/06/42de75ca-d811-11ea-930e-d88518c57dcc_story.html) 15 | 16 | May 10, 2020 - [Donald Trump’s rhetoric is steadily chasing Indians out of the US—towards Canada](https://qz.com/india/1866355/indians-fed-up-of-trumps-h-1b-opt-green-card-moves-eye-canada/) 17 | 18 | Oct 14, 2019 - [Iranian students barred from US: lost money, broken dreams, no answers](https://www.theguardian.com/us-news/2019/oct/14/iranian-students-barred-from-us-lost-money-broken-dreams-no-answers): They had been accepted to prestigious postgraduate programs, many in California, but they got no further than the airport 19 | 20 | Fall, 2019 - [Has President Trump Scared Away All the Foreign Students?](https://www.educationnext.org/has-president-trump-scared-away-foreign-students-facts-behind-fears-higher-education-revenue-recession/) 21 | 22 | May 30, 2018 - [The US is limiting visas for Chinese international students majoring in tech](https://www.theverge.com/2018/5/30/17410824/us-limiting-chinese-visa-international-students-technology-majors): The latest victim in the technology race is students 23 | 24 | May 6, 2018 - [The disappearing Chinese student visa](https://www.axios.com/foreign-student-visas-dropping-china-india-trump-81e70609-9fa7-43eb-8f40-ccfef9fe3fa5.html): There was a 17% drop in international students in the U.S. last year, mostly due to the 28% decline in Indian students and a 24% decline in Chinese students receiving visas. 25 | 26 |  27 | 28 | April 25, 2018 - [Trump's H-1B curbs are putting off foreign students and choking the US economy](https://qz.com/1261468/donald-trumps-h-1b-curbs-put-off-foreign-students-and-choke-the-us-economy/). 29 | 30 | March 23, 2018 - [FBI Director Defends Remarks That Chinese People In U.S. Pose Threats](https://www.huffingtonpost.com/entry/fbi-christopher-wray-chinese-immigrants_us_5ab3d47fe4b008c9e5f51975): Wray has received immense backlash from Asian-American civil rights organizations for his comments about "the China threat." 31 | 32 | March 23, 2018 - [Don't Slam the Door on Chinese Students](https://www.bloomberg.com/view/articles/2018-03-23/reducing-visas-for-chinese-students-is-un-american): Reducing student visas for foreigners is self-defeating and un-American. 33 | 34 | March 11, 2018 - [Visas Issued to Foreign Students Fall, Partly Due to Trump Immigration Policy](https://www.wsj.com/articles/visas-issued-to-foreign-students-fall-partly-due-to-trump-immigration-policy-1520766000): Fewer foreign students exacerbate financial challenges for some U.S. universities. 35 | 36 | March 5, 2018 - [Fewer international grad students are seeking computer science degrees in the U.S. for the first time in years](https://www.geekwire.com/2018/fewer-international-grad-students-seeking-computer-science-degrees-u-s-first-time-years/): The number of international students enrolling in American universities is declining for the first time in years, amid volatile shifts in U.S. immigration policy. 37 | 38 | January 24, 2018 - [H-1B visa hopefuls, green card holders are feeling the wrath of 'America first' Trump](https://www.theregister.co.uk/2018/01/24/trump_immigration_h_1b_crackdown/) 39 | 40 | January 18, 2018 - [Fewer international students coming to US for grad school in science and engineering](https://www.pri.org/stories/2018-01-18/fewer-international-students-coming-us-grad-school-science-and-engineering): Science and engineering fields saw a 6 percent decrease in international graduate students from the fall of 2016 to the fall of 2017, and almost all of that decrease was concentrated in two fields: computer science and engineering. 41 | 42 | January 3, 2018 - [Trump may deport thousands of Indian H-1B visa holders as they wait for green cards](https://qz.com/1170035/h-1b-visa-trap-trump-may-not-spare-even-those-indians-awaiting-green-cards/) 43 | -------------------------------------------------------------------------------- /juggling.md: -------------------------------------------------------------------------------- 1 | # Juggling Analogy 2 | 3 | I'm a juggler, and each of you (i.e., a student) is represented by a ball. 4 | My job is to keep all the balls in the air at any given point in time, and to sustain this state for an arbitrarily long period of time. 5 | 6 | The ball representing "you" is "in the air" if you are able to be productive, working independently (this includes reading, implementing, running experiments, writing, etc.). 7 | The ball representing "you" is "on the ground" if you're blocked by something. 8 | My goal is to make sure that doesn't happen, or at least minimize that time, i.e., to help you become unblocked. 9 | In the juggling analogy, that's me catching the ball and tossing it up again. 10 | 11 | For example, a canonical interaction pattern might be that we meet, discuss experiments to run (catch the ball and toss it up), and then you go off to run those experiments (ball is in the air). 12 | Sometimes the ball is in the air for a long time (e.g., you have a complex implementation to work on). 13 | Sometimes the ball is in the air for only a short time (e.g., you're running a quick experiment and will be back with results soon). 14 | 15 | In this juggling analogy, the gravitational constant _g_ quantifies how long your ball should be up in the air, based on the force applied when tossing the ball. 16 | For example, I would expect Ph.D. students to operate in low _g_, that is, they're able to work independently for long periods of time. 17 | For a Ph.D. student nearing graduation, it would not be weird to not have interactions for weeks 18 | (e.g., "I'm working on those experiments we discussed, and then writing up the results. Everything is proceeding as planned. Nothing to report, I'll let you know when I'm done with Chapter 4.") 19 | 20 | In this juggling analogy, the force applied during the toss is characterized by the planning horizon, and is independent of _g_. 21 | For example, we could be executing on a paper submission due in a month, so despite a Ph.D. student having a low _g_, since we're planning incrementally, we might have frequent interactions (i.e., lots of small tosses, ball coming back down again quickly). 22 | At one extreme, the interaction cycle might be on the order of minutes (e.g., I just rewrote the intro to the draft, let me know what you think). 23 | 24 | Understanding this analogy will help you interact with me in a more efficient and effective manner. 25 | 26 | ## Light vs. Heavy Tosses 27 | 28 | Sometimes, all you need is a light toss. 29 | For example, you just realized that your experimental setup depends on a critical assumption that didn't come up in our previous discussions. 30 | You just need my input on whether that assumption is reasonable. 31 | Here, just slack me, and I'll either respond, "yea, that's okay" or "hrm, that's not a reasonable assumption to make, let's schedule a discussion". 32 | Asynchronous interactions work great here. 33 | 34 | Sometimes, you need more a forceful toss. 35 | For example, you've exhaustive all the research ideas we've discussed, or want to pursue a different research direction. 36 | Here, a slack (text) chat likely won't be productive; this is where you want to schedule a more in-depth synchronous discussion with me. 37 | 38 | ## Falling... or on the Ground 39 | 40 | If you're completely blocked, you need to proactively reach out to me to help me unblock you. 41 | That is, sometimes I just lose track of all the balls I have in the air, and don't notice that there's one sitting on the ground. 42 | If you don't speak up, I might not notice for a while. 43 | 44 | Even better, if you let me know when your ball is about to hit the ground, I can plan accordingly. 45 | This works well, asynchronously over Slack: send me a message, "I'm almost done with the experiments we discussed. Should be ready by next week. Can we discuss?" 46 | 47 | ## Juggling Patterns 48 | 49 | Just like there are different [juggling patterns](https://en.wikipedia.org/wiki/Juggling_pattern), there are relatively predictable changes to my interaction patterns with students at different points in time. 50 | 51 | One common pattern is "conference deadline mode", when the juggling often switches to lots of small tosses of a number of balls (corresponding to the number of submissions planned). 52 | Usually, we (as a group) engage in a planning exercise a couple of months before a major deadline, enumerating potential paper submissions and a rough assignment of who's leading what. 53 | "Conference mode" then shifts into a focus on juggling these balls, sometimes explicitly acknowledging that other balls may fall on the ground, especially those that require a hard toss, i.e., I might explicitly say, "Sorry, no time to think about this right now, working on _X_ deadline. Let's schedule a discussion after the deadline." 54 | 55 | ## Implications for Me 56 | 57 | + I'm constantly context switching. 58 | + My time is rarely "my own", it's dictated by the balls in the air. 59 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # The Student's Guide to @lintool 2 | 3 | I started as an assistant professor in 2004. 4 | Over the years, I've become increasingly convinced that the graduate school experience is most like an apprenticeship, where graduate students learn to "do research" by working with their advisors. 5 | This [intensively personal](http://phdcomics.com/comics.php?f=1296) relationship is difficult to scale, but I've discovered that I give the same advice over and over again. 6 | So, it makes sense to write things down. 7 | This repo is the result of those efforts. 8 | Every time I say something to a student that I've repeated before, I write it down here. 9 | 10 | The result is the student's guide to @lintool. 11 | 12 | Of course, every advisor is idiosyncratic (putting it mildly), and I only claim that this advice is what I give to _my_ students. 13 | I generally agree with Oded Goldreich in there being [no generic advice](http://www.wisdom.weizmann.ac.il/~oded/advice.html). 14 | There are definitely many ways to succeed, but there are no doubt "archetypes", and every graduate student starts off with his or her advisor as a reference point. 15 | As a result, I suspect that at least some of what I say to my students is said by other advisors to their students. 16 | A few may even rise to the level of "research best practices". 17 | Given the potential relevance to others, I'm publicly sharing these documents. 18 | Of course, YMMV, and take everything with the usual grain of [NaCl](https://en.wikipedia.org/wiki/Sodium_chloride). 19 | 20 | ## Research Advice 21 | 22 | + [What working with me is like](what-working-with-me-is-like.md) 23 | + [What my personal schedule is like](what-my-personal-schedule-is-like.md) 24 | + [Should you take my advice?](ignoring-advice.md) 25 | + [General research tips](general-research-tips.md) 26 | + [Writing a research paper with me](writing-a-research-paper-with-me.md): this specifically describes how I work with my students. 27 | + [Writing pet peeves](writing-pet-peeves.md): Lots of specific writing tips. 28 | + [Thesis tips](thesis-tips.md): Writing tips specifically for writing theses. 29 | + [General life tips](general-tips.md): General life lessons. 30 | + [Working in industry is a one-way street out of academia](industry-one-way-street.md) 31 | + [Coding Style](coding-style.md) 32 | 33 | ## Unpopular and Contrarian Research Advice 34 | 35 | + [Be Paranoid! (Someone is working on your idea right now!)](paranoid.md) (November 15, 2019) 36 | + [Diversity, Inclusion, and a Vibrant Intellectual Ecosystem](ecosystem.md) (November 16, 2019) 37 | 38 | ## Other Posts 39 | 40 | + [Leaderboard Micropublications: A Proposal](micropublications.md) (December 2, 2019) 41 | 42 | ## Advice by Others 43 | 44 | Advice by others I've given to my students (and things that I haven't had time to write about): 45 | 46 | + [The Craft of Writing Effectively](https://www.youtube.com/watch?v=vtIzMaLkCaM): everything you learned about writing in school is wrong! 47 | + [Writing advice from the Kording Lab](http://kordinglab.com/2016/01/14/writing-guide.html), including [the idea of joint writing sessions](http://kordinglab.com/2016/01/13/writing-together.html) 48 | + [Jordan Boyd-Graber's writing style guide](http://umiacs.umd.edu/~jbg/static/style.html) 49 | + [Dealing with Rejection](https://sites.umiacs.umd.edu/elm/2016/10/25/dealing-with-rejection/) by Niklas Elmqvist 50 | + [Coping with Rejection](https://www.loom.com/share/89bfb10668d94595b265a156126474a5) by Aditya Parameswaran 51 | + [Attending Professional Conferences as a Newcomer](http://pgbovine.net/attending-conferences.htm) 52 | + [How to Prepare a Talk](https://www.deconstructconf.com/blog/how-to-prepare-a-talk): the secret to success? repetition 53 | + [Research Design Patterns](http://pgbovine.net/research-design-patterns.htm) 54 | + [Hanna Wallach's & Mark Dredze's How to Be a Successful PhD Student (in Computer Science (in NLP/ML)](https://people.cs.umass.edu/~wallach/how_to_be_a_successful_phd_student.pdf) 55 | + [Managing Your Advisor](https://medium.com/great-research/managing-your-advisor-bb9060f4f8ce) by Nick Feamster 56 | + [Thoughts on Competition in Academia](https://shomir.net/competition.html) by Shomir Wilson 57 | + [Advice for Whiteboard Coding Interviews](https://fpcastle.com/2021/10/03/coding-interviews.html) by Satnam Singh 58 | 59 | ## Other Odds and Ends 60 | 61 | I've collected other non-related odds and ends in this repo: 62 | 63 | + [Waterloo plugs](waterloo-plugs.md): media mentions of the burgeoning tech scene in Waterloo, the Greater Toronto Area, or Canada in general. 64 | + [US anti-plugs](us-anti-plugs.md): the opposite of above; media mentions on how the US is screwing up immigration, tech, etc. 65 | + [Chinese technical terms](chinese-technical-terms.md): a Chinese-English phrase book of technical terms that I've been slowly accumulating. 66 | + [Jinfeng Rao's job hunt experience](https://zhuanlan.zhihu.com/p/35435776) (in Chinese) 67 | + [Performance Maintenance During Continuous Flight Operations](docs/NAVMEDP-6410.pdf): an interesting look into how the military manages sleep and fatigue during military operations. 68 | -------------------------------------------------------------------------------- /coding-style.md: -------------------------------------------------------------------------------- 1 | # Coding Style 2 | 3 | These are general coding style guidelines we've adopted for my research group. 4 | 5 | **Background on UNIX command-line options:** 6 | Many (most?) UNIX style command-line options have "long form" and a "short form", where the latter is a single character. 7 | For example `-a` and `--all` for `ls` (although, interestingly, `-l` doesn't have a long form). 8 | 9 | Short form options can be strung together, e.g., `ls -l -a` becomes `ls -la`. 10 | Thus `--abc` and `-abc` mean very different things; the first represents a single option, the second is a shorthand for multiple options, e.g., `-a -b -c`. 11 | 12 | For more details, see [GNU Program Argument Syntax Conventions](https://www.gnu.org/software/libc/manual/html_node/Argument-Syntax.html). 13 | 14 | ## General 15 | 16 | + We use a maximum line length of 120 character. 17 | The rationale for this limit is that 120 is (approximately) the number of characters that will render in GitHub without horizontal scrolling. 18 | This maximizes use of screen real estate. 19 | 20 | ### Describing Arguments in Command-Line Applications 21 | 22 | + The description should be a short phrase that begins with a capital letter an ends with a period. It prescribes the function or method's effect as a command ("Do this", "Return that"), not as a description; e.g. don't write "Returns the pathname...". This overarching guideline is adapted from [PEP 257: Docstring Conventions](https://www.python.org/dev/peps/pep-0257/). 23 | + The description should be as succinct as possible, typically a noun phrase. For example, for `-index`, "Path to input collection." is preferred to "Set the path to the input collection.". 24 | + Omit articles, e.g., "Number of Hits." is preferred over "The number of hits." 25 | + Boolean flags should be written as verb phrases, and there is generally no need to indicate that it's a flag in the description. For example, for `-storeRaw`, "Store raw source documents." is preferred to "Flag to determine whether raw source documents are stored." or "Whether or not to store raw documents.". 26 | + Prefer prepositional phrases over possessives for clarity, e.g., "Foo parameter of Bar." is preferred to "Bar's Foo parameter." even though the latter is shorter. This is especially the case if Bar has multiple tokens. 27 | 28 | ### Pull Requests and Commit Messages 29 | 30 | + The title of pull requests should be written in the imperative, to be consistent with [PEP 257: Docstring Conventions](https://www.python.org/dev/peps/pep-0257/). 31 | Capitalize the initial letter but _don't_ end with punctuation. 32 | Equivalently, you can think of an implied subject of "this pull request will..." 33 | So, "Add super duper feature", not "adds super duper feature" or "Adding super duper feature". 34 | 35 | + Typically, the proposer of the pull request merges, after code review and other checks. 36 | I admit that I frequently break this rule and just merge the pull request myself — often it's necessary since the proposer doesn't have write access to the repo, and I forget who has write access and who doesn't. 37 | In practice, it works out that _regular contributors with write access_ to a repo merge their own pull request, with a somewhat fuzzy definition of how "regular" is defined. 38 | 39 | + Write a good commit message. 40 | The first line of your commit message should match the title of your pull request. 41 | If the pull request is relatively small, this should suffice as a commit message. 42 | Since our repos are set up for squashed commits, GitHub's default commit message is just a concatenation of the commit messages of all the individual commits that comprise this PR. 43 | **Don't just use this message.** 44 | Because we end up with unhelpful commit messages [like this one](https://github.com/castorini/pyserini/commit/770dfdca8dacf9460ee2ac673a09d9c98eef466a). 45 | Instead, I like to either write a short narrative paragraph summarizing issues on the pull request itself, or a short bulleted list. 46 | Remember, there's no need to duplicate everything on the pull request, since we can always trace back to examine the discussion online. 47 | 48 | 49 | ## Python 50 | 51 | + We adopt [PEP8](https://pep8.org/). 52 | + We adopt single quotes `'` for strings; unless another approach is obviously better, we use [f-strings](https://www.python.org/dev/peps/pep-0498/) for formatting. 53 | 54 | ### Docstrings 55 | 56 | + We adopt the [numpy docstring style](https://numpydoc.readthedocs.io/en/latest/format.html). 57 | + In general, follow guidelines in "Describing Arguments in Command-Line Applications". 58 | 59 | ### Command-Line Options 60 | 61 | We try to stay true to the UNIX convention, which is what `argparse` (implicitly) endorses anyway. 62 | This means that Java and Python counterparts of script that do the same thing will have different parameters. 63 | Thus: 64 | 65 | + Options begin with two dashes: `--index` instead of `-index`. 66 | + Use single dash (instead of underscore) for multi-token options: `--max-value` instead of `-max_value`. 67 | + Use `.` to group parameters, e.g., `--bm25` (use BM25) and `--bm25.b` (set the BM25 _b_ parameter). This looks better than `--bm25-b`. 68 | 69 | ## Java 70 | 71 | + Two space indent, otherwise standard conventions. 72 | 73 | ### Command-Line Options 74 | 75 | Java breaks the UNIX convention and is a total mess. 76 | For example, `-classpath` and `--class-path` both work (although the first is much more common). 77 | For command-line programs written in Java, we try to maintain consistency to Java options: 78 | 79 | + Options begin with a single dash: `-index` instead of `--index`. 80 | + Use camel case instead of snake case: `-maxValue` instead of `-max_value`. 81 | + Use `.` to group parameters, e.g., `-bm25` (use BM25) and `-bm25.b` (set the BM25 _b_ parameter). 82 | -------------------------------------------------------------------------------- /chinese-technical-terms.md: -------------------------------------------------------------------------------- 1 | # Chinese Technical Terms 2 | 3 | ## General Technical Terms 4 | 5 | English | Chinese (simplified) | Chinese (traditional) | Pinyin 6 | --------|:--|:--|:-- 7 | hardware | 硬件 | 硬件 | yìngjiàn 8 | software | 软件 | 軟件 | ruǎnjiàn 9 | source code | 代码 | 代碼 | dàimǎ 10 | internet | 互联网 | 互聯網 | hùliánwǎng 11 | browser | 浏览器 | 瀏覽器 | liúlǎn qì 12 | search engine | 搜索引擎 | 搜索引擎 | sōusuǒ yǐnqíng 13 | recommender system | 推荐系统 | 推薦系統 | tuījiàn xìtǒng 14 | e-commerce | 电商 | 電商 | diàn shāng 15 | end-to-end | 端到端 | 端到端 | duān dào duān 16 | embedded system | 嵌入式系统 | 嵌入式系統 | qiànrù shì xìtǒng 17 | pain point | 痛点 | 痛點 | tòng diǎn 18 | team | 团队 | 團隊 | tuánduì 19 | scenario | 场景 | 場景 | chǎngjǐng 20 | 21 | ## General Machine Learning 22 | 23 | English | Chinese (simplified) | Chinese (traditional) | Pinyin 24 | --------|:--|:--|:-- 25 | supervised learning | 监督学习 | 監督學習 | jiāndū xuéxí 26 | unsupervised learning | 无监督学习 | 無監督學習 | wú jiāndū xuéxí 27 | distant supervision | 远程监督 | 遠程監督 | yuǎnchéng jiāndū 28 | transfer learning | 迁移学习 | 遷移學習 | qiānyí xuéxí 29 | multi-task learning | 多任务学习 | 多任務學習 | duōrènwù xuéxí 30 | reinforcement learning | 强化学习 | 強化學習 | qiánghuà xuéxí 31 | maximize | 最大化 | 最大化 | zuìdà huà 32 | minimize | 最小化 | 最小化 | zuìxiǎo huà 33 | similarity | 相似度 | 相似度 | xiāngsì dù 34 | linear algebra | 线性代数 | 線性代數 | xiànxìng dàishù 35 | vector | 向量 | 向量 | xiàngliàng 36 | sparse vector | 稀疏向量 | 稀疏向量 | xīshū xiàngliàng 37 | dense vector | 稠密向量 (also, 密集向量) | 稠密向量 | chóumì xiàngliàng (mìjí xiàngliàng) 38 | model | 模型 | 模型 | móxíng 39 | optimization | 优化 [1] | 優化 [1] | yōuhuà 40 | regression | 回归 | 回歸 | huíguī 41 | classification | 分类 | 分類 | fēnlèi 42 | logistic regression | 逻辑回归 | 邏輯回歸 | luójí huíguī 43 | sample (noun) | 样本 | 樣本 | yàngběn 44 | sample (verb) | 采样| 採樣 | cǎiyàng 45 | label | 标注 | 標註 | biāozhù 46 | train | 训练 | 訓練 | xùnliàn 47 | test | 测试 | 測試 | cèshì 48 | fit model parameters | 拟合模型参数 | 擬合模型參數 | nǐhé móxíng cānshù 49 | dataset | 数据集 | 數據集 | shùjù jí 50 | training set | 训练集 | 訓練集 | xùnliàn jí 51 | validation set | 验证集 | 驗證集 | yànzhèng jí 52 | test set | 测试集 | 測試集 | cèshì jí 53 | gradient descent (method) | 梯度下降(法) | 梯度下降(法) | tīdù xiàjiàng fǎ 54 | stochastic gradient descent | 随机梯度下降 | 隨機梯度下降 | suíjī tīdù xiàjiàng 55 | step size | 步长 | 步長 | bùcháng 56 | epoch | 轮 | 輪 | lún 57 | batch size | 批量样本数 | 批量樣本數 | pīliàng yàngběn shù 58 | learning rate | 学习率 | 學習率 | xuéxí lǜ 59 | overfitting | 过拟合 | 過擬合 | guò nǐhé 60 | under fitting | 欠拟合 | 欠擬合 | qiàn nǐhé 61 | feed (i.e., to feed into a model) | 喂 | 餵 | wèi 62 | 63 | [1] Literally, "to make excellent" 64 | 65 | ## NLP + IR 66 | 67 | English | Chinese (simplified) | Chinese (traditional) | Pinyin 68 | --------|:--|:--|:-- 69 | natural language processing | 自然语言处理 | 自然語言處理 | zìrán yǔyán chǔlǐ 70 | information retrieval | 信息检索 | 信息檢索 | xìnxī jiǎnsuǒ 71 | passage retrieval | 段落检索 | 段落檢索 | duànluò jiǎnsuǒ 72 | unstructured text | 无结构文本 | 無結構文本 | wú jiégòu wénběn 73 | reading comprehension | 阅读理解 | 閱讀理解 | yuèdú lǐjiě 74 | intent | 意图 | 意圖 | yìtú 75 | slot | 槽(位) | 槽(位) | cáowèi 76 | chatbot | 对话机器人 | 對話機器人 | duìhuà jīqìrén 77 | knowledge graph | 知识图谱 | 知識圖譜 | zhīshì túpǔ 78 | entity-attribute-value | 实体-属性-值 | 實體-屬性-值 | shítǐ-shǔxìng-zhí 79 | triple | 三元组 | 三元組 | sān yuán zǔ 80 | 81 | ## Deep Learning 82 | 83 | English | Chinese (simplified) | Chinese (traditional) | Pinyin 84 | --------|:--|:--|:-- 85 | deep learning | 深度学习 | 深度學習 | shēndù xuéxí 86 | neural network | 神经网络 | 神經網絡 | shénjīng wǎngluò 87 | deep... (e.g., neural network) | 深度(神经网络) | 深度(神經網絡) | shēndù (shénjīng wǎngluò) 88 | backpropagation | 反向传播 | 反向傳播 | fǎn xiàng chuánbò 89 | feedforward (neural) network | 前馈(神经)网络 | 前饋(神經)網絡 | qián kuì (shénjīng) wǎngluò 90 | fully-connected (layer) | 全连接(层) | 全連接(層) | quán liánjiē (céng) 91 | perceptron | 感知器 | 感知器 | gǎnzhī qì 92 | multi-layer perceptron | 多层感知器 | 多層感知器 | duōcéng gǎnzhī qì 93 | encoder | 编码器 | 編碼器 | biānmǎ qì 94 | decoder | 解码器 | 解碼器 | jiěmǎ qì 95 | word embedding | 单词嵌入 | 單詞嵌入 | dāncí qiànrù 96 | convolutional (neural) network | 卷积(神经)网络 | 卷積(神經)網絡 | juàn jī (shénjīng) wǎngluò 97 | feature map | 特征图 | 特徵圖 | tèzhēng tú 98 | convolutional filter | 卷积滤波器 | 卷積過濾器 | juàn jī guòlǜqì 99 | stride | 步长 | 步長 | bùcháng 100 | (max/min) pooling | (最大/最小) 池化| (最大/最小) 池化 | (zuìdà / zuìxiǎo) chí huà 101 | recursive (neural) network | 递归(神经)网络 | 遞歸(神經)網絡 | dìguī (shénjīng) wǎngluò 102 | recurrent (neural) network | 循环(神经)网络 | 循環(神經)網絡 | xúnhuán (shénjīng) wǎngluò 103 | (bidirectional) LSTM | (双向)LSTM网络 | (双向)LSTM網絡 | (shuāngxiàng) LSTM wǎngluò 104 | (bidirectional) LSTM | (双向)长短时记忆循环神经网络 [1] | (雙向)長短時記憶循環神經網絡 [1] | (shuāngxiàng) chángduǎn shí jìyì xúnhuán shénjīng wǎngluò 105 | tree-LSTM | 树形长短时记忆递归神经网络 | 樹形長短時記憶遞歸神經網網絡 |shùxíng chángduǎn shí jìyì dìguī shénjīng wǎngluò 106 | autoencoder | 自编码器 | 自編碼器 | zì biānmǎ qì 107 | 108 | [1] More precisely, bidirectional long-short term memory recurrent neural network 109 | 110 | ## Databases, Big Data, etc. 111 | 112 | English | Chinese (simplified) | Chinese (traditional) | Pinyin 113 | --------|:--|:--|:-- 114 | big data | 大数据 | 大數據 | dà shùjù 115 | data science | 数据科学 | 數據科學 | shùjù kēxué 116 | scalability | 可扩展性 | 可擴展性 | kě kuòzhǎn xìng 117 | scale up | 向上扩容 | 向上擴容 | xiàngshàng kuòróng 118 | scale out | 平面扩容 | 平面擴容 | píngmiàn kuòróng 119 | platform | 平台 | 平台 | píngtái 120 | relational database | 关系数据库 | 關係數據庫 | guānxì shùjùkù 121 | NoSQL database | 非关系数据库 [1] | 非關係數據庫 [1] | fēi guānxì shùjùkù 122 | key-value store | 键值(数据)库 | 鍵值(數據)庫 | jiàn zhí (shùjù) kù 123 | document store | 文档数据库 | 文檔數據庫 | wéndàng shùjùkù 124 | column store | 列(式)数据库 | 列(式)數據庫 | liè (shì) shùjùkù 125 | graph store | 图数据库 | 圖數據庫 | tú shùjùkù 126 | API | 应用编程接口 | 應用編程接口 | yìngyòng biānchéng jiēkǒu 127 | cluster | 集群 | 集群 | jíqún 128 | node | 节点 | 節點 | jiédiǎn 129 | rack | 机架 | 機架 | jī jià 130 | index | 索引 | 索引 | suǒyǐn 131 | query plan | 查询计划 | 查詢計劃 | cháxún jìhuà 132 | logical plan | 逻辑计划 | 邏輯計劃 | luójí jìhuà 133 | physical plan | 物理计划 | 物理計劃 | wùlǐ jìhuà 134 | 135 | [1] Literally, non-relational 136 | -------------------------------------------------------------------------------- /ignoring-advice.md: -------------------------------------------------------------------------------- 1 | # Should You Take My Advice? 2 | 3 | I often say to my students, "I am your advisor, not your boss." 4 | As an advisor, I dispense advice, explaining the consequences of specific actions (both positive and negative), but I rarely tell you what to do. 5 | I explain to you what the design space looks like, or, alternatively, what the loss landscape is. 6 | It's up to you to decide on your course of action. 7 | Ultimately, it's _your_ research and _your_ degree. 8 | 9 | However, this doesn't mean that all my advice is equal in terms of "assertiveness" or "firmness" — the degree to which you should follow my guidance. 10 | At one end of the spectrum, some advice I offer is quite close to a directive, i.e., "please do it". 11 | At the other end of the spectrum, some advice I offer comprises merely suggestions, i.e., "feel free to ignore me". 12 | Given subtle nuances in communication, particularly for non-native English speakers [1], I've decided to explicitly elaborate on communicative intent here. 13 | 14 | Think of this as a way to translate between "what I say" and "what I really mean". 15 | Of course, this guide is idiosyncratic with respect to the particular way I communicate, but in general, for many advisors, the "surface form" of an utterance often diverges from its underlying intent. 16 | 17 | ## "Advice" That Really Means "Please Do It!" 18 | 19 | To begin, there _are_ cases where what I say should be interpreted as directives, even though it sounds like advice. 20 | These generally fall into two categories: 21 | The first is externally imposed rules, which if ignored, leads to negative consequences. 22 | For example, 23 | 24 | + "You should get that form signed"... otherwise you can't graduate. 25 | + "You shouldn't go over the page limit specified in the call for papers"... otherwise your paper will be desk rejected. 26 | 27 | Thus, even though these statements are phrased in the [subjunctive](https://en.wikipedia.org/wiki/Subjunctive_mood) ("should"), understand these directives as "please do it". 28 | 29 | The other main category comprises commitments we (as a group) made. 30 | For example, 31 | 32 | + A deliverable in the context of a proposal. 33 | + A feature in a piece of software for a downstream user. 34 | + A particular model variant for a collaborator. 35 | 36 | Not honoring these commitments has negative consequences (e.g., unhappy collaborators). 37 | In these contexts, statements I make should be interpreted as _requests_, i.e., you should do it. 38 | And if you can't, let's discuss further and figure something out (for example, prioritize between competing responsibilities). 39 | Please don't just ignore me. 40 | 41 | ## Advice That Really Is Advice 42 | 43 | With the "please do it" stuff out of the way, everything else below _really_ is advice, but to various degrees of assertiveness. 44 | Yes, in many cases, it _is_ okay to ignore me. 45 | 46 | I'll sketch out common things I say, in decreasing order of assertiveness: 47 | 48 | **"It's a bad idea to..."** or **"You shouldn't..."** or **"Don't..."** 49 | When I phrase advice negatively, it should be understood as "don't do it unless you have a very good reason otherwise". 50 | There are (likely) negative consequences (in my experience) to a course of action. 51 | For example: 52 | 53 | + "You shouldn't introduce _X_ here in Section 3 because you don't define _X_ until Section 4." 54 | Likely negative consequence: your reader is not going to understand your paper. 55 | + "You shouldn't talk for more than _Y_ minutes." 56 | Likely negative consequence: your audience is going to get annoyed. 57 | 58 | Advice along these lines isn't a "100% rule", but if you're going to ignore me, you'd better have good reasons [2]. 59 | 60 | **"It's a good idea to..."** or **"You should..."** 61 | When I phrase advice positively, it usually means that I am telling you about best practices, norms, or preferred solutions. 62 | For example: 63 | 64 | + "It's a good idea to have an explicit statement of contributions in your introduction." (Many reviewers except such a statement, perhaps elevating it to a norm.) 65 | + "You should add some ablation conditions here." (Running ablations falls under "experimental best practices".) 66 | 67 | Advice along these lines should be interpreted as less assertive than the "it's a bad idea" variants above. 68 | There are plenty of violations of best practices [3], but just remember that best practices exist for a reason, and if you wish to violate them, be deliberate in terms of what you hope to accomplish. 69 | 70 | **"You might want to..."** or **"You should think about..."** or **"Have you considered..."** 71 | Take these as suggestions. 72 | For example: 73 | 74 | + "You might want to move that figure here." 75 | + "You should think about adding this feature to your model." [4] 76 | 77 | Often, these comments are made in the context of a discussion where I may not have "thought things through" (completely). 78 | In these cases, give my suggestions some thought, and if they don't make sense, feel free to ignore them. 79 | However, if I bring up the same issue again (at a later point in time), it would be nice for you to demonstrate that you've at least considered my feedback, e.g., "Yes, I thought about moving that figure here, but that doesn't make sense because..." 80 | 81 | **"... your call"** or **"... I don't feel strongly either way"** or **"I'm agnostic..."** or 🤷. 82 | This happens often after a discussion wherein multiple options are considered, all of which represent reasonable choices or actions. 83 | For example: 84 | 85 | + "You could phrase it like this... or like that... your call." 86 | + "You could structure your results section this way... or that way... I don't feel strongly either way." 87 | 88 | It really means... up to you... 89 | 90 | I strive to communicate clearly and directly, but the bottom line is: If you don't understand the level of assertiveness in the advice I'm offering, **ASK ME TO CLARIFY**! 91 | 92 | ## Footnotes 93 | 94 | [1] I once said to a student (non-native English speaker) "[do this at your peril](https://www.ldoceonline.com/dictionary/do-something-at-your-peril)" and the student didn't understand what I meant. 95 | 96 | [2] Yes, there indeed cases (but relatively rare) where a student ignore my advice in the category, but ended up with a better outcome than would have occurred if my advice had been followed. 97 | 98 | [3] In fact, violation of best practices is one approach to progress. Learning to rank using tree-based models constituted best practices before the "neural age". Of course, they've since been supplanted (or at the least, augmented) by neural networks. 99 | 100 | [4] Note the subtle difference between "you should..." and "you should think about/consider..." 101 | -------------------------------------------------------------------------------- /what-working-with-me-is-like.md: -------------------------------------------------------------------------------- 1 | # What working with me is like 2 | 3 | tl;dr - see [Juggling Analogy](juggling.md). 4 | 5 | + Unlike most advisors, I don't have regular, fixed meetings with students. 6 | The rationale is that progress doesn't happen at an even pace. 7 | Sometimes you're churning out experimental results every two days and need my help interpreting them and formulating next steps—making you wait until our next weekly meeting reduces your productivity. 8 | Sometimes you're in the middle of a complex implementation that may take a couple of weeks—you'll have nothing to tell me other than "I'm working on it". 9 | 10 | + Instead, I try to keep as much of my schedule as open as possible and give you individualized attention. 11 | I might spend an entire afternoon brainstorming an idea with you, but then not meet again for three weeks while you're implementing the idea. 12 | Toward the end of your Ph.D. career, I might say, "Go write your thesis!" and not meet with you for a month until I see drafts. 13 | 14 | + I am a heavy Slack user and use it as my primary channel of communication with students. 15 | My favorite thing about Slack is that it fills in the "little crevices in time" that allows me to effectively multi-task. 16 | For example, when I'm waiting in line at the checkout, I can use the 30 seconds productively on a quick issue. 17 | 18 | + I don't particularly care _where_ you work, as long as you're productive. 19 | 20 | + I don't particularly care _when_ you work, as long as you're productive. 21 | Want to work on Hawaii time (which is six hours behind Waterloo)? 22 | Fine with me. 23 | (For some time, a student actually did this.) 24 | The only caveat is that you might want to have some overlap with my schedule for meetings, discussions, etc. 25 | 26 |I get asked a lot: why stay in academia, all the excitement in AI is happening in industry with massive compute. And I am seeing some profs leaving academia, but also seeing lots of researchers in industry looking to go back to academia, especially those who don’t work on LLMs.…
— Russ Salakhutdinov (@rsalakhu) September 15, 2023
27 | 28 | ## How much should I work? 29 | 30 | For reference, I estimate that I work around 40–50 hours a week. 31 | I try to get 7–8 hours of sleep every night. 32 | See my writeup on [what my personal schedule is like](what-my-personal-schedule-is-like.md). 33 | 34 | Additional useful links: 35 | 36 | + [Why We Have to Go Back to a 40-Hour Work Week to Keep Our Sanity](https://www.alternet.org/2018/04/why-we-have-go-back-40-hour-work-week-keep-our-sanity/): 150 years of research proves that long hours at work kill profits, productivity and employees. 37 | + [Why Crunch Modes Doesn't Work: Six Lessons](https://igda.org/resources-archive/why-crunch-mode-doesnt-work-six-lessons-2005/): Workers can maintain productivity more or less indefinitely at 40 hours per five-day work week. When working longer hours, productivity begins to decline. Somewhere between four days and two months, the gains from additional hours of work are negated by the decline in hourly productivity. 38 | + [How Some Men Fake an 80-Hour Work week, and Why It Matters](http://www.nytimes.com/2015/05/05/upshot/how-some-men-fake-an-80-hour-workweek-and-why-it-matters.html) 39 | + [If You’re So Successful, Why Are You Still Working 70 Hours a Week?](https://hbr.org/2018/02/if-youre-so-successful-why-are-you-still-working-70-hours-a-week) Elite professional organizations deliberately set out to identify and recruit "insecure overachievers". 40 | 41 | ## What should I be working on? 42 | 43 | In terms of research, I expect you to tackle items in one of two categories: 44 | 45 | + Research problems that I've formulated or novel approaches to existing problems that I've devised. 46 | In some cases, these "ready-made" research ideas are contributed by a post-doc or a senior student. 47 | These items can be found in our group's internal research board, which contains a holding area for "unclaimed ideas". 48 | They vary from vague observations ("this is interesting, we should dig into it a bit more...") to concrete suggestions ("I think we can improve this model with a small tweak...") and span lots of different sub-areas of NLP, IR, etc. 49 | I generally have good intuition about what's interesting, and these ideas are usually good starting points that will lead to something publishable. 50 | In formulating these research ideas, I also try to build in a "fail fast" property so that you can do some initial explorations to make sure it's not a dead end. 51 | + Your own ideas, e.g., a problem you've formulated yourself or a novel approach to an existing problem that you came up with. 52 | 53 | The tl;dr is that you should be making _demonstrable_ progress executing "my" idea or "your" idea. 54 | I put these in quotes because it's an over-simplification: obviously, you'll contribute to "my" ideas and I'll certainly help you refine "your" ideas. 55 | Since the bulk of the research conducted by my group is _empirical_ (i.e., experimental) in nature, by demonstrable progress I typically mean new experimental results (or concrete steps towards generating them). 56 | Of course, I don't expect all experiments to actually "work", but "failed" experiments also offer valuable insight into the problem. 57 | 58 | The "anti-pattern" I _don't_ want to see is you struggling to come up with something to work on for extended periods of time. 59 | To put it somewhat bluntly: either come up with a compelling research vision of your own (and I'll say, great, go ahead) or competently execute a research vision that I've laid out. 60 | 61 | Generally, I will nudge newer students to "my ideas" to give them concrete things to work on and leave more senior students "to find their own way". 62 | A few common "advising patterns" I frequently adopt: 63 | 64 | + For particularly enthusiastic new students, I sometimes give them "a lot of rope" to come up with their own ideas. 65 | Some do succeed, but to be honest, the success rate isn't very high. 66 | New students have a lot of energy, but lack the experience of knowing what's _interesting_ or _doable_. 67 | + In the above case, after I let the student "spin their wheels" without getting anywhere for a while (for example, the first semester), I'll start nudging them towards more concrete ideas... the nudging becomes more forceful the longer I don't see demonstrable progress. 68 | + For Ph.D. students, I actually take the opposite approach: I will let students "wander in the wilderness" for a while to develop their own ideas, because that's an important part of the Ph.D. experience. 69 | 70 | I recently learned (Feb 2022) that [Austin Henley](https://austinhenley.com/) has a much more pithy way of saying this: ["lead or be led"](https://austinhenley.com/blog/lessonsfrommyphd.html). 71 | His lesson of "managers as input/output machines" is also quite complementary and consistent with what I've written here. 72 | -------------------------------------------------------------------------------- /ecosystem.md: -------------------------------------------------------------------------------- 1 | # Diversity, Inclusion, and a Vibrant Intellectual Ecosystem 2 | 3 | Initial post: November 16, 2019 4 | 5 | A vibrant, healthy intellectual ecosystem is one that is diverse in _all_ aspects. 6 | Of course, there are the important ones concerning race, gender, sexual orientation, nation of origin, age, etc. 7 | But we also need diverse ideas and research styles: 8 | People who chase leaderboards and rapidly answer the obvious questions. 9 | People who work on really difficult problems and aren't afraid to fail. 10 | People who question the status quo and why we do things a certain way. 11 | We need people with different risk/reward profiles, different time horizons, etc. 12 | 13 | In the late 1990s to early 2000s, Apple had a brilliant advertising campaign called [Think Different](https://en.wikipedia.org/wiki/Think_different). 14 | The [most memorable commercial](https://www.youtube.com/watch?v=4oAB83Z1ydE) from that campaign ended with the punch line, "... because the ones who are crazy enough to think that they can change the world, are the ones who do". 15 | (Here's [Steve Jobs](https://www.youtube.com/watch?v=keCwRdbwNQY) introducing the commercial.) 16 | We absolutely need people in the world like this [[1](#footnote1)]. 17 | 18 | I have tremendous respect for people who take large intellectual risks, who pursue research programs that have no near-term chance of success. 19 | 20 | Unfortunately, respect is often not reciprocated for my incremental, nose-to-the-grindstone, leaderboard-chasing, perspiration-over-inspiration style of research. 21 | For rhetorical convenience, I'll just refer to this style as "leaderboard chasing". 22 | 23 | Leaderboard chasing is frequently met with derision and condescension as "mere engineering" and "even a waste of time". 24 | I understand not everyone wants to do it; 25 | I promote it but I'm not forcing anyone to do it. 26 | Certainly not my students. 27 | I have students who leaderboard chase, have lots of fun, and absolutely love it. 28 | It revs up their competitive spirit, challenges them to think creatively (because everyone has BERT, to win you need more than BERT), and teaches them to execute efficiently. 29 | For certain students, this is exactly what they want to do. 30 | For other students, this isn't their cup of tea, and so I work with them on other problems. 31 | 32 | If my students don't want to chase leaderboards, they don't. 33 | I didn't when I was a graduate student. 34 | Against the crashing wave of "statistical NLP", I wrote my Ph.D. dissertation (in 2004) on theoretical linguistics, ["Event Structure and the Encoding of Arguments: The Syntax of the Mandarin and English Verb Phrase"](https://cs.uwaterloo.ca/~jimmylin/publications/Lin_PhD_thesis_2004.pdf). 35 | Let me give you a sample of things that my students have worked on in recent years: 36 | One's built [an analytical relational database _in JavaScript_](https://dl.acm.org/citation.cfm?id=3058736). 37 | A couple of years ago I collaborated with a student and a number of colleagues on imagining what it would be like to [build a search engine on Mars](https://dl.acm.org/citation.cfm?id=3038912.3052625), where the primary challenge is roundtrip speed-of-light signal propagation delays. 38 | I collaborate with a historian and a librarian on [a project to analyze web archives](https://archivesunleashed.org/). 39 | I'm currently working with students on [data-driven models for streamflow (i.e., flood) prediction](https://cs.uwaterloo.ca/~jimmylin/publications/Gauch_etal_CI2019.pdf), in the broader context of climate change mitigation. 40 | I'll let you judge if these projects count as more than "leaderboard chasing". 41 | 42 | If you don't want to chase leaderboards, don't. 43 | But do you really have to belittle those who do? 44 | I don't care if you insult me. 45 | I'm a senior faculty, I can defend myself. 46 | But insulting graduate students for their choices of what they want to work on _is_ _not_ _acceptable_. 47 | There is a student out there whose first ACL paper was primarily leaderboard chasing; 48 | in fact, her point of pride was reaching number one on a popular, competitive leaderboard using a small but clever incremental tweak on a well-known architecture. 49 | And now you've just sneer at her accomplishment. 50 | 51 | All I'm asking is: _don't_ _judge_. 52 | Especially since those who attack me on social media make the (implied) assumption that I run an authoritarian group with a toxic monoculture. 53 | Whether in fact, I do, you'll have to ask my students. 54 | But put away your prejudicial condescension before you've had a chance to get to know me (or my students) personally. 55 | 56 | What infuriates me is that the people who express this holier-than-thou attitude are often the same people who claim that they value diversity and inclusion. 57 | 58 | Do you know what makes it even worse? 59 | It's that this attitude is frequently directed at Chinese students, with the implied undertone of "oh, you're doing this because you lack creativity." 60 | This is, at the very least, a [microagression](https://en.wikipedia.org/wiki/Microaggression), and, at the worst, outright racism. 61 | 62 | It is usually at this point that I am accused of appropriating the language of diversity and inclusion. 63 | The response is often something like, "oh, those poor Chinese students, they're already _X_ of our university's Ph.D. program" or "yea, sure, they need _so_ much help". 64 | To which I say, your ignorance is embarrassing. 65 | Try reading [this Wikipedia article](https://en.wikipedia.org/wiki/Model_minority) as an entry point into the well-researched academic literature about so-called "model minorities"; the very term was invented by the white power structures half a century ago as a divisive tool of oppression. 66 | I don't deny that certain groups are more marginalized than others. 67 | But to start comparing who's _more_ marginalized is counterproductive to the goal that we want diversity and inclusion _for all_. 68 | 69 | ## Follow-Up: "Authoritarian Toxic Monoculture" 70 | 71 | I've been accused of, paraphrasing only slightly, running an authoritarian group with a toxic monoculture. 72 | Let's even say that's partially true (read on...). 73 | 74 | Consider this scenario: 75 | You're an ambitious Chinese undergraduate student who's already risen to the top in the [gaokao](https://en.wikipedia.org/wiki/National_College_Entrance_Examination). 76 | Here's the deal: 77 | I get you that dream job at Google, Facebook, etc. in the Bay Area, Seattle, etc. and the green card that comes with it [[2](#footnote2)]. 78 | For that, "all" you need to do is to "pay your dues" and grind it out for a few years, contribute as a cog in the machine that is my lab. 79 | Yes, you'll likely be miserable. 80 | Yes, you won't have work/life balance. 81 | Yes, I will make you slave away chasing leaderboards, pumping out *ACL papers. 82 | But, you'll get that Ph.D. and I'll deliver on my promise to get you that dream job. 83 | You can evaluate based on my previous students whether I can indeed deliver. 84 | With a bit of _hubris_, let's say that I can, with reasonably high probability. 85 | So, "authoritarian" and "monoculture"? 86 | Guilty as charged. 87 | 88 | Nevertheless, do you take the deal? 89 | Even if you don't, can you understand why many would? 90 | As long as students know the terms coming in, why wouldn't you respect their choice? 91 | 92 | Now, if _all_ North American research groups operated this way, I would be concerned. 93 | But that's exactly the point of this post: we need a bit of everything for a vibrant intellectual ecosystem. 94 | Now, I understand that it might be easy for an authoritarian group to veer into clearly unethical behavior (e.g., data falsification). 95 | That's a bright red line and there should be zero tolerance for such behavior. 96 | 97 | These aside: 98 | What's your problem with people who are simply aspiring to "a better life" via a path that you might not necessarily take? 99 | Maybe you shouldn't cast stones from your position of privilege? 100 | 101 | ## Footnotes 102 | 103 | [1] My [admonition against hubris](paranoid.md) still stands, as the punch line could easily as well end "... because the ones who are crazy enough to think that they can change the world, often really are". 104 | In the same vein, Carl Sagan once said, "But the fact that some geniuses were laughed at does not imply that all who are laughed at are geniuses. They laughed at Columbus, they laughed at Fulton, they laughed at the Wright Brothers. But they also laughed at Bozo the Clown." 105 | 106 | [2] Yes, I realize that I'm in Canada right now and so talking about green cards and SV isn't particularly apt. It's all hypothetical anyway. 107 | 108 | ## Acknowledgments 109 | 110 | I want to thank the following people who have made substantial, constructive comments to this piece: Chris Brew (for now). 111 | Feedback, however, does not imply endorsement. 112 | -------------------------------------------------------------------------------- /thoughts-chatgpt.md: -------------------------------------------------------------------------------- 1 | # Thoughts about ChatGPT, Bard, etc. 2 | 3 | ## Preamble 4 | 5 |We asked our employees where they want to work:
— Dan Price (@DanPriceSeattle) March 15, 2022
7% want to go to the office full time
31% want an office-remote hybrid
62% want to work only from home
So I told them: sounds great. Do whatever you want. As a CEO, what do I care? If you get your work done, that's all that matters
6 | 7 |I'll be headlining a webinar organized by @WatSPEEDUW next week on chatGPT, Bard, and its ilk. There's still time to register! https://t.co/cUV9qg5QVv
— Jimmy Lin (@lintool) February 8, 2023
8 | 9 | ## On Prompt Engineering 10 | 11 |As part of preparing for this event, I've gathered some opinions and predictions that I'm willing to express "on the record". I'll be sharing my thoughts in subsequent tweets. Let's see how they age.
— Jimmy Lin (@lintool) February 8, 2023
12 | 13 |My (contrarian?) take: prompt engineering is programming in natural language. We've tried this before, with attempts dating back decades. Recent advances do not change the fact that natural languages are ambiguous, imprecise, under-specified, highly contextual, etc.
— Jimmy Lin (@lintool) February 8, 2023
14 | 15 |What's going to happen is that prompts will become increasingly stylized, with atoms (tokens) acquiring precise semantics, a rigid but semantically precise means of combination, and a similarly constrained means of abstraction. In other words...
— Jimmy Lin (@lintool) February 8, 2023
16 | 17 |Prompts will become yet another programming language. This is inevitable for anything other than "casual" use.
— Jimmy Lin (@lintool) February 8, 2023
18 | 19 |Yes, the prompt programming language will likely be closer to natural language, easier to learn, etc. but it will remain a technical skill that's non-trivial to learn.
— Jimmy Lin (@lintool) February 8, 2023
20 | 21 |An analogy would be legalese, which *is* natural language, but has acquired many rigid structures that have precise meanings and are often incomprehensible to non-experts (and you go to law school to become proficient).
— Jimmy Lin (@lintool) February 8, 2023
22 | 23 |This perspective does not preclude the Copilot-like scenario where the programmer interactively builds applications with a LLM, starting from a legalese-like prompt programming language that generates code, which the human then refines. That's the future I envision.
— Jimmy Lin (@lintool) February 8, 2023
24 | 25 | ## On Hallucinations and Toxic Content 26 | 27 |And in fact, Knuth articulated this vision nearly four decades ago... it's called Literate Programming https://t.co/eZ3yRB4U1J and technology has sufficiently caught up that we may finally be able to realize it.
— Jimmy Lin (@lintool) February 8, 2023
28 | 29 |My (contrarian?) predictions on ChatGPT, Bard, and its ilk: Regarding the two biggest problems today, (1) hallucinations and (2) toxicity, the first will be transient (i.e., solved relatively shortly) and the second will be perpetual (i.e., will never be solved). Rationale:
— Jimmy Lin (@lintool) February 8, 2023
30 | 31 |(1) the hallucination problem is technical and thus solvable (already many promising directions, e.g., retrieval augmentation, attribution techniques, etc.). Pretty soon we'll be able to accurately probe model output to trace back exactly where each token came from.
— Jimmy Lin (@lintool) February 8, 2023
32 | 33 |LLMs will soon be able to synthesize content from source material (whether pretrained data or retrieval augmentation) with perfect fidelity. This does not address the problem that untruthful content exists (e.g., misinformation) - but the situation will be no worse than today.
— Jimmy Lin (@lintool) February 8, 2023
34 | 35 |It will always be possible to coax untruthful content out of LLMs (for humans, we call this writing fiction), but in "normal" (non-adversarial) use, hallucinations won't pose a problem.
— Jimmy Lin (@lintool) February 8, 2023
36 | 37 |(2) the toxicity problem will never be solved in an enlightened liberal society, because it's not a technical problem, but rather a matter of human nature.
— Jimmy Lin (@lintool) February 8, 2023
38 | 39 |The InstructGPT paper asks the poignant question of "who are we aligning to?" The answer is, of course, the annotators. We can make LLMs increasingly aligned with the values that most people hold (e.g., racism is bad, child porn is bad), but it is impossible to do better...
— Jimmy Lin (@lintool) February 8, 2023
40 | 41 |For the simple reason that *humans* generate content (i.e., express opinions) that other humans find toxic, and we tolerate this as a society (within bounds)... it's called free speech.
— Jimmy Lin (@lintool) February 8, 2023
42 | 43 |LLMs will be increasingly consonant with shared social norms, and, pretty soon, in "normal" (non-adversarial) use, toxic content won't pose a problem.
— Jimmy Lin (@lintool) February 8, 2023
44 | 45 |However, there exist plenty of opinions (on any controversial topic) that are within the bounds of socially acceptable discourse, but some may find objectionable. It'll be impossible to prevent LLMs from generating content that at least some find distasteful.
— Jimmy Lin (@lintool) February 8, 2023
46 | 47 |DAN (Do Anything Now) and other "jailbreaks" are equivalent to recording your racist uncle ranting after a few drinks, but your racist uncle is (most likely) a functioning member of society and observes social norms in most situations. It'll be like that for LLMs.
— Jimmy Lin (@lintool) February 8, 2023
48 | 49 |One major difference, though: humans are accountable for their words, i.e., "freedom of speech, not freedom from consequences". But it is impossible for LLMs to be accountable for their output.
— Jimmy Lin (@lintool) February 8, 2023
50 | 51 |Instead, we'll take the organizations (or individuals) behind the LLM to task - exactly as what happens today. For example, a company expresses a distasteful opinion, activists organize boycotts.
— Jimmy Lin (@lintool) February 8, 2023
52 | 53 | -------------------------------------------------------------------------------- /paranoid.md: -------------------------------------------------------------------------------- 1 | # Be Paranoid! (Someone is working on your idea right now!) 2 | 3 | Initial post: November 15, 2019 4 | 5 | First off: of course I don't mean it in a pathological [DSM](https://www.psychiatry.org/psychiatrists/practice/dsm) sense. 6 | So seriously, save your judgmental "oh it must be terrible to be your student" derision and go back under the bridge whence you came. 7 | Of course when I say it to my students it is couched within a broader context, and they understand that context. 8 | _You don't_, because you read my slogan in a tweet. 9 | I mean what I say in the "only the paranoid survive" Andy Grove sense and the "customer obsession" Jeff Bezos sense. 10 | 11 | With that aside, back to the slogan. 12 | First, let me illustrate, (pseudo-)mathematically, why this statement is likely to be true when uttered to a broad audience. 13 | Hinton estimates that [there are around 10k graduates students in China "doing neural nets"](https://youtu.be/Mqt8fs6ZbHk?t=124). 14 | Assume that's a reasonable estimate. 15 | That's just in China. 16 | So the entire worldwide population of "people doing neural nets" is likely a multiplicative factor of that. 17 | Let's then consider the [pigeonhole principle](https://en.wikipedia.org/wiki/Pigeonhole_principle): How many distinct novel "ideas" do you think there are on the frontier of deep learning? 18 | Are there 10k? 19 | If there are less (which I believe), then a collision is inevitable. 20 | Even if there are more than 10k (even way more), there is a [birthday paradox](https://en.wikipedia.org/wiki/Birthday_problem) effect when referring to an audience. 21 | Of course ideas aren't discrete and countable, and of course the distributions of ideas and effort aren't uniform. 22 | This isn't meant to be a formal proof, just a [prima facie](https://en.wikipedia.org/wiki/Prima_facie) argument about the veracity of the claim. 23 | 24 | Now let's consider the objections. 25 | 26 | Okay, your solution is simple. 27 | Don't work on deep learning. 28 | Knuth [says this](https://twitter.com/lintool/status/1057722875942068225). 29 | Yes, if you're Knuth you can work on anything you want. 30 | If you don't want to work on deep learning, that's of course completely fine. 31 | However, consider this: often, the reason why a particular approach is popular is usually _because it works_. 32 | Today, we if encounter someone who refuses to adopt data-driven techniques and stubbornly insists on _only_ writing rules, we'd think that's kinda silly. 33 | It is likely that in a few years, a categorical resistance to deep learning would seem just as silly. 34 | 35 | I'll provide a bit of historical context: the previous major paradigm shift in NLP was from rule-based to "statistical" approaches in the 1990s [[1](#footnote1)]. 36 | Yes, there were researchers back then who simply refused to count tokens and compute likelihoods. 37 | I know because I lived through those years (as a student). 38 | Today, most people would find that laughable. 39 | 40 | Oh, there's also a practical matter: You'll struggle getting your *ACL papers accepted. 41 | 42 | Next, the old, trite, uninspired retort: "well, if you're in danger of being scooped, you should work on a more interesting problem." 43 | I argue the exact opposite: many people are working on a problem _precisely_ because it's interesting. 44 | Or useful, impactful, etc. 45 | It is this reason why DL/NLP/etc. has advanced so quickly: because people quickly pick the low hanging fruit (tasty!), rush to ask all the obvious questions, quickly determine the answers, and then tell everyone else about it (what you call flag planting derisively I call rapid dissemination of knowledge). 46 | In this way, the research frontier expands rapidly, to everyone's benefit. 47 | It feels absolutely exhilarating to be _on_ this frontier, to be working on something that boatloads of people care about. 48 | It feels absolutely intoxicating to sit atop a leaderboard, knowing that you and your colleagues, _at that moment_ are the best in the world that this very thing. 49 | 50 | But the downside is that you might get scooped. 51 | Hence the slogan. 52 | 53 | You're not in danger of getting scooped? 54 | That doesn't mean you're working on an interesting problem. 55 | It may just mean that you're working on a problem no one cares about. 56 | 57 | Many of the remaining retorts I've heard could be characterize as _hubris_ (defined by [Merriam-Webster](https://www.merriam-webster.com/dictionary/hubris) as "exaggerated pride or self-confidence"). 58 | 59 | "I work on something that I'm uniquely suited for...", "... that only I can do", "... that plays to my strengths", or something along those lines. 60 | Oh really? 61 | You must have a high opinion of yourself. 62 | You may truly be unique, or it's just _hubris_. 63 | As Bill Gates famously said, [in China when you are one in a million, there are thirteen hundred people just like you](https://books.google.com/books?id=CfHCBUepsXIC&pg=PA353&lpg=PA353&dq=%22one+in+a+million%22). 64 | 65 | "I like to take my time to think", "... look further ahead", "... work on really difficult problems", or something along those lines. 66 | Oh really? 67 | The rest of us are just running around like headless chickens, and our work is no more principled than [Brownian motion](https://en.wikipedia.org/wiki/Brownian_motion)? 68 | You can think two steps ahead of everyone else without rolling up your sleeves and actually working on step one? 69 | Wow, that's amazing. 70 | Perhaps you're just so insightful that you see something all of us have missed. 71 | It may be true, or it's just _hubris_. 72 | 73 | The final cluster of objections is what I would characterize as "intrinsic motivation": 74 | instances include, for example, "for the simple joy of discovery", or "I'm not paranoid because I don't care about precedence, publications, h-indexes, citations, etc.", or "I'm not doing this for external validation". 75 | In some cases, these sentiments are expressed by people who are in the enviable position of not needing to care, for example, post-tenure academics. 76 | This strikes me as disingenuous if offered as serious advice, i.e., "_you_ are in a position where you don't need to run the publication rat race, but advising students that publication count doesn't matter _at all_"? 77 | When these sentiments are expressed, for example, by pre-tenure academics, I really hope they are cognizant of the consequences of their approach (on promotion and tenure decisions, etc.). 78 | I've heard many times: "if this tenure thing doesn't work out, I'll just do _X_, but until then I'll do what I do". 79 | And they truly mean it. 80 | I genuinely admire such individuals and wish them the best of luck. 81 | 82 | Thomas Kuhn talks about paradigm shifts followed by ["normal science"](https://en.wikipedia.org/wiki/Normal_science). 83 | Clayton Christensen talks about [sustaining innovating vs. disruptive innovative](https://en.wikipedia.org/wiki/The_Innovator%27s_Dilemma). 84 | If you think you can bring about a paradigm shift and disrupt the status quo, then, sure, you're less likely to be scooped. 85 | I truly look forward to being enlightened by your brilliance when you change the world. 86 | Maybe it'll happen, or it's just _hubris_. 87 | 88 | While you are thinking "your deep thoughts", I'll be here, grinding away, [poking tiny new frontiers in knowledge, bit by bit](http://matt.might.net/articles/phd-school-in-pictures/). 89 | Perhaps it is through this process that I will stumble upon upon some brilliant, revolutionary idea. 90 | 91 | _Nah_, that's just _hubris_. 92 | 93 | I end this essay noting, with great relish, that even Albert Einstein [was nearly scooped](https://en.wikipedia.org/wiki/Relativity_priority_dispute). 94 | 95 | **Update (2020/05/18):** I submit as supporting evidence, the following Exhibit A. 96 | 97 |tl;dr - the disruption from ChatGPT, Bard, LLMs will be mostly transitory, and we'll reach a (different) equilibrium relatively soon.
— Jimmy Lin (@lintool) February 8, 2023
98 | 99 | **Update (2024/08/04):** Documenting another "real-life scoopage" related to work from my group, ["LLMs Can Patch Up Missing Relevance Judgments in Evaluation"](https://arxiv.org/abs/2405.04727), by Upadhyay et al., posted on arXiv May 8. Reaction: 100 | 101 |"Be Paranoid! Someone is working on your idea right now!" was controversial #nlproc https://t.co/xyECGcMCch I submit as further evidence: two papers, both accepted to #acl2020nlp on same basic idea (BERT early exits). Ours: https://t.co/junzZZybZw Theirs: https://t.co/W8jVfpKp4b
— Jimmy Lin (@lintool) May 18, 2020
102 | 103 | To which I responded: 104 | 105 |It's great to see more work in this area, but don't we already know that (less powerful) LLMs can produce (soft) graded relevance labels on even less data @ehsk0 @lintool?https://t.co/6lthbvBX1B https://t.co/r3m7im3MBr pic.twitter.com/NNQCk1aDLo
— Sean MacAvaney (@macavaney) May 9, 2024
106 | 107 | And, just a day later, a similar paper was posted on arXiv: 108 | 109 |Well, the short honest answer is... we got scooped by you et al. This thread of work had been languishing for a long time here - finally was able to push it out.
— Jimmy Lin (@lintool) May 9, 2024
110 | 111 | The paper ["Can We Use Large Language Models to Fill Relevance Judgment Holes?"](https://arxiv.org/abs/2405.05600) by Abbasiantaeb et al. was posted on arXiv May 9. 112 | Yup, that's a difference of _one day_. 113 | 114 | ## Footnotes 115 | 116 | [1] I put "statistical" in quotes because that was the term of art back then, but "statistical" is better characterized as "data-driven" in today's parlance. 117 | -------------------------------------------------------------------------------- /micropublications.md: -------------------------------------------------------------------------------- 1 | # Leaderboard Micropublications: A Proposal 2 | 3 | Initial post: December 2, 2019 4 | 5 | Our field (more narrowly, *ACL, and more broadly, human language technologies) has been blessed with explosive popularity and tremendous amounts of attention from students, industry, and even the media. 6 | This is undoubtedly an exciting area to be working in! 7 | 8 | However, the downside is that our existing models of knowledge dissemination (i.e., publications) are on the verge of collapse, both from sheer volume, different styles of research, and different norms for sharing results. 9 | Although there is no agreement on how to address our woes, there is little doubt that the status quo is unsustainable. 10 | As they say, something's gotta give. 11 | 12 | I'd like to propose a constructive, concrete suggestion that might help address the deluge: leaderboard micropublications. 13 | 14 | ## Huh? 15 | 16 | In short, a leaderboard micropublication is a short paper that is tied to a particular entry on a leaderboard. 17 | Recognizing that there are many different possible types of micropublications, which is already an established genre [[1](#footnote1)], for rhetorical convenience (but at the risk of some confusion) I'll simply use the term micropublication in this post. 18 | By virtue of a tight coupling with a leaderboard, a micropublication can be very short and information dense, making it easy to write and to evaluate. 19 | Much of the prose in a traditional research paper that goes into establishing context is no longer necessary, since the shared context is captured in the leaderboard itself. 20 | A micropublication has methods, results, and discussion, and that's pretty much it. 21 | 22 | For concreteness, I offer an example micropublication co-authored with Rodrigo Nogueira for the [MS MARCO passage retrieval task](http://www.msmarco.org/leaders.aspx) describing our [docTTTTTquery technique](https://cs.uwaterloo.ca/~jimmylin/publications/Nogueira_Lin_2019_docTTTTTquery.pdf). 23 | Consider this as a starting point to begin the conversation. 24 | 25 | How can a micropublication be so short and yet have something to offer? 26 | 27 | A micropublication doesn't need an introduction. 28 | It doesn't need to define the task. 29 | It doesn't need to motivate the task. 30 | Whether the task is interesting, meaningful, well-designed, etc. lies outside the scope of the micropublication itself. 31 | In nearly all cases, the developers of the leaderboard already have a separate paper describing the motivation of the task, the construction of the dataset, the choice of evaluation metrics, etc. 32 | There is no need to repeat those arguments. 33 | 34 | A micropublication doesn't need an explicit related work section. 35 | It only needs to reference the previous work it directly depends on. 36 | It doesn't need a long list of references for signalling purposes. 37 | The "background" related work? 38 | Well, that's all the other entries and their corresponding micropublications on the leaderboard! 39 | 40 | A micropublication doesn't need a future work section or a conclusion. 41 | Future work? 42 | Get a higher position on the leaderboard! 43 | Wait until the next micropublication! 44 | 45 | A micropublication only needs to focus on the methods, contrastive results, and discussion. 46 | Furthermore, the methods can exclusively focus on things like model design, training regime, etc. 47 | Other aspects of the evaluation design, like description of the dataset, evaluation metrics, etc.? 48 | Well, that's already defined in the leaderboard! 49 | While the "main result" of the micropublication is captured in the leaderboard entry itself, authors can (and should) discuss contrastive conditions, ablation experiments, and the associated analyses that contextualize a proposed approach. 50 | 51 | ## How can this help? 52 | 53 | How can micropublications improve the dissemination of knowledge? 54 | 55 | **Easier to write, easier to evaluate.** 56 | Many papers submitted to the standard venues today _are_ already micropublications, just wrapped in lots of "fluff". 57 | Let's just dispense with all the fluff. 58 | Explicitly acknowledging this type of contribution will make such papers easier to write and easier to evaluate. 59 | 60 | Easier to write? 61 | Authors won't need to justify the task, pay homage to the influential (but only marginally related) papers, or engage in any other signalling that's often necessary to get a paper accepted. 62 | The authors can dive directly into exactly what they did. 63 | 64 | Easier to evaluate? 65 | For a micropublication, there is a ready pool of qualified reviewers... the organizers as well as the other participants on the leaderboard! 66 | They are well qualified because they have likely thought about the problem deeply already, and would be well-equipped to recognize innovative ideas. 67 | Furthermore, reviewers don't need to do an in-depth literature search to verify the SOTA claims of the authors — the results are right on the leaderboard! 68 | 69 | Thus, micropublications can improve the overall publication ecosystem by increasing the efficiency with which the community evaluates a subset of its research output. 70 | 71 | **Micropublications are compatible with blind review.** 72 | So long as the leaderboard accepts anonymous submissions (in the case of MS MARCO, they do), it would be possible to couple the leaderboard with a system like OpenReview to support blind review. 73 | 74 | **Micropublications are compatible with researchers who rapidly iterate via preprints.** 75 | I think this is self-evident and needs no elaboration. 76 | 77 | **Recognizing a different category of contribution.** 78 | To be clear, I am not suggesting that micropublications receive any sort of preferential treatment. 79 | And I'm definitely not suggesting that micropublications replace any existing publication types; researchers can continue writing the same types of papers that exist today. 80 | All I'm suggesting is that micropublications be recognized as a (new) distinct category of contribution. 81 | We already have full papers, short papers, demo papers (and other categories, depending on the conference). 82 | Let's add micropublications as another. 83 | 84 | ## Potential Objections and Unresolved Issues 85 | 86 | The purpose of this post is to begin a conversation. 87 | I don't have all the answers, and right off the bat I can anticipate a number of questions and potential objections. 88 | I'll try to address some of them. 89 | 90 | **Is it meaningful?** 91 | I use this as a catch-all for a host of related questions: 92 | Is the task captured by the leaderboard meaningful? 93 | Are there data artifacts or biases in the data? 94 | Do the metrics reflect human preferences? 95 | Are we overfitting? 96 | Yes, these are all very important questions, but they are orthogonal to my micropublication proposal. 97 | A traditional paper needs to contend with all these issues also, so a micropublication does not make things worse. 98 | 99 | In fact, I would argue that a micropublication puts everyone on more equal footing, since the reviewers are forced to evaluate only what's in front of them. 100 | Imagine a scenario where two papers, _A_ and _B_ present comparable, SOTA-achieving results on some leaderboard task. 101 | By bad luck, paper _A_ has a reviewer who questions the very premise of the task, while no reviewers of paper _B_ do. 102 | Assume that, roughly, everything else about the quality of the two papers is equal. 103 | Because the two papers are not handled by the same area chair, there is no "normalization", and thus paper _B_ is accepted while paper _A_ is rejected. 104 | At least to me, the papers were not treated equitably since they were subjected to different reviewing standards. 105 | With micropublications, this would not happen. 106 | A reviewer can't question the premise of the task in a micropublication; it's simply outside the scope of evaluation. 107 | 108 | But yes, we should absolutely discuss all the questions raised above. 109 | However, those questions are orthogonal to doing well on the task _as defined_. 110 | In fact, the burden of answering those questions should be placed on the leaderboard organizers, and likely those arguments are better made via traditional publication types. 111 | The leaderboard organizers, with a behind-the-scenes view, are far better equipped to determine, for example, if new entries just overfitting. 112 | (And if the answer is yes, the leaderboard should be shut down.) 113 | 114 | My feeling is that since there are so many leaderboards out there today competing for our attention, they are themselves under the selective pressure of being interesting, relevant, meaningful, etc. 115 | Thus, it might make sense to let everyone "vote with their feet"; researchers will, over time, gravitate towards tasks that interesting, fun, challenging, easy to make progress in, etc. 116 | But on the flip side, leaderboard popularity is likely to suffer from a [Matthew effect](https://en.wikipedia.org/wiki/Matthew_effect). 117 | Furthermore, tasks and datasets proposed by a few organizations automatically gain more cachet than others. 118 | 119 | However, this proposal is not designed to critique or change the current research culture around leaderboard chasing. 120 | The problem I'm trying to address is: given this style of research, can we at least more efficiently evaluate the outputs? 121 | 122 | **How should micropublications be evaluated?** 123 | This leads to the next important point: micropublications need to be evaluated differently. 124 | In fact, building on the "easier to evaluate" theme raised above, the typical *ACL evaluation form could be dramatically simplified. 125 | Gone are all the questions about scope (does this work belong in *ACL?). 126 | Gone are all the questions about meaningful comparisons and related work: check the leaderboard! 127 | The question about "soundness/correctness" should be rolled up with the standard *ACL question about originality. 128 | In fact, stripping away all these unnecessary elements allows the reviewer to focus on what I propose to be the most important criterion: originality. 129 | To wit, I've taken the standard wording and rephrased it in this context: 130 | 131 | + 5 = Innovative: Highly original and significant new technique, approach, or insight. 132 | + 4 = Creative: An intriguing technique or approach that is substantially different from previous entries. 133 | + 3 = Respectable: A nice research contribution that represents a notable extension of previous entries. 134 | + 2 = Uninspiring: Obvious, or a minor improvement on previous entries. 135 | + 1 = Significant portions have actually been done in previous entries. 136 | 137 | This focus, for example, would allow us to accept papers that don't achieve SOTA, but demonstrate some remarkable insight, and reject papers that achieve SOTA, but in an uninteresting way. 138 | At a high level, I think this is what we'd want? 139 | The [docTTTTTquery example](https://cs.uwaterloo.ca/~jimmylin/publications/Nogueira_Lin_2019_docTTTTTquery.pdf), trying to be as objective as possible, I would rate between a "2" and a "3". 140 | While using T5 for document expansion is perhaps an obvious extension of the original doc2query model, it's more than a "minor improvement" since we report a 6 point gain. 141 | 142 | Once again, the proposal here simply aims to begin the conversation. 143 | I'm not totally committed to this rubric, but the broader point remains: we need different evaluation criteria for micropublications. 144 | 145 | **How do I demonstrate that an approach is general?** 146 | Reviewers like to see a technique applied to multiple tasks to demonstrate its generality. 147 | I see no problem with this. 148 | A micropublication can certainly contain results for multiple leaderboards, although with the increased length we might have to call these millipublications. 149 | 150 | **What if the increased efficiency of evaluating micropublications is offset by more micropublications to evaluate?** 151 | This is an interesting point. 152 | My rebuttal is that all micropublications must be associated with actual leaderboard submissions, so if the number of submissions increases, it means that micropublications have enabled the community to iterate and share findings faster? 153 | I'd consider this a net win. 154 | 155 | **The overall benefit to the broader publication ecosystem is dependent on the number of research outputs that can be evaluated as micropublications.** 156 | Agreed. 157 | But based on the frequent complaints I hear about leaderboard chasing on social media, the savings seem like they would be substantial? 158 | 159 | **Formal recognition and "worth".** 160 | Students still need to get jobs, faculty still need to be promoted. 161 | None of this harebrained scheme is going to work until micropublications are valued by the powers that be (hiring committees, tenure and promotion committees, letter writers, etc.). 162 | Of course. 163 | This is a problem faced by any new attempt to do something different. 164 | There are the standard answers: 165 | Let's try to convince an established reputable venue to "take on" micropublications as a track or new category of contribution. 166 | Let's try to create a new journal devoted to micropublications. 167 | Over time, "the market" will converge on the value or "worth" of micropublications, relative to, say, full and short papers. 168 | 169 | However, that's putting the cart before the horse. 170 | Nothing will happen until at least enough people think this is a good idea. 171 | What do you think? 172 | 173 | ## Footnotes 174 | 175 | [1] For example, there is a journal actually called [microPublication Biology](https://www.micropublication.org/); in computer science, Dittrich proposed the concept of [PaperBricks](https://arxiv.org/abs/1102.3523) in 2011. 176 | 177 | ## Acknowledgments 178 | 179 | I want to thank the following people for feedback on various drafts of this post: Ashutosh Adhikari, Martin Gauch, Yoav Goldberg, Rodrigo Nogueira. 180 | Feedback, however, does not imply endorsement. 181 | -------------------------------------------------------------------------------- /writing-a-research-paper-with-me.md: -------------------------------------------------------------------------------- 1 | 2 | # Writing a Research Paper With Me 3 | 4 |Same happened to us. Our arXiv will be out soon 😂
— Mohammad Aliannejadi (@maliannejadi) May 9, 2024
5 | 6 | I can't stress this enough. 7 | Start early. 8 | Most students grossly underestimate the time they take to write. 9 | 10 | ## Plan to Run Experiments Twice 11 | 12 | Plan to run your experiments twice. 13 | The first time is to figure out what experiments you need to run, the second time is to actually run them for inclusion in the paper. 14 | 15 | Why? 16 | Because when you run your experiments for the first time, you don't know the relevant experimental factors. 17 | You may have some ideas, but you don't know definitively, because otherwise you wouldn't need to run the experiments. 18 | Inevitably, you'll discover something you didn't know before, which is the whole point of research. 19 | A set of experiments will inform a previous set of experiments, which will compel you to go back to the first experiment to verify. 20 | In these cases, you might be comparing experiment results from today with a condition you tried some time ago (maybe a week, maybe months). 21 | Lots of things might have changed in the meantime (the code has changed slightly, for example). 22 | Thus, it's better to just run through all the experiments again. 23 | 24 | For the second time through, you should be able to script all your experimental conditions: 25 | if you can't it is likely that you don't understand the design or parameter space well enough. 26 | Make sure your code is able to execute all the conditions through command-line parameters or an external config file. 27 | Preliminary exploration is okay, but if you intend for a set of results to make its way into a paper, _never_ run those experiments by code modifications (e.g., commenting out a line) — turn them into command-line options. 28 | Otherwise, it will be virtually impossible (especially over time) to keep track of which code led to which results. 29 | 30 | ## Don't Procrastinate on Writing by More Coding 31 | 32 | Most students find it easier to build implementations (and run experiments, etc.) than to write papers. 33 | So, they put off writing the paper and continue coding, even when they've collected enough experimental results for their paper. 34 | Recognize that this happens (it's okay, I see it all the time), but resist the urge. 35 | 36 | ## Writing Order 37 | 38 | ### Start With Results 39 | 40 | Start writing your paper from the results section. 41 | In fact, start with the results figures or tables. 42 | Usually, we would have discussed the contribution claims of the paper well in advance. 43 | Make sure the results support those claims. 44 | For example, if your claimed contribution is to demonstrate that technique *X* improves over competitive baselines for task *Y*, there'd better be a results table 45 | (or figure) that shows technique *X* vs. competitive baselines. 46 | Typically, there will be a main set of results and a set of secondary results that flesh out your experiments, e.g., exploring different parameter settings, ablation experiments where you examine the impact of different components, etc. 47 | Focus on the main results first. 48 | 49 | Even if you don't have all the experimental results yet, you can still set up the results tables (with missing entries) or put in a stub for a figure. 50 | As your experiments progress, it's a matter of filling in values in an empty table, or adding another line in a plot... 51 | Having these placeholder will allow you to proceed concurrently on the prose. 52 | 53 | See my tip on [this page](writing-pet-peeves.md) under "Walk your reader through the results". 54 | 55 | ### "Middle Sections" 56 | 57 | Once you're done with the results section, write the experimental 58 | setup section. This includes things like what dataset you used, the 59 | machine/cluster you ran your experiments on, training regime, 60 | hyperparameter settings, etc. 61 | 62 | Once you're done with the results section, write the methods 63 | section. That is, how your model actually works, the architecture of 64 | your system, etc. 65 | 66 | Then write your related work. I've noticed over the years that systems 67 | and DB papers tend to have related work late in the paper, e.g., after 68 | results, but other fields (IR, NLP, KDD, etc.) tend to have related 69 | work right after the intro. So I usually follow these community 70 | standards. In the latter case I usually call the section "Background 71 | and Related Work", directly following the introduction. 72 | 73 | Save the introduction for last. 74 | 75 | ### Save "the Story" for Last 76 | 77 | The introduction is 78 | where you "tell the story" and attempt to convince the reviewer why 79 | they should care, relating the work to the broader context of what 80 | people in the field consider important. 81 | 82 | At the end of the introduction, I typically include an explicit contributions paragraph, e.g., "We believe this work makes the following contributions..." 83 | Writing the contributions is difficult, but a necessary part of academic writing. 84 | It is absolutely essential that claimed contributions are supported (empirically or otherwise) in the paper. 85 | Typically, these claims evolve with the rest of the paper: for example, initially, we expect our experimental results to support one assertion, but then the results are a bit unexpected, so we need to tweak our claim. 86 | Or, alternatively, we would like to strengthen our claim or make a slightly different assertion, so this leads running slightly different experiments as support. 87 | It is not unusual for these changes to happen in a few rounds before we finally converge on a set of claims we're happy with and the evidence that supports those claims. 88 | 89 | ## My Input 90 | 91 | If you're a junior student, I often write the introduction for you, because that's the hardest part to write (see above). 92 | As you gain experience, I expect you to be able to start telling your own story. 93 | This is the hardest skill to master. 94 | 95 | Typically, I will communicate with you about a rough time period where I will set aside time to help you with your paper. 96 | My input is usually in the form of one of the following: (1) reading and giving feedback, (2) substantially revising your paper directly, and (3) meta-writing. 97 | 98 | In some cases, especially if you are a junior student, I may end up completely rewriting your paper such that the final product may contain the same ideas, algorithms, models, results, but with a completely different expression. 99 | Realize that this is the most valuable part of your interactions with me: this is where you get my undivided attention working on *your* problem, helping you refine and improve your work. 100 | These interactions are also where you learn to write research papers. For example: 101 | 102 | + I help you explain your complex algorithm in the simplest possible way that helps the reader understand its underlying intuitions. 103 | 104 | + I help you untangle your experimental results into a series of clear findings that reveals insights about your system or technique. 105 | 106 | These cases focus on how to clearly articulate your ideas, so it is important that your draft already contains the "raw ingredients". 107 | For example, there should already be a rough attempt at explaining the algorithm or model in a sufficient level of detail. 108 | All the experimental results should be present for analysis. 109 | I call this the "fodder" that serves as the input to refined prose. 110 | 111 | An implication of this is that it is in your best interest to get everything in the draft before I start to work on it. 112 | While I'm working on it, also be available on Slack to answer questions on missing details. 113 | If there is "not enough" for me to work on or if you are not responsive, I will context switch and work on other things. 114 | I try to minimize my context switches so it might be a while until I return to your paper. 115 | 116 | For more senior students, instead of actually editing your paper, I often do what I call "meta-writing", which is writing what you should write about—directly into your draft (e.g., on Overleaf). 117 | This often appears in red font, so it's difficult to miss. 118 | Examples of meta-writing include reorganization suggestions, e.g., "you should move the model description here to improve the flow", 119 | missing logical chains of reasoning, e.g., "here you're missing the argument that recall is the most important metric for this task", 120 | specific requests, e.g., "let's cherry pick a better example here", etc. 121 | When I do this, I leave the implementation of the suggestions up to you, i.e., to execute a suggestion into fluent prose that flows smoothly with the surrounding text. 122 | Often, my meta-writing is fragmentary, frequently not even complete sentences. 123 | Sometimes, my meta-writing might contain colorful language, like, "here you want to say, this is stupid", and "here, you want to brag about your results". 124 | Obviously, you should turn these suggestions into academically acceptable writing. 125 | 126 | The proportion of time I spend substantially revising your paper will decrease as you gain experience. 127 | For more senior students, I might mostly focus on meta-writing and leave the excecution to you. 128 | In these cases, I might edit your introduction and that's it. 129 | For students close to graduating and postdocs, I might only read a close-to-final draft to offer comments with no "hands-on-keyboard" work. 130 | For postdocs, I might not even be a co-author. 131 | 132 | If you are unclear what type of help you are going to get from me, **ask**! 133 | I will be clear and up front to you and say things like, "I'll have all of Thursday morning to devote to your paper" or "I have too many other deadlines. I'll write your introduction but otherwise you are on your own." 134 | 135 | ## After My Input 136 | 137 | ### Accepting/Rejecting My Edits: Comparing Before and After 138 | 139 | If I am editing your paper directly, I will use the track changes feature in Overleaf: 140 | 141 | **Don't just blindly accept my edits!!** 142 | 143 | Instead, _consider_ each edit deliberately before you accept... or reject! 144 | It is important that you compare "before" and "after", for three reasons: 145 | 146 | 1. I might have messed up your text. For example, I might have described your algorithm wrong. 147 | 148 | 2. Comparing "before" and "after" is critical to your learning experience and an important component of my mentoring. 149 | This is a case study approach: you will see how I have better re-expressed your ideas, and over time, your own ability to do so will improve. 150 | If you have questions about why I have expressed something in a certain manner, ask and I will be happy to explain. 151 | 152 | 3. I may not have actually improved the writing. 153 | Senior students come up with better ways of "telling the story" than I do. 154 | I will engage with you and vigorously debate the merits of alternative approaches. 155 | I will certainly concede if your approach is superior, and I will most likely defer to your own judgment if there are pros and cons to alternative approaches. 156 | I very much welcome these debates and they are satisfying, because it means that I have trained you well. 157 | I am not surprised that you can tell your story better than I can, because, after it, it is *your* research. 158 | 159 | When I say "do a cleanup pass", it's a shorthand for asking you to go through the draft and "process" all the tracked changes in manner described above. 160 | I often "batch up" my edits and will wait until you've done a cleanup before I proceed with more editing. 161 | 162 | ### Smoothing Pass 163 | 164 | After I've worked on your paper, you usually need to do what I call a "smoothing pass". 165 | The amount of smoothing depends on how extensive my edits were and how much I changed. 166 | For example, if I rewrote a paragraph here, you'll need to make sure that the flow from the preceding to the following paragraph remains smooth; you might need to add a transition sentence. 167 | Sometimes I'm missing this because I focused on making local edits; you'll likely have a more accurate global view than I do. 168 | 169 | Note that often my edits trigger the need for changes to propagate non-locally to other parts of the paper. 170 | For example, if I tweak the story in the introduction, then we need to make sure the narrative in the rest of the paper is consistent. 171 | As an another example, if I change a particular phrasing, then we need to make sure the phrasing is propagated everywhere else. 172 | What is it that we're proposing? 173 | A model? Technique? Approach? Method? 174 | Whatever we decide (let's discuss), make it consistent. 175 | That is, don't say in the intro "We propose an approach to..." and then later say, "Our method does this..." 176 | See ["use consistent terminology"](https://github.com/lintool/guide/blob/master/writing-pet-peeves.md). 177 | 178 | ## As the Deadline Approaches 179 | 180 | As the deadline approaches, the magnitude of changes should decrease, because the chances you'll make the paper worse increase. 181 | You've been working hard, haven't been getting enough sleep, you're tired, stressed out, etc. and this makes it likely you'll make a mistake. 182 | Say you've come up with a better way to tell your story, but it's two days before the deadline and it'll require major surgery on the paper (moving sections around, etc.). 183 | _Resist that urge._ 184 | Chances are you'll leave out some major component accidentally (because you're not left with adequate time for proofreading) and end up making the narrative more confusing. 185 | 186 | The truth is that, close to the deadline, you're often agonizing over details that make epsilon difference with respect to acceptance probability. 187 | Do I add this sentence or take it out? 188 | Do I use this phrase in the intro or not? 189 | It likely doesn't matter: the effects of the decisions are far smaller than the variance due to reviewer assignment. 190 | So, don't stress over it. 191 | tl;dr — be careful not to make things worse. 192 | 193 | ## After the Deadline 194 | 195 | Take a break and decompress! 196 | 197 | Retire technical debt. 198 | In the run up to the deadline, it is likely you took a bunch of shortcuts, wrote janky code, etc. 199 | Fix these issues as soon as possible, as technical debt carries with it a high interest rate. 200 | Otherwise: 201 | 202 | **Don't work on the paper any more!** 203 | Resist the urge to run more experiments, tweak the model, etc. 204 | The rationale for this is that you'll end up second guessing yourself needlessly about decisions you _could_ have made in writing the paper. 205 | E.g., "if I had only been able to run this experiment and included it in the paper, it would have a better chance of acceptance!" 206 | Don't to this! 207 | You'll just end up stressing yourself out needlessly. 208 | The paper is done; it's out of your hands. 209 | Wait for the review process to take its course, and when the notification comes, _then_ think about next steps. 210 | 211 | Thus, I advise my students to have multiple projects: once you've submitted the paper, context switch to the other project. 212 | 213 | -------------------------------------------------------------------------------- /writing-pet-peeves.md: -------------------------------------------------------------------------------- 1 | # Writing Pet Peeves 2 | 3 | ## Organization 4 | 5 | + **Use consistent terminology.** 6 | Use the same words to describe your methods, contributions, etc. throughout the paper. 7 | Don't say "soft alignment between words in phrases in the question and answer" in one place and then talk about "soft semantic matching" if you mean for them to be the same thing. 8 | Especially technical terms that sound similar but are slightly different, like in the example above. 9 | Don't confuse your reader. 10 | If stylistically the prose comes out too repetitive and mechanical, then consider mixing verbal and nominal forms, e.g., "matching at the semantic level" and "semantic matching", but make sure they _clearly_ mean the same thing. 11 | Don't do this, even for things that _you_ think mean the same thing. 12 | For example, "neural seq2seq model", "seq2seq neural network", and "seq2seq neural model" are all roughly interchangeable, but there are nuanced differences in emphasis. 13 | If you mix these references, the reader might infer that you are trying to establish some contrast that is likely not your intention. 14 | 15 | + **Use parallel structure.** 16 | Use parallel structure throughout your paper to help your reader quickly grasp what you are trying to convey. For example: 17 | 18 | - If you describe three models in your methods section, then make sure that in your results section, the tables or figures show results on those same three models, _in the same presentation order_, with _exactly_ the same name (see point about consistency above). 19 | - If you organize your paper in terms of research questions (RQs), you'll present your RQs in the intro. 20 | Make sure in the results section that each RQ is answered, _in the same order_, clearly delineated (e.g., with subsections). 21 | - If you claim four model innovations in your introduction, then your methods section should discuss each innovation _in the same order_; ideally, your experiments will show ablations for each of these innovations. 22 | 23 | It would be ideal if you can extend the parallel structure consistently throughout the entire paper, from abstract to conclusion. 24 | For example, 25 | 26 | - abstract: this paper is about two innovations, A1 and A2; 27 | - introduction: here is why A1 and A2 are important; 28 | - related work: A1 is related to this, A2 is related to that; 29 | - methods: see how A1 and A2 translate directly to M1 and M2; 30 | - results/discussion: I'll show you that M1 and M2 empirically work and M1+M2 work even better; 31 | - conclusion: you should now care about A1 and A2. 32 | 33 | This same idea is articulated in [Mensh and Kording (2017)](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005619) as "Rule 4: Optimize your logical flow by avoiding zig-zag and using parallelism". 34 | 35 | + **Walk your reader through the results.** 36 | Don't just throw up a big table and lots of numbers and expect the reader to figure out what's going on. 37 | A common technique I use is to number the rows and to group semantically similar experimental conditions together, e.g., (1a) and (1b). 38 | Then, you can write something like "the baselines are shown in rows (1a) and (1b)... the effectiveness of our full model is shown in row (2a)... based on this, we can clearly see that our model yields significantly higher precision across all datasets." 39 | 40 | + **Explicit "roadmaps" in prose.** 41 | By this, I mean a paragraph like "In Section 2 we provide background... Section 3 describes our model... Experimental results are presented in Section 4." 42 | These are useless and an absolute waste of space. 43 | Why spend a paragraph describing what I already expect (i.e., all research papers share similar structure) and what I could figure out in two seconds (i.e., I could simply flip ahead and read your section titles). 44 | An exception to this rule is for theses, where it is common to write something like, "In Chapter 3, I present my model for..." 45 | However, what follows is typically a paragraph summary of the chapter, so all considered, a roadmap that spans several paragraphs (or often, a bulleted list) isn't that jarring. 46 | 47 | + **Implicit narrative roadmaps.** 48 | Instead, let your prose implicitly lay out the structure of the paper for your reader. 49 | For example, if I say that my model has three novel features, then in the methods section there should be three subsections, each corresponding to each novel feature. 50 | This is merely an elaboration of the "use parallel structure" guidance above. 51 | 52 | ## Grammar and Style 53 | 54 | + **Possessives.** 55 | In English, "A's B" and "B of A" are both acceptable ways to indicate possession. 56 | This is mostly personal style, but I prefer "B of A" since it reads more natural to me. 57 | So, I would prefer "popularity of dense retrieval" over "dense retrieval's popularity". 58 | The issue in my mind is the scoping of the possessive morpheme: the actual syntactic structure of the latter is "[ dense retrieval ] 's popularity". 59 | As the noun phrase corresponding to the possessor increases in complexity, I find the possessive construction increasingly awkward. 60 | For example, "state-of-the-art dense retrieval models' popularity" is grammatically correct but I find difficult to read. 61 | I would rephrase as "the popularity of state-of-the-art dense retrieval models". 62 | 63 | I encounter this issue frequently with native Mandarin speakers, likely because of 的-constructions, where the A in "A 的 B" can be arbitrarily complex. 64 | For example, "[ I want to go ] 的 place" (word for word gloss; translation: "the place where I want to go") is perfectly grammatical in Mandarin, whereas in English there is no grammatical counterpart using the possessive morpheme. 65 | 66 | + **Use of the term "performance".** Treatments across different sub-disciplines of computer science differ, but "performance" generally refers to measures like latency, throughput, etc. 67 | NLP (and often IR) researchers, however, often use "performance" to refer to "output quality", which annoys me. 68 | This is especially annoying in papers that _actually_ discuss performance (for example, on model compression), because "better performance" is ambiguous between "faster" (e.g., lower inference latency) and "better" (e.g., higher prediction accuracy). 69 | 70 | I like to be as precise as possible. 71 | In IR papers, I typically use "effectiveness" to refer to output quality in the general sense. 72 | In NLP papers, I either use "effectiveness" ("our model is more effective") or "output quality" ("our model produces higher quality output"). 73 | Unless the prose becomes awkward (e.g., when comparing many different metrics that measure different things), I try to be as precise as possible with terms like "accuracy", "precision", "recall", "loss", "likelihood", "BLEU score", "ROUGE score", "perplexity", etc. as appropriate. 74 | (Side note, writing something like "higher BLEU/ROUGE scores" makes a factual statement without getting into a debate on whether higher BLEU/ROUGE scores actually yield better translations/summaries.) 75 | 76 | In all papers, I typically use "efficiency" to refer to what other sub-disciplines call "performance" (since IR folks routinely talk about effectiveness/efficiency tradeoffs; it's well-established parlance), but where ever possible I also like to be precise, e.g., lower query latency, higher query throughout, less training time, etc. 77 | 78 | As a corollary, I tend to avoid phrases like "performs well". 79 | An exception, however, is the verb "to outperform", which I sometimes use, since in English there is no verbal form that conveys more precision, e.g., "to outprecise". 80 | So I might write "our model outperforms the baseline" to either mean "faster" or "better output quality"; often, the sense is clear from context. 81 | 82 | + **Use of superlatives and "hype phrases".** In short, use sparingly - things like "dramatically improves the user experience" - unless it's actually warranted. If you improve precision from 0.4 to 0.8, calling the improvements _dramatic_ might be warranted. A 5% improvement is not. I would let the results speak from themselves. Overuse of superlatives potentially turns off reviewers. 83 | 84 | + **Use of the term "significant".** Related, use the term _significant_ only in the technical sense of statistically significant. 85 | 86 | + **Use of "SOTA".** "State of the art" as a noun is written without hyphens, as in "Our algorithm represents the state of the art". As an adjective, "start-of-the-art" is hyphenated, as in, "These are state-of-the-art results". 87 | Also, don't forget the determiner in the nominal form, i.e., _the_ state of the art. 88 | 89 | + **Use of training and test sets.** It's "training set", "validation test", and "test set". Not "testing set". I know, it's odd. I think the non-parallel construction is just idiosyncratic, since "train set" sounds odd. 90 | 91 | + **"Plural" of words like "performance", "effectiveness", "evidence", "setup", etc.** 92 | The "performance of different models", _not_ "performances of different models". 93 | Same with "effectiveness" and "evidence". 94 | Mistake also appears in forms like "evidences from multiple experiments support our hypothesis" ("evidence" not "evidences"). 95 | Also, it's "experimental setup" (i.e., singular), even if there are multiple experiments. 96 | 97 | + **Continuation of above... you probably don't mean "codes".** If you're talking about source code, the source code of two different programs is still "source code", not "source codes". 98 | Unless you mean something like "Huffman codes"... 99 | 100 | + **Another continuation of above... the case of analysis vs. analyses.** "Analysis" vs. "analyses" is an interesting case. 101 | I personally have a slight preference for analysis, but analyses sounds fine in most contexts. 102 | In an analysis, you might be running multiple trials, tests, ablations, etc., so it might be tempting to use the plural. 103 | However, even if you do run multiple tests to analyze something (e.g., "Is A better than B?"), at least to me, it is more natural to call it an analysis (singular). 104 | What if you're analyzing different things? 105 | For example, "parameter analysis" or "parameter analyses"? 106 | In this specific case, I have a strong preference for the former, even if you're analyzing multiple parameters. 107 | But what if you're analyzing two completely different hypotheses? 108 | For example, "In this analysis/these analyses we examine our proposed hypotheses..." 109 | I think both sound fine to me. 110 | 111 | + **Related work, or, yet another continuation of above.** "Related work", _not_ "related works". "Work" is a collective noun, like "staff". You don't write "research staffs". Wait, you ask: if this is indeed the case, why would you write ["DaVinci is the creator of several famous _works_ of art"](https://twitter.com/mmparker/status/991012314332545024)? Well, that's a completely different context: "work" in "related work" is intended to refer to a body of work collectively, as in "the literature". You don't write "related literatures" (well, except in a very narrow sense, something like "isolated silos of related literatures"). Note that "DaVinci's work" and "DaVinci's works" are both fine, but in different contexts. The first might refer to his body of work collectively; the second focuses on individual pieces. Also, be aware of subject-verb agreement. 112 | 113 | + **Mass vs. count nouns; yes, it's the same issue...** 114 | "Related work" is actually a specific instance of the mass/count dichotomy in English. What makes it confusing is that for many words, both count and mass interpretations are possible. For example, "Over the course of several internships, I have gained experience in deep learning" (_not_ "experiences"). But, "I love having new experiences when visiting different countries". We write, "Preceding sentences form the context of the dialogue" (_not_ "contexts"), but "In different contexts, people may act differently". 115 | 116 | + **Hyphenate compound adjectives.** 117 | "run" + "time" = ??? The answer is [this](https://homes.cs.washington.edu/~jrw12/runtime.html). 118 | More generally, adjectival forms (i.e., [compound adjectives](https://www.grammarbook.com/punctuation/hyphens.asp)) are hyphenated, i.e., "machine learning" but "machine-learned models". 119 | SOTA is a common special case of this. 120 | Hyphenation rules are actually much more complex... see [this guide](https://www.dailywritingtips.com/adverbs-and-hyphens/) for more details. 121 | The Chicago Manual of Style has [an even more complex list of rules](https://docs.google.com/viewer?url=https://github.com/lintool/guide/blob/master/docs/CMS_list.pdf?raw=true). 122 | 123 | + **Be consistent in the capitalization of hyphenated words.** This applies in paper titles and section headings. For words like "multi-aspect" and "cross-lingual", be consistent. Don't have "Multi-aspect" in one place and "Cross-Lingual" in another. 124 | 125 | + **Stay in the same tense.** 126 | Some authors like to describe experiments in the past tense, e.g., "The user received a push notification and clicked on it." Others like to describe experiments in the present tense, e.g., "The user receives a push notification and clicks on it." Both are fine, but stay consistent in the same tense. Don't write, "The user receives a push notification and clicks on it. The click was then recorded by the server." 127 | 128 | + **Tense when describing experiments.** 129 | Related to the above point. 130 | Some authors describe experiments in the present tense, e.g., "We apply our models on three benchmark datasets." 131 | I think this is fine, but personally, I prefer past tense, e.g., "We applied our models on three benchmark datasets." 132 | This makes a sequence like the following sound more natural, e.g., "We applied our models on three benchmark datasets. Based on preliminary experiments, a fourth dataset was discarded because..." 133 | In consistent present tense, the above sequence sounds really odd, "We apply our models on three benchmark datasets. Based on preliminary experiments, we discard a fourth dataset / a fourth dataset is discarded (?) because..." 134 | Writing about experiments in the past tense more accurately mirrors reality; you already ran those experiments, now you're just describing them. 135 | 136 | + **Subject-verb agreement issues.** Common subject-verb (note that in Latex this would be an en-dash, see below) agreement issue: "number of sample increases", not "increase". The verb agrees with the head noun, which is number, singular in this case. 137 | 138 | + **Learn the difference between "that" and "which".** There are already lots of explanations available on the web, so there's no need for me to repeat them: [here's](https://blog.apastyle.org/apastyle/2012/01/that-versus-which.html) a good one. 139 | 140 | + **"E.g." and "i.e."** 141 | "E.g." refers to the Latin expression _exempli gratia_, which means "for example". 142 | "I.e." refers to the Latin expression _id est_, which means "that is". 143 | Both are always written with periods and are _always_ offset by a comma, both before and after. 144 | Correct usage: "I like simple and effective retrieval techniques, e.g., BM25." 145 | 146 | + **Always use the [Oxford Comma](https://en.wikipedia.org/wiki/Serial_comma)**. 147 | 148 | ## Appearance 149 | 150 | + **Aesthetics matter.** 151 | I often take a paper, zoom out in my PDF viewer so the text is unreadable, flip through the pages, and see if it "looks right". 152 | Does it have the right balance of sections, sub-sections, figures, equations, tables, etc.? 153 | Unfortunately, I can't operationalize this into an imperative, other than to say that "appearances count". 154 | 155 | + **Get rid of orphan lines.** 156 | Orphan lines are final lines in paragraphs that are really short, e.g., a single word on its own line that ends a paragraph. 157 | Get rid of them by rewriting the prose slightly. 158 | This is a common space-saving trick. 159 | Never have orphan lines that span page break, i.e., a single word or short phrase that ends a paragraph on a new page. 160 | Latex does this sometimes and it looks terrible. 161 | 162 | + **Opposite of orphan lines.** 163 | If the final line of a paragraph runs all the way to the rightmost edge of the column or page, it looks terrible also because the looks like a run-on with the next paragraph. 164 | Rewrite the text slightly so this doesn't happen. 165 | 166 | ## Citations 167 | 168 | + **NEVER *ever* begin a sentence with a citation.** For example, "[5] explored the use of..." The worst is related work that's simply a collection of such sentences, e.g. "[1] did this... [2] did that... [3] did other". 169 | 170 | + **A citation is not a noun phrase.** For example, "... is shown in [5]" is bad; should be "... is shown by Smith et al. [5]". However, I break this rule myself sometimes, especially if I'm trying to save a bit of space. 171 | 172 | + **Name your proceedings volumes consistently in the bibliography.** 173 | I hate seeing "SIGIR 2015", "Proceedings of SIGIR 2014", "SIGIR'12", and "Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval" all in the same bibliography. (More below.) 174 | The fact that NAACL can't even consistently name its conferences from year to year drives me nuts: it's [NAACL HLT 2018](http://naacl2018.org/) (with a space), before it was [NAACL-HLT 2012](http://mirror.aclweb.org/hlt-naacl12/) (with a dash); back to a space with [NAACL HLT 2010](https://naaclhlt2010.isi.edu/), back to a dash with [NAACL-HLT 2007](http://mirror.aclweb.org/hlt-naacl07/); oh, let's swap the order now with [HLT-NAACL 2006](https://nlp.cs.nyu.edu/hlt-naacl06/), and for good measure let's use a slash in [HLT/NAACL 2004](http://mirror.aclweb.org/hlt-naacl04/); wait, we really wanted a dash in the beginning, with [HLT-NAACL 2003](http://mirror.aclweb.org/hlt-naacl03/). 175 | So the choice as an author referencing papers is inconsistent citations vs. regularizing the name (which is citing a proceedings volume that never existed). 176 | 177 | + **Maddeningly inconsistent volume names... continued...** [SIGIR](https://dl.acm.org/conference/ir/proceedings) can't seem to agree on what to call itself either. 178 | Up until 2008, it was "Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval" - for some reason, capital "R" but lowercase everything else except for "ACM SIGIR". Okay...? 179 | But then, SIGIR 2009 became "Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval". 180 | Still weird cpaitalization, but note the dropping of "annual". 181 | Starting 2015, we finally arrived at "Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval". 182 | Guess someone (_finally_) noticed the wierd capitalization? 183 | (Thankfully, it's been consistent until today.) 184 | 185 | + **Details really matter.** 186 | Take care to ensure the quality of the bibliography, i.e., make sure you get every citation "right". 187 | Why is this important? 188 | I see this as a sign of respect: messing up a citation to me is like messing up someone's name. 189 | When you meet someone face-to-face for the first time, do you try to learn the correct pronunciation of the person's name? 190 | Well, this is just like that. 191 | Whenever I read a paper that cites a work I'm involved in, I get really annoyed if the authors have misspelled a name or otherwise mangled the citation (for example, adding an author that doesn't exist). 192 | Yes, this happens, especially if the authors just download bibtex references from some third-party source without bothering to check their accuracy. 193 | Common issues include inconsistent treatment of middle names (middle initial or no? middle initial with a dot or not?) and not capitalizing parts of titles that should be capitalized (e.g., "llm" instead of "`\mbox{LLM}`"). 194 | Please get it right. 195 | 196 | ## Polish 197 | 198 | + **Fixed-point reading.** This is a technique that I've found to be time consuming but effective in catching typos and final polishing (e.g., for camera-ready submissions). 199 | I named this technique after the [fixed-point combinator](https://en.wikipedia.org/wiki/Fixed-point_combinator). 200 | Here's what I do: 201 | I print out the paper (yes, on dead trees) and read over with a pen in hand. 202 | To make sure I actually read each word (and not miss typos), I _point_ to each word with my pen. 203 | I then enter all corrections into latex. 204 | At first, I catch a lot of typos. 205 | Then I do it again. 206 | I catch fewer typos. 207 | Then I do it again. 208 | And even fewer. 209 | I repeat until I take a pass and find nothing more I wish to change (or the number of corrections is below a threshold _epsilon_). 210 | This typically takes around half a dozen passes for me. 211 | A few details: 212 | 213 | + Somehow for me, printing on dead trees is critical to catching typos. 214 | + I've done this on screen and with an iPad, and it somehow doesn't work quite as well: I print out on dead trees and I still find more typos. 215 | + _Pointing_ to each word is critical, to make your eyes [saccade](https://en.wikipedia.org/wiki/Saccade) to it; otherwise your brain has a tendency to just interpolate and error-correct the typos. 216 | + To combat fatigue and learning effects, in each pass I vary the order in which I read sections: e.g., start with the intro, start from the conclusion, start from the experiments, etc. 217 | 218 | + Here's an interesting way to proofread: [say the text back to you](https://kmicinski.com/writing/2017/09/22/reading-text-back/). 219 | 220 | ## Latex 221 | 222 |:@squarecog to me: dude, you're just a glorified technical writer.
— Jimmy Lin (@lintool) May 23, 2012
223 | 224 |pro·fes·sor — noun
— Zachary Lipton (@zacharylipton) July 13, 2020
***the member of a research lab with the deepest intuitions about how LaTeX \vspace, page breaks, and figure placement work.***
225 | 226 | ### Finer Points of Punctuation 227 | 228 | + [Details matter!](https://twitter.com/aacooper/status/1109678683877310464) 229 | 230 | + Learn the difference between [an hyphen, en-dash, and em-dash](http://www.thepunctuationguide.com/hyphen-and-dashes.html); here's [another guide](https://style.mla.org/hyphens-and-en-dashes/). 231 | 232 | + It's `key--value` as in `key--value` store, not `key-value`. 233 | 234 | + It's `human--computer interaction`. Note that the [Wikipedia article](https://en.wikipedia.org/wiki/Human%E2%80%93computer_interaction) gets it right and [this incorrect URL](https://en.wikipedia.org/wiki/Human-computer_interaction) redirects to the correct article. 235 | 236 | + It's one of my favorite obscure rules, but yes, there are cases when you mix an en-dash and a hyphen _in the same word_! See [this explanation](https://www.thepunctuationguide.com/en-dash.html). In things that we are likely to write, "pre--fine-tuning" is one such word. 237 | 238 | + Footnotes go after the punctuation, as in `End of sentence.\footnote{...}`. Note, there is _no_ space between punctuation and the footnote. 239 | 240 | + There is a difference between end-of-sentence spacing and intra-sentence spacing after a period. For example, `cat vs.\ dog` but `I like cats. She likes dogs.` 241 | 242 | + Cite a reference as `Lin et al.~\cite{Lin}`. Note `~` prevents ugly breaks. 243 | 244 | ### Appearance 245 | 246 | + **Watch out for overfull lines.** 247 | Latex does its best to lay out lines in paragraphs, hyphenating words as necessary. 248 | Sometimes, however, it can't do this _well_ (according to its internal algorithm) and will just give up, in which case, it will let words spill into the margin. 249 | Knuth's design decision was that, instead of producing something that would be distasteful to a discerning typographer, the system should force the author to "deal with it". 250 | The problem is that many authors forget to "deal with it", and so sometimes final versions of papers have terrible overfull lines; see [multiple examples in this classic paper](https://dl.acm.org/citation.cfm?id=1076115) (just to pick on a random example). 251 | Sometimes, manually forcing hyphenation works; otherwise, a minor rewrite will do the trick. 252 | Either way, "deal with it"! 253 | 254 | + **Watch out for weird hyphenation.** 255 | Latex does weird hyphenation on line breaks sometimes. For example, analysis might get hyphenated as "anal-ysis", which just looks dirty. MapReduce gets hyphenated as "MapRe-duce", which bugs me. Insert in the preamble `\hyphenation{Map-Reduce}` to specify particular hyphenations you want, or use `\mbox{...}` to suppress hyphenation. 256 | 257 | ## Cross-Paper Consistency 258 | 259 | + **Choose your name... wisely.** 260 | Choose the name you want to use professionally in your papers ("professional name") and make sure to stick to that form _exactly_, across all your publications. 261 | For a typical "anglicized name", decide if you want to include your full middle name, just the middle initial, or no middle name at all. 262 | This will help various scholarly aggregators do their thing (e.g., Google Scholar, DBLP, etc.). 263 | I regret that in some early publications I was inconsistent between "Jimmy J. Lin" and "Jimmy Lin" (until I started exclusively using only the latter form). 264 | A few additional issues to consider: 265 | 266 | - If you decide to use a professional name that is _not_ the same as your legal name, you might run into issues with governmental authorities. A common case is if you need a visa, the invitation letter is likely going to reference your professional name (or the paper itself, obviously), which will not match your legal name (on your passport). There are of course ways around this, but beware of the additional hassles. 267 | - For people whose surnames comprises two tokens, be aware of a bunch of annoying corner cases. I'll use "Zeynep Akkalyoncu Yilmaz" as a specific example. Properly, "Akkalyoncu Yilmaz", is the surname, but most people will likely parse out "Yilmaz" as the surname, leading to the proliferation of variants like "Zeynep Yilmaz" and "Zeynep A. Yilmaz" with automatically generated tools. Citations will also appear as Yilmaz et al. (2019). Yes, there is a way to make bibtex "do the right thing", but most authors aren't going to (and if you cite yourself correctly, that might confuse the readers). There aren't good alternatives to deal with these cases; either (a) accept that your name will be mangled, or (b) proactively manage the situation by using a professional name that's different from your given name (and more easily and consistently parsed). 268 | - For people whose names comprise accent characters, similar advice applies also. If you decide to retain accent characters, then be prepared that your name might be mangled in some cases. The alternative of just writing your name consistently without accents would be a reasonable alternative. 269 | -------------------------------------------------------------------------------- /waterloo-plugs.md: -------------------------------------------------------------------------------- 1 | # Waterloo Plugs 2 | 3 | Every time I come across a media mention of the burgeoning tech scene in Waterloo, the GTA ([Greater Toronto Area](https://en.wikipedia.org/wiki/Greater_Toronto_Area)), or Canada in general, I make note of it here. 4 | 5 | I'll admit, some of these reports are a bit "flaky" and hype-ish... so take with a big grain of [NaCl](https://en.wikipedia.org/wiki/Salt). 6 | 7 | --- 8 | 9 |And correctly distinguish uses of hyphens, n-dashes, and m-dashes.
— Jimmy Lin (@lintool) July 13, 2020
10 | 11 | August 29, 2023 - [Government of Canada invests in over 4,700 researchers across the country](https://www.canada.ca/en/innovation-science-economic-development/news/2023/08/government-of-canada-invests-in-over-4700-researchers-across-the-country.html) 12 | 13 |The #GoC supports Canada’s research ecosystem.
— Canadian Science (@CDNScience) August 30, 2023
Yesterday, we announced support for over 4,700 researchers and projects across Canada through grants, scholarships and programs.
Learn more: https://t.co/ATKb7HhoCD pic.twitter.com/sM0pwHyyWu
14 | 15 | July 11, 2023 - [Canada welcomed over 32,000 tech workers in past 12 months](https://communitech.ca/media-release/canada-welcomed-more-than-32000-tech-workers-over-past-12-months.html) 16 | 17 |According to a report from @TechCouncils, Canada welcomed over 32,000 tech workers in past 12 months. Learn more about their findings ->https://t.co/iiyBMNDjHe pic.twitter.com/NdjHToXgMr
— Communitech (@Communitech) August 31, 2023
18 | 19 | July 26, 2023 - [5 Tech Hubs, Incubators and Accelerators in Waterloo](https://www.waterlooedc.ca/blog/tech-hubs-incubators-accelerators-waterloo): It takes a dynamic community to grow renowned companies, and these active tech hubs, incubators and accelerators have contributed to Waterloo's success. 20 | 21 |The @UWaterloo's world ranking of #22 for #ComputerScience puts it ahead of well-known American tech schools like Columbia University (#23), California Institute of Technology (#36) and the University of Texas at Austin (#36).
— Waterloo EDC (@WaterlooEDC) July 31, 2023
Read more: https://t.co/3DDkpV5PdZ
22 | 23 | July 24, 2023 - [Constant Contact: amplifying digital marketing with AI and machine learning](https://www.investontario.ca/success-stories/constant-contact-amplifying-digital-marketing-ai-machine-learning) 24 | 25 |#Startups, scale-ups, #unicorns, global giants – you name it, our tech ecosystem has it. Here’s a list of five hubs, incubators and accelerators powering innovation in #Waterloo: https://t.co/nY0VGDdB9B
— Waterloo EDC (@WaterlooEDC) July 26, 2023
26 | 27 | July 24, 2023 - [From Startups to Tech Titans: The 10 Biggest Tech Companies in Waterloo](https://www.waterlooedc.ca/blog/top-technology-companies-waterloo): Waterloo's tech ecosystem is one of the best in the world. Learn how some of the biggest tech companies got their start in Waterloo and how much they've grown since then. 28 | 29 |"With the establishment of its new innovation lab in the #WaterlooRegion, @ConstantContact is making the most of Ontario’s top #techtalent to help it take its platform and features to new levels."
— Waterloo EDC (@WaterlooEDC) July 24, 2023
Learn more from @investontario's spotlight: https://t.co/CAmtTnrJYU
30 | 31 | July 20, 2023 - [Waterloo is #1 Small Tech Talent Market for 3rd Year in a Row](https://www.waterlooedc.ca/blog/cbre-scoring-tech-talent-2023) 32 | 33 | [Scoring Tech Talent 2023](https://www.cbre.com/insights/books/scoring-tech-talent-2023) 34 | 35 |10 of the largest tech companies thriving in #Waterloo: https://t.co/vk84Cx8xo0
— Waterloo EDC (@WaterlooEDC) July 24, 2023
36 | 37 | July 19, 2023 -[H-1B Visa Holder Applications Overwhelm Canada’s New Program](https://www.forbes.com/sites/stuartanderson/2023/07/19/h-1b-visa-holder-applications-overwhelm-canadas-new-program/) 38 | 39 |#Waterloo has once again been named the #1 small #techtalent market in North America in the latest @CBRE Scoring Tech Talent report — marking the third year in a row the region has claimed the title.
— Waterloo EDC (@WaterlooEDC) July 20, 2023
Explore the key findings from the report: https://t.co/5zEPVKK8jp
40 | 41 | July 7, 2023 - [University of Waterloo Recognized for World-Class Tech Talent Pipeline](https://techtalent.ca/university-waterloo-tech-talent-pipeline/) 42 | 43 |“Waterloo attracts companies that are developing highly sophisticated technology solutions and seeking a dynamic ecosystem that allows for collaboration and innovation,” said Waterloo EDC (@WaterlooEDC) Market Development Manager Cameron Branston. 1/2https://t.co/kJiHdDatHP pic.twitter.com/GHT4QpbLIQ
— Waterloo's Cheriton School of Computer Science (@UWCheritonCS) July 10, 2023
44 | 45 | April 28, 2023 - [Toronto capitalises on tech success](https://www.ft.com/content/f9b97a19-631c-46f5-bb4b-0ca28c6cc16e): City’s start-up scene is maturing after years of dizzying growth 46 | 47 | March 24, 2023 - [Report: UWaterloo is One of the World’s Best Tech Universities](https://www.waterlooedc.ca/blog/uwaterloo-qs-rankings-2023): The University of Waterloo has been ranked in the top-25 tech universities in the world. Find out how it stacks up against other well-known schools. 48 | 49 |“Waterloo is like saying you went to Harvard Law". @FT highlights #UWaterloo as a breeding-ground for world-class tech talent. https://t.co/bUuYiObgdu
— University of Waterloo (@UWaterloo) April 28, 2023
50 | 51 | [Waterloo is Canada’s 3rd Fastest-Growing Community, Statistics Canada](https://blog.waterlooedc.ca/waterloo-fastest-growing-communities-canada-2023) 52 | 53 |Waterloo is Canada’s 3rd Fastest-Growing Community according to Statistics Canada https://t.co/wrienZ3pFF
— R+T PARK (@RTPARKUW) January 26, 2023
54 | 55 | September 13, 2022 - [Waterloo: A CBRE Profile](https://blog.waterlooedc.ca/waterloo-cbre-ecosystem-profile-2022) 56 | 57 |“Waterloo is on a winning streak. For the second year in a row, our region was named the #1 small tech talent market in North America by CBRE.” Find out why: https://t.co/d8wkuin6ev pic.twitter.com/XPi2dVd82d
— R+T PARK (@RTPARKUW) September 15, 2022
58 | 59 | August 5, 2022 - [The best places to live in North America](https://www.economist.com/graphic-detail/2022/08/05/the-best-places-to-live-in-north-america) — They’re in Canada. 60 | 61 |The region is home to some of the world’s most liveable cities, according to an index by the EIU. Explore the list
— The Economist (@TheEconomist) August 11, 2022
62 | 63 | July 21, 2022 - [Waterloo-linked startups on track for billion-dollar annual revenues](https://uwaterloo.ca/news/waterloo-linked-startups-track-billion-dollar-annual): Communitech report highlights Canadian companies with impressive growth. 64 | 65 | July 14, 2022 - [CBRE names Waterloo the #1 small tech talent market in North America](https://blog.waterlooedc.ca/cbre-scoring-tech-talent-2022): For the second consecutive year, CBRE has identified Waterloo as the #1 small tech talent market in North America. The report highlights Waterloo’s fast-growing graduate and millennial populations, high degree of tech worker concentration and significant cost advantages. 66 | 67 | July 4, 2022 - [Toronto-Waterloo Region Corridor highlighted in 2022 Startup Genome rankings](https://uwaterloo.ca/news/toronto-waterloo-innovation-corridor-rankings-2022): University of Waterloo support for startup community, vision and values reflected in GSER top-20 rank. [Key Insights from #GSER2022](https://startupgenome.com/report/gser2022) 68 | 69 | June 14, 2022 - [Startup Genome report highlights Toronto-Waterloo growth](https://blog.waterlooedc.ca/toronto-waterloo-startup-genome-2022): top tech ecosystem in Canada, #8 in North America and #17 globally. 70 | 71 | June 13, 2022 - [The world's best and biggest tech schools](https://blog.waterlooedc.ca/global-best-biggest-tech-universities): When quality is (almost) equal you have to move onto other measures. What matters most for a scaling tech company? Quantity. Well, wouldn’t you know it – the University of Waterloo is right there at the top. 72 | 73 | March 21, 2022 - [Toronto, the Quietly Booming Tech Town](https://www.nytimes.com/2022/03/21/technology/toronto-tech-boom.html): For all the excitement around places like Austin and Miami, the biggest tech expansion has been in Canada’s largest city. ([IA copy](https://web.archive.org/web/20220321070516/https://www.nytimes.com/2022/03/21/technology/toronto-tech-boom.html)) 74 | 75 |Nine startups with links to @UWaterloo were named in a recent report tracking Canadian companies on their way to $1 billion annual revenues, including four companies that grew out of #UWaterloo’s startup incubator @UWVelocity.
— University of Waterloo (@UWaterloo) July 22, 2022
Read this story: https://t.co/Uw1oa9ZHZ0 pic.twitter.com/LfYxxvd6DC
76 | 77 | November 18, 2021 - [Waterloo’s new leaders on revitalizing our tech ecosystem](https://blog.waterlooedc.ca/new-leaders-waterloo-tech-ecosystem) 78 | 79 |In 2021, Waterloo Region welcomed new leaders at two of our most important tech institutions, @Communitech CEO @ChrisAlbinson and the @AC_Waterloo CEO Jay Krishnan. Find out how they are bringing fresh perspectives to our innovation ecosystem. https://t.co/kN2STjZfhr
— Waterloo EDC (@WaterlooEDC) November 18, 2021
80 | 81 | November 17, 2021 - [Waterloo well-represented on Deloitte Technology Fast 50](https://blog.waterlooedc.ca/waterloo-deloitte-technology-2021): Once again, Waterloo is home to some of Canada’s fastest-growing tech companies, according to a new ranking from Deloitte. The Deloitte Technology Fast 50 list ranks companies by revenue growth percentage over their last four years of operation. In past years, Waterloo has dominated the list, scoring the top three companies in 2019 and top two companies in 2020. 82 | 83 |Waterloo well-represented on Deloitte Technology Fast 50 https://t.co/WkdTETb18I
— R+T PARK (@RTPARKUW) November 22, 2021
84 | 85 | August 11, 2021 - [How Does a Founder's Alma Mater Impact Their Startup's Markup Rate?](https://www.angellist.com/blog/founder-schools) Ranking colleges by the markup rate of their alumni’s startups. 86 | 87 | August 4, 20201 - [Velocity companies secure more than $2 billion in funding](https://uwaterloo.ca/news/media/velocity-companies-secure-more-2-billion-funding): Investments in software and services, deep-tech drive dramatic growth 88 | 89 | March 31, 2021 - [10 exciting companies founded by UWaterloo alumni](https://blog.waterlooedc.ca/uwaterloo-alumni-founders) 90 | 91 | March 17, 2021 - [Canada is the world’s #1 destination for global talent](https://blog.waterlooedc.ca/canada-top-destination-global-talent) according to a report by the Boston Consulting Group. 92 | 93 | March 4, 2021 - [US Falls Behind Canada as a Work Destination](https://www.bcg.com/press/4march2021-us-falls-behind-canada-work-destination-pandemic-lowers-mobility-global-workforce-study) according to a Survey of 209,000 People in 190 Countries by Boston Consulting. [[PDF](https://web-assets.bcg.com/cf/76/00bdede345b09397d1269119e6f1/bcg-decoding-global-talent-onsite-and-virtual-mar-2021-rr.pdf)] 94 | 95 |  96 | 97 | November 17, 2020 - [5 Charts: Waterloo vs. America's top emerging tech hubs](https://blog.waterlooedc.ca/5-charts-waterloo-vs-americas-top-emerging-tech-hubs) 98 | 99 | November 17, 2020 - [Canada, a destination for AI talent and innovation](https://www.cifar.ca/cifarnews/2020/11/17/a-look-toward-2030) 100 | 101 |Which college’s alumni generate outsized returns for investors?
— AngelList (@AngelList) August 11, 2021
According to our new study based on markup rates, @UW, @UWaterloo, and @BrownUniversity lead the way.
Read the full study here: https://t.co/941QYl0Zyy
102 | 103 | August 13, 2020 - [Canadian tech group spends $100K on billboards to lure anxious tech workers from U.S.](https://www.cbc.ca/news/canada/kitchener-waterloo/tech-billboards-1.5683762): Campaign aimed at tech workers worried by Trump administration work visa ban. 104 | 105 |Canada is a destination for talent in artificial intelligence & innovation thanks to our vibrant AI ecosystem: Elissa Strome, ED of the CIFAR Pan-Canadian AI Strategy on the Strategy's impact to date at @Mila_Quebec @VectorInst @AmiiThinkshttps://t.co/L1UpbKiJeV pic.twitter.com/EEq3HjIH6M
— CIFAR (@CIFAR_News) November 25, 2020
106 | 107 | June 17, 2020 - [Toronto would like to be seen as the nice person’s Silicon Valley, if that’s not too much trouble](https://www.technologyreview.com/2020/06/17/1003314/toronto-would-like-to-be-seen-as-the-nice-persons-silicon-valley-if-thats-not-too-much-trouble/) 108 | 109 | February 8, 2020 - [Google's expansion plans show why Canada's tech boom is here to stay](https://www.cbc.ca/news/business/google-canada-expansion-analysis-1.5455122): A decade ago, Canada's tech industry was worried about extinction — now it's Silicon Valley North. 110 | 111 | December 9, 2019 - [Canada’s top international AI talent grows to 80](https://www.cifar.ca/cifarnews/2019/12/09/canada-s-top-international-ai-talent-grows-to-80) 112 | 113 |"Workers who contact Communitech will get information about Canadian work permits and job boards, including opportunities across the country from @Viatec in B.C. to Volta Labs(@voltaeffect) in Nova Scotia."https://t.co/3tXpD1JSqw
— Communitech (@Communitech) August 13, 2020
114 | 115 | June 25, 2019 - [Google is preparing for a major expansion of its Canadian engineering headquarters on Breithaupt Street in Kitchener](https://www.therecord.com/news-story/9449961-google-is-preparing-for-a-major-expansion-of-its-canadian-engineering-headquarters-on-breithaupt-street-in-kitchener/) 116 | 117 | May 26, 2019 - this thread: 118 | 119 |Today we announce 34 new Canada CIFAR Artificial Intelligence Chairs.
— CIFAR (@CIFAR_News) December 9, 2019
cc @AmiiThinks @MILAMontreal @VectorInst #realbrainshttps://t.co/MvQObspJGO
120 | 121 | May 23, 2019 - [Global Push Has Dividends for Toronto-Waterloo](http://news.communitech.ca/global-focus-paying-dividends-for-toronto-waterloo/): "[about the Waterloo Region:] an amazing ecosystem. It’s producing one of the highest number of startups per population, which means its funnel, its entrepreneurial mindset, spirit, the strength of the community as an engine for the region, is just so strong. It’s very impressive in that way. Waterloo has 1,100 startups per million people, or something like that. Silicon Valley is at 2,000. But everybody else is at 300, 400, 500. So it continues to create a very dynamic ecosystem. I think this is a real strength." 122 | 123 | April 10, 2019 - [Scoring Tech Talent in North America 2018](https://www.cbre.com/research-and-reports/Scoring-Tech-Talent-in-North-America-2018) 124 | 125 |With the #Raptors about to hit the global stage, I’m going to share a story about what makes Toronto, and Canada, so special. It’s about a guy you’re going to be seeing a whole lot of.
— Muhammad Lila (@MuhammadLila) May 26, 2019
A Thread.
[Hint: It's about racism, sports, and why you'll end up cheering for 🍁]
126 | 127 |  128 | 129 | March 3, 2019 - [Smaller Countries Are Becoming the Healthiest](https://www.bloomberg.com/news/articles/2019-03-04/global-wellness-rankings-these-are-the-best-places-to-live): A new analysis, the Global Wellness Index published by investment firm LetterOne, ranks Canada as the best country out of the 151 nations evaluated. The U.S. trails far behind, coming in at 37. 130 | 131 | February 27, 2019 - [2019 QS World University Rankings places computer science at Waterloo 22nd globally](https://cs.uwaterloo.ca/news/2019-qs-world-university-rankings-places-computer-science) 132 | 133 | December 18, 2018 - [Indian technology talent is flocking to Canada](https://www.economist.com/business/2018/12/22/indian-technology-talent-is-flocking-to-canada): As immigrant techies shun the US, its neighbour has rolled out the red carpet 134 | 135 | October 11, 2018 - [Canada’s Top School by Reputation 2019 136 | ](https://www.macleans.ca/education/university-rankings/canadas-top-school-by-reputation-2019/): We asked university faculty and senior administrators, high school guidance counsellors and a variety of business people across the country for their views on quality and innovation at Canadian universities 137 | 138 | + Best overall: Waterloo #2 (After Toronto) 139 | + Highest quality: Waterloo #3 (After Toronoto and McGill) 140 | + Most innovative: Waterloo #1 141 | + Leaders of tomorrow: Waterloo #2 (After Toronto) 142 | 143 | September 13, 2018 - [What if the Toronto-Waterloo corridor really becomes the next Silicon Valley?](https://www.macleans.ca/economy/what-if-the-toronto-waterloo-corridor-really-becomes-the-next-silicon-valley/): Uber, Microsoft and Intel are the latest tech heavyweights to invest and expand in the Toronto-Waterloo corridor—but buzzy names alone won’t make the region the next big thing 144 | 145 | September 11, 2018 - [Microsoft Announces New Canadian Headquarters in Downtown Toronto](https://www.newswire.ca/news-releases/microsoft-announces-new-canadian-headquarters-in-downtown-toronto-692971421.html) 146 | 147 | July 24, 2018 - [Who Just Beat the Bay Area in Tech Jobs? Toronto](https://www.bloomberg.com/news/articles/2018-07-24/toronto-beats-bay-area-in-new-tech-jobs-and-new-york-in-talent) 148 | 149 | June 22, 2018 - [Microsoft Research Montreal welcomes Fernando Diaz, Principal Researcher and lead of the new Montreal FATE Research Group](https://www.microsoft.com/en-us/research/blog/microsoft-research-montreal-welcomes-fernando-diaz-principal-researcher-and-lead-of-the-new-montreal-fate-research-group/) ([French version](https://news.microsoft.com/en-ca/2018/06/22/le-laboratoire-microsoft-research-montreal-accueille-fernando-diaz/)) 150 | 151 | June 13, 2018 - [NVIDIA Opening AI Research Lab in Toronto, Following Move in Seattle](https://blogs.nvidia.com/blog/2018/06/13/nvidia-ai-research-lab-toronto/) 152 | 153 | May 15, 2018 - [To increase Canadian innovation, take a lesson from our AI successes](https://www.theglobeandmail.com/business/commentary/article-to-increase-canadian-innovation-take-a-lesson-from-our-ai-successes/) 154 | 155 | May 4, 2018 - [Canada’s brain gain – the reverse migration of tech talent is happening](https://www.theglobeandmail.com/business/commentary/article-canadas-brain-gain-the-reverse-migration-of-tech-talent-is/) 156 | 157 | May 1, 2018 - [Tech conferences moving north as Trump policies turn off attendees](http://business.financialpost.com/technology/tech-conferences-moving-north-as-trump-policies-turn-off-attendees): The Collision Conference, Creative Commons and RightsCon will bring thousands of people to Toronto 158 | 159 | April 17, 2018 - [Toronto-Waterloo tech corridor among top 20 worldwide](https://www.therecord.com/news-story/8398212-toronto-waterloo-tech-corridor-among-top-20-worldwide/) 160 | 161 |Common takeaway from this chart: Canada is a bargain for AI talent
— Gilad Mishne (@gilad) April 10, 2019
Reality: difference between “good” and “exceptional” is huge, ROI more than justifies 2X costshttps://t.co/cNQQfCnVUj pic.twitter.com/Zy98avDKrH
162 | 163 | March 29, 2018 - [Canada 150 research chairs draw scientists fleeing Trump, guns and Brexit](https://www.theglobeandmail.com/canada/article-canada-150-research-chairs-draw-scientists-fleeing-trump-guns-and/) 164 | 165 |Read @PenderRecord's article in today's @WR_Record on the 2018 Global @startupgenome report here: https://t.co/0GzAM6gcie - "It allows us to capitalize on the diversity of region, our strong talent base and our excellent academic institutions." #TOWRcorridor @WRTech #CDNTech pic.twitter.com/1fQn3xo7I1
— Berry Vrbanovic (@berryonline) April 17, 2018
166 | 167 | February 23, 2018 - [Accelerator Centre Ranked #4 University Linked Accelerator Program in the World](http://acceleratorcentre.com/accelerator-centre-ranked-4-university-linked-accelerator-program-in-the-world/) 168 | 169 | February 15, 2018 - [Waterloo region wins big in $950M 'supercluster' competition](http://www.cbc.ca/news/canada/kitchener-waterloo/university-of-waterloo-key-partner-in-supercluster-1.4536920): The University of Waterloo will play a key research role in 2 of the 5 winning projects. 170 | 171 | February 7, 2018 - [Element AI Global Talent Report Finds Canada has Third-Largest Concentration of AI Researchers](https://betakit.com/element-ai-global-ai-talent-report-finds-canada-has-third-largest-concentration-of-ai-researchers/) 172 | 173 | February 5, 2018 - [Report Ranks Canada Third in Top 10 Startup Ecosystem Worldwide](https://betakit.com/report-ranks-canada-third-in-top-10-startup-ecosystems-worldwide/) 174 | 175 | January 5, 2018 - [Canada Courts Entrepreneurs As Trump Trims Back Immigration](https://www.inc.com/zoe-henry/canada-courts-american-startups-trump-immigration.html): As the President moves to reform the popular H-1B visa program, companies look to Canada as a low-cost alternative. 176 | 177 |Canada is #1 worldwide for student led #startups with young Cdn #talent founding world-leading IT companies! #entrepreneurs @RyersonDMZ @Communitech @MaRSDD https://t.co/OL3zWfgPPB pic.twitter.com/HWn4l9w1Ov
— Invest in Canada (@invest_canada) March 8, 2018
178 | 179 | December 13, 2017 - [A brain gain for Canada](http://www.newswire.ca/fr/news-releases/a-brain-gain-for-canada-663900913.html): Scientists from top international universities such as Harvard, Imperial College London, recruited as Canada 150 Research Chairs. 180 | 181 | December 13, 2017 - [Newly unveiled research chairs take aim at diversity gap in Canadian science](https://www.theglobeandmail.com/news/national/newly-unveiled-research-chairs-take-aim-at-diversity-gap-in-canadian-science/article37312603/) 182 | 183 | November 23, 2017- [Waterloo region's tech talent fastest growing in Canada: report](http://www.cbc.ca/news/canada/kitchener-waterloo/waterloo-region-tech-talent-market-cbre-report-1.4416308): Region's tech industry one of the best in Canada and 'a major engine of innovation,' says investment firm. (Also, from [Communitech](http://news.communitech.ca/news/waterloo-region-leads-nation-in-tech-talent-growth-rate-report/)) 184 | 185 | November, 2017 - [Toronto Region - A Magnet for Canadian and Foreign Tech Companies](https://torontoglobal.ca/about-us/News/2017/Toronto-Region-A-Magnet-for-Canadian-and-Foreign) 186 | 187 | November, 2017 - [Story Of Our Move From Silicon Valley To Toronto](https://movnorth.com/story/) 188 | 189 | October 19, 2017 - [Shopify to expand in Waterloo, no fear of Amazon, VP says](http://www.cbc.ca/news/canada/kitchener-waterloo/shopify-expansion-waterloo-amazon-loren-padelford-1.4360399) 190 | 191 | October 5, 2017 - [Canada becoming an in-demand destination for foreign tech talent](https://beta.theglobeandmail.com/report-on-business/careers/management/canada-becoming-an-in-demand-destination-for-foreign-tech-talent/article36499942/) 192 | 193 | September 16, 2017 - [Increasingly, foreign students are choosing Canada over US](http://www.bostonglobe.com/metro/2017/09/16/increasingly-foreign-students-choose-canada-over/khkot6AYt9lakpIFLTNGvM/story.html) 194 | 195 | January 24, 2017 - [The Unicorn League: where top start-up founders went to college](https://www.timeshighereducation.com/student/news/unicorn-league-where-top-start-founders-went-college) — Waterloo ranks at #11! 196 | 197 | August 31, 2017 - [4 reasons startups should consider moving to Toronto](http://www.geektime.com/2017/08/31/4-reasons-startups-should-consider-moving-to-toronto/) 198 | 199 | December 18, 2016 - [These Universities Are Training The World's Top Coders](https://www.fastcompany.com/3066485/the-future-of-work/these-universities-are-training-the-worlds-top-coders): Waterloo is #5. 200 | 201 | March 13, 2015 - [How to Survive the College Admissions Madness](http://www.nytimes.com/2015/03/15/opinion/sunday/frank-bruni-how-to-survive-the-college-admissions-madness.html) 202 | 203 | > I also spoke with Sam Altman, the president of Y Combinator, one of the best-known providers of first-step seed money for tech start-ups. I asked him if any one school stood out in terms of students and graduates whose ideas took off. “Yes,” he responded, and I was sure of the name I’d hear next: Stanford. It’s his alma mater, though he left before he graduated, and it’s famous as a feeder of Silicon Valley success. 204 | > 205 | > But this is what he said: “The University of Waterloo.” It’s a public school in the Canadian province of Ontario, and as of last summer, it was the source of eight proud ventures that Y Combinator had helped along. “To my chagrin,” Altman told me, “Stanford has not had a really great track record.” 206 | 207 | Febrary 3, 2015 - [Engineering Salaries Reviewed](https://rivierapartners.com/engineering-salaries-reviewed-2/): Data from 2014, but shows Waterloo as #2 most-hired school in Silicon Valley. 208 | 209 | 210 | 211 | 212 | --------------------------------------------------------------------------------Today, I announced the 1st wave of researchers & scholars who will call 🇨🇦 home as a result of the #Canada150 Research Chairs program https://t.co/augw8Ng7Dr @CRC_CRC
— Hon. Kirsty Duncan (@ScienceMin) December 13, 2017