├── .gitignore ├── CONTRIBUTORS.md ├── DEVELOPERS.md ├── Dockerfile.cpu ├── Dockerfile.gpu ├── LICENSE ├── README.md ├── download_model.py ├── encode.py ├── gpt_finetune.ipynb ├── requirements.txt ├── samples └── unconditional-topk15.txt ├── src ├── __init__.py ├── accumulate.py ├── encoder.py ├── generate_unconditional_samples.py ├── interactive_conditional_samples.py ├── load_dataset.py ├── model.py └── sample.py ├── train.py └── twitterbot.py /.gitignore: -------------------------------------------------------------------------------- 1 | data 2 | models 3 | 4 | *.ipynb 5 | twitter_secrets.py 6 | .ipynb_checkpoints 7 | __pycache__ 8 | *.log 9 | -------------------------------------------------------------------------------- /CONTRIBUTORS.md: -------------------------------------------------------------------------------- 1 | # Contributors (alphabetically) 2 | 3 | * **[madisonmay](https://github.com/madisonmay)** 4 | 5 | Added Dockerfiles 6 | 7 | * **[Margaret Mitchell et al](https://arxiv.org/abs/1810.03993)** 8 | 9 | Our [usage](./readme#usage) writeup was loosely inspired by the paper 10 | [Model Cards for Model Reporting](https://arxiv.org/abs/1810.03993) 11 | and related conversations with some of the authors. 12 | 13 | * **[webproduktion01](https://github.com/webproduktion01)** 14 | 15 | Ported download script to python. 16 | 17 | **[Full code contributors list](https://github.com/openai/gpt-2/contributors).** 18 | -------------------------------------------------------------------------------- /DEVELOPERS.md: -------------------------------------------------------------------------------- 1 | # Installation 2 | 3 | Git clone this repository, and `cd` into directory for remaining commands 4 | ``` 5 | git clone https://github.com/openai/gpt-2.git && cd gpt-2 6 | ``` 7 | 8 | Then, follow instructions for either native or Docker installation. 9 | 10 | ## Native Installation 11 | 12 | All steps can optionally be done in a virtual environment using tools such as `virtualenv` or `conda`. 13 | 14 | Install tensorflow 1.12 (with GPU support, if you have a GPU and want everything to run faster) 15 | ``` 16 | pip3 install tensorflow==1.12.0 17 | ``` 18 | or 19 | ``` 20 | pip3 install tensorflow-gpu==1.12.0 21 | ``` 22 | 23 | Install other python packages: 24 | ``` 25 | pip3 install -r requirements.txt 26 | ``` 27 | 28 | Download the model data 29 | ``` 30 | python3 download_model.py 117M 31 | ``` 32 | 33 | ## Docker Installation 34 | 35 | Build the Dockerfile and tag the created image as `gpt-2`: 36 | ``` 37 | docker build --tag gpt-2 -f Dockerfile.gpu . # or Dockerfile.cpu 38 | ``` 39 | 40 | Start an interactive bash session from the `gpt-2` docker image. 41 | 42 | You can opt to use the `--runtime=nvidia` flag if you have access to a NVIDIA GPU 43 | and a valid install of [nvidia-docker 2.0](https://github.com/nvidia/nvidia-docker/wiki/Installation-(version-2.0)). 44 | ``` 45 | docker run --runtime=nvidia -it gpt-2 bash 46 | ``` 47 | 48 | # Running 49 | 50 | | WARNING: Samples are unfiltered and may contain offensive content. | 51 | | --- | 52 | 53 | Some of the examples below may include Unicode text characters. Set the environment variable: 54 | ``` 55 | export PYTHONIOENCODING=UTF-8 56 | ``` 57 | to override the standard stream settings in UTF-8 mode. 58 | 59 | ## Unconditional sample generation 60 | 61 | To generate unconditional samples from the small model: 62 | ``` 63 | python3 src/generate_unconditional_samples.py | tee /tmp/samples 64 | ``` 65 | There are various flags for controlling the samples: 66 | ``` 67 | python3 src/generate_unconditional_samples.py --top_k 40 --temperature 0.7 | tee /tmp/samples 68 | ``` 69 | 70 | To check flag descriptions, use: 71 | ``` 72 | python3 src/generate_unconditional_samples.py -- --help 73 | ``` 74 | 75 | ## Conditional sample generation 76 | 77 | To give the model custom prompts, you can use: 78 | ``` 79 | python3 src/interactive_conditional_samples.py --top_k 40 80 | ``` 81 | 82 | To check flag descriptions, use: 83 | ``` 84 | python3 src/interactive_conditional_samples.py -- --help 85 | ``` 86 | -------------------------------------------------------------------------------- /Dockerfile.cpu: -------------------------------------------------------------------------------- 1 | FROM tensorflow/tensorflow:1.12.0-py3 2 | 3 | ENV LANG=C.UTF-8 4 | RUN mkdir /gpt-2 5 | WORKDIR /gpt-2 6 | ADD . /gpt-2 7 | RUN pip3 install -r requirements.txt 8 | RUN python3 download_model.py 117M 9 | -------------------------------------------------------------------------------- /Dockerfile.gpu: -------------------------------------------------------------------------------- 1 | FROM tensorflow/tensorflow:1.12.0-gpu-py3 2 | 3 | # nvidia-docker 1.0 4 | LABEL com.nvidia.volumes.needed="nvidia_driver" 5 | LABEL com.nvidia.cuda.version="${CUDA_VERSION}" 6 | 7 | # nvidia-container-runtime 8 | ENV NVIDIA_VISIBLE_DEVICES=all \ 9 | NVIDIA_DRIVER_CAPABILITIES=compute,utility \ 10 | NVIDIA_REQUIRE_CUDA="cuda>=8.0" \ 11 | LANG=C.UTF-8 12 | 13 | RUN mkdir /gpt-2 14 | WORKDIR /gpt-2 15 | ADD . /gpt-2 16 | RUN pip3 install -r requirements.txt 17 | RUN python3 download_model.py 117M 18 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 OpenAI 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # gpt-2 2 | 3 | Code and samples from the paper ["Language Models are Unsupervised Multitask Learners"](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf). 4 | 5 | For now, we have only released a smaller (117M parameter) version of GPT-2. 6 | 7 | See more details in our [blog post](https://blog.openai.com/better-language-models/). 8 | 9 | ## Usage 10 | 11 | This repository is meant to be a starting point for researchers and engineers to experiment with GPT-2-117M. While GPT-2-117M is less proficient than GPT-2-1.5B, it is useful for a wide range of research and applications which could also apply to larger models. 12 | 13 | ### Some caveats 14 | 15 | - GPT-2-117M robustness and worst case behaviors are not well-understood. As with any machine-learned model, carefully evaluate GPT-2-117M for your use case, especially if used without fine-tuning or in safety-critical applications where reliability is important. 16 | - The dataset our GPT-2-117M was trained on contains many texts with [biases](https://twitter.com/TomerUllman/status/1101485289720242177) and factual inaccuracies, and thus GPT-2-117M is likely to be biased and inaccurate as well. 17 | - To avoid having samples mistaken as human-written, we recommend clearly labeling samples as synthetic before wide dissemination. Our models are often incoherent or inaccurate in subtle ways, which takes more than a quick read for a human to notice. 18 | 19 | ### Work with us 20 | 21 | Please [let us know](mailto:languagequestions@openai.com) if you’re doing interesting research with or working on applications of GPT-2-117M! We’re especially interested in hearing from and potentially working with those who are studying 22 | - Potential malicious use cases and defenses against them (e.g. the detectability of synthetic text) 23 | - The extent of problematic content (e.g. bias) being baked into the models and effective mitigations 24 | 25 | ## Development 26 | 27 | See [DEVELOPERS.md](./DEVELOPERS.md) 28 | 29 | ## Contributors 30 | 31 | See [CONTRIBUTORS.md](./CONTRIBUTORS.md) 32 | 33 | ### Fine tuning on custom datasets 34 | 35 | To retrain GPT-2 117M model on a custom text dataset: 36 | 37 | ``` 38 | PYTHONPATH=src ./train.py --dataset 39 | ``` 40 | 41 | If you want to precompute the dataset's encoding for multiple runs, you can instead use: 42 | 43 | ``` 44 | PYTHONPATH=src ./encode.py /path/to/encoded.npz 45 | PYTHONPATH=src ./train.py --dataset /path/to/encoded.npz 46 | ``` 47 | 48 | To do distributed on multiple GPUs or machines using Horovod: 49 | 50 | ``` 51 | mpirun -np 4 \ 52 | -H localhost:4 \ 53 | -bind-to none -map-by slot \ 54 | -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH \ 55 | -x PYTHONPATH=src \ 56 | -mca pml ob1 -mca btl ^openib \ 57 | /home/jovyan/gpt-2/train-horovod.py --dataset encoded.npz 58 | ``` 59 | 60 | ## GPT-2 samples 61 | 62 | | WARNING: Samples are unfiltered and may contain offensive content. | 63 | | --- | 64 | 65 | While we have not yet released GPT-2 itself, you can see some samples from it in the `gpt-2-samples` folder. 66 | We show unconditional samples with default settings (temperature 1 and no truncation), with temperature 0.7, and with truncation with top_k 40. 67 | We show conditional samples, with contexts drawn from `WebText`'s test set, with default settings (temperature 1 and no truncation), with temperature 0.7, and with truncation with top_k 40. 68 | 69 | ## Citation 70 | 71 | Please use the following bibtex entry: 72 | ``` 73 | @article{radford2019language, 74 | title={Language Models are Unsupervised Multitask Learners}, 75 | author={Radford, Alec and Wu, Jeff and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya}, 76 | year={2019} 77 | } 78 | ``` 79 | 80 | ## Future work 81 | 82 | We may release code for evaluating the models on various benchmarks. 83 | 84 | We are still considering release of the larger models. 85 | 86 | ## License 87 | 88 | [MIT](./LICENSE) 89 | -------------------------------------------------------------------------------- /download_model.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import requests 4 | from tqdm import tqdm 5 | 6 | if len(sys.argv) != 2: 7 | print('You must enter the model name as a parameter, e.g.: download_model.py 117M') 8 | sys.exit(1) 9 | 10 | model = sys.argv[1] 11 | 12 | subdir = os.path.join('models', model) 13 | if not os.path.exists(subdir): 14 | os.makedirs(subdir) 15 | subdir = subdir.replace('\\','/') # needed for Windows 16 | 17 | for filename in ['checkpoint','encoder.json','hparams.json','model.ckpt.data-00000-of-00001', 'model.ckpt.index', 'model.ckpt.meta', 'vocab.bpe']: 18 | 19 | r = requests.get("https://storage.googleapis.com/gpt-2/" + subdir + "/" + filename, stream=True) 20 | 21 | with open(os.path.join(subdir, filename), 'wb') as f: 22 | file_size = int(r.headers["content-length"]) 23 | chunk_size = 1000 24 | with tqdm(ncols=100, desc="Fetching " + filename, total=file_size, unit_scale=True) as pbar: 25 | # 1k for chunk_size, since Ethernet packet size is around 1500 bytes 26 | for chunk in r.iter_content(chunk_size=chunk_size): 27 | f.write(chunk) 28 | pbar.update(chunk_size) 29 | -------------------------------------------------------------------------------- /encode.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Usage: 3 | # PYTHONPATH=src ./encode.py /path/to/output.npz 4 | # PYTHONPATH=src ./train --dataset /path/to/output.npz 5 | 6 | import argparse 7 | import numpy as np 8 | 9 | import encoder 10 | from load_dataset import load_dataset 11 | 12 | parser = argparse.ArgumentParser( 13 | description='Pre-encode text files into tokenized training set.', 14 | formatter_class=argparse.ArgumentDefaultsHelpFormatter) 15 | parser.add_argument('--model_name', metavar='MODEL', type=str, default='117M', help='Pretrained model name') 16 | parser.add_argument('--combine', metavar='CHARS', type=int, default=50000, help='Concatenate files with <|endoftext|> separator into chunks of this minimum size') 17 | parser.add_argument('in_txt', metavar='PATH', type=str, help='Input file, directory, or glob pattern (utf-8 text).') 18 | parser.add_argument('out_npz', metavar='OUT.npz', type=str, help='Output file path') 19 | 20 | def main(): 21 | args = parser.parse_args() 22 | enc = encoder.get_encoder(args.model_name) 23 | print('Reading files') 24 | chunks = load_dataset(enc, args.in_txt, args.combine) 25 | print('Writing', args.out_npz) 26 | np.savez_compressed(args.out_npz, *chunks) 27 | 28 | 29 | if __name__ == '__main__': 30 | main() 31 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | fire>=0.1.3 2 | regex==2017.4.5 3 | requests==2.21.0 4 | tqdm==4.31.1 5 | tweepy==3.7.0 6 | -------------------------------------------------------------------------------- /samples/unconditional-topk15.txt: -------------------------------------------------------------------------------- 1 | When we are born we love to be a blessing, and when we die we love to be a trouble. 2 | I don't think of myself as a poor deprived ghetto girl who made good. I have decided that this will never last. 3 | Good judgement comes from experience. Experience comes from bad judgement. 4 | I've decided to stick to fair dealing. I don't fight it. I don't pay it. I don't watch it. I'm not looking for glory, and I ain't nothing else but beating on its walls. I ain't got no magical transformation process. I ain't got nothing wrong with me. 5 | When your time comes to die, be not like those whose hearts are filled with fear of the dead, but like those whose eyes are filled with hope for the living, or who may give grief to someone who looks forward to their lives ending in happiness. 6 | When I was young I didn't appreciate fear, as that was a cliche at that point, but I do have a love for it. I have seen so many heads fall in love when they got tober ring I just lay there and watched it on TV. 7 | If a man walks in the woods for no reason other than to satisfy himself that he is a pure human being, there is absolutely no hope of him benefitting anyone else. 8 | A man is who he is not made to be. He has no fear of it. He has fear of death to keep from running into it. 9 | A liberal may well fear the punishment of a sympathetic man he will not moralize his friend. 10 | I don't fear death in the slightest. 11 | I do not fear fools. 12 | Bond is like a death sentence one must not say it in the expression of hatred one does not fear death, but frightens them. 13 | Death, like the promise of freedom, is forgotten when souls are charred to death. 14 | I don't fear death because I welcome the death of others. 15 | It is a sad thing when women entertain the notion that they can live without hope. I am a latecomer to the party and have been through it pretty badly. 16 | Death is caused by swallowing small amounts of saliva over a long period of time. 17 | For the coward never lets his fellows get ahead of him. 18 | I have no fear of flying, but I have been afraid to fly for more than ten miles, and I have always had a fear of being robbed of my life. 19 | The fear of death follows from the fear of life. A man is afraid of flying. He must get out of his body before his spirit can lift him. 20 | There are a lot of movies that are ideal for this, but it's hard for me to get attached to a lot of stuff. I grew up on a candy store shelf. I'm tempted by all the candy you get, even down in the aisle. So it's important to love the stuff you love. 21 | I was raised by a single mom who made it easy for me to get good music education. Now I'm a mother mom at a very small town. 22 | You cannot learn to like yourself just when you are going through the ringer for the fear of offending the bullies. 23 | Fear is not in the habit of speaking ill of others, but in leaning upon them in advance of his own. 24 | To me, money is just another kind of anxiety. I'd be a beggar if I couldn't fear money. 25 | What you're doing now is not the way you started it all. There was a real sense of renewal. There's still a lot of frustration and depression about the current political situation. 26 | I believe that unarmed Americans standing for God in the dark should have the right to protest and say what is unconstitutional. 27 | We are very comfortable with minority status quo. We feel we need a president with a compassionate, decent heart and is not afraid to make some moves. We believe the White House is Nixon to Republicans bows down to the scribes and drags. 28 | When you're a mom and you have three kids, you have to be the focus. If you don't take responsibility seriously, you can very quickly get in the way of the whole education that you were attending. 29 | I'm not scared of technology. My kids may say that's evil, but the truth is it can be horrific. 30 | I believe that a leader can pick you, get you somewhere, and lead you to a fulfilling life. You must have the attitude where you are not afraid and that the purpose of life is to give them everything. 31 | I don't have the same sense of respect or empathy for the music business that a nerd does. I'm just much more uncomfortable when reading records. 32 | Do your damnedest in the quiet of a book, and you'll find you have the most loved enemy in history. 33 | The history of progress is written in the ears of children. 34 | History repeats itself, but the special call of an art that repeats itself. 35 | History has taught me everything. I count it a difficult lesson, 36 | A lot of the time I'll be standing on a street corner looking at people and I don't care about their faces. I just want to be me and smile. 37 | I think there are a lot of things that make a great smile, and one of the ones that I find most rewarding about the project is just seeing the smile on people's minds that get blown to scenes that make people laugh or even when they don't laugh because there's a lot of heart left in the joke. 38 | There is no smile more intense than that of an army of intelligent people. 39 | My mother's whole life had been trying to start a business. She could have had a baby at any hour, but my chances were not that good. So I went to New York and worked, and then I moved to California and waited for my turn. Then I moved to try school. 40 | I think God has done good to me. I like my life. I like that he doesn't have to worry about me worrying about him. It's not so bad. 41 | I have never seen a worse monster than a man. He is the good man caught in the middle of all the bad things. Look at the faces of the Democrats. They are so angry that they are accusing each other of hate. They have no heart. 42 | But I did not consider myself a composer, nor even involved in costume, but just an individual who could smile and say whatever I wanted to say. 43 | It was sort of a, let's face it, relief seeing this, rather than almost a positive feeling, I was back home campaigning. This wasn't a mandate of the Labour party so this was a kind of a relief campaign. 44 | I think the positive things you get from running for office is from people you genuinely love. 45 | I hope to be a part of young people not just at large but shaping our future as political leaders. It is important for people to know what their future holds. 46 | You can't mix sports with politics. 47 | Sometimes in British politics you realise you are doing something extraordinary. You are the one who loses the last round of your ballot papers to the most recent one. 48 | One lesson you better learn if you want to be in politics is that social programmes don't look new in the Anglo-American culture gap. 49 | What the public wants, in politics or even in any other realm of experience, is not the privacy of the person or private information but the self-restraint, the self-reliance, in keeping as much of that person's private information as possible. 50 | I've always been involved in politics and everything that surrounds it at the end. 51 | If you look at the art of politics as a art form, you see it being combined with music. And you also see it being taught side by side, side by side, side by side. It certainly has alumni in Chicago. 52 | I don't think I've ever been a politician. But I do know how to use politics. I've been in politics my whole life, and the last thing I'm doing is focusing on the political process. 53 | You don't know who the next Conservative is going to be until you have set your sights on winning. Politics is the needle in the hay, and God is just what the beast feeds on. 54 | In politics, what begins in anger is the exact spot on the blame map. 55 | Well there are some things you just don'tolve it's still you, it's not your fault somebody else made a mistake. But in politics sometimes things start getting bad, and you instinctually make the worst decisions. 56 | Politics is a field kids can get into, and I fortunately have a great team of parents to help guide them. 57 | The secret of politics? Make a good one. 58 | I don't understand politics, unfortunately. I always think of it as between the two parties in the contest. 59 | Politics is not a problem to arrange and you can arrange it in a mountain of deceit, where undecided voters can't even agree. 60 | Politics is about the improvement of people's lives. It's about advancing our dreams. 61 | Good music is better in politics than bad music. 62 | I don't understand politics, insofar as American politics is concerned. 63 | There was always a political aspect to my life, but I was attracted to politics - politics as an interrupter of truth. 64 | You know that feeling when you've just finished your day's work and you're about to leave? It's like, this is the emotion that animates all my work, and this is the result of hours and hours of my life -this is the feeling that other people have, and that's one of truth. 65 | My parents were keen for the young people in my generation to have ideas, to have peace, and they saw that the art of politics was the business of humanity, that's all it is. And it didn't apply to them. 66 | I don't know if there is one sure-fire definition of politics, but there is probably one sure-fire definition. 67 | I've made quite the career by chewing through large amounts of gummy bears while driving. When you're in the driver's seat are you making a giant, deformed living? Of devoting yourself to any role but that worn by having a bad back? Of not knowing anything? 68 | I've had a bit of success because I's time machine has done its job. 69 | The secret to success is taking care of the people you love and liking yourself. 70 | Most success springs from getting a certain level of failure. 71 | Success is always a leap of faith sometimes made by people who have very little faith in the certainty of their own opinions. 72 | My mission is not to teach the way the world forward, it is to educate, love and call upon the powers that be, to dare, to fight for a better world in which truth, knowledge and progress are forever a part of the common destiny. 73 | In the long history of the world, only a few generations have been granted the role of defending freedom in its hour of maximum danger. I do not shrink from this responsibility - I welcome it. 74 | We must use good will to keep our courage bright and to sacrifice life for the day. 75 | The keys to success are:Be patient and considerately considerate of othersBe patient and considerate of yourselfBe patient and considerate of others and bear with yourselfBear and embrace with your wounded soulandcome true to what you have. 76 | We are taught to be embarrassed by our success, embarrassed only by our failureCome true to yourself and join the dots. 77 | To succeed, one must consider how he can help others. 78 | If you don't want to use, then don't take it because it'll make you feel better. 79 | Success consists of getting hold of what you don't need, and then grabbing it. 80 | There is no failure except in no longer trying. 81 | It's really a lot easier to change yourself from what you are than to change someone else. 82 | Success is built on confidence, on being able to say no and being intolerably opposed to those you don't like. 83 | You can never set a objective, objectively verifiable measure of your success or failure, it is simply a matter of making a compelling case that what you are doing proves that you are invincible. 84 | We always succeed because of our own effort, a strange, miraculous success. 85 | When you rise in the morning, give full consideration to the whole business of being up and about. It is then breakfast of immortality. 86 | I hate to be a failure and can't be, but I feel like a great man for a top job. 87 | I don't think of myself as a poor girl who made a purchase. My motivation come from being on the top food chain. 88 | If you really want to be great, if you really love yourself, then get out. And help someone else in their struggle. 89 | Definitely the 'Off the Wall' stuff is a lie. There will be critics. But it's a general lie and not personal attacks. 90 | The real secret to success is awakening inside you and becoming more selfless. 91 | If you want to be rich, get rich and make yourself. If you want to be powerful, get high and rich. 92 | The rate of personal success is determined by not how much we make of art, but how much we make of it. 93 | The key to success is to keep making mistakes, but also to not repeat them. 94 | If you don't quit, you don't do real success. 95 | To succeed, one must not just quit, but learn what quit is. Everyone has to find a way to quit. 96 | We succeed in life because we succeed in taking us to narrative endowments that transcend life itself. 97 | I think you can have great success without really trying. It's just more focused on the process and not getting into people's lives. I still go to parties with my head down because I want to give out the effort for what I'm doing. 98 | Success is dangerous. One will never grow old without suffering. 99 | The key to success is to keep growing in faith. 100 | An entrepreneur assumes leadership, yet he often makes it look so easy. Because if he doesn't, he will lose everything he has achieved... or so is the common saying. 101 | Why do I talk about the global race drive? That's my daddy issue. Wanting to push me to do something that I never before had believed, thinking that something will one day make me do it - that makes meieu. 102 | The secret of success is learning how to treat them like family. 103 | Success is always just an evil it is only neutralized by the aroma of truth. 104 | If you're the type of person who has to fulfill your dreams and go through the hardships, you've made a terrible impact on our society. 105 | There is no end to education. It is not that you learn more learn what to do, and how to survive. The end is certain. 106 | There is nothing nobler or greater in the world than when 107 | My friends think I've inherited my greatest fear. I don't. I'm a lapsed compulsive reader. My wife and I have both been readers since I was 12. I grew up with books and shared common interests in science and literature. I read to both of my parents as a teen, and reading her comic stories helped motivate me to write them. 108 | My fear is that, as soon as I get married and have kids that I'd rather raise them side by side with my son, versus relying on my husband for support and also for book publishing. 109 | It is no longer a fear of death to attempt it, or fear of dying without having done it. 110 | I have no fear of making mistakes, being under the influence of any while, ever achieving anything even about the right amount. 111 | I was terrified to express my own front teeth because people would ask, 'What's the big deal?' With my soundtracks, you don't do that anymore. So, I had to try and cut out a little edge for my music without getting so much as a laugh. 112 | I don't have any fear of going to the bathroom after getting married because I don't want that stinkin' beast in my bathroom with my tight pants. 113 | I was so afraid to even try. I was finally allowed to play soccer practice with my kids. It was a great experience and I will never have to return to. 114 | I have been afraid all my life to go to school. 115 | My biggest fear in life is to be poor. The rest could turn to history. 116 | I have no fear of childbirth. I have no fear of maternal pain. I have no fear of artificial birth devices. 117 | You can't begin to control your urges or forget them once they have invaded your life. You have to find a balance. By forgetting people are out of your league. By destroying all that you can exercise, you can have peace. 118 | The fear of burglars is also the fear of getting into the wrong ground. 119 | I fear those who make fun of the elderly more than the fear of the disease he suffers alone. 120 | You have to find a way to express yourself through music. Without it you are afraid. 121 | I don't think rap will ever overtake gospel - gospel record making is required like all successful thing in the industrial world - but the problem is that today nothing prepares you to do it. 122 | I have to remind you that I have never felt that the attitude I have for myself or others has changed in the 20 years since I became involved with the cause. In that time I have never had a person in my life that would actually like me or like what I do. 123 | I have a huge respect for writers and directors who tell me a story and tell a story well. I stopped believing this kind of thing. You can't take a bad story down - I do appreciate it. It's a great compliment, but it's also a great compliment, because I'm telling you the truth. 124 | When I had my first success I was 19 and I was in my twenties, and it still exists - although I rate less highly now - but I think it exists more fairly and persistently. And so I have a great deal of respect for it and a great deal of resentment towards it. 125 | The fear of the dark should never diminish. A man should be half the age who is in his twenties contented only with his appearances, in spite of his age. 126 | We do not want anyone to speak for the poor, the exiled, the helpless and the very unfortunate, for fear of running afoul of the laws or of being shut out of schools. We welcome anyone to talk for himself if fear is his worst fear. 127 | My biggest fear ever was to be involved in a plane crash, so I didn't want to go into training like a hamster on a camping trip. 128 | I like all the angels around because they protect me and my guitar. I mean, if you're wild, they protect you all right. 129 | I've never had a fear of flying, so flying to Vegas was just beyond my wildest expectations. 130 | I'll never have the fear of being fired at my career starts I was scheduled to retire, because I don't want to sound like a member of the opposite sex. 131 | If you've got to my age you truly believe that what you believe will happen is wise. All you've got to believe is that you're going to live your life and if you don't, you should! 132 | I don't have any fear of going to bed because of the fear that I'd be awoken by. 133 | We should never confirm our loves even when we have lost our temper. Such a man would surely destroy all respect. 134 | It is part of a good man to have wit and bravery. But a good man to have fear and to be wise in none of his dealings. 135 | I think fear is always a very schizophrenic thing... A man who has a troubled and sinister past has the virtue and the fear of his 136 | You can't just take a hint and expect it to work out. 137 | I hope for the best in everything everything everything. 138 | Life and death are one great road, and those to connect are that which niggles through the brain like a spider's web, which leads to our unconsciousness. 139 | In the practical art of war it is generally believed that you can have as much as you want, but only if you put down the proper manhood which will give you the victory. 140 | The moment you think you understand a good scare story, you don't. 141 | The moment you think you understand, you're in for the next thing. The moment you think you're good, you're not. 142 | It is better to get down to the nitty gritty and die prematurely, early, than to have your heart be the death of your child in a car accident. 143 | Life is one grand, sweet death, or prom of a soul. 144 | How blessed are we, who to consider our children only to our children, and by that we have but one life to fill. 145 | For a long time I tried to manage Sotheby's, but it'd make me miserable. So I'm always trying to save money on charity runs. 146 | No matter how good you are, you're still competing with people who are just as good. The same thing is true of Soul Joe, who's good and who's bad. 147 | Life is the art of drawing sufficient conclusions from insufficient premises. 148 | If you die you're still walking along in the cemetery, with your soul. 149 | I'm sure my best work was not in my sleep, but somewhere in my wildest childhood I made up my mind that I would make music. 150 | I think the best thing I can do is to be a distraction. A husband goes crazy, and he doesn't have a job. He doesn't have a friend. And he can become a nightmare to his children and do their parents no good. 151 | If God would have wanted us to live in, say, a British village, he or he could have chosen somewhere else. 152 | The best way I know to kill a man is to kill his children. 153 | The best way to choose a companion is to choose them as you love to be carried on your shoulders. 154 | The best things in life are unexpected, rare and never used up. 155 | I think it's best to me that the people I like be in charge. They're not in charge themselves, and they have to make the best of it. 156 | The world is the best it can be ignorant. 157 | He who is the best in his profession falls short of what most of his friends are. 158 | I'm not a professional I'm just a tourist. 159 | I think the best thing I can be is to be a bit of a idiot. 160 | The best things in life are unexpected, mild and long. 161 | There are only two types of women: those who think they are beautiful and those who think they're not. 162 | I think life is very interesting but it's also a reflection of what happens on a daily basis. And I sort of see the worst that can happen day-to-day. 163 | We were raised in an Italian-American household, although we didn't speak Spanish until we spoke Spanish and related cultural things. And through the drawings, as much as I could, I adored my Spanish. 164 | My mother's killing thing is true. She was the worst mom. My dad was a bartender until she was very young and my mom worked late and she worked late to raise my grandmother. 165 | I just want to do good work. I despise bad work. So I am a bit jealous of what others think of me. So I have always had jealous thoughts of who could do who work well and work well and what could I do to deserve such attention. 166 | You know how you wake up and, look at me look at you? I deserve to be incredibly blessed look like me and look like my mother. It makes me feel quite poor. 167 | If the people in charge must change hands many times, it is because the people are controlling the way they act. If you don't change you are going to keep controlling. 168 | I'm the cat person. That's a fantastic phrase. I don't like it. I like it when someone speaks it really well. They have a great sense of history and they bring it to the girls. 169 | It seems like the most logical thing in the world to believe in God, to believe that you are the chosen instrument of God, to believe that something is kindled. 170 | The best medicine you can give your soul to be free is to accept the faults of others. 171 | There is only one happiness in this life and that is to feel Free. 172 | There is no such thing as an eternal life. 173 | It's a very interesting situation, with a lot of amazing writers, how they all write. It doesn't make any sense to me, but it's very much an interesting situation. 174 | To be an 175 | God has placed no limits to the freedom of man. He can do what he likes, but only the freedom of man can solve his problems and be satisfied with the successes he has seen. 176 | Freedom is the 'first condition' of success. 177 | Freedom and order are the two claims of present civilization. 178 | Freedom and honesty are two sides of the same coin. 179 | Freedom and generality are never lost who excel in distinction, and imagine where their rights fail. 180 | It is a strange fact that freedom itself, as our present economic system exploits it, and as our educational system and infrastructure abound it could very well be that something is amiss. 181 | Freedom of opinion can only exist when freedom is also sought not by doing, but by acting. 182 | In spite of everything I still want to be a Free Man. I simply can't escape the reality of what it is to which I am entitled. 183 | It is a fact many of us do not have enough freedom to pursue our interests fully enough to achieve the political promises we have wished for years. 184 | Freedom and liberty have a magical talisman, but when freedom and liberty are denied, the results are certain. 185 | The Declaration of Independence is not the declaration of political positions. The freedom that we enjoy as Americans comes from the bedrock principle that believes us most inside, we believe we are the genuine source of justice. 186 | Freedom... is the promise of liberty upon which the nations have always depended. 187 | Human beings the world over need freedom and security that they are prepared to face their fears and misunderstand their limitations. 188 | I am a fighter for a freedom without political tyranny. 189 | The struggle for freedom is a long one, indeed it is only a few years when we were still living under colonial rule and the suspicion of colonial rule. But the movement grew and developed and as we strengthened our ties to some of the nations whovered in similar struggle. 190 | One of the rewards of success is the ability to become a greater person when one is at their best. To find this person is the great reward of life. 191 | Freedom means the freedom to be what we want to be. 192 | The framers of the Constitution were relatively unsentimental in their optimism about the American society. They believed that the government would want to attract a demographic and moral demand. As a practical matter, they believed this demographic and moral demand will dictate the terms of their lives, the terms of their marriage and the length of their marriage. 193 | You don't have freedom when you buy the right to run your life. 194 | Freedom can occur only through scientific change and management of the economy of the future. 195 | The future is not Big Government. Self-serving politicians. Powerful bureaucrats. This has been tried, tested throughout history. The results have been outsized. 196 | The fact is that all human beings have an innate capacity for negative emotional emotion, for finding something of ourselves in literature, in movies, in books. We are born into this environment, and we develop our capacity to develop that capacity over years in the way that we think and feel. 197 | Our political system provides us with endless opportunities to have some personal say in how government is run. The advantages of the ballot are always the greatest when the good people of New York and Chicago have an urge to call on their elected representatives. 198 | The Republican Party has had a track record of giving women more power at the state and federal levels. So, you know, if a Democrat is in the White House and a Republican is in the White House, there is a certain base of folks who believe that there ought to be some kind of order or some sort of direction for our government. 199 | The most important political act of the American Negro is before the world knows what it is about us, about America, that makes us strong. 200 | We have enslaved more men than any other nation in this history. Slave-holders, of course, want all the credit for that. They themselves are to blame. 201 | We believe in expanding opportunities for American men and women to make the world a better place to live. We must assure our children that we will welcome them to show their face and our people as wide as possible. 202 | When you're president, you have to make sure that everyone, the president, is safe in the knowledge that there are safe people for the president to work with. 203 | I wish I had as much freedom as I have now. I didn't want to have school and school and school. 204 | Freedom is contagious. 205 | Freedom passes into the ideal of perfect equality, which, in the case of Africa, is a kind of epidemic justice. 206 | Human rights are human rights. They cannot be raised by a bag of acorns. They cannot be enforced on the social safety net. And that is why the question of freedom of expression is an important question, not simply a question of granting the excuse of freedom. 207 | I don't think that, as an American, I would be elected if I weren't sufficiently conscious of the dangers of taking part in the effort to normalize the relationship with a country that has 208 | I can't find my way out of my writing. Sometimes I get out of myself and write something else. Sometimes I just do something else. Sometimes I do not make a full recovery. 209 | I have had bad nights, when unusual or inexplicable circumstances make me despair of things I cannot see. Sometimes I have gotten good nights, or bad nights, when things I have predicted would happen if I had been more prepared. 210 | I've decided to hang onto my craft and keep trying out things. I'm finding that it never quite comes together. 211 | The art of living does not consist in preserving and clinging to a particular mode of happiness, but in allowing liberty to the occasional accidents of life. 212 | You can't take a talent you put in your character and it lasts a lifetime. 213 | A man filled with the love of God may be incapable of fitness for any office or position. In this respect it is a paradox. 214 | Life is made up: the things we do make up what we think. 215 | A successful man is one who can devote himself to his objects in season and out of time. 216 | I'm positive and I smile a lot, and I'm kind of a banana, but serious work just seems to find me, so I'm not going to argue with it. 217 | Youth isn't made of love thoughts are made of regrets. 218 | Youth is what happens when resources are limited. 219 | There is only one path and that is the world of entertainment, which teaches us to give each others credit for our successes and failures. The world of work is great but the world of life is better than the world of work. 220 | The greatest honor you can enjoy in life, and that of a man or woman, is to have your dreams realized. 221 | The world is made up of families, friends and family, and I have friends in different industries, but I don't really care about friends. I care about what they do. I don't really care who the editor is, the sports editor, the football coach, the firefighter - everyone has different friends in different industries. 222 | One of the darkest places you can go to in your life is have your parents smile at you. 223 | If the only interest you have towards something you are doing will pay off, you are screwed. If you don't produce value for money, you are screwed. And the end is not in sight. 224 | When you have had that dream, take that chance, take it any day of the week, no matter what the reason. The more you sacrifice the past, the present and future, the more exciting the present is. 225 | The world is governed more by personality than by accomplishments. 226 | There's no such thing is prejudice, everything is prejudice. 227 | There is no prejudice against women in America. There is only an epidemic form of it. 228 | I'm not only a manager, but a thinker, and I'm thinking. 229 | If a building becomes a work of art, the importance that an architect attaches to it must become clear to the worker. The task must become a task for the engineer. 230 | I'm not interested in facelifts. This kills the creativity. This is too much work. 231 | The joys of work are impelled largely by the fear of not finding what is. 232 | The work of art is to photograph what is known and what is not. 233 | The artist cannot express what he is, comprehend it by imagery, or define it by something else. 234 | What language could be more abstract than the abstract? Yet it is language, that separates art from history. 235 | I think art history is a time when we talk about where our point of view is, and we tend to focus more on cultural and aesthetic. 236 | The artist is not his matter is what he is, his essence. 237 | A work of art is generally of the same price as a play. 238 | It is an artist's duty. 239 | Every art which is great and unique, every masterpiece that appears in a man's memory, is an artist's treasure. 240 | A sculptor is a person who is interested in his trade and yet is able to retain what he had, in order to forge new works of art. 241 | Painting is by nature an exquisite expression of power than many pictures are. 242 | Art is a crude reflection on the real world. 243 | A sculptor is a person who is interested in his trade and yet is able to keep what he has. 244 | In fact, I don't think an artist is good unless the parts he needs are also what will make up his personality. 245 | Art is the knowing that gives you the most without looking back. 246 | An artist is only better than a sculptor once he has realized him. 247 | The sculptor does not examine the artist's work in order to make a figure - he merely subtract the sculptor's power from his power. 248 | Painting and music are very simple things. You can't understand them. 249 | The sculptor does not examine the artist's work in order to make a figure - he merely subtracts the power 250 | There are a lot of things that people wish I would tell you about success. Some days it is great, some days it is difficult and some days it is hard work. But I think there is one thing I canstate. That being said, I Will Tell You about my Folly Of Memories. Now In My Heart I Am, One Of The Nation's Greatest Hits. Now In My Hand I was a Young River State River State. And It Wasn't a Flower And There was a Tree. 251 | There is no meaning to life except happiness. Then happiness is the world. 252 | It is evident that happiness, on flowers, may be heard from heaven, but in never life comes thought necessary, which never does. 253 | Happiness is not a goal but a sublimity of mind. 254 | There is nothing which can stop thinking. Its start and finish is uncertain. 255 | The beginning and end of times are determined by mind. 256 | Happiness? There's nothing wrong with that. Just try to work from the standpoint of happiness. 257 | You must not give up that which you do not need, in order to be happy you must not waste time or money. 258 | The trouble with always telling stories is that you never quite get the happiness you want. 259 | Your successes and happiness are locked in your lap you must constantly move and burnuate your efforts until they feel worthy of your thoughts. 260 | Happiness is nothing more than bringing that which is impossible to get, or getting that which is easy to find. 261 | There is no progress or change until one is ready to feel one's pain. Pain is the beginning of discontent. 262 | The beginning and end of times is tied to everyone. 263 | The happiness of your life depends upon the quality of your thoughts: therefore, guard accordingly, and take care that you entertain no notions unsuitable to the virtue of your own life. 264 | To know one's defects leads to the discovery of your inmost treasure. 265 | I don't think it's a big deal to be happy when there's nothing but happiness in your life. 266 | If you want to be happy, do yourself a favor and know what you're going to get. If you just want to be happy, do yourself a favour and know what you're going to be at some point in the future. 267 | Life is essentially a cheat and its healthy results only temporary comfort and self-respect only temporary prosperity. 268 | The pursuit of happiness, the pursuit of peace, and the pursuit of happiness by burning your ego in the bonfire are two different things. 269 | Happiness is not something you postpone for the future it is something you design for the present. 270 | If you are unhappy, whether you are happy or not, it's probably best to know what you're doing and to try to deal with it from your old age. 271 | I tremble for my country when I hear its grotesan and when I feel the murmur of the water. Because this is my country, this is my country. 272 | The happiness which is known, is esteemed for its wealth and the content of its misery. 273 | Happiness is not a matter of events, but of beings. 274 | Happiness is not a matter of ideas, but a state of mind. It cannot be said to have feelings of joy or depression. Its fulfillment is not assured. 275 | Money can't buy happiness, but it can buy a lot of happiness. 276 | Happiness cannot be found when the body is tired. It must be played. 277 | To give without caring that has never been tried is like giving without caring that has never been said. 278 | I've been playing against this new U.S. standard stuff, which is that you cannot take his value system as far outside the U.S. as Canada is concerned. There you have it. We're a country that doesn't discriminate. We don't discriminate on the basis of sex or anything like that. 279 | We're born in a moment, we're born out of many opportunities in this life, and most people's hopes are inflated before they are really born. 280 | I don't know what to say to that, Stanton. I don't know whether to like you or not. I don't know whether you like me or not. I'm not trying to modify my mother's love for my father. What I am saying is that life is almost as important as a child, and you really don't want to be born without a parent. 281 | My life has run from past trouble to past trouble. It has seen times, periods of great feeling, emotions of great pain, great joy, sad and sometimes violent moments of triumph. 282 | A life lived for poetry was a refuge from the horrible and the infinite. 283 | If you're going to poetry, as long as you get a living, then you've got it. 284 | Poetry is a way of taking life completely by surprise and changing it into something beautiful. 285 | Poetry is love, love is poetry, poetry is life. 286 | Poetry is love, poetry is peace. 287 | All art since 288 | My mom always said that the reason why I didn't get married was because I didn't wear a dress that day. It would probably stick up in the bathroom, so I wasn't allowed to wear anything. 289 | I grew up in a family of Republicans and Democrats and most especially to Blues-Hound and Howard Stern. For six years and I lived in a house where Republicans could talk down to the players, and the players talked down to the fans. 290 | I don't know anything that puts an indeterminate amount of power in the hands of hard working people. Maybe they have faith but they don't know everything. 291 | There is a gigantic shift being an actor taking age seriously. I'm interested in old fashioned, old fashioned. There isn't a complicated system that could hold against me. I have such faith in women's development that I wouldn't ever change anything. 292 | The type of training that I'm interested in entering into is kind of like a boxing ring. It's a waiting room for people who want to get treatment. It's a gradually getting better computerised system for giving information. 293 | There's a computer bug that makes it more likely that me going to the store after making inquiries is going to change everything for everyone. I just don't care if the store has a copy of something I want to know more than me. 294 | You can change your life by changing what you think. 295 | The only way you can make your dreams come true is to face your dreams and work at them. 296 | Don't wait see if you could get a record deal or whatever it is. You couldn't make a record and work. You had to work and hope. 297 | The United States strongly condemns the illegal disclosure of classified information. If you're caught, you should be immediately put on administrative leave and told that you can file a lawsuit as a defense attorney. That would be of immense help to the victims of illegal disclosure. 298 | If we as a people, and the American people, as a people want to change, we should all listen to our constituents. They are entitled to know and they are entitled to be educated. They are entitled to have an honest life. 299 | You listen to 'Glee' and it's pretty much the same as the mornings when I'm not on the phone. People change, but they don't change much. 300 | I'm a person that if you're healthy, you feel good. If you're injured, you feel good. I live my life to keep my injuries to myself. 301 | I was so obsessed by this problem that I was thinking about it constantly for years - even though I knew I had a solution. It was about six years ago, a little bit after I started looking into psychology. 302 | It seems that when you have freedom you magnify everything, you do things that you never thought you could do. 303 | I hope that I'll work more to the last four people, completely and painfully, with the kind of clarity that I showed my children on - kids feeling, I didn't dare kid you, you have to know, about the future, right now, it's going to be real quick. I don't want to be cocky. I don't want to be the son of a man, I don't want to be the daughter of a man. 304 | My goal is to get to the NBA. When you're trying to make it, you want it to make you. Last game of the year? Whatever, get your job done. Last minute of the night? Whatever. Whatever gives you the courage to work hard and get in the game, don't miss. 305 | I was terrified of being someone who funny in my family. I was one of those people. I had to deal with all those anxiety phases, and pain. There was never a time I didn't have a fear of being male. 306 | I am a type-2 diabetic, and I did everything that could be to dangerous by the time I was 18. I ate medication to prevent the most likely cause, shame, which means no one can shame you - weight gain. And that's fine if you have a face of some kind, like Listerine. It works great, I'm a diabetic, so I started getting little pimples on my fingers and toes. 307 | The thing about boxing is that you get the best fighter first... because they're not hungry or tired because those seconds matter. If they are, they're great. If they don't, they're not great. 308 | I love how she always holds up a little mirror to me and says, 'I'm just a kid. I suck at life.' I like her to be true to herself and not keep on dragging it along. I always have a niece who's like my niece. 309 | Life is very interesting, but you have to have fear, because life is not fair. 310 | I always say that if you want to work at being successful, you should go out into the world and teach yourself the language of Leibniz and Einstein. And 311 | My friends say, 'Remember, Christmas is over.' Remember, Dad said, 'The next time you need a miracle, remind the nuns.' 312 | I used to be such a perky girl with glamorous parents that when I had to go to Vegas I would plink into charity. I even had one sock from the ice cream cone that had the face of a cat. It was a shock - the same thing two years later when I was nine. 313 | You know, I always had three cats, Whistler, and Greyie. That's some serious kitty glam, man. 314 | My childhood sweet - not so very happy, but she still loves me, and she loves me now. My childhood aristocratically - miserable people go about their business like they do now, and they come to money and commodities but they never loved, and they never are. 315 | You may not be able to change a situation, but with humor you can change your attitude about it. 316 | As the true object of your suffering becomes known, so the true object of your courage will soon take center stage. 317 | When I was a kid I didn't feel like I fit in because my father used to sell my albums and my mom and my stepdad lived off my money. 318 | I can make a firm pledge, under my plan, no senior affecting the policy or raising public policy or raising public policy any higher than it already is. 319 | It is not death or destruction that is the great tragedy of our lives, but the fear of death which is our punishment for sin. 320 | I've been writing and reading a lot and I find interesting writing and interesting reading all the time. Sometimes I find myself writing at night, or occasionally during my nap. Sometimes I don't. 321 | I'm interested in spirituality and in religion and our relationship to the divine. I don't think I've fit in as a Catholic, but I am part of the social/political animal. 322 | We are biological creatures, born, bred, and die by what we make. That is our nature. However, we have become little pieces of our complex, never-shrinking and often unnatural intelligence, multiplied and multiplied and multiplied. 323 | The art of medicine consists in fondness for the shop door and a kind of moral affection for the door. 324 | One of the great problems with women is we have such fastidious feet of picketing that we fight so hard to get what we got. 325 | I always preferred goldfish. They show up in most places and the fishermen lost their jobs. But the girl fancier has a fish and wants to tell the fish she has. That made women unnecessary and had to be reduced to a fashion. 326 | When I was trying to impress Kate I was trying to make it in, and you're hooking up with the right guy and the right guy is the right person, so there will always be the same person. I think it's I, not him, but there's a connection that makes it happen. 327 | When you make it not be a matter of making money, you can forgive, but it can't keep you from going on. From now on you're on your own. If you just keep on trying and working, you'll have money in your bank that can be very important in your career. 328 | I think the world is going to figure out how we get along with each other, so we can move the flag forward for women from every corner of the globe. 329 | I would say, if you lived in either 1991 or 1997, you were living in blockhouses. It was all stuff you'd expect to see in the life of a miners' family: drunken, violent, in love, poor, male, protector of the poor, godless. 330 | I got a lot of stuff, but I also made some fantastic mistakes. 331 | You do the right thing as an artist. That's something you learn from. Paint with passion, make mistakes, keep to the real. And that's what I meant when I said it was I went to learn and to keep trying. 332 | A radical transformation of corporate culture could begin on the 16th, with the creation of a new finance ministry, and on the basis of a new sovereign wealth fund. 333 | I have a lot of money I put into smaller films, but nothing more. My little brothers and I were working on never earning beginnings. 334 | It is a melancholy truth that even great men have their poor relations. 335 | You know that when you're pretty, you don't need anybody to tell you how to do it. I don't mean I don't want to be a model - I just fear model - but I mean when you're pretty, you don't need anybody to tell you how to do it. 336 | I love the music of Tarantino. He really made me hate everything - the enemy of all good music and today I don't like it. I hate it when people talk about my character, when I make music, what kind of music I make, how bad it is. 337 | If you 338 | I've never had a relationship with a record, until I put it out. Then I got a record, I heard your groove, and decided that it was a relationship, and then when I heard your records, I made you mine. 339 | I don't need anyone to make friendship stronger just for me and my family. I'm not about to let anybody take the relationship away from me or destroy it. 340 | It's very hard for actors to play their relationship, it's not too difficult to do. But the audience sees it like a performance, and the copyright holder knows what it means. The audience sees it as some kind of contract. 341 | I love romance. All relationships are beautiful. 342 | I'm a very passionate guy. I love the idea of love, the idea of freedom, and the fun of love. The whole animal. I have a dark relationship to sex, than to metal, or blue, and I have a good relationship to romance. 343 | When I was in my 20s, I was in therapy for a lot of years. I didn't feel like I was in a good relationship. 344 | In a relationship, you have to survive on your own. It's sort of like your car, except you never run out of gas. 345 | I was lucky enough to be the love of my life, but I taught my son how to be a better man since the age of 16, and he came back and was like, 'I'm so lucky I got that key.' I'm like, 'You got to go.' 346 | If I wasn't a writer, I would be in rehab. I wouldn't feel happy, I wouldn't feel fulfilled, and that's a lot of selfish people in a lot of ways. But I'd like to find a relationship that willlast, you know, maybe even be the girl at the bar one day. No lies, little selfish dreams. 347 | It's a luxury to be able to retire with a clear conscience. I feel that's wrong and I would like a relationship that takes the reader on a journey. I need company, being able to share wisdom with myself and my inner person. 348 | My grandmother and I had a baby at 18 years old, and nearly died by that age, you know. So my own life was changed on that day. My entire life, I was no longer in a relationship with my grandmother. 349 | My grandparents got together only in 1948. That's a little hard because you have to start somewhere and work your way into your twenties, or your early thirties. 350 | If you're the type of person who has to fulfill a relationship, you don't have that. 351 | It's not like I had plans D, E, or F. My mom and my dad built them. I had enough, or not, unfortunately. But it gave me a measure of security: I could have got by being a better person, but I didn't have enough. 352 | My parents' love had a way of making me see the world as long as I lived. My brother and sister, for example, had a desire for girls their whole lives, so even if they didn't see it coming I had hopes that my brother and I would look on him with adoration. 353 | But we had both our parents have tattoos, and we both had a pretty extensive criminal history. So I was in a very vulnerable phase of my life. As an actor I was able to relate to people and still be vulnerable. 354 | I think I'm vulnerable to people asking me out. My dad says you go out with the girls and have fun, but I say you don't come with our bodies. He says we should stay in our place on 'course of action.' My dad says you go in and have fun, but that won't be enough. 355 | The human body and mind are in fact the great obstacles to curing cancer, but it is the clothes and shoes worn in surgeries that pose the greatest risk to our health. 356 | For all my career as an actor, I've never been comfortable with closed captions on movies so I had my movies screened alone in order to review them. 357 | I mean, I'm old, I have a lot of health, I've got a good body. 358 | But I think also, when you know, some actors, whether I've known them or not, they just seem to be very patient and you know, and the relationship that we have as a band and as a team, it always goes like this, you know? 359 | People keep asking me about it, but I haven't even studied that far back. Why should actors just go do that? Why business? 360 | In between records I like to sing about nature and nature and maybe even about nature and maybe even spirituality. Nature is a bit of a go-getter for me. 361 | The person who needs drama in their career, but who loves it, is the one who needs drama in his career. 362 | There's an internal battle, whether it's your agency, whether it's your 363 | It is said that the will is always stronger than the power. How loud would be the sound of a clacking clock to name just two powerful things: the clacking clock and the ticking time. 364 | If you have enough money to make it as a photographer, you are going to pay attention. But if you don't have enough money, you are going to pay too. 365 | The power of the harasser, the abuser, the rapist depends above all on the silence of women. 366 | I decided to make weight training a major part of my life and I feel that having a body that is feminine is indispensable. 367 | If the power to think was my biggest sin - that's what it took to get me to stop worrying about it. 368 | It is with an idea as large as its power) as it is with little power as to itsminess. 369 | I do not believe in the God of theology who rewards good and punishes evil. 370 | I like being alone. I like being alone. I want to go out and live my life alone. I don't know if it's good for anybody but me. 371 | I think you can calculate happiness by having somebody you know who is really miserable, who is not working would happily be happy if so, provided you could get along rather than each going somewhere else. 372 | I'm at the age where only somebody I know understands how to write a song is happy if they're happy to deliver a song on time, and if you can just take the best words of your mouth and put them into your song, that's my happiness. 373 | I love being alone. I'm too afraid to stay in love. 374 | It is not by muscle, speed, or skill that people should be drawn to illusion it is by muscle, speed, and skill that they are seduced by. Herding or moving near ones self makes it a place of wonder, while looking for any movement which will stir them. 375 | But men are so busy writing about women they deal with very little else. 376 | I believe children are our future. Teach them well and teach them well. 377 | It is not by the power of rhetoric that we can make our human beings feel that they have a confident place for peace. The important work of improving the lives of the human race is beyond their reach. 378 | The happiness in this world depends on the will power: the longer a soul can think, the greater will be its will to live. 379 | No man ever achieved worth-while happiness who did not rely on temporary comfort and security. 380 | You don't want to die like some kids after a hard day's lessons, you want to get up and try to do something new. 381 | Those who can write will rarely give up, nor will their memory be forever altered by the demands of writing. 382 | The art of being happy lies in the individual alone. 383 | The artist does not express in words his own nature, but expresses in the way in which nature is related to him. 384 | What is important is to make an image of what you are and to take it to your head. So an 18-year-old will learn how to succeed in the United States. He can see that the main obstacle to his success lies in being under the illusion that there is a better than barrier between him and the elite. 385 | Success is 99.99999% correct. Failure is absolute. 386 | Happiness is getting up every single time, being overwhelmed by the experience, and finding yourself again with butterflies when you wake up. 387 | The wise, when they first meet a fool, often ask what folly, for fools make no error return seeking to be wise in all their manifestations to which it is most wise, or most stupid. 388 | You can never give up. You can't give up your time. Your time is your money. 389 | If you have a chance like I had had a hundred times, you are lucky in that you don't interrupt once. But I have been lucky. 390 | Most men never dream of being something they cannot love. 391 | One should never criticize men and women. They are quick to do and quick to change. 392 | There are only two events on a man's mind - breaking of the law and finding justice. 393 | I think everybody is destined for this. What we are here to isful success. If everybody else is destined for this, then everybody will be here someday. 394 | The hardest people reach their real goal, which is to be who they are. 395 | I love climbing, I love photography, I love dance, and I want to get to high school and do gymnastics. All of them are trees in the open air. 396 | All real success and all of our real achievements take preparation. 397 | To my disappointment the men I love most in life are pessimists. They see this world as being basically in this rut, forever striving for the impossible. 398 | There are only two types of people: idealists who envision this world as being absolutely doomed, and critics who imagine the possibilities to which the world will inevitably pass. 399 | It is 400 | -------------------------------------------------------------------------------- /src/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jingw222/twitterbot-gpt2/5c9b8f502e45e9dbe45bb6426bec3cdd80ad8db2/src/__init__.py -------------------------------------------------------------------------------- /src/accumulate.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import json 3 | import os 4 | import numpy as np 5 | import tensorflow as tf 6 | import time 7 | 8 | 9 | class AccumulatingOptimizer(object): 10 | def __init__(self, opt, var_list): 11 | self.opt = opt 12 | self.var_list = var_list 13 | self.accum_vars = {tv : tf.Variable(tf.zeros_like(tv.initialized_value()), trainable=False) 14 | for tv in var_list} 15 | self.total_loss = tf.Variable(tf.zeros(shape=[], dtype=tf.float32)) 16 | self.count_loss = tf.Variable(tf.zeros(shape=[], dtype=tf.float32)) 17 | 18 | def reset(self): 19 | updates = [tv.assign(tf.zeros_like(tv)) for tv in self.accum_vars.values()] 20 | updates.append(self.total_loss.assign(tf.zeros(shape=[], dtype=tf.float32))) 21 | updates.append(self.count_loss.assign(tf.zeros(shape=[], dtype=tf.float32))) 22 | with tf.control_dependencies(updates): 23 | return tf.no_op() 24 | 25 | def compute_gradients(self, loss): 26 | grads = self.opt.compute_gradients(loss, self.var_list) 27 | updates = [self.accum_vars[v].assign_add(g) for (g,v) in grads] 28 | updates.append(self.total_loss.assign_add(loss)) 29 | updates.append(self.count_loss.assign_add(1.0)) 30 | with tf.control_dependencies(updates): 31 | return tf.no_op() 32 | 33 | def apply_gradients(self): 34 | grads = [(g,v) for (v,g) in self.accum_vars.items()] 35 | with tf.control_dependencies([self.opt.apply_gradients(grads)]): 36 | return self.total_loss / self.count_loss 37 | -------------------------------------------------------------------------------- /src/encoder.py: -------------------------------------------------------------------------------- 1 | """Byte pair encoding utilities""" 2 | 3 | import os 4 | import json 5 | import regex as re 6 | from functools import lru_cache 7 | 8 | basedir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) 9 | 10 | 11 | @lru_cache() 12 | def bytes_to_unicode(): 13 | """ 14 | Returns list of utf-8 byte and a corresponding list of unicode strings. 15 | The reversible bpe codes work on unicode strings. 16 | This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. 17 | When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. 18 | This is a signficant percentage of your normal, say, 32K bpe vocab. 19 | To avoid that, we want lookup tables between utf-8 bytes and unicode strings. 20 | And avoids mapping to whitespace/control characters the bpe code barfs on. 21 | """ 22 | bs = list(range(ord("!"), ord("~")+1))+list(range(ord("¡"), ord("¬")+1))+list(range(ord("®"), ord("ÿ")+1)) 23 | cs = bs[:] 24 | n = 0 25 | for b in range(2**8): 26 | if b not in bs: 27 | bs.append(b) 28 | cs.append(2**8+n) 29 | n += 1 30 | cs = [chr(n) for n in cs] 31 | return dict(zip(bs, cs)) 32 | 33 | def get_pairs(word): 34 | """Return set of symbol pairs in a word. 35 | 36 | Word is represented as tuple of symbols (symbols being variable-length strings). 37 | """ 38 | pairs = set() 39 | prev_char = word[0] 40 | for char in word[1:]: 41 | pairs.add((prev_char, char)) 42 | prev_char = char 43 | return pairs 44 | 45 | class Encoder: 46 | def __init__(self, encoder, bpe_merges, errors='replace'): 47 | self.encoder = encoder 48 | self.decoder = {v:k for k,v in self.encoder.items()} 49 | self.errors = errors # how to handle errors in decoding 50 | self.byte_encoder = bytes_to_unicode() 51 | self.byte_decoder = {v:k for k, v in self.byte_encoder.items()} 52 | self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges)))) 53 | self.cache = {} 54 | 55 | # Should haved added re.IGNORECASE so BPE merges can happen for capitalized versions of contractions 56 | self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""") 57 | 58 | def bpe(self, token): 59 | if token in self.cache: 60 | return self.cache[token] 61 | word = tuple(token) 62 | pairs = get_pairs(word) 63 | 64 | if not pairs: 65 | return token 66 | 67 | while True: 68 | bigram = min(pairs, key = lambda pair: self.bpe_ranks.get(pair, float('inf'))) 69 | if bigram not in self.bpe_ranks: 70 | break 71 | first, second = bigram 72 | new_word = [] 73 | i = 0 74 | while i < len(word): 75 | try: 76 | j = word.index(first, i) 77 | new_word.extend(word[i:j]) 78 | i = j 79 | except: 80 | new_word.extend(word[i:]) 81 | break 82 | 83 | if word[i] == first and i < len(word)-1 and word[i+1] == second: 84 | new_word.append(first+second) 85 | i += 2 86 | else: 87 | new_word.append(word[i]) 88 | i += 1 89 | new_word = tuple(new_word) 90 | word = new_word 91 | if len(word) == 1: 92 | break 93 | else: 94 | pairs = get_pairs(word) 95 | word = ' '.join(word) 96 | self.cache[token] = word 97 | return word 98 | 99 | def encode(self, text): 100 | bpe_tokens = [] 101 | for token in re.findall(self.pat, text): 102 | token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8')) 103 | bpe_tokens.extend(self.encoder[bpe_token] for bpe_token in self.bpe(token).split(' ')) 104 | return bpe_tokens 105 | 106 | def decode(self, tokens): 107 | text = ''.join([self.decoder[token] for token in tokens]) 108 | text = bytearray([self.byte_decoder[c] for c in text]).decode('utf-8', errors=self.errors) 109 | return text 110 | 111 | def get_encoder(model_name): 112 | with open(os.path.join(basedir, 'models', model_name, 'encoder.json'), 'r') as f: 113 | encoder = json.load(f) 114 | with open(os.path.join(basedir, 'models', model_name, 'vocab.bpe'), 'r', encoding="utf-8") as f: 115 | bpe_data = f.read() 116 | bpe_merges = [tuple(merge_str.split()) for merge_str in bpe_data.split('\n')[1:-1]] 117 | return Encoder( 118 | encoder=encoder, 119 | bpe_merges=bpe_merges, 120 | ) 121 | -------------------------------------------------------------------------------- /src/generate_unconditional_samples.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import fire 4 | import json 5 | import os 6 | import numpy as np 7 | import tensorflow as tf 8 | 9 | from src import model, sample, encoder 10 | 11 | basedir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) 12 | 13 | def sample_model( 14 | model_name='117M', 15 | seed=None, 16 | nsamples=1, 17 | batch_size=1, 18 | length=None, 19 | temperature=1.0, 20 | top_k=40, 21 | ): 22 | """ 23 | Run the sample_model 24 | :model_name=117M : String, which model to use 25 | :seed=None : Integer seed for random number generators, fix seed to 26 | reproduce results 27 | :nsamples=20 : Number of samples to return, if 0, continues to 28 | generate samples indefinately. 29 | :batch_size=1 : Number of batches (only affects speed/memory). 30 | :length=None : Number of tokens in generated text, if None (default), is 31 | determined by model hyperparameters 32 | :temperature=1 : Float value controlling randomness in boltzmann 33 | distribution. Lower temperature results in less random completions. As the 34 | temperature approaches zero, the model will become deterministic and 35 | repetitive. Higher temperature results in more random completions. 36 | :top_k=40 : Integer value controlling diversity. 1 means only 1 word is 37 | considered for each step (token), resulting in deterministic completions, 38 | while 40 means 40 words are considered at each step. 0 (default) is a 39 | special setting meaning no restrictions. 40 generally is a good value. 40 | """ 41 | enc = encoder.get_encoder(model_name) 42 | hparams = model.default_hparams() 43 | with open(os.path.join(basedir, 'models', model_name, 'hparams.json')) as f: 44 | hparams.override_from_dict(json.load(f)) 45 | 46 | if length is None: 47 | length = hparams.n_ctx 48 | elif length > hparams.n_ctx: 49 | raise ValueError("Can't get samples longer than window size: %s" % hparams.n_ctx) 50 | 51 | with tf.Session(graph=tf.Graph()) as sess: 52 | np.random.seed(seed) 53 | tf.set_random_seed(seed) 54 | 55 | output = sample.sample_sequence( 56 | hparams=hparams, length=length, 57 | start_token=enc.encoder['<|endoftext|>'], 58 | batch_size=batch_size, 59 | temperature=temperature, top_k=top_k 60 | )[:, 1:] 61 | 62 | saver = tf.train.Saver() 63 | ckpt = tf.train.latest_checkpoint(os.path.join(basedir, 'models', model_name)) 64 | saver.restore(sess, ckpt) 65 | 66 | generated = 0 67 | while nsamples == 0 or generated < nsamples: 68 | out = sess.run(output) 69 | for i in range(batch_size): 70 | generated += batch_size 71 | text = enc.decode(out[i]) 72 | print(text) 73 | 74 | return text 75 | 76 | if __name__ == '__main__': 77 | fire.Fire(sample_model) 78 | 79 | -------------------------------------------------------------------------------- /src/interactive_conditional_samples.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import fire 4 | import json 5 | import os 6 | import numpy as np 7 | import tensorflow as tf 8 | 9 | import model, sample, encoder 10 | 11 | def interact_model( 12 | model_name='117M', 13 | seed=None, 14 | nsamples=1, 15 | batch_size=1, 16 | length=None, 17 | temperature=1.0, 18 | top_k=20, 19 | ): 20 | """ 21 | Interactively run the model 22 | :model_name=117M : String, which model to use 23 | :seed=None : Integer seed for random number generators, fix seed to reproduce 24 | results 25 | :nsamples=1 : Number of samples to return total 26 | :batch_size=1 : Number of batches (only affects speed/memory). Must divide nsamples. 27 | :length=None : Number of tokens in generated text, if None (default), is 28 | determined by model hyperparameters 29 | :temperature=1 : Float value controlling randomness in boltzmann 30 | distribution. Lower temperature results in less random completions. As the 31 | temperature approaches zero, the model will become deterministic and 32 | repetitive. Higher temperature results in more random completions. 33 | :top_k=0 : Integer value controlling diversity. 1 means only 1 word is 34 | considered for each step (token), resulting in deterministic completions, 35 | while 40 means 40 words are considered at each step. 0 (default) is a 36 | special setting meaning no restrictions. 40 generally is a good value. 37 | """ 38 | if batch_size is None: 39 | batch_size = 1 40 | assert nsamples % batch_size == 0 41 | 42 | enc = encoder.get_encoder(model_name) 43 | hparams = model.default_hparams() 44 | with open(os.path.join('models', model_name, 'hparams.json')) as f: 45 | hparams.override_from_dict(json.load(f)) 46 | 47 | if length is None: 48 | length = hparams.n_ctx // 2 49 | elif length > hparams.n_ctx: 50 | raise ValueError("Can't get samples longer than window size: %s" % hparams.n_ctx) 51 | 52 | with tf.Session(graph=tf.Graph()) as sess: 53 | context = tf.placeholder(tf.int32, [batch_size, None]) 54 | np.random.seed(seed) 55 | tf.set_random_seed(seed) 56 | output = sample.sample_sequence( 57 | hparams=hparams, length=length, 58 | context=context, 59 | batch_size=batch_size, 60 | temperature=temperature, top_k=top_k 61 | ) 62 | 63 | saver = tf.train.Saver() 64 | ckpt = tf.train.latest_checkpoint(os.path.join('models', model_name)) 65 | saver.restore(sess, ckpt) 66 | 67 | while True: 68 | raw_text = input("Model prompt >>> ") 69 | while not raw_text: 70 | print('Prompt should not be empty!') 71 | raw_text = input("Model prompt >>> ") 72 | context_tokens = enc.encode(raw_text) 73 | generated = 0 74 | for _ in range(nsamples // batch_size): 75 | out = sess.run(output, feed_dict={ 76 | context: [context_tokens for _ in range(batch_size)] 77 | })[:, len(context_tokens):] 78 | for i in range(batch_size): 79 | generated += 1 80 | text = enc.decode(out[i]) 81 | print("=" * 40 + " SAMPLE " + str(generated) + " " + "=" * 40) 82 | print(text) 83 | print("=" * 80) 84 | 85 | if __name__ == '__main__': 86 | fire.Fire(interact_model) 87 | 88 | -------------------------------------------------------------------------------- /src/load_dataset.py: -------------------------------------------------------------------------------- 1 | import glob 2 | import numpy as np 3 | import os 4 | import random 5 | import tensorflow as tf 6 | import tqdm 7 | 8 | 9 | def load_dataset(enc, path, combine): 10 | paths = [] 11 | if os.path.isfile(path): 12 | # Simple file 13 | paths.append(path) 14 | elif os.path.isdir(path): 15 | # Directory 16 | for (dirpath, _, fnames) in os.walk(path): 17 | for fname in fnames: 18 | paths.append(os.path.join(dirpath, fname)) 19 | else: 20 | # Assume glob 21 | paths = glob.glob(path) 22 | 23 | token_chunks = [] 24 | raw_text = '' 25 | for path in tqdm.tqdm(paths): 26 | if path.endswith('.npz'): 27 | # Pre-encoded 28 | with np.load(path) as npz: 29 | for item in npz.files: 30 | token_chunks.append(npz[item]) 31 | else: 32 | # Plain text 33 | with open(path, 'r') as fp: 34 | raw_text += fp.read() 35 | if len(raw_text) >= combine: 36 | tokens = np.stack(enc.encode(raw_text)) 37 | token_chunks.append(tokens) 38 | raw_text = '' 39 | else: 40 | raw_text += '<|endoftext|>' 41 | if raw_text: 42 | tokens = np.stack(enc.encode(raw_text)) 43 | token_chunks.append(tokens) 44 | return token_chunks 45 | 46 | 47 | def binary_search(f, lo, hi): 48 | if f(lo) or not f(hi): 49 | return None 50 | while hi > lo + 1: 51 | mid = (lo + hi) // 2 52 | if f(mid): 53 | hi = mid 54 | else: 55 | lo = mid 56 | return hi 57 | 58 | 59 | class Sampler(object): 60 | """Fairly samples a slice from a set of variable sized chunks. 61 | 62 | 'Fairly' means that the distribution is the same as sampling from one concatenated chunk, 63 | but without crossing chunk boundaries.""" 64 | 65 | def __init__(self, chunks): 66 | self.chunks = chunks 67 | self.total_size = sum(chunk.shape[0] for chunk in chunks) 68 | self.boundaries = [0] 69 | for i in range(len(chunks)): 70 | self.boundaries.append(self.boundaries[-1] + chunks[i].shape[0]) 71 | 72 | def sample(self, length): 73 | assert length < self.total_size // len( 74 | self.chunks 75 | ), "Dataset files are too small to sample {} tokens at a time".format( 76 | length) 77 | while True: 78 | index = random.randint(0, self.total_size - length - 1) 79 | i = binary_search(lambda j: self.boundaries[j] > index, 0, 80 | len(self.boundaries) - 1) - 1 81 | if self.boundaries[i + 1] > index + length: 82 | within_chunk = index - self.boundaries[i] 83 | return self.chunks[i][within_chunk:within_chunk + length] 84 | -------------------------------------------------------------------------------- /src/model.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tensorflow as tf 3 | from tensorflow.contrib.training import HParams 4 | 5 | def default_hparams(): 6 | return HParams( 7 | n_vocab=0, 8 | n_ctx=1024, 9 | n_embd=768, 10 | n_head=12, 11 | n_layer=12, 12 | ) 13 | 14 | def shape_list(x): 15 | """Deal with dynamic shape in tensorflow cleanly.""" 16 | static = x.shape.as_list() 17 | dynamic = tf.shape(x) 18 | return [dynamic[i] if s is None else s for i, s in enumerate(static)] 19 | 20 | def softmax(x, axis=-1): 21 | x = x - tf.reduce_max(x, axis=axis, keepdims=True) 22 | ex = tf.exp(x) 23 | return ex / tf.reduce_sum(ex, axis=axis, keepdims=True) 24 | 25 | def gelu(x): 26 | return 0.5*x*(1+tf.tanh(np.sqrt(2/np.pi)*(x+0.044715*tf.pow(x, 3)))) 27 | 28 | def norm(x, scope, *, axis=-1, epsilon=1e-5): 29 | """Normalize to mean = 0, std = 1, then do a diagonal affine transform.""" 30 | with tf.variable_scope(scope): 31 | n_state = x.shape[-1].value 32 | g = tf.get_variable('g', [n_state], initializer=tf.constant_initializer(1)) 33 | b = tf.get_variable('b', [n_state], initializer=tf.constant_initializer(0)) 34 | u = tf.reduce_mean(x, axis=axis, keepdims=True) 35 | s = tf.reduce_mean(tf.square(x-u), axis=axis, keepdims=True) 36 | x = (x - u) * tf.rsqrt(s + epsilon) 37 | x = x*g + b 38 | return x 39 | 40 | def split_states(x, n): 41 | """Reshape the last dimension of x into [n, x.shape[-1]/n].""" 42 | *start, m = shape_list(x) 43 | return tf.reshape(x, start + [n, m//n]) 44 | 45 | def merge_states(x): 46 | """Smash the last two dimensions of x into a single dimension.""" 47 | *start, a, b = shape_list(x) 48 | return tf.reshape(x, start + [a*b]) 49 | 50 | def conv1d(x, scope, nf, *, w_init_stdev=0.02): 51 | with tf.variable_scope(scope): 52 | *start, nx = shape_list(x) 53 | w = tf.get_variable('w', [1, nx, nf], initializer=tf.random_normal_initializer(stddev=w_init_stdev)) 54 | b = tf.get_variable('b', [nf], initializer=tf.constant_initializer(0)) 55 | c = tf.reshape(tf.matmul(tf.reshape(x, [-1, nx]), tf.reshape(w, [-1, nf]))+b, start+[nf]) 56 | return c 57 | 58 | def attention_mask(nd, ns, *, dtype): 59 | """1's in the lower triangle, counting from the lower right corner. 60 | 61 | Same as tf.matrix_band_part(tf.ones([nd, ns]), -1, ns-nd), but doesn't produce garbage on TPUs. 62 | """ 63 | i = tf.range(nd)[:,None] 64 | j = tf.range(ns) 65 | m = i >= j - ns + nd 66 | return tf.cast(m, dtype) 67 | 68 | 69 | def attn(x, scope, n_state, *, past, hparams): 70 | assert x.shape.ndims == 3 # Should be [batch, sequence, features] 71 | assert n_state % hparams.n_head == 0 72 | if past is not None: 73 | assert past.shape.ndims == 5 # Should be [batch, 2, heads, sequence, features], where 2 is [k, v] 74 | 75 | def split_heads(x): 76 | # From [batch, sequence, features] to [batch, heads, sequence, features] 77 | return tf.transpose(split_states(x, hparams.n_head), [0, 2, 1, 3]) 78 | 79 | def merge_heads(x): 80 | # Reverse of split_heads 81 | return merge_states(tf.transpose(x, [0, 2, 1, 3])) 82 | 83 | def mask_attn_weights(w): 84 | # w has shape [batch, heads, dst_sequence, src_sequence], where information flows from src to dst. 85 | _, _, nd, ns = shape_list(w) 86 | b = attention_mask(nd, ns, dtype=w.dtype) 87 | b = tf.reshape(b, [1, 1, nd, ns]) 88 | w = w*b - tf.cast(1e10, w.dtype)*(1-b) 89 | return w 90 | 91 | def multihead_attn(q, k, v): 92 | # q, k, v have shape [batch, heads, sequence, features] 93 | w = tf.matmul(q, k, transpose_b=True) 94 | w = w * tf.rsqrt(tf.cast(v.shape[-1].value, w.dtype)) 95 | 96 | w = mask_attn_weights(w) 97 | w = softmax(w) 98 | a = tf.matmul(w, v) 99 | return a 100 | 101 | with tf.variable_scope(scope): 102 | c = conv1d(x, 'c_attn', n_state*3) 103 | q, k, v = map(split_heads, tf.split(c, 3, axis=2)) 104 | present = tf.stack([k, v], axis=1) 105 | if past is not None: 106 | pk, pv = tf.unstack(past, axis=1) 107 | k = tf.concat([pk, k], axis=-2) 108 | v = tf.concat([pv, v], axis=-2) 109 | a = multihead_attn(q, k, v) 110 | a = merge_heads(a) 111 | a = conv1d(a, 'c_proj', n_state) 112 | return a, present 113 | 114 | 115 | def mlp(x, scope, n_state, *, hparams): 116 | with tf.variable_scope(scope): 117 | nx = x.shape[-1].value 118 | h = gelu(conv1d(x, 'c_fc', n_state)) 119 | h2 = conv1d(h, 'c_proj', nx) 120 | return h2 121 | 122 | 123 | def block(x, scope, *, past, hparams): 124 | with tf.variable_scope(scope): 125 | nx = x.shape[-1].value 126 | a, present = attn(norm(x, 'ln_1'), 'attn', nx, past=past, hparams=hparams) 127 | x = x + a 128 | m = mlp(norm(x, 'ln_2'), 'mlp', nx*4, hparams=hparams) 129 | x = x + m 130 | return x, present 131 | 132 | def past_shape(*, hparams, batch_size=None, sequence=None): 133 | return [batch_size, hparams.n_layer, 2, hparams.n_head, sequence, hparams.n_embd // hparams.n_head] 134 | 135 | def expand_tile(value, size): 136 | """Add a new axis of given size.""" 137 | value = tf.convert_to_tensor(value, name='value') 138 | ndims = value.shape.ndims 139 | return tf.tile(tf.expand_dims(value, axis=0), [size] + [1]*ndims) 140 | 141 | def positions_for(tokens, past_length): 142 | batch_size = tf.shape(tokens)[0] 143 | nsteps = tf.shape(tokens)[1] 144 | return expand_tile(past_length + tf.range(nsteps), batch_size) 145 | 146 | 147 | def model(hparams, X, past=None, scope='model', reuse=False): 148 | with tf.variable_scope(scope, reuse=reuse): 149 | results = {} 150 | batch, sequence = shape_list(X) 151 | 152 | wpe = tf.get_variable('wpe', [hparams.n_ctx, hparams.n_embd], 153 | initializer=tf.random_normal_initializer(stddev=0.01)) 154 | wte = tf.get_variable('wte', [hparams.n_vocab, hparams.n_embd], 155 | initializer=tf.random_normal_initializer(stddev=0.02)) 156 | past_length = 0 if past is None else tf.shape(past)[-2] 157 | h = tf.gather(wte, X) + tf.gather(wpe, positions_for(X, past_length)) 158 | 159 | # Transformer 160 | presents = [] 161 | pasts = tf.unstack(past, axis=1) if past is not None else [None] * hparams.n_layer 162 | assert len(pasts) == hparams.n_layer 163 | for layer, past in enumerate(pasts): 164 | h, present = block(h, 'h%d' % layer, past=past, hparams=hparams) 165 | presents.append(present) 166 | results['present'] = tf.stack(presents, axis=1) 167 | h = norm(h, 'ln_f') 168 | 169 | # Language model loss. Do tokens 4 | 5 | import argparse 6 | import json 7 | import os 8 | import numpy as np 9 | import tensorflow as tf 10 | import time 11 | 12 | import model, sample, encoder 13 | from load_dataset import load_dataset, Sampler 14 | from accumulate import AccumulatingOptimizer 15 | 16 | CHECKPOINT_DIR = 'checkpoint' 17 | SAMPLE_DIR = 'samples' 18 | 19 | 20 | parser = argparse.ArgumentParser( 21 | description='Fine-tune GPT-2 on your custom dataset.', 22 | formatter_class=argparse.ArgumentDefaultsHelpFormatter) 23 | 24 | parser.add_argument('--dataset', metavar='PATH', type=str, required=True, help='Input file, directory, or glob pattern (utf-8 text, or preencoded .npz files).') 25 | parser.add_argument('--model_name', metavar='MODEL', type=str, default='117M', help='Pretrained model name') 26 | parser.add_argument('--combine', metavar='CHARS', type=int, default=50000, help='Concatenate input files with <|endoftext|> separator into chunks of this minimum size') 27 | 28 | parser.add_argument('--batch_size', metavar='SIZE', type=int, default=1, help='Batch size') 29 | parser.add_argument('--learning_rate', metavar='LR', type=float, default=0.0001, help='Learning rate for Adam') 30 | parser.add_argument('--accumulate_gradients', metavar='N', type=int, default=5, help='Accumulate gradients across N minibatches.') 31 | 32 | parser.add_argument('--restore_from', type=str, default='latest', help='Either "latest", "fresh", or a path to a checkpoint file') 33 | parser.add_argument('--run_name', type=str, default='run1', help='Run id. Name of subdirectory in checkpoint/ and samples/') 34 | parser.add_argument('--sample_every', metavar='N', type=int, default=100, help='Generate samples every N steps') 35 | parser.add_argument('--sample_length', metavar='TOKENS', type=int, default=1023, help='Sample this many tokens') 36 | parser.add_argument('--sample_num', metavar='N', type=int, default=1, help='Generate this many samples') 37 | parser.add_argument('--save_every', metavar='N', type=int, default=1000, help='Write a checkpoint every N steps') 38 | 39 | 40 | def maketree(path): 41 | try: 42 | os.makedirs(path) 43 | except: 44 | pass 45 | 46 | 47 | def main(): 48 | args = parser.parse_args() 49 | enc = encoder.get_encoder(args.model_name) 50 | hparams = model.default_hparams() 51 | with open(os.path.join('models', args.model_name, 'hparams.json')) as f: 52 | hparams.override_from_dict(json.load(f)) 53 | 54 | if args.sample_length > hparams.n_ctx: 55 | raise ValueError( 56 | "Can't get samples longer than window size: %s" % hparams.n_ctx) 57 | 58 | config = tf.ConfigProto() 59 | config.gpu_options.allow_growth = True 60 | with tf.Session(config=config) as sess: 61 | context = tf.placeholder(tf.int32, [args.batch_size, None]) 62 | output = model.model(hparams=hparams, X=context) 63 | loss = tf.reduce_mean( 64 | tf.nn.sparse_softmax_cross_entropy_with_logits( 65 | labels=context[:, 1:], logits=output['logits'][:, :-1])) 66 | 67 | tf_sample = sample.sample_sequence( 68 | hparams=hparams, 69 | length=args.sample_length, 70 | context=context, 71 | batch_size=args.batch_size, 72 | temperature=1.0, 73 | top_k=40) 74 | 75 | train_vars = [v for v in tf.trainable_variables() if 'model' in v.name] 76 | if args.accumulate_gradients > 1: 77 | opt = AccumulatingOptimizer( 78 | opt=tf.train.AdamOptimizer(learning_rate=args.learning_rate), 79 | var_list=train_vars) 80 | opt_reset = opt.reset() 81 | opt_compute = opt.compute_gradients(loss) 82 | opt_apply = opt.apply_gradients() 83 | summary_loss = tf.summary.scalar('loss', opt_apply) 84 | else: 85 | opt_apply = tf.train.AdamOptimizer( 86 | learning_rate=args.learning_rate).minimize( 87 | loss, var_list=train_vars) 88 | summary_loss = tf.summary.scalar('loss', loss) 89 | 90 | summary_log = tf.summary.FileWriter( 91 | os.path.join(CHECKPOINT_DIR, args.run_name)) 92 | 93 | saver = tf.train.Saver( 94 | var_list=train_vars, 95 | max_to_keep=5, 96 | keep_checkpoint_every_n_hours=2) 97 | sess.run(tf.global_variables_initializer()) 98 | 99 | if args.restore_from == 'latest': 100 | ckpt = tf.train.latest_checkpoint( 101 | os.path.join(CHECKPOINT_DIR, args.run_name)) 102 | if ckpt is None: 103 | # Get fresh GPT weights if new run. 104 | ckpt = tf.train.latest_checkpoint( 105 | os.path.join('models', args.model_name)) 106 | elif args.restore_from == 'fresh': 107 | ckpt = tf.train.latest_checkpoint( 108 | os.path.join('models', args.model_name)) 109 | else: 110 | ckpt = tf.train.latest_checkpoint(args.restore_from) 111 | print('Loading checkpoint', ckpt) 112 | saver.restore(sess, ckpt) 113 | 114 | print('Loading dataset...') 115 | chunks = load_dataset(enc, args.dataset, args.combine) 116 | data_sampler = Sampler(chunks) 117 | print('dataset has', data_sampler.total_size, 'tokens') 118 | print('Training...') 119 | 120 | counter = 1 121 | counter_path = os.path.join(CHECKPOINT_DIR, args.run_name, 'counter') 122 | if os.path.exists(counter_path): 123 | # Load the step number if we're resuming a run 124 | # Add 1 so we don't immediately try to save again 125 | with open(counter_path, 'r') as fp: 126 | counter = int(fp.read()) + 1 127 | 128 | def save(): 129 | maketree(os.path.join(CHECKPOINT_DIR, args.run_name)) 130 | print( 131 | 'Saving', 132 | os.path.join(CHECKPOINT_DIR, args.run_name, 133 | 'model-{}').format(counter)) 134 | saver.save( 135 | sess, 136 | os.path.join(CHECKPOINT_DIR, args.run_name, 'model'), 137 | global_step=counter) 138 | with open(counter_path, 'w') as fp: 139 | fp.write(str(counter) + '\n') 140 | 141 | def generate_samples(): 142 | context_tokens = data_sampler.sample(1) 143 | all_text = [] 144 | index = 0 145 | while index < args.sample_num: 146 | out = sess.run( 147 | tf_sample, 148 | feed_dict={context: args.batch_size * [context_tokens]}) 149 | for i in range(min(args.sample_num - index, args.batch_size)): 150 | text = enc.decode(out[i]) 151 | text = '======== SAMPLE {} ========\n{}\n'.format( 152 | index + 1, text) 153 | all_text.append(text) 154 | index += 1 155 | print(text) 156 | maketree(os.path.join(SAMPLE_DIR, args.run_name)) 157 | with open( 158 | os.path.join(SAMPLE_DIR, args.run_name, 159 | 'samples-{}').format(counter), 'w') as fp: 160 | fp.write('\n'.join(all_text)) 161 | 162 | def sample_batch(): 163 | return [data_sampler.sample(1024) for _ in range(args.batch_size)] 164 | 165 | avg_loss = (0.0, 0.0) 166 | start_time = time.time() 167 | 168 | try: 169 | while True: 170 | if counter % args.save_every == 0: 171 | save() 172 | if counter % args.sample_every == 0: 173 | generate_samples() 174 | 175 | if args.accumulate_gradients > 1: 176 | sess.run(opt_reset) 177 | for _ in range(args.accumulate_gradients): 178 | sess.run( 179 | opt_compute, feed_dict={context: sample_batch()}) 180 | (v_loss, v_summary) = sess.run((opt_apply, summary_loss)) 181 | else: 182 | (_, v_loss, v_summary) = sess.run( 183 | (opt_apply, loss, summary_loss), 184 | feed_dict={context: sample_batch()}) 185 | 186 | summary_log.add_summary(v_summary, counter) 187 | 188 | avg_loss = (avg_loss[0] * 0.99 + v_loss, 189 | avg_loss[1] * 0.99 + 1.0) 190 | 191 | print( 192 | '[{counter} | {time:2.2f}] loss={loss:2.2f} avg={avg:2.2f}' 193 | .format( 194 | counter=counter, 195 | time=time.time() - start_time, 196 | loss=v_loss, 197 | avg=avg_loss[0] / avg_loss[1])) 198 | 199 | counter += 1 200 | except KeyboardInterrupt: 201 | print('interrupted') 202 | save() 203 | 204 | 205 | if __name__ == '__main__': 206 | main() 207 | -------------------------------------------------------------------------------- /twitterbot.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | import os 4 | import random 5 | import logging 6 | import tweepy 7 | from twitter_secrets import Secrets 8 | from src.generate_unconditional_samples import sample_model 9 | 10 | logfilename = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'bot.log') 11 | logging.basicConfig(level=logging.DEBUG, filename=logfilename, format='%(asctime)s %(name)s %(levelname)s:%(message)s') 12 | logger = logging.getLogger(__name__) 13 | logger.info(f'log file path: {logfilename}') 14 | 15 | results = sample_model() 16 | logger.info('Sampling model succeeds') 17 | 18 | results = results.split('\n')[1:-1] 19 | logger.info(f'Fetched {len(results)} samples') 20 | 21 | results = [line for line in results if len(line) <= 140] 22 | 23 | auth = tweepy.OAuthHandler(Secrets.CONSUMER_KEY, Secrets.CONSUMER_KEY_SECRET) 24 | auth.set_access_token(Secrets.ACCESS_TOKEN, Secrets.ACCESS_TOKEN_SECRET) 25 | api = tweepy.API(auth) 26 | 27 | api.update_status(random.choice(results)) 28 | --------------------------------------------------------------------------------